Trace execution preparation and durable response persistence (#16419)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The server prepares an execution environment before it starts an
agent.
> - Operators need to distinguish preparation, provider wait, and
response persistence.
> - Existing preparation phases start separate traces, and the native
session trace ends before the final response is saved.
> - This pull request adds an execution-attempt trace and detailed spans
at the existing work boundaries.
> - The benefit is a measurable startup and settlement path without
changing execution policy.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Operator-configured OpenTelemetry tracing for agent preparation,
startup, and durable response storage.

**Subsystem affected**

Server orchestration, native Runner startup, and the Honeycomb run link.

**Current behavior**

Preparation phases start independent roots. Workspace lock waits,
authorization groups, runtime asset work, provider-pack fallback, and
the gap after provider completion are hard to attribute.

**Proposed behavior**

Use a server-owned async context for each execution attempt. Parent
nested work to that attempt. Link the separately sampled native session
because its ingress and queue history can begin earlier. Record
transaction entry, lock waits, grouped authorization, asset preparation,
artifact fallback, copyback, and response commit.

**Reason and benefit**

Operators can identify expensive work from recorded boundaries instead
of inferring the cause from elapsed time. Nested and concurrent
durations remain visible and must not be added together.

**Breaking changes**

No API or execution-policy change. New Observability span names and
closed count attributes are documented. The existing local
run-performance schema also records the additional provider-pack and
harness measurements.

Refs: #16178, #16205, #16330.

## What Changed

- Add `task.execution_attempt` with same-run async ownership,
independent root sampling, and native-session links.
- Measure workspace transaction entry and settlement, advisory and row
waits, source scans, reader groups, notifications, and writes.
- Measure runtime selection and each concurrent instructions, skills,
and MCP branch. Separate manifest, cache, materialization, and
verification work.
- Measure provider-pack discovery, verified reuse or upload, final
verification, and remote harness state checks.
- Measure workspace settlement, accounting, liveness, response
authorization, comment commit, and publication enqueue.
- Preserve the silent endpoint-off path, including when another library
has registered a tracer. Bound repeated detail spans to 16 per name per
attempt.
- Extend the Honeycomb run link and document the actual coverage and
remaining gaps.

## Verification

- After resolving the conflict with retained-reader batching on
`b98cdaea531c8447637cd4dcc19c61b3f5f41844`, all 61 targeted query,
authorization, workspace-preparation and trace-boundary tests passed.
The server typecheck passed. Reader tests still bound query count, avoid
loading retained run payloads and enforce fresh project grants on every
publication attempt.
- Before that base integration, the tracing/exporter group passed 855
tests with 2 optional-SDK skips across 10 files and no inherited OTLP
endpoint. It covers real PostgreSQL waits/rollback, durable response
commit, assembled heartbeat preparation, concurrent parentage, sampling
and tracer failure isolation. Native workspace/history tests passed 23
cases. These results remain supporting evidence for the unchanged
tracing paths.
- Full workspace typecheck and build passed during implementation. Full
sharded CI, release validation, E2E and Apex 5/5 passed on prior head
`8cebad80f3856dbcff6f3a6d5d34880ff77a20f2`. Fresh CI, security and Apex
review are required on the current head before merge.
- `pnpm check:token-gates` passed. No styles changed.
- Earlier CI runs exposed exact-source-expression assertions changed by
wrappers, missing run hashes on transaction-boundary spans, and sampling
fixtures without an enabled endpoint. The fixes preserve the calls, add
the declared hash and span cap, and enable tracing explicitly in the
sampling fixtures. Focused repairs passed; the full prior-head CI passed
after those repairs. Earlier failed run:
https://github.com/paperclipai/paperclip/actions/runs/38729131366.
- The full local `pnpm test:run` was stopped after the base advanced
during that process. Its mixed-version results are not final-head
evidence. A full local-suite pass is not claimed.
- The old-head Superagent scan stalled at 35%. The current head has a
fresh scan; no security pass is inferred from the old pending result.
- No deployed Honeycomb acceptance is claimed by this code change.

## Risks

- Enabled tracing creates more spans. Repeated detail spans are bounded,
and inventories use group spans.
- The attempt and native session remain linked traces. Historical queue
time can precede the attempt, and nested intervals overlap.
- A committed comment proves durable server storage. Publication enqueue
and realtime dispatch do not prove external delivery or browser paint.
- New custom attributes use only the existing run hash and validated
file/byte counts. No SQL, paths, permissions, content, or exception text
is added.

## Model Used

OpenAI GPT-6 through Codex. The serving model ID and context window size
are not exposed in this session. The agent used reasoning, tools, code
editing, and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-10-15 08:13:53 -05:00
1 parent 7420af26fc
commit cccde7122b
32 files changed
+1187 -514

No files matched your search

+58 -11
View File
@@ -179,13 +179,15 @@ connection instructions, and agent identity. Their names are
`execution_configuration`; do not add them to its duration. They remain available
when a later setup step fails before the provider lifecycle trace starts.
Each phase starts a new root trace and links to the active trigger span when
one exists. An unsampled HTTP request must not suppress diagnostics for the
background run it starts. Root traces still follow the operator's configured
sampler. The trigger's trace remains unchanged, and overlapping runs keep
their own phase and child contexts.
`task.execution_attempt` owns one invocation of the server executor, from its
admission reads through preparation, provider dispatch, completion, reply persistence,
and cleanup. It starts a new root trace with a link to the active trigger. It
therefore follows the operator's root sampler even when the HTTP trigger was not
sampled. Async-local ownership parents same-run preparation and nested phases to
the current attempt. Concurrent runs have separate contexts. A phase invoked
without a live matching attempt still starts an independent linked root.
These spans exist before provider dispatch, including setup failures. Each has
These spans exist before provider dispatch, including setup failures. Preparation phases have
only the existing `paperclip.task.run.run_id` attribute: the first 12 hexadecimal
characters of the run UUID's SHA-256 hash. Failed operations set error status
without recording exception messages, paths, inputs, or returned content.
@@ -203,8 +205,8 @@ project repositories, and recovery-root reservation. These intervals are inside
Do not add these intervals to the containing workspace duration. They use the
same run hash and contain no workspace paths, commands, or returned records.
The `paperclip.native-controller` tracer records separate
`native.controller.<phase>` root spans around GitHub access setup, wake binding,
The `paperclip.native-controller` tracer records
`native.controller.<phase>` spans around GitHub access setup, wake binding,
backend construction, connector assignments, assigned MCP discovery, tool
authority, companion selection and Runner binary resolution. It also records
GitHub shutdown, coordinator release, instruction checkpoints and collection,
@@ -240,9 +242,51 @@ once known, `paperclip.native_reconciliation.candidates` (a count). A failed
pass marks the pass span and the phase that was running with an error status.
Faster passes record nothing.
### Detailed execution phases
The `paperclip.execution` tracer adds these content-free spans beneath the current
attempt or owning phase:
| Span family | Measured work |
| --- | --- |
| `workspace.transaction`, `.entry`, `.commit`, `.rollback` | Transaction lifetime, connection acquisition plus BEGIN, and settlement after the callback returns. Entry cannot distinguish pool wait from BEGIN. |
| `workspace.provenance_lock`, `workspace.row_lock` | The awaited advisory-lock statement and current workspace row read with its row lock. |
| `workspace.sources.*` | Source inventory queries, canonicalization, one filesystem alias scan, repository scan, and configured project scan. |
| `workspace.readers`, `workspace.readers.*`, `workspace.access.*` | Retained-reader groups, identity, trust, project and source authorization, privacy and inventory reads. |
| `workspace.provenance`, `workspace.notification`, `workspace.write` | Preserved ownership, notification statements, and record writes. |
| `runtime_selection.*` | Completion contracts, session/recovery reads, profiles, interaction materialization, backend discovery, bootstrap, run/coordinator writes, transaction entry/settlement, and workspace synchronization. |
| `runtime_context.instructions`, `.skills`, `.mcp` | The three concurrent runtime-context branches. Their durations overlap. |
| `runtime_context.asset_*` | Source reads, manifest hashing, cache lookup, cold materialization, and exact file verification on cold and warm paths. |
| `workspace_sync.*` | Existing Git and baseline snapshot, pack, seed verification, transfer, and restore operations. Pack and verification can be inside transfer; do not add nested durations. |
| `execution.runtime` | Runtime dispatch, usage, workspace finalization and runtime cleanup; this includes work after provider execution. |
| `execution.workspace_ownership`, `.workspace_ownership_wait`, `.workspace_restore`, `.workspace_commit`, `.native_finalize` | Ownership coordination including its action, acquisition before the action, copyback, final snapshot/stamp/descriptor persistence, and native settlement. |
| `completion.*` | Completion handler, run-log finalization, terminal persistence, accounting, liveness, and continuation summaries. |
| `response.resolve`, `.authorize`, `.add_comment`, `.transaction*`, `.row_lock`, `.write`, `.publish` | Response selection, existing authorization, comment transaction including commit or rollback, row wait, insert, and external publication enqueue. |
| `execution.realtime` | Synchronous live-event and plugin publication dispatch. It does not measure browser receipt or paint. |
Detail phases emit at most 16 spans of each name per attempt. Larger reader and
asset inventories continue under the containing group without unbounded per-row
or per-file spans. Work, query order, authorization, and locks are unchanged.
Manual transaction and dispatch boundaries retain their actual lifetime.
Asset writes and verification may add only non-negative integer
`paperclip.execution.fileCount` and `paperclip.execution.bytes` attributes.
The closed keys are declared in `EXECUTION_PREPARATION_SPAN_ATTRS` in
`packages/adapter-utils/src/acpx-engine/startup-timing.ts`.
Other execution phases carry only the existing hashed run ID; manual boundaries
have no attributes. Exceptions set status without exporting their messages.
A committed comment transaction establishes durable server storage. A publication
enqueue establishes pending delivery, and realtime dispatch establishes a server
notification call. None establishes browser paint or external delivery. The attempt
can continue through later bookkeeping and resource cleanup. A later recovery
invocation has its own attempt; this does not manufacture one continuous trace
across queueing, process restarts, or an unavailable controller.
## Native Runner Trace Spans
Paperclip Runner task runs emit a single foldable OpenTelemetry trace. This is
The native provider lifecycle emits one foldable OpenTelemetry trace, linked to
the server attempt described above. This is
the native-run trace schema version `3`. `task.provider_session` is the native
provider lifecycle root; it includes preparation and provider execution, but
ends before outer workspace persistence. The heartbeat-owned local run-log
@@ -251,12 +295,15 @@ every other native span carries a real OpenTelemetry parent context rather than
only a descriptive `parentName` field.
The native lifecycle root uses the operator's root sampling policy. It links
to a valid active trigger trace without inheriting its sampling decision.
to the matching live server attempt (or a valid active trigger) without inheriting
its sampling decision. This is a link because historical ingress and queue spans
can predate this invocation. It does not reparent earlier work beneath a later span.
An unsampled HTTP request therefore cannot suppress an otherwise sampled
background run. An operator who disables root sampling still gets no spans.
Without an OTLP endpoint, tracing remains silent and does not export data.
The run link in Honeycomb matches both `task.provider_session` for native runs and `task.run` for legacy adapters. It retains the exact hashed run-ID and service filters and groups by trace ID. The URL builder uses Honeycomb’s supported `in` operator for the two root names; see the [query specification](https://github.com/honeycombio/honeycomb-mcp#query-specification-for-run_query). This changes only the query link. The operator endpoint gate and span attribute allowlist remain unchanged.
The run link in Honeycomb matches `task.provider_session` for native runs, `task.run` for legacy adapters, and
`task.execution_attempt` for server preparation and settlement. It retains the exact hashed run-ID and service filters and groups by trace ID. The URL builder uses Honeycomb’s supported `in` operator for the three root names; see the [query specification](https://github.com/honeycombio/honeycomb-mcp#query-specification-for-run_query). This changes only the query link. The operator endpoint gate and span attribute allowlist remain unchanged.
The canonical lifecycle is:
+14
View File
@@ -5,6 +5,20 @@ Run-log events write to the `heartbeat_run_events` table
Paperclip Telemetry events, and they are not OpenTelemetry exports. A run-log
event needs no operator endpoint.
## Provider-pack preparation timing
The existing `run.performance.span` event also records `provider_pack.select`,
`provider_pack.discover`, `provider_pack.verify`, `provider_pack.verify_preinstalled`,
`provider_pack.link_preinstalled`, `provider_pack.clear_stale`, `provider_pack.upload`,
and `provider_pack.verify_uploaded`. Selection records `stateSource` as `staged`,
`preinstalled`, or `uploaded`. Its closed reason is `verified_pack_reused` or
`cached_packs_unverified`. A verification failure describes an unusable candidate;
it does not identify a particular compatibility mismatch or a failed run.
Harness startup also records `harness_state.inspect`, `harness_state.path_check`,
and `harness_state.launch_material` for remote state inspection and fresh launch
material. These events retain the existing schema and carry no paths or exception strings.
They also become native Observability spans only with an operator OTLP endpoint.
## Native PRP Run-Log Events
`native.recovery.execution_reclaimed` records an atomic same-run recovery claim
@@ -47,6 +47,12 @@ export function normalizeProviderFamily(key: string | undefined): string {
*/
export const SANDBOX_STARTUP_SPAN_ATTR_PREFIX = "paperclip.sandbox.startup.";
export const EXECUTION_PREPARATION_SPAN_ATTRS = {
runId: "paperclip.task.run.run_id",
fileCount: "paperclip.execution.fileCount",
bytes: "paperclip.execution.bytes",
} as const;
/**
* The closed attribute-name contract for every sandbox-startup span. This is
* the single source of truth for the harness span attributes. Each name uses
+4
View File
@@ -380,6 +380,10 @@ describe("redaction", () => {
"runner.session.bootstrap",
"runner.turn.submit",
"runner.session.startup",
"provider_pack.select",
"provider_pack.discover",
"provider_pack.verify_uploaded",
"provider_pack.upload",
"provider.turn.queue",
"question_response.to_run_created",
"native.session.execute",
+3 -1
View File
@@ -39,7 +39,7 @@ import { checkExactPeerVersions } from "./peer-version-check.js";
export { checkExactPeerVersions } from "./peer-version-check.js";
const endpoint = process.env.OTEL_EXPORTER_OTLP_ENDPOINT;
const endpoint = process.env.OTEL_EXPORTER_OTLP_ENDPOINT?.trim();
let sdkShutdown: (() => Promise<void>) | null = null;
let shutdownPromise: Promise<void> | null = null;
@@ -96,6 +96,7 @@ let tracerApiLoadFailed = false;
* local no-op tracer (fail open).
*/
export function getStartupTracer(name = "paperclip.startup"): StartupTracerHandle {
if (!process.env.OTEL_EXPORTER_OTLP_ENDPOINT?.trim()) return NOOP_TRACER;
try {
const require = createRequire(import.meta.url);
const api = require("@opentelemetry/api") as {
@@ -151,6 +152,7 @@ let traceContextApiLoadFailed = false;
* local no-op trace context (fail open).
*/
export function getStartupTraceContext(name = "paperclip.startup"): StartupTraceContextHandle {
if (!process.env.OTEL_EXPORTER_OTLP_ENDPOINT?.trim()) return NOOP_TRACE_CONTEXT;
try {
const require = createRequire(import.meta.url);
const api = require("@opentelemetry/api") as {
+11
View File
@@ -328,6 +328,17 @@ const NATIVE_RUN_SPAN_NAMES = new Set([
"provider.time_to_first_agent_event",
"provider.turn.queue",
"question_response.to_run_created",
"harness_state.inspect",
"harness_state.path_check",
"harness_state.launch_material",
"provider_pack.select",
"provider_pack.discover",
"provider_pack.verify",
"provider_pack.verify_preinstalled",
"provider_pack.verify_uploaded",
"provider_pack.link_preinstalled",
"provider_pack.clear_stale",
"provider_pack.upload",
"runner.artifact.discover",
"runner.artifact.prepare",
"runner.prp.authenticate",
+7
View File
@@ -1,3 +1,4 @@
import { beginRunPhase } from "./execution-preparation-timing.js";
import type { ActivityPublication } from "../types/activity-publication.js";
export type { ActivityPublication } from "../types/activity-publication.js";
import { randomUUID } from "node:crypto";
@@ -146,12 +147,18 @@ export async function resolveResponsibleUserIdForActivity(db: Db, input: LogActi
}
export function publishActivity(publication: ActivityPublication) {
const finish = beginRunPhase("execution.realtime");
try {
publishLiveEvent({
companyId: publication.companyId,
type: "activity.logged",
payload: publication.payload,
});
if (publication.pluginEvent) publishPluginDomainEvent(publication.pluginEvent);
} catch (error) {
finish(true);
throw error;
} finally { finish(); }
}
export async function persistActivity(db: Db, input: LogActivityInput) {
@@ -247,7 +247,7 @@ describe("chat publication commit signals", () => {
'eventType: "run.presentation.resolved"',
);
const committedComment = completionSource.lastIndexOf(
"await issuesSvc.addComment",
"issuesSvc.addComment",
presentationMarker,
);
expect(presentationMarker).toBeGreaterThanOrEqual(0);
@@ -1,5 +1,7 @@
import { AsyncLocalStorage } from "node:async_hooks";
import { EXECUTION_PREPARATION_SPAN_ATTRS, runWithRuntimeParent } from "@paperclipai/adapter-utils/acpx-engine/startup-timing";
import { createHash } from "node:crypto";
import { isSpanContextValid, ROOT_CONTEXT, SpanStatusCode, trace } from "@opentelemetry/api";
import { isSpanContextValid, ROOT_CONTEXT, SpanStatusCode, trace, type Context } from "@opentelemetry/api";
type PreparationPhase = "environment_selection" | "execution_configuration" | "session"
| "workspace" | "github_environment" | "github_launchers" | "run_output"
@@ -27,37 +29,125 @@ type PreparationPhase = "environment_selection" | "execution_configuration" | "s
| "github_stop" | "coordinator_release" | "instruction_checkpoint"
| "instruction_changes" | "warm_release" | "instruction_collection";
export type ExecutionPhase =
| "workspace.transaction" | "workspace.transaction.entry" | "workspace.transaction.commit" | "workspace.transaction.rollback"
| "workspace.identity_read" | "workspace.row_lock" | "workspace.provenance" | "workspace.notification" | "workspace.write"
| "workspace.sources" | "workspace.sources.canonicalize" | "workspace.sources.inventory" | "workspace.sources.alias_scan"
| "workspace.sources.repository_scan" | "workspace.sources.project_scan" | "workspace.provenance_lock"
| "workspace.readers" | "workspace.readers.identity" | "workspace.readers.trust" | "workspace.readers.projects" | "workspace.readers.sources"
| "workspace.access" | "workspace.access.privacy" | "workspace.access.inventory" | "workspace.access.authorization"
| "runtime_selection.review" | "runtime_selection.contract_read" | "runtime_selection.contract_ensure"
| "runtime_selection.resume" | "runtime_selection.recovery_profile" | "runtime_selection.profile" | "runtime_selection.interactions"
| "runtime_selection.provider_discovery" | "runtime_selection.bootstrap" | "runtime_selection.run_write" | "runtime_selection.finalization_write"
| "runtime_selection.transaction" | "runtime_selection.transaction.entry" | "runtime_selection.transaction.commit" | "runtime_selection.transaction.rollback"
| "runtime_selection.row_lock" | "runtime_selection.workspace_sync"
| "runtime_context.instructions" | "runtime_context.skills" | "runtime_context.mcp"
| "runtime_context.asset_read" | "runtime_context.asset_manifest" | "runtime_context.asset_cache" | "runtime_context.asset_materialize" | "runtime_context.asset_verify"
| "workspace_sync.snapshot_git" | "workspace_sync.snapshot_baseline" | "workspace_sync.pack" | "workspace_sync.seed_verify" | "workspace_sync.stage_workspace" | "workspace_sync.restore"
| "execution.runtime" | "execution.workspace_ownership" | "execution.workspace_restore" | "execution.workspace_commit"
| "execution.workspace_ownership_wait" | "execution.realtime"
| "execution.usage" | "execution.native_finalize"
| "completion.run" | "completion.output" | "completion.terminal_write" | "completion.cost" | "completion.liveness" | "completion.summary"
| "response.transaction" | "response.transaction.entry" | "response.transaction.commit" | "response.transaction.rollback" | "response.row_lock" | "response.write"
| "response.resolve" | "response.authorize" | "response.add_comment" | "response.activity" | "response.publish";
type RunScope = { runId: string; parent?: Context; ended: boolean; details: Map<string, number> };
const ownership = new AsyncLocalStorage<RunScope>();
const endpointEnabled = () => Boolean(process.env.OTEL_EXPORTER_OTLP_ENDPOINT?.trim());
export function getRunExecutionTraceContext(runId: string): Context | undefined {
const owned = ownership.getStore();
return endpointEnabled() && owned?.runId === runId && !owned.ended ? owned.parent : undefined;
}
function measure<T>(runId: string, name: string, work: () => T | Promise<T>, independent = false, counts: { fileCount?: number; bytes?: number } = {}): Promise<T> {
let operation: Promise<T> | undefined;
const run = (span?: { setStatus(status: { code: number }): void; end(): void }, parent?: Context) => {
if (operation) return operation;
const previous = ownership.getStore();
const owned: RunScope = { runId,
parent: parent ?? (!independent && previous?.runId === runId && !previous.ended ? previous.parent : undefined),
ended: false,
details: !independent && previous?.runId === runId ? previous.details : new Map<string, number>(),
};
operation = Promise.resolve().then(() =>
ownership.run(owned, () => runWithRuntimeParent(owned.parent, work))).then(value => {
try { span?.end(); } catch {}
owned.ended = true;
return value;
}, error => {
try { span?.setStatus({ code: SpanStatusCode.ERROR }); } catch {}
try { span?.end(); } catch {}
owned.ended = true;
throw error;
});
return operation;
};
if (!endpointEnabled()) return (async () => work())();
try {
const parent = independent ? undefined : getRunExecutionTraceContext(runId);
const trigger = parent ? undefined : trace.getActiveSpan()?.spanContext();
const traced = trace.getTracer(name.startsWith("heartbeat.prepare.") ? "paperclip.heartbeat-preparation"
: name.startsWith("native.controller.") ? "paperclip.native-controller" : "paperclip.execution").startActiveSpan(name, {
attributes: { [EXECUTION_PREPARATION_SPAN_ATTRS.runId]: createHash("sha256").update(runId).digest("hex").slice(0, 12),
...Object.fromEntries((["fileCount", "bytes"] as const).flatMap(key =>
typeof counts[key] === "number" && Number.isSafeInteger(counts[key]) && counts[key]! >= 0
? [[EXECUTION_PREPARATION_SPAN_ATTRS[key], counts[key]!]] : [])),
},
...(trigger && isSpanContextValid(trigger) ? { links: [{ context: trigger }] } : {}),
}, parent ?? ROOT_CONTEXT, span => {
let childContext: Context | undefined;
try { childContext = trace.setSpan(parent ?? ROOT_CONTEXT, span); } catch {}
return run(span, childContext);
});
void Promise.resolve(traced).catch(() => undefined);
} catch {}
return operation ?? run();
}
export function withRunExecutionTiming<T>(runId: string, work: () => T | Promise<T>): Promise<T> {
return measure(runId, "task.execution_attempt", work, true);
}
export async function measureRunPhase<T>(phase: ExecutionPhase, work: () => T | Promise<T>, counts?: { fileCount?: number; bytes?: number }): Promise<T> {
const owned = ownership.getStore();
if (!endpointEnabled() || !owned || owned.ended) return work();
const count = owned.details.get(phase) ?? 0;
if (count >= 16) return work();
owned.details.set(phase, count + 1);
return measure(owned.runId, phase, work, false, counts);
}
export function beginRunPhase(phase: ExecutionPhase): (failed?: boolean) => void {
const owned = ownership.getStore();
if (!endpointEnabled() || !owned?.parent || owned.ended) return () => {};
const count = owned.details.get(phase) ?? 0;
if (count >= 16) return () => {};
owned.details.set(phase, count + 1);
try {
const span = trace.getTracer("paperclip.execution").startSpan(phase, {
attributes: { [EXECUTION_PREPARATION_SPAN_ATTRS.runId]: createHash("sha256").update(owned.runId).digest("hex").slice(0, 12) },
}, owned.parent);
let ended = false;
return failed => {
if (ended) return;
ended = true;
if (failed) { try { span.setStatus({ code: SpanStatusCode.ERROR }); } catch {} }
try { span.end(); } catch {}
};
} catch { return () => {}; }
}
export function createRunPreparationTiming(runId: string, scope: "heartbeat.prepare" | "native.controller" = "heartbeat.prepare") {
let runIdHash: string | undefined;
return <T>(phase: PreparationPhase, work: () => T | Promise<T>): Promise<T> => {
let operation: Promise<T> | undefined;
const run = (span?: { setStatus(status: { code: number }): void; end(): void }) => {
operation ??= (async () => {
try { return await work(); }
catch (error) {
try { span?.setStatus({ code: SpanStatusCode.ERROR }); } catch {}
throw error;
} finally {
try { span?.end(); } catch {}
}
})();
return operation;
};
if (!process.env.OTEL_EXPORTER_OTLP_ENDPOINT?.trim()) return run();
try {
runIdHash ??= createHash("sha256").update(runId).digest("hex").slice(0, 12);
const trigger = trace.getActiveSpan()?.spanContext();
const traced = trace.getTracer(scope === "heartbeat.prepare" ? "paperclip.heartbeat-preparation" : "paperclip.native-controller").startActiveSpan(
`${scope}.${phase}`,
{
attributes: { "paperclip.task.run.run_id": runIdHash },
...(trigger && isSpanContextValid(trigger) ? { links: [{ context: trigger }] } : {}),
},
ROOT_CONTEXT,
span => run(span),
);
void Promise.resolve(traced).catch(() => undefined);
} catch {}
return operation ?? run();
return <T>(phase: PreparationPhase, work: () => T | Promise<T>): Promise<T> => measure(runId, `${scope}.${phase}`, work);
}
export function measureWorkspaceSyncPhase<T>(name: string, work: () => Promise<T>): Promise<T> {
const phases: Readonly<Record<string, ExecutionPhase>> = {
"snapshot.git": "workspace_sync.snapshot_git", "snapshot.baseline": "workspace_sync.snapshot_baseline",
pack: "workspace_sync.pack", "seed.verify": "workspace_sync.seed_verify",
"stage.workspace": "workspace_sync.stage_workspace", "restore.workspace": "workspace_sync.restore",
"workspace.restore": "execution.workspace_restore",
};
return Object.prototype.hasOwnProperty.call(phases, name) ? measureRunPhase(phases[name]!, work) : work();
}
@@ -0,0 +1,192 @@
import { getActiveStepContext } from "@paperclipai/adapter-utils/acpx-engine/startup-timing";
import { createHash } from "node:crypto";
import { context, trace, TraceFlags } from "@opentelemetry/api";
import { AsyncLocalStorageContextManager } from "@opentelemetry/context-async-hooks";
import { InMemorySpanExporter, SimpleSpanProcessor, TracerProvider } from "@opentelemetry/sdk-trace";
import { afterEach, beforeEach, expect, it, vi } from "vitest";
import { getStartupTraceContext, getStartupTracer } from "../instrumentation.js";
import { prepareVerifiedRemoteProviderPack } from "./native-runtime/remote-provider-pack.js";
import { createNativeRunTrace } from "./native-runtime/native-run-trace.js";
import { beginRunPhase, createRunPreparationTiming, getRunExecutionTraceContext, measureRunPhase, withRunExecutionTiming } from "./execution-preparation-timing.js";
let exporter: InMemorySpanExporter;
let provider: TracerProvider;
let manager: AsyncLocalStorageContextManager;
beforeEach(() => {
vi.stubEnv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://collector.test");
exporter = new InMemorySpanExporter();
provider = new TracerProvider({ spanProcessors: [new SimpleSpanProcessor({ exporter })] });
manager = new AsyncLocalStorageContextManager().enable();
expect(context.setGlobalContextManager(manager)).toBe(true);
expect(trace.setGlobalTracerProvider(provider)).toBe(true);
});
afterEach(async () => {
vi.restoreAllMocks();
vi.unstubAllEnvs();
trace.disable();
context.disable();
manager.disable();
await provider.shutdown();
});
it("parents overlapping preparation and response work to their own execution attempts independently of ingress sampling", async () => {
const trigger = { traceId: "1".repeat(32), spanId: "2".repeat(16), traceFlags: TraceFlags.NONE, isRemote: true };
await context.with(trace.setSpanContext(context.active(), trigger), () => Promise.all(["one", "two"].map(runId =>
withRunExecutionTiming(runId, async () => {
await createRunPreparationTiming(runId)("workspace", () => Promise.all([
measureRunPhase("workspace.sources", async () => {
await Promise.resolve();
await measureRunPhase("workspace.sources.alias_scan", () => {
const parent = getActiveStepContext()?.parentContext;
expect(parent).toBeDefined();
expect(trace.getSpan(parent as import("@opentelemetry/api").Context)?.spanContext()).toEqual(trace.getActiveSpan()?.spanContext());
expect(getRunExecutionTraceContext(runId === "one" ? "two" : "one")).toBeUndefined();
});
}),
measureRunPhase("workspace.readers", async () => { await Promise.resolve(); }),
]));
await measureRunPhase("response.add_comment", () => undefined);
}),
)));
const spans = exporter.getFinishedSpans();
const attempts = spans.filter(span => span.name === "task.execution_attempt");
expect(attempts).toHaveLength(2);
expect(new Set(attempts.map(span => span.spanContext().traceId)).size).toBe(2);
for (const runId of ["one", "two"]) {
const attempt = attempts.find(span => span.attributes["paperclip.task.run.run_id"] === createHash("sha256").update(runId).digest("hex").slice(0, 12))!;
expect(attempt.parentSpanContext).toBeUndefined();
expect(attempt.links[0].context).toEqual(trigger);
const owned = spans.filter(span => span.spanContext().traceId === attempt.spanContext().traceId);
const phase = owned.find(span => span.name === "heartbeat.prepare.workspace")!;
const source = owned.find(span => span.name === "workspace.sources")!;
const aliases = owned.find(span => span.name === "workspace.sources.alias_scan")!;
const readers = owned.find(span => span.name === "workspace.readers")!;
const reply = owned.find(span => span.name === "response.add_comment")!;
expect(phase.parentSpanContext?.spanId).toBe(attempt.spanContext().spanId);
expect(source.parentSpanContext?.spanId).toBe(phase.spanContext().spanId);
expect(readers.parentSpanContext?.spanId).toBe(phase.spanContext().spanId);
expect(aliases.parentSpanContext?.spanId).toBe(source.spanContext().spanId);
expect(reply.parentSpanContext?.spanId).toBe(attempt.spanContext().spanId);
const ns = (time: readonly [number, number]) => time[0] * 1e9 + time[1];
expect(ns(readers.startTime)).toBeLessThanOrEqual(ns(source.endTime));
expect(ns(source.startTime)).toBeLessThanOrEqual(ns(readers.endTime));
expect(owned).toHaveLength(6);
}
expect(getRunExecutionTraceContext("one")).toBeUndefined();
});
it("uses a link for a native session whose queue history predates the current attempt", async () => {
await withRunExecutionTiming("native", async () => {
const native = createNativeRunTrace({ runId: "native", startedAtMs: Date.now() - 10_000 });
await native.measure("agent.turn", async () => {});
await native.finish("ok");
await measureRunPhase("response.add_comment", () => undefined);
});
const spans = exporter.getFinishedSpans();
const attempt = spans.find(span => span.name === "task.execution_attempt")!;
const session = spans.find(span => span.name === "task.provider_session")!;
const response = spans.find(span => span.name === "response.add_comment")!;
expect(session.parentSpanContext).toBeUndefined();
expect(session.links.map(link => link.context.spanId)).toContain(attempt.spanContext().spanId);
expect(response.parentSpanContext?.spanId).toBe(attempt.spanContext().spanId);
expect(session.startTime[0]).toBeLessThan(attempt.startTime[0]);
});
it("caps repeated detail spans and ends failed scopes without content or exception text", async () => {
const failure = new Error("private path and credential");
const work = vi.fn(() => undefined);
await expect(withRunExecutionTiming("bounded", async () => {
for (let index = 0; index < 100; index++) await measureRunPhase("workspace.readers.identity", work);
for (let index = 0; index < 100; index++) beginRunPhase("workspace.transaction.entry")();
await measureRunPhase("workspace.readers.trust", () => { throw failure; });
})).rejects.toBe(failure);
expect(work).toHaveBeenCalledTimes(100);
const spans = exporter.getFinishedSpans();
expect(spans.filter(span => span.name === "workspace.readers.identity")).toHaveLength(16);
expect(spans.filter(span => span.name === "workspace.transaction.entry")).toHaveLength(16);
for (const span of spans) expect(Object.keys(span.attributes)).toEqual(["paperclip.task.run.run_id"]);
expect(spans.filter(span => span.status.code === 2).map(span => span.name).sort()).toEqual(["task.execution_attempt", "workspace.readers.trust"]);
expect(JSON.stringify(spans.map(span => ({ attributes: span.attributes, events: span.events, status: span.status })))).not.toContain(failure.message);
expect(getRunExecutionTraceContext("bounded")).toBeUndefined();
});
it.each([undefined, "", " "])("does not consult tracers or warn without endpoint %s even with a registered provider", async endpoint => {
vi.stubEnv("OTEL_EXPORTER_OTLP_ENDPOINT", endpoint);
const lookup = vi.spyOn(trace, "getTracer");
const warnings = (["log", "info", "warn", "error", "debug"] as const).map(method => vi.spyOn(console, method));
await expect(withRunExecutionTiming("off", () => measureRunPhase("response.add_comment", () => 42))).resolves.toBe(42);
getStartupTracer().startSpan("disabled").end();
getStartupTraceContext().tracer.startSpan("disabled").end();
const native = createNativeRunTrace({ runId: "off" });
await native.measure("agent.turn", async () => undefined);
await native.finish("ok");
expect(lookup).not.toHaveBeenCalled();
expect(exporter.getFinishedSpans()).toEqual([]);
for (const warning of warnings) expect(warning).not.toHaveBeenCalled();
});
it.each(["staged", "preinstalled", "uploaded", "failed"] as const)("records provider-pack selection %s with closed source and fallback reasons", async source => {
const failure = new Error("private manifest path and hash");
const native = createNativeRunTrace({ runId: "pack" });
const select = prepareVerifiedRemoteProviderPack({
trace: native,
verifyStaged: async () => { if (source !== "staged") throw failure; },
usePreinstalled: async () => source === "preinstalled",
stageAndVerify: async () => { if (source === "failed") throw failure; },
});
if (source === "failed") await expect(select).rejects.toBe(failure);
else await expect(select).resolves.toBe(source);
await native.finish(source === "failed" ? "failed" : "ok");
const span = exporter.getFinishedSpans().find(span => span.name === "provider_pack.select")!;
expect(span.status.code).toBe(source === "failed" ? 2 : 0);
if (source !== "failed") expect(span.attributes["paperclip.native.span.stateSource"]).toBe(source);
expect(JSON.stringify({ attributes: span.attributes, events: span.events })).not.toContain(failure.message);
});
it("exports only the declared count attributes and rejects unsafe numbers or extra fields", async () => {
await withRunExecutionTiming("counts", async () => {
await measureRunPhase("runtime_context.asset_materialize", () => undefined,
{ fileCount: 3, bytes: 99, privatePath: "/private/fixture" } as { fileCount: number; bytes: number });
await measureRunPhase("runtime_context.asset_verify", () => undefined, { fileCount: -1, bytes: Infinity });
});
const spans = exporter.getFinishedSpans();
const materialize = spans.find(span => span.name === "runtime_context.asset_materialize")!;
expect(materialize.attributes).toEqual({
"paperclip.task.run.run_id": createHash("sha256").update("counts").digest("hex").slice(0, 12),
"paperclip.execution.fileCount": 3,
"paperclip.execution.bytes": 99,
});
const verify = spans.find(span => span.name === "runtime_context.asset_verify")!;
expect(Object.keys(verify.attributes)).toEqual(["paperclip.task.run.run_id"]);
});
it("does not inherit another run when a new attempt's tracer lookup fails", async () => {
await withRunExecutionTiming("outer", async () => {
const lookup = vi.spyOn(trace, "getTracer").mockImplementationOnce(() => { throw new Error("tracer unavailable"); });
try {
await withRunExecutionTiming("inner", () => measureRunPhase("workspace.sources", () => 42));
} finally { lookup.mockRestore(); }
});
const spans = exporter.getFinishedSpans();
const outer = spans.find(span => span.name === "task.execution_attempt")!;
const inner = spans.find(span => span.name === "workspace.sources")!;
expect(inner.parentSpanContext).toBeUndefined();
expect(inner.spanContext().traceId).not.toBe(outer.spanContext().traceId);
expect(inner.attributes["paperclip.task.run.run_id"]).toBe(createHash("sha256").update("inner").digest("hex").slice(0, 12));
});
it("does not attach native children to an ambient run when native context construction fails", async () => {
await withRunExecutionTiming("ambient", async () => {
const native = createNativeRunTrace({ runId: "native-context-failure", traceContext: {
tracer: getStartupTracer("broken-native-context"),
contextWithSpan() { throw new Error("context unavailable"); },
} });
await native.measure("agent.turn", async () => undefined);
await native.finish("ok");
});
const spans = exporter.getFinishedSpans();
const ambient = spans.find(span => span.name === "task.execution_attempt")!;
const child = spans.find(span => span.name === "agent.turn")!;
expect(child.parentSpanContext).toBeUndefined();
expect(child.spanContext().traceId).not.toBe(ambient.spanContext().traceId);
});
@@ -0,0 +1,127 @@
import { context, trace } from "@opentelemetry/api";
import { AsyncLocalStorageContextManager } from "@opentelemetry/context-async-hooks";
import { InMemorySpanExporter, SimpleSpanProcessor, TracerProvider } from "@opentelemetry/sdk-trace";
import { companies, createDb, executionWorkspaces, issueComments, issues, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS } from "@paperclipai/db";
import { eq, sql } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js";
import { measureRunTransaction } from "./execution-transaction-timing.js";
import { withRunExecutionTiming } from "./execution-preparation-timing.js";
import { issueService } from "./issues.js";
import { logActivity } from "./activity-log.js";
import { executionWorkspaceService } from "./execution-workspaces.js";
import { lockExecutionWorkspaceRootProvenance } from "./execution-workspace-repositories.js";
import { assertTaskWorkspaceAccess } from "./task-workspace-source-access.js";
const support = await getEmbeddedPostgresTestSupport();
(support.supported ? describe : describe.skip)("execution timing with real database waits", () => {
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let db: ReturnType<typeof createDb>;
let companyId: string, workspaceId: string;
let exporter: InMemorySpanExporter, provider: TracerProvider, manager: AsyncLocalStorageContextManager;
beforeAll(async () => {
temporary = await startEmbeddedPostgresTestDatabase("paperclip-execution-timing-");
db = createDb(temporary.connectionString);
const [company] = await db.insert(companies).values({ name: "Timing fixture", issuePrefix: "TIME" }).returning();
companyId = company.id;
const [workspace] = await db.insert(executionWorkspaces).values({ companyId, name: "Files", mode: "shared_workspace", strategyType: "task_directory" }).returning();
workspaceId = workspace.id;
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
beforeEach(() => {
vi.stubEnv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://collector.test");
exporter = new InMemorySpanExporter();
provider = new TracerProvider({ spanProcessors: [new SimpleSpanProcessor({ exporter })] });
manager = new AsyncLocalStorageContextManager().enable();
expect(context.setGlobalContextManager(manager)).toBe(true);
expect(trace.setGlobalTracerProvider(provider)).toBe(true);
});
afterEach(async () => {
vi.unstubAllEnvs();
trace.disable(); context.disable(); manager.disable();
await provider.shutdown();
});
afterAll(async () => { await temporary?.cleanup(); });
it.each(["advisory", "row"] as const)("attributes a controlled %s lock wait to its leaf and commits the original update", async kind => {
let release!: () => void, acquired!: () => void;
const released = new Promise<void>(resolve => { release = resolve; });
const ready = new Promise<void>(resolve => { acquired = resolve; });
const blocker = db.transaction(async tx => {
if (kind === "advisory") await lockExecutionWorkspaceRootProvenance(tx, companyId);
else await tx.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspaceId)).for("update");
acquired();
await released;
});
await ready;
const update = withRunExecutionTiming(`wait-${kind}`, () => executionWorkspaceService(db).update(workspaceId, { metadata: { verified: kind } }));
try {
await vi.waitFor(async () => {
const pending = await db.execute(sql`select count(*)::int as count from pg_locks where not granted and locktype = ${kind === "advisory" ? "advisory" : "transactionid"}`);
expect(pending[0]?.count).toBeGreaterThan(0);
});
expect(exporter.getFinishedSpans().some(span => span.name === (kind === "advisory" ? "workspace.provenance_lock" : "workspace.row_lock"))).toBe(false);
} finally {
release();
await blocker;
await update;
}
const [saved] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspaceId));
expect(saved.metadata).toMatchObject({ verified: kind });
const spans = exporter.getFinishedSpans();
const transaction = spans.find(span => span.name === "workspace.transaction")!;
const wait = spans.find(span => span.name === (kind === "advisory" ? "workspace.provenance_lock" : "workspace.row_lock"))!;
expect(wait.parentSpanContext?.spanId).toBe(transaction.spanContext().spanId);
expect(wait.duration[0] * 1e9 + wait.duration[1]).toBeGreaterThan(0);
expect(spans.map(span => span.name)).toEqual(expect.arrayContaining(["workspace.transaction.entry", "workspace.transaction.commit", "workspace.notification", "workspace.write"]));
expect(spans.some(span => span.name === "workspace.transaction.rollback")).toBe(false);
expect(JSON.stringify(spans.map(span => span.attributes))).not.toContain(workspaceId);
});
it("preserves transaction rollback and the original error without replaying writes", async () => {
const [before] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspaceId));
const failure = new Error("private rollback fixture");
let calls = 0;
await expect(withRunExecutionTiming("rollback", () => measureRunTransaction(db, "workspace", async tx => {
calls++;
await tx.update(executionWorkspaces).set({ metadata: { rolledBack: true } }).where(eq(executionWorkspaces.id, workspaceId));
throw failure;
}))).rejects.toBe(failure);
expect(calls).toBe(1);
const [after] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspaceId));
expect(after.metadata).toEqual(before.metadata);
const spans = exporter.getFinishedSpans();
expect(spans.find(span => span.name === "workspace.transaction.rollback")?.status.code).toBe(2);
expect(spans.some(span => span.name === "workspace.transaction.commit")).toBe(false);
expect(JSON.stringify(spans.map(span => ({ attributes: span.attributes, events: span.events })))).not.toContain(failure.message);
});
it("records durable response commit and server publication under the same attempt", async () => {
const [issue] = await db.insert(issues).values({ companyId, title: "Response timing" }).returning();
const body = "Private response fixture";
const comment = await withRunExecutionTiming("response", async () => {
const saved = await issueService(db).addComment(issue.id, body, { userId: "local-board" });
await logActivity(db, { companyId, actorType: "user", actorId: "local-board", action: "issue.comment_added", entityType: "issue", entityId: issue.id });
return saved;
});
const [stored] = await db.select().from(issueComments).where(eq(issueComments.id, comment.id));
expect(stored.body).toBe(body);
const spans = exporter.getFinishedSpans();
const attempt = spans.find(span => span.name === "task.execution_attempt")!;
const transaction = spans.find(span => span.name === "response.transaction")!;
const commit = spans.find(span => span.name === "response.transaction.commit")!;
expect(transaction.parentSpanContext?.spanId).toBe(attempt.spanContext().spanId);
expect(commit.parentSpanContext?.spanId).toBe(transaction.spanContext().spanId);
expect(spans.map(span => span.name)).toEqual(expect.arrayContaining(["response.row_lock", "response.write", "execution.realtime"]));
expect(spans.every(span => span.spanContext().traceId === attempt.spanContext().traceId)).toBe(true);
expect(JSON.stringify(spans.map(span => span.attributes))).not.toContain(body);
});
it("preserves a real access denial and exports only failed status", async () => {
await expect(withRunExecutionTiming("denied", () => assertTaskWorkspaceAccess(db,
{ type: "board", userId: "not-a-company-member" }, companyId, workspaceId,
))).rejects.toMatchObject({ status: 403 });
const spans = exporter.getFinishedSpans();
expect(spans.find(span => span.name === "task.execution_attempt")?.status.code).toBe(2);
expect(JSON.stringify(spans.map(span => ({ attributes: span.attributes, events: span.events })))).not.toContain("not-a-company-member");
});
});
@@ -0,0 +1,31 @@
import type { Db } from "@paperclipai/db";
import { beginRunPhase, measureRunPhase } from "./execution-preparation-timing.js";
type Transaction = Parameters<Parameters<Db["transaction"]>[0]>[0];
export function measureRunTransaction<T>(db: Db, scope: "workspace" | "runtime_selection" | "response", work: (tx: Transaction) => Promise<T>): Promise<T> {
return measureRunPhase(`${scope}.transaction`, async () => {
const entry = beginRunPhase(`${scope}.transaction.entry`);
let settle: ((failed?: boolean) => void) | undefined;
let failed = false;
try {
return await db.transaction(async tx => {
entry();
try {
const value = await work(tx);
settle = beginRunPhase(`${scope}.transaction.commit`);
return value;
} catch (error) {
settle = beginRunPhase(`${scope}.transaction.rollback`);
throw error;
}
});
} catch (error) {
failed = true;
throw error;
} finally {
entry(failed);
settle?.(failed);
}
});
}
@@ -1,3 +1,5 @@
import { measureRunTransaction } from "./execution-transaction-timing.js";
import { measureRunPhase } from "./execution-preparation-timing.js";
import { createHash, randomUUID } from "node:crypto";
import { execFile as execFileCallback } from "node:child_process";
import { isDeepStrictEqual, promisify } from "node:util";
@@ -65,10 +67,10 @@ export function assertRepositoryRetryMatches(row: Pick<RepositoryRow, "repositor
async function resolveRepositoryRootSources(tx: Db | DbTransaction, input: {
companyId: string; cwd: string; sourceWorkspaceId?: string; inspectCurrentAliases?: boolean; preparingCopyProjectIds?: Set<string>;
}) {
const canonicalCwd = await canonicalRepositoryRoot(input.cwd);
const canonicalCwd = await measureRunPhase("workspace.sources.canonicalize", () => canonicalRepositoryRoot(input.cwd));
const rootKey = createHash("sha256").update(canonicalCwd).digest("hex");
const companyEntries = await tx.select().from(executionWorkspaceRepositoryAccess)
.where(eq(executionWorkspaceRepositoryAccess.companyId, input.companyId));
const companyEntries = await measureRunPhase("workspace.sources.inventory", async () => tx.select().from(executionWorkspaceRepositoryAccess)
.where(eq(executionWorkspaceRepositoryAccess.companyId, input.companyId)));
const entries = companyEntries.filter(entry => entry.rootKey === rootKey);
const overlappingEntries = companyEntries.filter(entry => repositoryRootsOverlap(entry.canonicalCwd, canonicalCwd));
if (entries.some(entry => entry.canonicalCwd !== canonicalCwd)
@@ -94,22 +96,24 @@ async function resolveRepositoryRootSources(tx: Db | DbTransaction, input: {
const isolatedSourceIds = new Set(overlappingEntries.filter(entry => entry.kind === "isolated_source").map(entry => entry.scopeId));
const sourceIds = new Set([...sharedSourceIds, ...isolatedSourceIds]);
if (input.sourceWorkspaceId) { sourceIds.add(input.sourceWorkspaceId); sharedSourceIds.add(input.sourceWorkspaceId); addSharedSource(input.sourceWorkspaceId, canonicalCwd); }
const workspaces = await tx.select({ id: executionWorkspaces.id, cwd: executionWorkspaces.cwd }).from(executionWorkspaces)
.where(eq(executionWorkspaces.companyId, input.companyId));
for (const workspace of workspaces) {
if (!workspace.cwd) continue;
const peerRoot = await canonicalRepositoryRoot(workspace.cwd).catch((error: NodeJS.ErrnoException) => {
if (["ENOTDIR", "EACCES", "EPERM", "ELOOP", "ENAMETOOLONG"].includes(error.code ?? "")) return null;
throw error;
});
if (peerRoot && repositoryRootsOverlap(peerRoot, canonicalCwd)) {
sourceIds.add(workspace.id); sharedSourceIds.add(workspace.id); addSharedSource(workspace.id, peerRoot);
const workspaces = await measureRunPhase("workspace.sources.inventory", async () => tx.select({ id: executionWorkspaces.id, cwd: executionWorkspaces.cwd }).from(executionWorkspaces)
.where(eq(executionWorkspaces.companyId, input.companyId)));
await measureRunPhase("workspace.sources.alias_scan", async () => {
for (const workspace of workspaces) {
if (!workspace.cwd) continue;
const peerRoot = await canonicalRepositoryRoot(workspace.cwd!).catch((error: NodeJS.ErrnoException) => {
if (["ENOTDIR", "EACCES", "EPERM", "ELOOP", "ENAMETOOLONG"].includes(error.code ?? "")) return null;
throw error;
});
if (peerRoot && repositoryRootsOverlap(peerRoot, canonicalCwd)) {
sourceIds.add(workspace.id); sharedSourceIds.add(workspace.id); addSharedSource(workspace.id, peerRoot);
}
}
}
const inventory = await tx.select({ id: executionWorkspaceRepositories.id, workspaceId: executionWorkspaces.id, cwd: executionWorkspaces.cwd,
});
const inventory = await measureRunPhase("workspace.sources.inventory", async () => tx.select({ id: executionWorkspaceRepositories.id, workspaceId: executionWorkspaces.id, cwd: executionWorkspaces.cwd,
identity: executionWorkspaceRepositories.repositoryIdentity }).from(executionWorkspaceRepositories)
.innerJoin(executionWorkspaces, eq(executionWorkspaces.id, executionWorkspaceRepositories.executionWorkspaceId))
.where(and(eq(executionWorkspaceRepositories.companyId, input.companyId), eq(executionWorkspaces.companyId, input.companyId)));
.where(and(eq(executionWorkspaceRepositories.companyId, input.companyId), eq(executionWorkspaces.companyId, input.companyId))));
const namespace = path.join(canonicalCwd, ".paperclip-repositories");
const namespaceEntry = await fs.lstat(namespace).catch((error: NodeJS.ErrnoException) => {
if (error.code === "ENOENT") return null;
@@ -122,37 +126,39 @@ async function resolveRepositoryRootSources(tx: Db | DbTransaction, input: {
}
for (const name of await fs.readdir(namespace)) if (name.startsWith("task-repo-")) publishedNames.add(name);
}
const repositoryOrigins = await tx.select({ id: executionWorkspaceRepositoryAccess.scopeId }).from(executionWorkspaceRepositoryAccess)
.where(and(eq(executionWorkspaceRepositoryAccess.companyId, input.companyId), eq(executionWorkspaceRepositoryAccess.kind, "repository")));
const repositoryOrigins = await measureRunPhase("workspace.sources.inventory", async () => tx.select({ id: executionWorkspaceRepositoryAccess.scopeId }).from(executionWorkspaceRepositoryAccess)
.where(and(eq(executionWorkspaceRepositoryAccess.companyId, input.companyId), eq(executionWorkspaceRepositoryAccess.kind, "repository"))));
const knownOrigins = new Set(repositoryOrigins.map(origin => origin.id));
const recognizedNames = new Set<string>();
for (const peer of inventory) {
const name = path.basename(taskRepositoryRelativePath(peer.identity));
const published = publishedNames.has(name);
const peerRoot = peer.cwd ? await fs.realpath(peer.cwd).catch((error: NodeJS.ErrnoException) => {
if (["ENOENT", "ENOTDIR", "EACCES", "EPERM", "ELOOP", "ENAMETOOLONG"].includes(error.code ?? "")) return null;
throw error;
}) : null;
const matches = sourceIds.has(peer.workspaceId) || (peerRoot !== null && repositoryRootsOverlap(peerRoot, canonicalCwd))
|| (published && peerRoot === null && !knownOrigins.has(peer.id))
|| (peerRoot === null && !entries.some(entry => entry.kind === "reader") && peer.cwd && await legacyRepositoryRootMayMatch(peer.cwd, canonicalCwd));
if (matches) {
sourceIds.add(peer.workspaceId);
if (!isolatedSourceIds.has(peer.workspaceId) || (peerRoot && repositoryRootsOverlap(peerRoot, canonicalCwd))) {
sharedSourceIds.add(peer.workspaceId); addSharedSource(peer.workspaceId, peerRoot ?? canonicalCwd);
await measureRunPhase("workspace.sources.repository_scan", async () => {
for (const peer of inventory) {
const name = path.basename(taskRepositoryRelativePath(peer.identity));
const published = publishedNames.has(name);
const peerRoot = peer.cwd ? await fs.realpath(peer.cwd).catch((error: NodeJS.ErrnoException) => {
if (["ENOENT", "ENOTDIR", "EACCES", "EPERM", "ELOOP", "ENAMETOOLONG"].includes(error.code ?? "")) return null;
throw error;
}) : null;
const matches = sourceIds.has(peer.workspaceId) || (peerRoot !== null && repositoryRootsOverlap(peerRoot, canonicalCwd))
|| (published && peerRoot === null && !knownOrigins.has(peer.id))
|| (peerRoot === null && !entries.some(entry => entry.kind === "reader") && peer.cwd && await legacyRepositoryRootMayMatch(peer.cwd, canonicalCwd));
if (matches) {
sourceIds.add(peer.workspaceId);
if (!isolatedSourceIds.has(peer.workspaceId) || (peerRoot && repositoryRootsOverlap(peerRoot, canonicalCwd))) {
sharedSourceIds.add(peer.workspaceId); addSharedSource(peer.workspaceId, peerRoot ?? canonicalCwd);
}
if (published) recognizedNames.add(name);
}
if (published) recognizedNames.add(name);
}
}
});
if ([...publishedNames].some(name => !recognizedNames.has(name))) throw forbidden("Published repository has no retained workspace provenance");
const configured = await tx.select({ projectId: projectWorkspaces.projectId, cwd: projectWorkspaces.cwd, repoUrl: projectWorkspaces.repoUrl })
.from(projectWorkspaces).where(eq(projectWorkspaces.companyId, input.companyId));
const configured = await measureRunPhase("workspace.sources.inventory", async () => tx.select({ projectId: projectWorkspaces.projectId, cwd: projectWorkspaces.cwd, repoUrl: projectWorkspaces.repoUrl })
.from(projectWorkspaces).where(eq(projectWorkspaces.companyId, input.companyId)));
const sourceProjectIds = new Set(overlappingEntries.filter(entry => (entry.kind === "project_source" || entry.kind === "isolated_project_source")).map(entry => entry.scopeId));
const sharedProjectIds = new Set(overlappingEntries.filter(entry => entry.kind === "shared_project_source" && !isCopiedAncestor(entry.scopeId, entry.canonicalCwd)).map(entry => entry.scopeId));
const isolatedProjectIds = new Set(overlappingEntries.filter(entry => entry.kind === "isolated_project_source").map(entry => entry.scopeId));
if (sourceIds.size) {
const owners = await tx.select({ id: executionWorkspaces.id, projectId: executionWorkspaces.projectId, strategyType: executionWorkspaces.strategyType }).from(executionWorkspaces).where(and(
eq(executionWorkspaces.companyId, input.companyId), or(...[...sourceIds].map(id => eq(executionWorkspaces.id, id)))));
const owners = await measureRunPhase("workspace.sources.inventory", async () => tx.select({ id: executionWorkspaces.id, projectId: executionWorkspaces.projectId, strategyType: executionWorkspaces.strategyType }).from(executionWorkspaces).where(and(
eq(executionWorkspaces.companyId, input.companyId), or(...[...sourceIds].map(id => eq(executionWorkspaces.id, id))))));
for (const owner of owners) if (owner.projectId) {
sourceProjectIds.add(owner.projectId);
if (owner.strategyType === "git_worktree" || !sharedSourceIds.has(owner.id)
@@ -160,13 +166,15 @@ async function resolveRepositoryRootSources(tx: Db | DbTransaction, input: {
else sharedProjectIds.add(owner.projectId);
}
}
for (const source of configured) {
if (sharedProjectIds.has(source.projectId)) continue;
const sourceRoot = await configuredWorkspaceRoot(source, true);
if (!sourceRoot || !repositoryRootsOverlap(sourceRoot, canonicalCwd)) continue;
sourceProjectIds.add(source.projectId);
if (!isCopiedAncestor(source.projectId, sourceRoot)) sharedProjectIds.add(source.projectId);
}
await measureRunPhase("workspace.sources.project_scan", async () => {
for (const source of configured) {
if (sharedProjectIds.has(source.projectId)) continue;
const sourceRoot = await configuredWorkspaceRoot(source, true);
if (!sourceRoot || !repositoryRootsOverlap(sourceRoot, canonicalCwd)) continue;
sourceProjectIds.add(source.projectId);
if (!isCopiedAncestor(source.projectId, sourceRoot)) sharedProjectIds.add(source.projectId);
}
});
for (const entry of overlappingEntries) if (entry.kind === "project_source" && !isolatedProjectIds.has(entry.scopeId)) {
sharedProjectIds.add(entry.scopeId);
}
@@ -195,9 +203,9 @@ async function assertRootProjectsReadAccessForActors(db: Db | DbTransaction, act
}
async function retainedRootReader(tx: DbTransaction, companyId: string, reader: Pick<typeof executionWorkspaceRepositoryAccess.$inferSelect, "scopeId" | "issueId" | "agentId" | "responsibleUserId">) {
const [run] = await tx.select(rootReaderRunColumns).from(heartbeatRuns).where(and(eq(heartbeatRuns.id, reader.scopeId), eq(heartbeatRuns.companyId, companyId)));
const [agent] = reader.agentId ? await tx.select({ id: agents.id }).from(agents).where(and(eq(agents.id, reader.agentId), eq(agents.companyId, companyId))) : [];
const [issue] = reader.issueId ? await tx.select().from(issues).where(and(eq(issues.id, reader.issueId), eq(issues.companyId, companyId))) : [];
const [run] = await measureRunPhase("workspace.sources.inventory", async () => tx.select(rootReaderRunColumns).from(heartbeatRuns).where(and(eq(heartbeatRuns.id, reader.scopeId), eq(heartbeatRuns.companyId, companyId))));
const [agent] = reader.agentId ? await measureRunPhase("workspace.sources.inventory", async () => tx.select({ id: agents.id }).from(agents).where(and(eq(agents.id, reader.agentId!), eq(agents.companyId, companyId)))) : [];
const [issue] = reader.issueId ? await measureRunPhase("workspace.sources.inventory", async () => tx.select().from(issues).where(and(eq(issues.id, reader.issueId!), eq(issues.companyId, companyId)))) : [];
if (!run || !agent || !issue || run.agentId !== reader.agentId || run.responsibleUserId !== reader.responsibleUserId
|| (run.nativeIssueId ?? run.contextSnapshot?.issueId ?? run.contextSnapshot?.taskId ?? run.issueId) !== reader.issueId) {
throw forbidden("Repository root retains a reader whose identity is no longer available");
@@ -208,7 +216,7 @@ async function retainedRootReader(tx: DbTransaction, companyId: string, reader:
}
export async function lockExecutionWorkspaceRootProvenance(tx: DbTransaction, companyId: string) {
await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${`task-repository-roots:${companyId}`}, 0))`);
await measureRunPhase("workspace.provenance_lock", () => tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${`task-repository-roots:${companyId}`}, 0))`));
}
export async function preserveConfiguredProjectWorkspaceRootProvenance(tx: DbTransaction, input: {
@@ -217,12 +225,12 @@ export async function preserveConfiguredProjectWorkspaceRootProvenance(tx: DbTra
const canonicalCwd = await configuredWorkspaceRoot(input, false);
if (!canonicalCwd) return;
const rootKey = createHash("sha256").update(canonicalCwd).digest("hex");
const entries = await tx.select().from(executionWorkspaceRepositoryAccess).where(and(
eq(executionWorkspaceRepositoryAccess.companyId, input.companyId), eq(executionWorkspaceRepositoryAccess.rootKey, rootKey)));
const entries = await measureRunPhase("workspace.sources.inventory", async () => tx.select().from(executionWorkspaceRepositoryAccess).where(and(
eq(executionWorkspaceRepositoryAccess.companyId, input.companyId), eq(executionWorkspaceRepositoryAccess.rootKey, rootKey))));
if (entries.some(entry => entry.canonicalCwd !== canonicalCwd)) throw forbidden("Repository root identity conflicts with its retained access record");
for (const kind of ["project_source", "shared_project_source"] as const) {
if (!entries.some(entry => entry.kind === kind && entry.scopeId === input.projectId)) {
await tx.insert(executionWorkspaceRepositoryAccess).values({ companyId: input.companyId, canonicalCwd, rootKey, kind, scopeId: input.projectId });
await measureRunPhase("workspace.write", () => tx.insert(executionWorkspaceRepositoryAccess).values({ companyId: input.companyId, canonicalCwd, rootKey, kind, scopeId: input.projectId }));
}
}
}
@@ -237,13 +245,13 @@ export async function preserveExecutionWorkspaceRootProvenance(tx: DbTransaction
});
if (!directory) return;
if (!directory.isDirectory()) throw forbidden("Workspace files cannot be detached from unverifiable provenance");
const { canonicalCwd, rootKey, entries, sharedSourceIds, isolatedSourceIds, sourceProjectIds, sharedProjectIds, isolatedProjectIds } = await resolveRepositoryRootSources(tx, {
companyId: input.companyId, cwd: input.cwd, sourceWorkspaceId: input.workspaceId, inspectCurrentAliases: true,
});
const { canonicalCwd, rootKey, entries, sharedSourceIds, isolatedSourceIds, sourceProjectIds, sharedProjectIds, isolatedProjectIds } = await measureRunPhase("workspace.sources", () => resolveRepositoryRootSources(tx, {
companyId: input.companyId, cwd: input.cwd!, sourceWorkspaceId: input.workspaceId, inspectCurrentAliases: true,
}));
for (const [kind, ids] of [["source", sharedSourceIds], ["isolated_source", isolatedSourceIds], ["project_source", sourceProjectIds], ["shared_project_source", sharedProjectIds], ["isolated_project_source", isolatedProjectIds]] as const) {
for (const scopeId of ids) {
if (!entries.some(entry => entry.kind === kind && entry.scopeId === scopeId)) {
await tx.insert(executionWorkspaceRepositoryAccess).values({ companyId: input.companyId, canonicalCwd, rootKey, kind, scopeId });
await measureRunPhase("workspace.write", () => tx.insert(executionWorkspaceRepositoryAccess).values({ companyId: input.companyId, canonicalCwd, rootKey, kind, scopeId }));
}
}
}
@@ -252,9 +260,9 @@ export async function preserveExecutionWorkspaceRootProvenance(tx: DbTransaction
export async function assertExecutionWorkspaceRootReadAccess(db: Db | DbTransaction, input: {
companyId: string; cwd: string; actor: AuthorizationActor; sourceWorkspaceId?: string;
}): Promise<void> {
const { sourceIds, sourceProjectIds } = await resolveRepositoryRootSources(db, { ...input, inspectCurrentAliases: true });
await assertTaskWorkspacesReadAccess(db, input.actor, input.companyId, sourceIds);
await assertRootProjectAccess(db, input.actor, input.companyId, sourceProjectIds);
const { sourceIds, sourceProjectIds } = await measureRunPhase("workspace.sources", () => resolveRepositoryRootSources(db, { ...input, inspectCurrentAliases: true }));
await measureRunPhase("workspace.readers.sources", () => assertTaskWorkspacesReadAccess(db, input.actor, input.companyId, sourceIds));
await measureRunPhase("workspace.readers.projects", () => assertRootProjectAccess(db, input.actor, input.companyId, sourceProjectIds));
}
export type IsolatedWorkspaceSourceAccess = {
@@ -267,20 +275,20 @@ export type IsolatedWorkspaceSourceAccess = {
async function retainGitMetadataOwners(tx: DbTransaction, input: {
companyId: string; commonRoot: string; workspaceIds: Set<string>; projectIds: Set<string>;
}): Promise<void> {
const closure = await resolveRepositoryRootSources(tx, { companyId: input.companyId, cwd: input.commonRoot, inspectCurrentAliases: true });
const closure = await measureRunPhase("workspace.sources", () => resolveRepositoryRootSources(tx, { companyId: input.companyId, cwd: input.commonRoot, inspectCurrentAliases: true }));
const workspaceIds = new Set([...closure.sourceIds, ...input.workspaceIds]);
const projectIds = new Set([...closure.sourceProjectIds, ...input.projectIds]);
const readerActors: AuthorizationActor[] = [];
for (const reader of closure.overlappingEntries.filter(entry => entry.kind === "reader")) {
const { actor } = await retainedRootReader(tx, input.companyId, reader);
const { actor } = await measureRunPhase("workspace.readers.identity", () => retainedRootReader(tx, input.companyId, reader));
readerActors.push(actor);
}
await assertRootProjectsReadAccessForActors(tx, readerActors, input.companyId, projectIds);
await assertTaskWorkspacesReadAccessForActors(tx, readerActors, input.companyId, workspaceIds);
await measureRunPhase("workspace.readers.projects", () => assertRootProjectsReadAccessForActors(tx, readerActors, input.companyId, projectIds));
await measureRunPhase("workspace.readers.sources", () => assertTaskWorkspacesReadAccessForActors(tx, readerActors, input.companyId, workspaceIds));
for (const [kind, ids] of [["source", workspaceIds], ["project_source", projectIds]] as const) {
for (const scopeId of ids) if (!closure.entries.some(entry => entry.kind === kind && entry.scopeId === scopeId)) {
await tx.insert(executionWorkspaceRepositoryAccess).values({ companyId: input.companyId,
canonicalCwd: closure.canonicalCwd, rootKey: closure.rootKey, kind, scopeId });
await measureRunPhase("workspace.write", () => tx.insert(executionWorkspaceRepositoryAccess).values({ companyId: input.companyId,
canonicalCwd: closure.canonicalCwd, rootKey: closure.rootKey, kind, scopeId }));
}
}
}
@@ -289,14 +297,14 @@ export async function assertIsolatedWorkspaceSourceReadAccess(db: Db, input: {
companyId: string; sourceCwd: string; actor: AuthorizationActor; copyKind?: WorkspaceSourceCopyKind;
}): Promise<IsolatedWorkspaceSourceAccess> {
const { roots, commonRoot } = await isolatedWorkspaceSourceRoots(db, input.companyId, input.sourceCwd, input.copyKind);
return db.transaction(async tx => {
return measureRunTransaction(db, "workspace", async tx => {
await lockExecutionWorkspaceRootProvenance(tx, input.companyId);
const workspaceIds = new Set<string>(), projectIds = new Set<string>();
for (const cwd of roots) {
const closure = await resolveRepositoryRootSources(tx, { companyId: input.companyId, cwd, inspectCurrentAliases: true });
await assertTaskWorkspacesReadAccess(tx, input.actor, input.companyId, closure.sourceIds);
const closure = await measureRunPhase("workspace.sources", () => resolveRepositoryRootSources(tx, { companyId: input.companyId, cwd, inspectCurrentAliases: true }));
await measureRunPhase("workspace.readers.sources", () => assertTaskWorkspacesReadAccess(tx, input.actor, input.companyId, closure.sourceIds));
for (const id of closure.sourceIds) workspaceIds.add(id);
await assertRootProjectAccess(tx, input.actor, input.companyId, closure.sourceProjectIds);
await measureRunPhase("workspace.readers.projects", () => assertRootProjectAccess(tx, input.actor!, input.companyId, closure.sourceProjectIds));
for (const id of closure.sourceProjectIds) projectIds.add(id);
}
await retainGitMetadataOwners(tx, { companyId: input.companyId, commonRoot, workspaceIds, projectIds });
@@ -313,31 +321,31 @@ export async function retainIsolatedWorkspaceSourceAccess(db: Db, input: {
const roots = new Set(input.captured.roots);
try { for (const cwd of (await isolatedWorkspaceSourceRoots(db, input.companyId, input.sourceCwd, input.copyKind)).roots) roots.add(cwd); }
catch (error) { sourceFailure = error; }
const canonicalCwd = await canonicalRepositoryRoot(input.targetCwd);
const canonicalCwd = await measureRunPhase("workspace.sources.canonicalize", () => canonicalRepositoryRoot(input.targetCwd));
const rootKey = createHash("sha256").update(canonicalCwd).digest("hex");
const denied = await db.transaction(async tx => {
const denied = await measureRunTransaction(db, "workspace", async tx => {
await lockExecutionWorkspaceRootProvenance(tx, input.companyId);
const existing = await tx.select().from(executionWorkspaceRepositoryAccess).where(and(
eq(executionWorkspaceRepositoryAccess.companyId, input.companyId), eq(executionWorkspaceRepositoryAccess.rootKey, rootKey)));
const existing = await measureRunPhase("workspace.sources.inventory", async () => tx.select().from(executionWorkspaceRepositoryAccess).where(and(
eq(executionWorkspaceRepositoryAccess.companyId, input.companyId), eq(executionWorkspaceRepositoryAccess.rootKey, rootKey))));
if (existing.some(entry => entry.canonicalCwd !== canonicalCwd)) throw forbidden("Repository root identity conflicts with its retained access record");
const workspaceIds = new Set(input.captured.workspaceIds), projectIds = new Set(input.captured.projectIds);
let failure = sourceFailure;
for (const cwd of roots) {
try {
const closure = await resolveRepositoryRootSources(tx, { companyId: input.companyId, cwd, inspectCurrentAliases: true });
const closure = await measureRunPhase("workspace.sources", () => resolveRepositoryRootSources(tx, { companyId: input.companyId, cwd, inspectCurrentAliases: true }));
for (const id of closure.sourceIds) workspaceIds.add(id);
for (const id of closure.sourceProjectIds) projectIds.add(id);
} catch (error) { failure ??= error; }
}
try { await assertTaskWorkspacesReadAccess(tx, input.actor, input.companyId, workspaceIds); }
try { await measureRunPhase("workspace.readers.sources", () => assertTaskWorkspacesReadAccess(tx, input.actor, input.companyId, workspaceIds)); }
catch (error) { failure ??= error; }
for (const id of projectIds) {
try { await assertRootProjectAccess(tx, input.actor, input.companyId, new Set([id])); }
try { await measureRunPhase("workspace.readers.projects", () => assertRootProjectAccess(tx, input.actor!, input.companyId, new Set([id]))); }
catch (error) { failure ??= error; }
}
for (const [kind, ids] of [["isolated_source", workspaceIds], ["isolated_project_source", projectIds]] as const) {
for (const scopeId of ids) if (!existing.some(entry => entry.kind === kind && entry.scopeId === scopeId)) {
await tx.insert(executionWorkspaceRepositoryAccess).values({ companyId: input.companyId, rootKey, canonicalCwd, kind, scopeId });
await measureRunPhase("workspace.write", () => tx.insert(executionWorkspaceRepositoryAccess).values({ companyId: input.companyId, rootKey, canonicalCwd, kind, scopeId }));
}
}
const phase = input.phase ?? "preparing";
@@ -360,23 +368,23 @@ export async function retainIsolatedWorkspaceSourceAccess(db: Db, input: {
}
} catch (error) { failure ??= error; }
try {
const target = await resolveRepositoryRootSources(tx, { companyId: input.companyId, cwd: canonicalCwd, preparingCopyProjectIds });
await assertRootProjectTrust(tx, { companyId: input.companyId, actor: input.actor, issueId: input.issueId, projectIds: target.sourceProjectIds });
await assertTaskWorkspacesReadAccess(tx, input.actor, input.companyId, target.sourceIds);
await assertRootProjectAccess(tx, input.actor, input.companyId, target.sourceProjectIds,
{ sharedProjectIds: target.sharedProjectIds, issueId: input.issueId });
const target = await measureRunPhase("workspace.sources", () => resolveRepositoryRootSources(tx, { companyId: input.companyId, cwd: canonicalCwd, preparingCopyProjectIds }));
await measureRunPhase("workspace.readers.trust", () => assertRootProjectTrust(tx, { companyId: input.companyId, actor: input.actor, issueId: input.issueId, projectIds: target.sourceProjectIds }));
await measureRunPhase("workspace.readers.sources", () => assertTaskWorkspacesReadAccess(tx, input.actor, input.companyId, target.sourceIds));
await measureRunPhase("workspace.readers.projects", () => assertRootProjectAccess(tx, input.actor!, input.companyId, target.sourceProjectIds,
{ sharedProjectIds: target.sharedProjectIds, issueId: input.issueId }));
const readerActors: AuthorizationActor[] = [];
for (const reader of target.overlappingEntries.filter(entry => entry.kind === "reader")) {
const { actor, issue } = await retainedRootReader(tx, input.companyId, reader);
const { actor, issue } = await measureRunPhase("workspace.readers.identity", () => retainedRootReader(tx, input.companyId, reader));
readerActors.push(actor);
await assertRootProjectTrust(tx, { companyId: input.companyId, actor, issueId: issue.id, projectIds: target.sourceProjectIds });
await measureRunPhase("workspace.readers.trust", () => assertRootProjectTrust(tx, { companyId: input.companyId, actor, issueId: issue.id, projectIds: target.sourceProjectIds }));
}
await assertRootProjectsReadAccessForActors(tx, readerActors, input.companyId, target.sourceProjectIds);
await assertTaskWorkspacesReadAccessForActors(tx, readerActors, input.companyId, target.sourceIds);
await measureRunPhase("workspace.readers.projects", () => assertRootProjectsReadAccessForActors(tx, readerActors, input.companyId, target.sourceProjectIds));
await measureRunPhase("workspace.readers.sources", () => assertTaskWorkspacesReadAccessForActors(tx, readerActors, input.companyId, target.sourceIds));
} catch (error) { failure ??= error; }
if (!failure && verifiedCopyProjectIds) {
for (const scopeId of verifiedCopyProjectIds) if (!existing.some(entry => entry.kind === "isolated_copy" && entry.scopeId === scopeId)) {
await tx.insert(executionWorkspaceRepositoryAccess).values({ companyId: input.companyId, rootKey, canonicalCwd, kind: "isolated_copy", scopeId });
await measureRunPhase("workspace.write", () => tx.insert(executionWorkspaceRepositoryAccess).values({ companyId: input.companyId, rootKey, canonicalCwd, kind: "isolated_copy", scopeId }));
}
}
return failure;
@@ -464,10 +472,10 @@ async function assertRootProjectTrust(db: Db | DbTransaction, input: {
export async function assertExecutionWorkspaceRootWriteAccess(db: Db | DbTransaction, input: {
companyId: string; cwd: string; actor: AuthorizationActor; sourceWorkspaceId?: string; taskCreation?: TaskWorkspaceCreationTrustContext;
} & Omit<NonNullable<Parameters<typeof assertTaskWorkspaceSourceProjectAccess>[4]>, "write">): Promise<void> {
const { sourceIds, sourceProjectIds, sharedProjectIds } = await resolveRepositoryRootSources(db, { ...input, inspectCurrentAliases: true });
await assertTaskWorkspacesReadAccess(db, input.actor, input.companyId, sourceIds);
await assertRootProjectAccess(db, input.actor, input.companyId, sourceProjectIds, { ...input, sharedProjectIds });
await assertRootProjectTrust(db, { ...input, projectIds: sourceProjectIds });
const { sourceIds, sourceProjectIds, sharedProjectIds } = await measureRunPhase("workspace.sources", () => resolveRepositoryRootSources(db, { ...input, inspectCurrentAliases: true }));
await measureRunPhase("workspace.readers.sources", () => assertTaskWorkspacesReadAccess(db, input.actor, input.companyId, sourceIds));
await measureRunPhase("workspace.readers.projects", () => assertRootProjectAccess(db, input.actor, input.companyId, sourceProjectIds, { ...input, sharedProjectIds }));
await measureRunPhase("workspace.readers.trust", () => assertRootProjectTrust(db, { ...input, projectIds: sourceProjectIds }));
}
export function executionWorkspaceRepositoryService(db: Db) {
@@ -501,10 +509,10 @@ export function executionWorkspaceRepositoryService(db: Db) {
if (await requiresAnonymousAcquisition(db, { issue, workspace, agentId: input.agentId, runId: input.runId })) throw forbidden(controllerAcquisitionMessage);
}
async function reserveRootPolicy(tx: DbTransaction, companyId: string, cwd: string, mode: "anonymous_only" | "managed") {
const canonicalCwd = await canonicalRepositoryRoot(cwd);
const canonicalCwd = await measureRunPhase("workspace.sources.canonicalize", () => canonicalRepositoryRoot(cwd));
const rootKey = createHash("sha256").update(canonicalCwd).digest("hex");
const reservations = await tx.select().from(executionWorkspaceRepositoryRoots)
.where(eq(executionWorkspaceRepositoryRoots.companyId, companyId));
const reservations = await measureRunPhase("workspace.sources.inventory", async () => tx.select().from(executionWorkspaceRepositoryRoots)
.where(eq(executionWorkspaceRepositoryRoots.companyId, companyId)));
const existing = reservations.find(entry => entry.rootKey === rootKey);
const overlapping = reservations.filter(entry => repositoryRootsOverlap(entry.canonicalCwd, canonicalCwd));
if ((existing && existing.canonicalCwd !== canonicalCwd)
@@ -512,10 +520,10 @@ export function executionWorkspaceRepositoryService(db: Db) {
throw forbidden(lowTrustRepositoryMessage);
}
if (mode === "anonymous_only") {
const managed = await tx.select({ cwd: executionWorkspaces.cwd, repositoryIdentity: executionWorkspaceRepositories.repositoryIdentity }).from(executionWorkspaceRepositories)
const managed = await measureRunPhase("workspace.sources.inventory", async () => tx.select({ cwd: executionWorkspaces.cwd, repositoryIdentity: executionWorkspaceRepositories.repositoryIdentity }).from(executionWorkspaceRepositories)
.innerJoin(executionWorkspaces, eq(executionWorkspaces.id, executionWorkspaceRepositories.executionWorkspaceId))
.where(and(eq(executionWorkspaceRepositories.companyId, companyId), eq(executionWorkspaces.companyId, companyId),
isNotNull(executionWorkspaceRepositories.catalogRepositoryId)));
isNotNull(executionWorkspaceRepositories.catalogRepositoryId))));
if (managed.length) {
const namespace = path.join(canonicalCwd, ".paperclip-repositories");
const namespaceEntry = await fs.lstat(namespace).catch((error: NodeJS.ErrnoException) => {
@@ -539,79 +547,83 @@ export function executionWorkspaceRepositoryService(db: Db) {
if (peer.cwd && await legacyRepositoryRootMayMatch(peer.cwd, canonicalCwd)) throw forbidden(lowTrustRepositoryMessage);
}
}
if (!existing) await tx.insert(executionWorkspaceRepositoryRoots).values({ companyId, canonicalCwd, rootKey, mode });
if (!existing) await measureRunPhase("workspace.write", () => tx.insert(executionWorkspaceRepositoryRoots).values({ companyId, canonicalCwd, rootKey, mode }));
return canonicalCwd;
}
type RootReader = { runId: string; issueId: string; agentId: string };
async function authorizeRootAccess(tx: DbTransaction, input: {
companyId: string; cwd: string; sourceWorkspaceId?: string; reader?: RootReader; actor?: AuthorizationActor; issueId?: string; preserveOwnAdmission?: boolean;
}) {
const { canonicalCwd, rootKey, entries, overlappingEntries, sourceIds, sharedSourceIds, isolatedSourceIds, sourceProjectIds, sharedProjectIds, isolatedProjectIds } = await resolveRepositoryRootSources(tx, input);
const { canonicalCwd, rootKey, entries, overlappingEntries, sourceIds, sharedSourceIds, isolatedSourceIds, sourceProjectIds, sharedProjectIds, isolatedProjectIds } = await measureRunPhase("workspace.sources", () => resolveRepositoryRootSources(tx, input));
const readers = overlappingEntries.filter(entry => entry.kind === "reader");
if (input.reader && !entries.some(entry => entry.kind === "reader" && entry.scopeId === input.reader!.runId)) {
const [run] = await tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.id, input.reader.runId), eq(heartbeatRuns.companyId, input.companyId)));
const [run] = await measureRunPhase("workspace.sources.inventory", async () => tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.id, input.reader!.runId), eq(heartbeatRuns.companyId, input.companyId))));
readers.push({ id: randomUUID(), companyId: input.companyId, rootKey, canonicalCwd, kind: "reader", scopeId: input.reader.runId,
issueId: input.reader.issueId, agentId: input.reader.agentId, responsibleUserId: run?.responsibleUserId ?? null, createdAt: new Date(), updatedAt: new Date() });
}
const ownAdmissionOnly = input.preserveOwnAdmission && input.reader && input.sourceWorkspaceId
&& sourceIds.size === 1 && sourceIds.has(input.sourceWorkspaceId)
&& readers.every(reader => reader.issueId === input.reader!.issueId);
const sourceProjects = sourceIds.size ? await tx.selectDistinct({ projectId: executionWorkspaces.projectId }).from(executionWorkspaces).where(and(
eq(executionWorkspaces.companyId, input.companyId), inArray(executionWorkspaces.id, [...sourceIds]))) : [];
const sourceProjects = sourceIds.size ? await measureRunPhase("workspace.sources.inventory", async () => tx.selectDistinct({ projectId: executionWorkspaces.projectId }).from(executionWorkspaces).where(and(
eq(executionWorkspaces.companyId, input.companyId), inArray(executionWorkspaces.id, [...sourceIds])))) : [];
const sourceActors: AuthorizationActor[] = [];
const projectReaders: AuthorizationActor[] = [];
for (const reader of readers) {
const { actor, issue } = await retainedRootReader(tx, input.companyId, reader);
await assertRootProjectTrust(tx, { companyId: input.companyId, projectIds: sourceProjectIds, actor, issueId: issue.id,
sourceWorkspaceId: input.reader?.runId === reader.scopeId ? input.sourceWorkspaceId : undefined });
if (input.reader?.runId === reader.scopeId) {
await assertRootProjectAccess(tx, actor, input.companyId, sourceProjectIds, { sharedProjectIds, issueId: issue.id });
} else projectReaders.push(actor);
if (ownAdmissionOnly) {
const [ownSource] = await tx.select({ id: executionWorkspaces.id }).from(executionWorkspaces).where(and(
eq(executionWorkspaces.companyId, input.companyId), eq(executionWorkspaces.id, input.sourceWorkspaceId!),
await executionWorkspaceReadSqlCondition(tx, { ...actor, onBehalfOfUserId: null }),
actor.onBehalfOfUserId ? await executionWorkspaceReadSqlCondition(tx, {
type: "board", userId: actor.onBehalfOfUserId, ignoreInstanceAdmin: true,
}) : undefined));
if (!ownSource) throw forbidden("Task workspace access is no longer available");
} else sourceActors.push(actor);
for (const sourceWorkspace of sourceProjects.length ? sourceProjects : [null]) {
if (await requiresAnonymousAcquisition(tx, { issue, workspace: sourceWorkspace, agentId: reader.agentId!, runId: reader.scopeId })) {
await reserveRootPolicy(tx, input.companyId, canonicalCwd, "anonymous_only");
await measureRunPhase("workspace.readers", async () => {
for (const reader of readers) {
const { actor, issue } = await measureRunPhase("workspace.readers.identity", () => retainedRootReader(tx, input.companyId, reader));
await measureRunPhase("workspace.readers.trust", () => assertRootProjectTrust(tx, { companyId: input.companyId, projectIds: sourceProjectIds, actor, issueId: issue.id,
sourceWorkspaceId: input.reader?.runId === reader.scopeId ? input.sourceWorkspaceId : undefined }));
if (input.reader?.runId === reader.scopeId) {
await measureRunPhase("workspace.readers.projects", () => assertRootProjectAccess(tx, actor, input.companyId, sourceProjectIds, { sharedProjectIds, issueId: issue.id }));
} else projectReaders.push(actor);
if (ownAdmissionOnly) {
const [ownSource] = await measureRunPhase("workspace.sources.inventory", async () => tx.select({ id: executionWorkspaces.id }).from(executionWorkspaces).where(and(
eq(executionWorkspaces.companyId, input.companyId), eq(executionWorkspaces.id, input.sourceWorkspaceId!),
await executionWorkspaceReadSqlCondition(tx, { ...actor, onBehalfOfUserId: null }),
actor.onBehalfOfUserId ? await executionWorkspaceReadSqlCondition(tx, {
type: "board", userId: actor.onBehalfOfUserId, ignoreInstanceAdmin: true,
}) : undefined)));
if (!ownSource) throw forbidden("Task workspace access is no longer available");
} else sourceActors.push(actor);
for (const sourceWorkspace of sourceProjects.length ? sourceProjects : [null]) {
if (await requiresAnonymousAcquisition(tx, { issue, workspace: sourceWorkspace, agentId: reader.agentId!, runId: reader.scopeId })) {
await reserveRootPolicy(tx, input.companyId, canonicalCwd, "anonymous_only");
}
}
}
}
await assertRootProjectsReadAccessForActors(tx, projectReaders, input.companyId, sourceProjectIds);
await measureRunPhase("workspace.readers.projects", () => assertRootProjectsReadAccessForActors(tx, projectReaders, input.companyId, sourceProjectIds));
});
if (input.actor) {
if (!input.issueId) throw forbidden("Workspace write authorization requires a task");
await assertRootProjectTrust(tx, { ...input, projectIds: sourceProjectIds, actor: input.actor });
await assertRootProjectAccess(tx, input.actor, input.companyId, sourceProjectIds, { sharedProjectIds, issueId: input.issueId });
await measureRunPhase("workspace.readers.trust", () => assertRootProjectTrust(tx, { ...input, projectIds: sourceProjectIds, actor: input.actor! }));
await measureRunPhase("workspace.readers.projects", () => assertRootProjectAccess(tx, input.actor!, input.companyId, sourceProjectIds, { sharedProjectIds, issueId: input.issueId }));
sourceActors.push(input.actor);
}
await assertTaskWorkspacesReadAccessForActors(tx, sourceActors, input.companyId, sourceIds);
await measureRunPhase("workspace.readers.sources", () => assertTaskWorkspacesReadAccessForActors(tx, sourceActors, input.companyId, sourceIds));
for (const [kind, ids] of [["source", sharedSourceIds], ["isolated_source", isolatedSourceIds], ["project_source", sourceProjectIds],
["shared_project_source", sharedProjectIds], ["isolated_project_source", isolatedProjectIds]] as const) {
for (const scopeId of ids) {
if (!entries.some(entry => entry.kind === kind && entry.scopeId === scopeId)) {
await tx.insert(executionWorkspaceRepositoryAccess).values({ companyId: input.companyId, rootKey, canonicalCwd, kind, scopeId });
await measureRunPhase("workspace.write", () => tx.insert(executionWorkspaceRepositoryAccess).values({ companyId: input.companyId, rootKey, canonicalCwd, kind, scopeId }));
}
}
}
for (const reader of readers) {
if (reader.rootKey === rootKey && !entries.some(entry => entry.kind === "reader" && entry.scopeId === reader.scopeId)) {
await tx.insert(executionWorkspaceRepositoryAccess).values(reader);
await measureRunPhase("workspace.readers", async () => {
for (const reader of readers) {
if (reader.rootKey === rootKey && !entries.some(entry => entry.kind === "reader" && entry.scopeId === reader.scopeId)) {
await measureRunPhase("workspace.write", () => tx.insert(executionWorkspaceRepositoryAccess).values(reader));
}
}
}
});
}
async function reserveRootAccess(input: Parameters<typeof authorizeRootAccess>[1]) {
return db.transaction(async tx => {
return measureRunTransaction(db, "workspace", async tx => {
await lockExecutionWorkspaceRootProvenance(tx, input.companyId);
await authorizeRootAccess(tx, input);
});
}
async function reserveAdmissionRoot(companyId: string, cwd: string, mode: "anonymous_only" | "managed") {
return db.transaction(async tx => {
return measureRunTransaction(db, "workspace", async tx => {
await lockExecutionWorkspaceRootProvenance(tx, companyId);
return reserveRootPolicy(tx, companyId, cwd, mode);
});
@@ -637,16 +649,16 @@ export function executionWorkspaceRepositoryService(db: Db) {
}
const available = await catalog(input.companyId, userId, localTrusted);
const requestedRef = parsed.ref ?? "HEAD";
const receipt = await db.transaction(async tx => {
const receipt = await measureRunTransaction(db, "workspace", async tx => {
await lockExecutionWorkspaceRootProvenance(tx, input.companyId);
const [issue] = await tx.select().from(issues).where(and(eq(issues.id, input.issueId), eq(issues.companyId, input.companyId), await issueReadSqlCondition(tx, input.actor))).for("no key update");
const [issue] = await measureRunPhase("workspace.sources.inventory", async () => tx.select().from(issues).where(and(eq(issues.id, input.issueId), eq(issues.companyId, input.companyId), await issueReadSqlCondition(tx, input.actor))).for("no key update"));
if (!issue) throw notFound("Task not found");
const currentDecision = await authorizationService(tx).decide({ actor: input.actor, action: "issue:mutate", resource: issueResource(issue) });
if (!currentDecision.allowed) throw forbidden("Task workspace modification is not allowed");
if (input.actor.type === "agent" && !["standard", "skill_test"].includes(issue.workMode)) throw forbidden("Repository preparation is unavailable in Ask or Plan mode");
if (!issue.executionWorkspaceId) throw conflict("The task has no admitted workspace yet. Start it once before preparing a repository.");
if (!(await canActorReadExecutionWorkspace(db, input.actor, issue.executionWorkspaceId))) throw notFound("Execution workspace not found");
const [workspace] = await tx.select().from(executionWorkspaces).where(and(eq(executionWorkspaces.id, issue.executionWorkspaceId), eq(executionWorkspaces.companyId, input.companyId), taskWorkspaceSelectableCondition())).for("no key update");
const [workspace] = await measureRunPhase("workspace.sources.inventory", async () => tx.select().from(executionWorkspaces).where(and(eq(executionWorkspaces.id, issue.executionWorkspaceId!), eq(executionWorkspaces.companyId, input.companyId), taskWorkspaceSelectableCondition())).for("no key update"));
if (!workspace) throw conflict("Execution workspace is unavailable");
await assertTaskWorkspaceAccess(tx, input.actor, input.companyId, workspace.id, { write: true, issueId: input.issueId });
const anonymousOnly = await requiresAnonymousAcquisition(tx, { issue, workspace,
@@ -659,7 +671,7 @@ export function executionWorkspaceRepositoryService(db: Db) {
if (!workspace.cwd) throw conflict("The task workspace has no directory for repository preparation");
await authorizeRootAccess(tx, { companyId: input.companyId, cwd: workspace.cwd, sourceWorkspaceId: workspace.id, actor: input.actor, issueId: input.issueId });
if (anonymousOnly) throw forbidden(controllerAcquisitionMessage);
const existing = await tx.select().from(executionWorkspaceRepositories).where(and(eq(executionWorkspaceRepositories.executionWorkspaceId, workspace.id), or(eq(executionWorkspaceRepositories.repositoryIdentity, source.repositoryIdentity), sql`${executionWorkspaceRepositories.requestKeys} ? ${parsed.requestKey}`)));
const existing = await measureRunPhase("workspace.sources.inventory", async () => tx.select().from(executionWorkspaceRepositories).where(and(eq(executionWorkspaceRepositories.executionWorkspaceId, workspace.id), or(eq(executionWorkspaceRepositories.repositoryIdentity, source.repositoryIdentity), sql`${executionWorkspaceRepositories.requestKeys} ? ${parsed.requestKey}`))));
for (const row of existing) {
assertAcquisitionReceipt(row, anonymousOnly);
assertRepositoryRetryMatches(row, source.repositoryIdentity, requestedRef);
@@ -672,11 +684,11 @@ export function executionWorkspaceRepositoryService(db: Db) {
}
return row;
}
const [row] = await tx.insert(executionWorkspaceRepositories).values({ companyId: input.companyId, executionWorkspaceId: workspace.id, requestedByIssueId: input.issueId, ...source, relativePath: taskRepositoryRelativePath(source.repositoryIdentity), requestedRef, requestKey: parsed.requestKey, requestKeys: [parsed.requestKey] }).returning();
const canonicalCwd = await canonicalRepositoryRoot(workspace.cwd);
await tx.insert(executionWorkspaceRepositoryAccess).values({ companyId: input.companyId,
rootKey: createHash("sha256").update(canonicalCwd).digest("hex"), canonicalCwd, kind: "repository", scopeId: row.id });
await tx.insert(activityLog).values({ companyId: input.companyId, actorType: input.actor.type === "agent" ? "agent" : "user", actorId: input.actor.agentId ?? input.actor.userId ?? "local-board", action: "workspace.repository.requested", entityType: "execution_workspace", entityId: workspace.id, runId: input.actor.runId ?? null, responsibleUserId: userId, details: { repositoryId: row.id, issueId: input.issueId, applies: "next_normal_admission" } });
const [row] = await measureRunPhase("workspace.write", () => tx.insert(executionWorkspaceRepositories).values({ companyId: input.companyId, executionWorkspaceId: workspace.id, requestedByIssueId: input.issueId, ...source, relativePath: taskRepositoryRelativePath(source.repositoryIdentity), requestedRef, requestKey: parsed.requestKey, requestKeys: [parsed.requestKey] }).returning());
const canonicalCwd = await measureRunPhase("workspace.sources.canonicalize", () => canonicalRepositoryRoot(workspace.cwd!));
await measureRunPhase("workspace.write", () => tx.insert(executionWorkspaceRepositoryAccess).values({ companyId: input.companyId,
rootKey: createHash("sha256").update(canonicalCwd).digest("hex"), canonicalCwd, kind: "repository", scopeId: row.id }));
await measureRunPhase("workspace.write", () => tx.insert(activityLog).values({ companyId: input.companyId, actorType: input.actor.type === "agent" ? "agent" : "user", actorId: input.actor.agentId ?? input.actor.userId ?? "local-board", action: "workspace.repository.requested", entityType: "execution_workspace", entityId: workspace.id, runId: input.actor.runId ?? null, responsibleUserId: userId, details: { repositoryId: row.id, issueId: input.issueId, applies: "next_normal_admission" } }));
return row;
});
return { kind: "requires_next_admission" as const, applies: "next_normal_admission" as const, operationId: receipt.id, repository: receipt, reason: "immutable_admitted_workspace" };
@@ -686,24 +698,24 @@ export function executionWorkspaceRepositoryService(db: Db) {
}
type PublicationInput = { companyId: string; issueId: string; workspaceId: string; cwd: string; agentId: string; runId: string };
async function publicationBinding(tx: DbTransaction, input: PublicationInput) {
const [issue] = await tx.select().from(issues).where(and(eq(issues.id, input.issueId), eq(issues.companyId, input.companyId)));
const [workspace] = await tx.select().from(executionWorkspaces).where(and(eq(executionWorkspaces.id, input.workspaceId), eq(executionWorkspaces.companyId, input.companyId)));
const [issue] = await measureRunPhase("workspace.sources.inventory", async () => tx.select().from(issues).where(and(eq(issues.id, input.issueId), eq(issues.companyId, input.companyId))));
const [workspace] = await measureRunPhase("workspace.sources.inventory", async () => tx.select().from(executionWorkspaces).where(and(eq(executionWorkspaces.id, input.workspaceId), eq(executionWorkspaces.companyId, input.companyId))));
if (!issue || !workspace || issue.executionWorkspaceId !== workspace.id || !workspace.cwd
|| await canonicalRepositoryRoot(workspace.cwd) !== await canonicalRepositoryRoot(input.cwd)) {
|| await measureRunPhase("workspace.sources.canonicalize", () => canonicalRepositoryRoot(workspace.cwd!)) !== await measureRunPhase("workspace.sources.canonicalize", () => canonicalRepositoryRoot(input.cwd))) {
throw forbidden("Task repository publication binding is unavailable");
}
await requiresAnonymousAcquisition(tx, { issue, workspace, agentId: input.agentId, runId: input.runId });
const [run] = await tx.select({ responsibleUserId: heartbeatRuns.responsibleUserId }).from(heartbeatRuns).where(and(eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId)));
const { actor } = await retainedRootReader(tx, input.companyId, {
const [run] = await measureRunPhase("workspace.sources.inventory", async () => tx.select({ responsibleUserId: heartbeatRuns.responsibleUserId }).from(heartbeatRuns).where(and(eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId))));
const { actor } = await measureRunPhase("workspace.readers.identity", () => retainedRootReader(tx, input.companyId, {
scopeId: input.runId, issueId: input.issueId, agentId: input.agentId, responsibleUserId: run?.responsibleUserId ?? null,
});
}));
return { workspace, actor };
}
async function guardForPublication(input: PublicationInput) {
await db.transaction(async tx => {
await measureRunTransaction(db, "workspace", async tx => {
await lockExecutionWorkspaceRootProvenance(tx, input.companyId);
const { workspace } = await publicationBinding(tx, input);
await authorizeRootAccess(tx, { companyId: input.companyId, cwd: input.cwd, sourceWorkspaceId: workspace.id, reader: input });
await authorizeRootAccess(tx, { companyId: input.companyId, cwd: input.cwd!, sourceWorkspaceId: workspace.id, reader: input });
});
}
async function prepareConfiguredRepositorySource(input: PublicationInput & { sourceCwd: string }): Promise<() => Promise<void>> {
@@ -717,35 +729,35 @@ export function executionWorkspaceRepositoryService(db: Db) {
commonRoot = source.commonRoot;
for (const cwd of source.roots) roots.add(cwd);
} catch (error) { sourceFailure = error; }
const denied = await db.transaction(async tx => {
const denied = await measureRunTransaction(db, "workspace", async tx => {
await lockExecutionWorkspaceRootProvenance(tx, input.companyId);
const { workspace, actor } = await publicationBinding(tx, input);
const canonicalCwd = await canonicalRepositoryRoot(input.cwd);
const canonicalCwd = await measureRunPhase("workspace.sources.canonicalize", () => canonicalRepositoryRoot(input.cwd));
const rootKey = createHash("sha256").update(canonicalCwd).digest("hex");
const existing = await tx.select().from(executionWorkspaceRepositoryAccess).where(and(
eq(executionWorkspaceRepositoryAccess.companyId, input.companyId), eq(executionWorkspaceRepositoryAccess.rootKey, rootKey)));
const existing = await measureRunPhase("workspace.sources.inventory", async () => tx.select().from(executionWorkspaceRepositoryAccess).where(and(
eq(executionWorkspaceRepositoryAccess.companyId, input.companyId), eq(executionWorkspaceRepositoryAccess.rootKey, rootKey))));
if (existing.some(entry => entry.canonicalCwd !== canonicalCwd)) throw forbidden("Repository root identity conflicts with its retained access record");
let failure = sourceFailure;
for (const cwd of roots) {
try {
const source = await resolveRepositoryRootSources(tx, { companyId: input.companyId, cwd });
const source = await measureRunPhase("workspace.sources", () => resolveRepositoryRootSources(tx, { companyId: input.companyId, cwd }));
for (const id of source.sourceIds) workspaceIds.add(id);
for (const id of source.sourceProjectIds) projectIds.add(id);
} catch (error) { failure ??= error; }
}
try { await assertTaskWorkspacesReadAccess(tx, actor, input.companyId, workspaceIds); }
try { await measureRunPhase("workspace.readers.sources", () => assertTaskWorkspacesReadAccess(tx, actor, input.companyId, workspaceIds)); }
catch (error) { failure ??= error; }
for (const id of projectIds) {
try { await assertRootProjectAccess(tx, actor, input.companyId, new Set([id])); }
try { await measureRunPhase("workspace.readers.projects", () => assertRootProjectAccess(tx, actor, input.companyId, new Set([id]))); }
catch (error) { failure ??= error; }
}
for (const [kind, ids] of [["isolated_source", workspaceIds], ["isolated_project_source", projectIds]] as const) {
for (const scopeId of ids) if (!existing.some(entry => entry.kind === kind && entry.scopeId === scopeId)) {
await tx.insert(executionWorkspaceRepositoryAccess).values({ companyId: input.companyId, rootKey, canonicalCwd, kind, scopeId });
await measureRunPhase("workspace.write", () => tx.insert(executionWorkspaceRepositoryAccess).values({ companyId: input.companyId, rootKey, canonicalCwd, kind, scopeId }));
}
}
try {
await authorizeRootAccess(tx, { companyId: input.companyId, cwd: input.cwd, sourceWorkspaceId: workspace.id,
await authorizeRootAccess(tx, { companyId: input.companyId, cwd: input.cwd!, sourceWorkspaceId: workspace.id,
actor, issueId: input.issueId });
} catch (error) { failure ??= error; }
if (!failure && commonRoot) {
+33 -31
View File
@@ -1,3 +1,5 @@
import { measureRunTransaction } from "./execution-transaction-timing.js";
import { measureRunPhase } from "./execution-preparation-timing.js";
import { assertTaskWorkspaceAccess, assertTaskWorkspaceSourceProjectAccess } from "./task-workspace-source-access.js";
import { verifiedRecordedIsolatedRemoteSource } from "./isolated-remote-workspace-source.js";
import { parseObject } from "../adapters/utils.js";
@@ -251,9 +253,9 @@ async function preserveWorkspaceOwnershipForPatch(
tx: DbTransaction, current: typeof executionWorkspaces.$inferSelect, patch: Partial<typeof executionWorkspaces.$inferInsert>,
) {
if (workspaceSourceKeys.some(key => patch[key] !== undefined && patch[key] !== current[key])) {
await preserveExecutionWorkspaceRootProvenance(tx, {
await measureRunPhase("workspace.provenance", () => preserveExecutionWorkspaceRootProvenance(tx, {
companyId: current.companyId, workspaceId: current.id, cwd: current.cwd,
});
}));
}
const nextPatch = { ...patch };
const retainedSources = {
@@ -1531,8 +1533,8 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
async function bindTaskWorkspace(companyId: string, issueId: string, workspaceId: string, patch: TaskWorkspaceBindingPatch = {}) {
return db.transaction(async tx => {
const [task] = await tx.select().from(issues).where(and(eq(issues.id, issueId), eq(issues.companyId, companyId))).for("no key update");
const [workspace] = await tx.select().from(executionWorkspaces).where(and(eq(executionWorkspaces.id, workspaceId), eq(executionWorkspaces.companyId, companyId))).for("no key update");
const [task] = await measureRunPhase("workspace.row_lock", () => tx.select().from(issues).where(and(eq(issues.id, issueId), eq(issues.companyId, companyId))).for("no key update"));
const [workspace] = await measureRunPhase("workspace.row_lock", () => tx.select().from(executionWorkspaces).where(and(eq(executionWorkspaces.id, workspaceId), eq(executionWorkspaces.companyId, companyId))).for("no key update"));
if (!task || !workspace) throw notFound("Task workspace unavailable");
await tx.update(executionWorkspaces).set({ metadata: { ...workspace.metadata,
_issuePrivacySources: { ...(workspace.metadata?._issuePrivacySources as Record<string, unknown> ?? {}), [task.id]: true } } }).where(eq(executionWorkspaces.id, workspaceId));
@@ -1572,9 +1574,9 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
async function prepareTaskWorkspaceUpdate(input: {
companyId: string; issueId: string; patch: TaskWorkspaceBindingPatch;
}, tx: DbTransaction) {
const [task] = await tx.select().from(issues).where(and(
const [task] = await measureRunPhase("workspace.row_lock", () => tx.select().from(issues).where(and(
eq(issues.id, input.issueId), eq(issues.companyId, input.companyId),
)).for("no key update");
)).for("no key update"));
if (!task) throw notFound("Task not found");
const fields = ["executionWorkspaceId", "projectWorkspaceId", "executionWorkspacePreference", "executionWorkspaceSettings"] as const;
if (!fields.some(field => input.patch[field] !== undefined && !isDeepStrictEqual(input.patch[field], task[field]))) return {};
@@ -1929,7 +1931,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
}
return input.onSkip();
}
await notifyTerminalWorkspaceWork(tx, `workspace:${input.workspaceId}`);
await measureRunPhase("workspace.notification", () => notifyTerminalWorkspaceWork(tx, `workspace:${input.workspaceId}`));
return input.write({ tx, fresh });
});
}
@@ -3582,36 +3584,36 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
},
create: async (data: typeof executionWorkspaces.$inferInsert) => {
const row = await db.transaction(async tx => {
await notifyTerminalWorkspaceWork(tx, data.id ? `workspace:${data.id}`
: data.sourceIssueId ? `issue:${data.sourceIssueId}` : undefined);
return tx.insert(executionWorkspaces).values(data).returning().then(rows => rows[0] ?? null);
const row = await measureRunTransaction(db, "workspace", async tx => {
await measureRunPhase("workspace.notification", () => notifyTerminalWorkspaceWork(tx, data.id ? `workspace:${data.id}`
: data.sourceIssueId ? `issue:${data.sourceIssueId}` : undefined));
return measureRunPhase("workspace.write", () => tx.insert(executionWorkspaces).values(data).returning().then(rows => rows[0] ?? null));
});
return row ? toExecutionWorkspace(row) : null;
},
update: async (id: string, patch: Partial<typeof executionWorkspaces.$inferInsert>) => {
if (patch.metadata === undefined && !workspaceSourceKeys.some(key => patch[key] !== undefined)) {
const row = await db.transaction(async tx => {
await notifyTerminalWorkspaceWork(tx, `workspace:${id}`);
return tx.update(executionWorkspaces).set({ ...patch, updatedAt: new Date() })
.where(eq(executionWorkspaces.id, id)).returning().then(rows => rows[0] ?? null);
const row = await measureRunTransaction(db, "workspace", async tx => {
await measureRunPhase("workspace.notification", () => notifyTerminalWorkspaceWork(tx, `workspace:${id}`));
return measureRunPhase("workspace.write", () => tx.update(executionWorkspaces).set({ ...patch, updatedAt: new Date() })
.where(eq(executionWorkspaces.id, id)).returning().then(rows => rows[0] ?? null));
});
return row ? toExecutionWorkspace(row) : null;
}
const [identity] = await db.select({ companyId: executionWorkspaces.companyId }).from(executionWorkspaces)
.where(eq(executionWorkspaces.id, id));
const [identity] = await measureRunPhase("workspace.identity_read", () => db.select({ companyId: executionWorkspaces.companyId }).from(executionWorkspaces)
.where(eq(executionWorkspaces.id, id)));
if (!identity) return null;
return db.transaction(async tx => {
return measureRunTransaction(db, "workspace", async tx => {
await lockExecutionWorkspaceRootProvenance(tx, identity.companyId);
const [current] = await tx.select().from(executionWorkspaces).where(and(
const [current] = await measureRunPhase("workspace.row_lock", () => tx.select().from(executionWorkspaces).where(and(
eq(executionWorkspaces.id, id), eq(executionWorkspaces.companyId, identity.companyId),
)).for("no key update");
)).for("no key update"));
if (!current) return null;
const nextPatch = await preserveWorkspaceOwnershipForPatch(tx, current, patch);
await notifyTerminalWorkspaceWork(tx, `workspace:${id}`);
const [row] = await tx.update(executionWorkspaces).set({ ...nextPatch, updatedAt: new Date() })
.where(and(eq(executionWorkspaces.id, id), eq(executionWorkspaces.companyId, current.companyId))).returning();
const nextPatch = await measureRunPhase("workspace.provenance", () => preserveWorkspaceOwnershipForPatch(tx, current, patch));
await measureRunPhase("workspace.notification", () => notifyTerminalWorkspaceWork(tx, `workspace:${id}`));
const [row] = await measureRunPhase("workspace.write", () => tx.update(executionWorkspaces).set({ ...nextPatch, updatedAt: new Date() })
.where(and(eq(executionWorkspaces.id, id), eq(executionWorkspaces.companyId, current.companyId))).returning());
return row ? toExecutionWorkspace(row) : null;
});
},
@@ -3866,7 +3868,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
// generation so a queued cleanup that captured the old generation does
// nothing. Clear cleanupEligibleAt so the reaper does not destroy the
// half-built worktree while a later reopen retries.
await notifyTerminalWorkspaceWork(tx, `workspace:${row.id}`);
await measureRunPhase("workspace.notification", () => notifyTerminalWorkspaceWork(tx, `workspace:${row.id}`));
await tx
.update(executionWorkspaces)
.set({
@@ -3917,7 +3919,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
// flag stops the terminal reaper and the archive route from archiving and
// destroying the rebuilt worktree in that window.
const activeMetadata = setMetadataReopenPendingConsumption(nextMetadata, now());
await notifyTerminalWorkspaceWork(tx, `workspace:${row.id}`);
await measureRunPhase("workspace.notification", () => notifyTerminalWorkspaceWork(tx, `workspace:${row.id}`));
const activeRow = await tx
.update(executionWorkspaces)
.set({
@@ -4085,12 +4087,12 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
return db.transaction(async (tx) => {
await lockExecutionWorkspaceRootProvenance(tx, identity.companyId);
await acquireExecutionWorkspaceLifecycleLock(tx, input.id);
const fresh = await tx
const fresh = await measureRunPhase("workspace.row_lock", () => tx
.select()
.from(executionWorkspaces)
.where(and(eq(executionWorkspaces.id, input.id), eq(executionWorkspaces.companyId, identity.companyId)))
.for("no key update")
.then((rows) => rows[0] ?? null);
.then((rows) => rows[0] ?? null));
if (!fresh || isClosedExecutionWorkspaceStatus(fresh.status)) {
// The row is missing or already closed by a concurrent path. Do not
// archive again.
@@ -4103,14 +4105,14 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
// rebuilt worktree. The route maps this to HTTP 409.
return { outcome: "reopen_pending" };
}
const nextPatch = await preserveWorkspaceOwnershipForPatch(tx, fresh, input.patch);
const nextPatch = await measureRunPhase("workspace.provenance", () => preserveWorkspaceOwnershipForPatch(tx, fresh, input.patch));
const baseMetadata =
(nextPatch.metadata as Record<string, unknown> | null | undefined)
?? (fresh.metadata as Record<string, unknown> | null);
const archiveMetadata = clearMetadataReopenPendingConsumption(
bumpExecutionWorkspaceLifecycleGeneration(baseMetadata),
);
await notifyTerminalWorkspaceWork(tx, `workspace:${input.id}`);
await measureRunPhase("workspace.notification", () => notifyTerminalWorkspaceWork(tx, `workspace:${input.id}`));
const archived = await tx
.update(executionWorkspaces)
.set({
@@ -4375,7 +4377,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
updatePatch.name = inspection.toBranch;
}
await notifyTerminalWorkspaceWork(tx, `workspace:${lockedWorkspace.id}`);
await measureRunPhase("workspace.notification", () => notifyTerminalWorkspaceWork(tx, `workspace:${lockedWorkspace.id}`));
const [branchUpdatedRow] = await tx
.update(executionWorkspaces)
.set(updatePatch)
+13 -5
View File
@@ -1,3 +1,4 @@
import { withRunExecutionTiming, measureRunPhase, measureWorkspaceSyncPhase } from "./execution-preparation-timing.js";
import { createHeartbeatPreparationTiming } from "./heartbeat/preparation-timing.js";
import { createHeartbeatInstructionCopyRelease, createHeartbeatInstructionWorkingCopy } from "./heartbeat/instruction-working-copy.js";
import { prepareHeartbeatRuntimeServiceIntents, startHeartbeatRuntimeServices, recordHeartbeatAdapterRuntimeServices } from "./heartbeat/runtime-services.js";
@@ -1795,6 +1796,13 @@ export function heartbeatService(
nativeLeaseOwner?: string;
nativeRestartRecovery?: NativeRestartRecoveryClaim;
} = {},
) {
return withRunExecutionTiming(runId, () => executeRunAttempt(runId, runOptions));
}
async function executeRunAttempt(
runId: string,
runOptions: { nativeLeaseOwner?: string; nativeRestartRecovery?: NativeRestartRecoveryClaim },
) {
const attemptStartedAtMs = Date.now();
let attestedQuestionResponseAtMs: number | null = null;
@@ -3304,7 +3312,7 @@ export function heartbeatService(
},
onPhase: async (name, work) => {
const started = Date.now();
try { return await work(); }
try { return await measureWorkspaceSyncPhase(name, work); }
finally { await appendRunEvent(run, { eventType: "lifecycle", stream: "system", level: "info",
message: name, payload: { phase: name, durationMs: Date.now() - started } }); }
},
@@ -3339,7 +3347,7 @@ export function heartbeatService(
if (!runtimeSelection.selected) return;
const { nativeExecution, nativeRunnerInstanceId, getNativeFreshSessionHandoff } = runtimeSelection;
nativeWorkspaceSync = runtimeSelection.nativeWorkspaceSync;
const execution = await executeHeartbeatRuntime(db, {
const execution = await measureRunPhase("execution.runtime", () => executeHeartbeatRuntime(db, {
run, agent, options,
task: {
issueRef,
@@ -3436,7 +3444,7 @@ export function heartbeatService(
onProviderResourceDisposition: (value) => { providerResourceDispositionForRun = value; },
onWorkspaceRestoreFailure: (evidence) => { requiredWorkspaceRestoreEvidence = evidence; },
},
});
}));
if (!execution.dispatched) return;
const { adapterResult } = execution;
// Reconcile the referenced-project set against the real remote staging outcome. A referenced
@@ -3479,7 +3487,7 @@ export function heartbeatService(
run, agent, issueId, issueRef, executionWorkspace, context, issuesSvc, onLog,
adapterResult, runtimeServices,
});
await runCompletion.completeRun({
await measureRunPhase("completion.run", () => runCompletion.completeRun({
run, agent, issueId, issueRef, signal: executionControl.controller.signal,
output: runOutput,
session: { runtimeForAdapter, previousSessionParams, taskKey, configuredModel, sessionConfigMetadata },
@@ -3489,7 +3497,7 @@ export function heartbeatService(
providerTraceCapture: providerTraceCapture !== null,
onProviderTraceFinalized: () => { providerTraceFinalized = true; },
issueContext, onLog,
});
}));
} catch (err) {
await persistUsageCaptureFailure?.();
if (err instanceof NativeControllerDetachedForRestartError) {
+29 -28
View File
@@ -1,3 +1,4 @@
import { measureRunPhase } from "../execution-preparation-timing.js";
import { NON_RETRYABLE_PREFLIGHT_FAILURE_CODES, isHeartbeatRunTerminalStatus } from "./run-lifecycle.js";
import { ChatControlRecoveryUnresolvedError } from "./queue.js";
import { readTransientRecoveryContractFromRun, readHeartbeatRunErrorFamily, isTransientWorkspaceGitScanCode } from "./retries.js";
@@ -465,7 +466,7 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC
compressed: boolean;
} | null = null;
if (handle) {
logSummary = await runLogStore.finalize(handle);
logSummary = await measureRunPhase("completion.output", () => runLogStore.finalize(handle));
}
const finalLogBytes = logSummary?.bytes;
if (outputProgressState.pending && typeof finalLogBytes === "number") {
@@ -599,12 +600,12 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC
logSha256: logSummary?.sha256,
logCompressed: logSummary?.compressed ?? false,
};
const persistedRunWrite = await setRunStatusIfRunning(
const persistedRunWrite = await measureRunPhase("completion.terminal_write", () => setRunStatusIfRunning(
run.id,
status,
finalRunPatch,
{ adapterErrorMeta: adapterResult.errorMeta, secretValues: readFailureReportSecrets() },
);
));
let persistedRun: typeof heartbeatRuns.$inferSelect | null =
persistedRunWrite.run;
if (!persistedRunWrite.updated) {
@@ -645,7 +646,7 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC
.then((rows) => rows[0] ?? null);
}
if (!persistedRun) {
await accountRunCost(db, run.id, budgetHooks).catch((err) => {
await measureRunPhase("completion.cost", () => accountRunCost(db, run.id, budgetHooks)).catch((err) => {
logger.error({ err, runId: run.id }, "Late run accounting queued for recovery");
});
logger.info(
@@ -661,14 +662,14 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC
}
if (persistedRun) {
// Accounting recovery is independent of workspace or issue finalization.
await accountRunCost(db, persistedRun.id, budgetHooks).catch((err) => {
await measureRunPhase("completion.cost", () => accountRunCost(db, persistedRun!.id, budgetHooks)).catch((err) => {
logger.error({ err, runId: run.id }, "Run accounting queued for recovery");
});
persistedRun =
(await classifyAndPersistRunLiveness(
persistedRun,
(await measureRunPhase("completion.liveness", () => classifyAndPersistRunLiveness(
persistedRun!,
persistedResultJson,
)) ?? persistedRun;
))) ?? persistedRun;
}
await setWakeupStatus(
@@ -712,25 +713,25 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC
);
}
const livenessRun = finalizedRun;
await refreshContinuationSummaryForRun(livenessRun, agent);
await measureRunPhase("completion.summary", () => refreshContinuationSummaryForRun(livenessRun, agent));
const skipRunIssueComment =
parseObject(livenessRun.contextSnapshot).skipIssueComment === true;
let resolvedPresentationDecision: RunPresentationDecision | null =
null;
try {
const existingRunComment = issueId
? await findRunIssueComment(
? await measureRunPhase("response.resolve", () => findRunIssueComment(
livenessRun.id,
livenessRun.companyId,
issueId,
persistedResultJson,
)
))
: null;
const finalAgentMessage =
await findLatestCompletedFinalAgentMessage(
await measureRunPhase("response.resolve", () => findLatestCompletedFinalAgentMessage(
livenessRun.id,
livenessRun.companyId,
);
));
const externalChatPresentationCandidate =
isExternalChatPresentationContext(livenessRun.contextSnapshot) ||
parseObject(livenessRun.contextSnapshot).source === "tool_action_review" ||
@@ -738,11 +739,11 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC
parseObject(livenessRun.contextSnapshot).source === "issue.update";
const externalChatPresentationAuthorization =
issueId && externalChatPresentationCandidate
? await resolveChatRunPresentationAuthorizationReason(db, {
? await measureRunPhase("response.authorize", () => resolveChatRunPresentationAuthorizationReason(db, {
companyId: livenessRun.companyId,
issueId,
runId: livenessRun.id,
})
}))
: null;
const externalChatPresentationContext = isExternalChatPresentationContext(
livenessRun.contextSnapshot,
@@ -781,17 +782,17 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC
// authorize that narrow presentation as the provider reply;
// ordinary internal runs retain the private default.
const presentationAuthorizationReason =
await resolveChatRunPresentationAuthorizationReason(db, {
await measureRunPhase("response.authorize", () => resolveChatRunPresentationAuthorizationReason(db, {
companyId: livenessRun.companyId,
issueId,
runId: livenessRun.id,
});
const comment = await issuesSvc.addComment(
issueId,
resolved.text,
}));
const comment = await measureRunPhase("response.add_comment", () => issuesSvc.addComment(
issueId!,
resolved.text!,
{ agentId: agent.id, runId: livenessRun.id },
{ authorizationReason: presentationAuthorizationReason, completionReply: true },
);
));
presentationDecision = {
...presentationDecision,
commentId: comment.id,
@@ -800,7 +801,7 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC
"resolved_response_materialized",
],
};
await logActivity(db, {
await measureRunPhase("response.activity", () => logActivity(db, {
companyId: livenessRun.companyId,
actorType: "agent",
actorId: agent.id,
@@ -819,7 +820,7 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC
source: "run_presentation_resolver",
presentationSource: presentationDecision.chosenSource,
},
});
}));
} else if (presentationDecision.commentAction === "create") {
presentationDecision = {
...presentationDecision,
@@ -1092,7 +1093,7 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC
} | null = null;
if (handle) {
try {
logSummary = await runLogStore.finalize(handle);
logSummary = await measureRunPhase("completion.output", () => runLogStore.finalize(handle));
} catch (finalizeErr) {
logger.warn(
{ err: finalizeErr, runId },
@@ -1114,7 +1115,7 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC
const stoppedDuringFailure = signal.aborted;
const stopSnapshot = stoppedDuringFailure ? await getRun(run.id) : null;
const failureOutcome = stoppedDuringFailure ? "cancelled" : "failed";
const failedRunWrite = await setRunStatusIfRunning(run.id, failureOutcome, {
const failedRunWrite = await measureRunPhase("completion.terminal_write", () => setRunStatusIfRunning(run.id, failureOutcome, {
error: message,
errorCode: stopSnapshot?.errorCode ?? failureErrorCode,
finishedAt: new Date(),
@@ -1143,7 +1144,7 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC
logBytes: logSummary?.bytes,
logSha256: logSummary?.sha256,
logCompressed: logSummary?.compressed ?? false,
}, { error: err, phase: "execute", secretValues: readFailureReportSecrets() });
}, { error: err, phase: "execute", secretValues: readFailureReportSecrets() }));
if (
!failedRunWrite.updated &&
!(
@@ -1175,7 +1176,7 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC
message,
});
const livenessRun =
(await classifyAndPersistRunLiveness(failedRun)) ?? failedRun;
(await measureRunPhase("completion.liveness", () => classifyAndPersistRunLiveness(failedRun))) ?? failedRun;
try {
await completeSkillTestRunForHeartbeatOutcome({
run: livenessRun,
@@ -1190,7 +1191,7 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC
"failed to complete skill test run after heartbeat adapter failure",
);
}
await refreshContinuationSummaryForRun(livenessRun, agent);
await measureRunPhase("completion.summary", () => refreshContinuationSummaryForRun(livenessRun, agent));
if (
!isWorkspaceValidationFailedRun(livenessRun) &&
!isConfigurationIncompleteFailedRun(livenessRun)
@@ -1,3 +1,4 @@
import { measureRunPhase } from "../execution-preparation-timing.js";
import { parseNativeSessionGoalControl } from "./scheduling.js";
import { isHeartbeatRunTerminalStatus } from "./run-lifecycle.js";
import {
@@ -890,7 +891,7 @@ export async function executeHeartbeatRuntime(db: Db, input: HeartbeatRuntimeExe
if (instructionSave) adapterResult.resultJson = { ...adapterResult.resultJson, instructionSave };
if (parseObject(adapterResult.executionRecovery).providerWorkStarted !== false) {
const captured = await usageRecorder.complete(adapterResult);
const captured = await measureRunPhase("execution.usage", () => usageRecorder.complete(adapterResult));
adapterResult = { ...adapterResult, ...captured, usageComplete: captured.complete };
} else {
// Stop may already own the terminal result. Preserve its metadata
@@ -916,10 +917,10 @@ export async function executeHeartbeatRuntime(db: Db, input: HeartbeatRuntimeExe
)).limit(1);
if (exported.length) adapterFinalizeOutcome = "succeeded";
else await recordWorkspaceFinalize(hasWorkspaceRestoreFailure(adapterResult.resultJson) ? "failed" : "succeeded", undefined,
async () => { await restoreNativeWorkspaceBestEffort({
async () => { await measureRunPhase("execution.workspace_restore", () => restoreNativeWorkspaceBestEffort({
db, runId: run.id, assertOwnership: ownership?.assertHeld,
restore: () => nativeWorkspaceSync!.restoreWorkspace(ownership?.assertHeld),
}); });
})); });
}
await ownership?.assertHeld();
await db
@@ -937,12 +938,12 @@ export async function executeHeartbeatRuntime(db: Db, input: HeartbeatRuntimeExe
adapterResult.nativeFinalization.workspaceFinalizeStatus =
workspaceFinalizeStatus;
try {
const finalized = await finalizeNativeRun({
const finalized = await measureRunPhase("execution.native_finalize", () => finalizeNativeRun({
db,
runId: run.id,
workspaceFinalizeStatus,
preserveProviderAttempt: Boolean(nativeWorkspaceSync),
});
}));
await dispatchPendingNativeStatusWakeups({
companyId: run.companyId,
});
@@ -965,9 +966,9 @@ export async function executeHeartbeatRuntime(db: Db, input: HeartbeatRuntimeExe
}
};
if (nativeWorkspaceSync) {
const owned = await withNativeWorkspaceFinalizationOwnership({
const owned = await measureRunPhase("execution.workspace_ownership", () => withNativeWorkspaceFinalizationOwnership({
db, companyId: run.companyId, runId: run.id,
}, completeWorkspace);
}, completeWorkspace));
if (!owned.acquired) throw new NativeWorkspaceFinalizationBusyError();
} else {
await completeWorkspace();
@@ -1115,11 +1116,11 @@ export async function executeHeartbeatRuntime(db: Db, input: HeartbeatRuntimeExe
);
}
try {
await finalizeNativeRun({
await measureRunPhase("execution.native_finalize", () => finalizeNativeRun({
db,
runId: run.id,
workspaceFinalizeStatus: "failed",
});
}));
await dispatchPendingNativeStatusWakeups({
companyId: run.companyId,
});
@@ -1,3 +1,5 @@
import { measureRunTransaction } from "../execution-transaction-timing.js";
import { measureRunPhase } from "../execution-preparation-timing.js";
import {
ConfigurationIncompleteFailure,
type buildPaperclipWakePayload,
@@ -273,10 +275,10 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele
const nativeExecutionWorkspaceId =
persistedExecutionWorkspace?.id ?? run.id;
const nativeReviewContext = readNativeReviewAssignmentContext(context);
const nativeReview = nativeReviewContext ? await getNativeReviewAssignment(db, {
const nativeReview = nativeReviewContext ? await measureRunPhase("runtime_selection.review", () => getNativeReviewAssignment(db, {
companyId: agent.companyId, issueId: issueRef.id, agentId: agent.id,
contextSnapshot: nativeReviewContext,
}) : null;
})) : null;
if (nativeReviewContext && !nativeReview) throw new Error("native_review_assignment_no_longer_available");
const nativeReviewRequest = nativeReview
? buildNativeReviewRequest({
@@ -286,18 +288,18 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele
})
: null;
const persistedContract = run.completionContractId
? await db
? await measureRunPhase("runtime_selection.contract_read", () => db
.select()
.from(completionContracts)
.where(
and(
eq(completionContracts.id, run.completionContractId),
eq(completionContracts.id, run.completionContractId!),
eq(completionContracts.companyId, agent.companyId),
eq(completionContracts.issueId, issueRef.id),
),
)
.limit(1)
.then((rows) => rows[0] ?? null)
.then((rows) => rows[0] ?? null))
: null;
// Only a server-verified human resolution may supply a current answer
// reference. Tool/agent results and generated summaries stay evidence.
@@ -359,7 +361,7 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele
row: persistedContract,
contract: persistedContract.contractJson as never,
}
: await ensureNativeCompletionContract({
: await measureRunPhase("runtime_selection.contract_ensure", () => ensureNativeCompletionContract({
db,
companyId: agent.companyId,
issue: issueRef,
@@ -380,7 +382,7 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele
humanResponseId: currentHumanResponseId,
immediateRequests: immediateCompletion.requests,
immediateRequestSources: immediateCompletion.sources,
});
}));
const taskNativeSessionId = !taskSessionCredentialCompatible ? null : readNonEmptyString(
taskSessionDecodedParams?.sessionId,
);
@@ -390,7 +392,7 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele
// replacement makes the ownership ambiguous and therefore ineligible.
const legacyRetrySource =
run.retryOfRunId && !isFailedChatRunRetry
? await db
? await measureRunPhase("runtime_selection.resume", () => db
.select({
id: heartbeatRuns.id,
companyId: heartbeatRuns.companyId,
@@ -404,19 +406,19 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele
.from(heartbeatRuns)
.where(
and(
eq(heartbeatRuns.id, run.retryOfRunId),
eq(heartbeatRuns.id, run.retryOfRunId!),
eq(heartbeatRuns.companyId, agent.companyId),
eq(heartbeatRuns.agentId, agent.id),
),
)
.limit(1)
.then((rows) => rows[0] ?? null)
.then((rows) => rows[0] ?? null))
: null;
const nativeBootstrapHasProviderEvidence =
legacyRetrySource ||
run.nativeSessionId ||
persistedNativeExecutionInput
? await db
? await measureRunPhase("runtime_selection.resume", () => db
.select({ id: heartbeatRunEvents.id })
.from(heartbeatRunEvents)
.where(
@@ -434,7 +436,7 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele
),
)
.limit(1)
.then((rows) => rows.length > 0)
.then((rows) => rows.length > 0))
: false;
const compatibleLegacyRetrySource =
!managedAiRuntime && !isConversation(issueContext) && context.forceFreshSession !== true && isUnusedLegacyNativeRetryReplacement({
@@ -467,14 +469,14 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele
run,
nativeBootstrapHasProviderEvidence,
)
? await findNativeSessionResumeRun(db, {
? await measureRunPhase("runtime_selection.resume", () => findNativeSessionResumeRun(db, {
companyId: agent.companyId,
agentId: agent.id,
issueId: issueRef.id,
normalizedSessionId: requestedNativeSessionId,
currentRunId: run.id,
beforeCreatedAt: run.createdAt,
})
}))
: null;
nativeRunnerInstanceId =
previousNativeRun?.runnerInstanceId &&
@@ -569,12 +571,13 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele
});
}
if (nativeExecution.provider.kind === "claude_managed") {
const recoveryProfile = await managedAgentProfileService(
const recoveryProfileId = nativeExecution.provider.managedProfile.profileId;
const recoveryProfile = await measureRunPhase("runtime_selection.profile", () => managedAgentProfileService(
db,
).requireQualified(
agent.companyId,
nativeExecution.provider.managedProfile.profileId,
);
recoveryProfileId,
));
assertManagedProfileRecoveryBinding({
adapterConfig: agent.adapterConfig,
snapshot: nativeExecution.provider.managedProfile,
@@ -582,13 +585,14 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele
});
}
if (nativeExecution.provider.kind === "aws_agentcore") {
const recoveryProfile = await remoteAgentProfileService(
const recoveryProfileId = nativeExecution.provider.agentCoreProfile.profileId;
const recoveryProfile = await measureRunPhase("runtime_selection.profile", () => remoteAgentProfileService(
db,
).requireQualified(
agent.companyId,
nativeExecution.provider.agentCoreProfile.profileId,
recoveryProfileId,
"aws_bedrock_agentcore_harness",
);
));
assertAgentCoreProfileRecoveryBinding({
snapshot: nativeExecution.provider.agentCoreProfile,
stored: recoveryProfile,
@@ -599,7 +603,7 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele
const interactionResponses = context[
EXTERNAL_CHAT_QUESTION_RESPONSE_KEY
]
? await materializeExternalChatQuestionResponseInput({
? await measureRunPhase("runtime_selection.interactions", () => materializeExternalChatQuestionResponseInput({
db,
binding: {
companyId: agent.companyId,
@@ -608,8 +612,8 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele
agentId: agent.id,
},
contextSnapshot: context,
})
: await materializeNativeInteractionResponses({
}))
: await measureRunPhase("runtime_selection.interactions", () => materializeNativeInteractionResponses({
db,
companyId: agent.companyId,
issueId: issueRef.id,
@@ -627,27 +631,27 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele
: interactionId
? [interactionId]
: [],
});
}));
const runnerAdapterConfig = parseObject(agent.adapterConfig);
const managedProfile =
nativeRuntimeResolution.profile.backend ===
"claude_managed_agents_api"
? await managedAgentProfileService(db).requireQualified(
? await measureRunPhase("runtime_selection.profile", () => managedAgentProfileService(db).requireQualified(
agent.companyId,
readNonEmptyString(runnerAdapterConfig.managedProfileId) ??
"",
)
))
: null;
const agentCoreProfile =
nativeRuntimeResolution.profile.backend ===
"aws_agentcore_harness_api"
? await remoteAgentProfileService(db).requireQualified(
? await measureRunPhase("runtime_selection.profile", () => remoteAgentProfileService(db).requireQualified(
agent.companyId,
readNonEmptyString(
runnerAdapterConfig.agentCoreProfileId,
) ?? "",
"aws_bedrock_agentcore_harness",
)
))
: null;
if (managedProfile) {
const rawApiKeyBinding = parseObject(
@@ -827,9 +831,9 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele
},
runtimeContext: nativeRuntimeContext,
});
const backendDescriptor = await describeRunnerdNativeSessionBackend(buildExecution({
const backendDescriptor = await measureRunPhase("runtime_selection.provider_discovery", () => describeRunnerdNativeSessionBackend(buildExecution({
normalizedSessionId: nativeSessionId, resumedSession: previousNativeRun !== null,
}));
})));
const nativeExecutionWithCheckpoint = buildNativeExecutionWithCheckpoint({
previousRun: previousNativeRun,
normalizedSessionId: nativeSessionId,
@@ -888,14 +892,14 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele
: undefined;
onProviderResourceDisposition(providerResourceDispositionForRun);
await beforeNativeRuntimeSelection?.(run.id);
const nativeSelected = await db.transaction(async (tx) => {
const lockedRun = await tx
const nativeSelected = await measureRunTransaction(db, "runtime_selection", async (tx) => {
const lockedRun = await measureRunPhase("runtime_selection.row_lock", () => tx
.select()
.from(heartbeatRuns)
.where(eq(heartbeatRuns.id, run.id))
.for("update")
.limit(1)
.then((rows) => rows[0] ?? null);
.then((rows) => rows[0] ?? null));
if (!lockedRun) throw new Error("native_runtime_run_missing");
// Cancellation and runtime selection serialize on this row. A
// stopped preparation must never create a new native coordinator.
@@ -913,13 +917,13 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele
}
const lockedProfile = parseObject(lockedRun.runnerProfileJson);
const persistedNativeSessionId =
await prepareNativeSessionBootstrapPersistence(tx, {
await measureRunPhase("runtime_selection.bootstrap", () => prepareNativeSessionBootstrapPersistence(tx, {
run: lockedRun,
selectedSessionId: nativeSessionId,
execution: nativeExecution!,
restoringCheckpoint: nativeResumeCheckpoint !== null,
});
await tx
}));
await measureRunPhase("runtime_selection.run_write", () => tx
.update(heartbeatRuns)
.set({
runtimeMode: "native",
@@ -1005,8 +1009,8 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele
lockedRun.nativePhaseUpdatedAt ?? new Date(),
updatedAt: new Date(),
})
.where(eq(heartbeatRuns.id, run.id));
await tx
.where(eq(heartbeatRuns.id, run.id)));
await measureRunPhase("runtime_selection.finalization_write", () => tx
.insert(nativeRunFinalizations)
.values({
runId: run.id,
@@ -1014,12 +1018,12 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele
issueId: issueRef.id,
phase: "observed",
})
.onConflictDoNothing();
.onConflictDoNothing());
return true;
});
if (!nativeSelected) return { selected: false };
stopControllerLease();
nativeWorkspaceSync = await prepareNativeWorkspaceSync({
nativeWorkspaceSync = await measureRunPhase("runtime_selection.workspace_sync", () => prepareNativeWorkspaceSync({
db,
runId: run.id,
companyId: agent.companyId,
@@ -1032,7 +1036,7 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele
resourceDisposition: providerResourceDispositionForRun,
onCheckpoint: input.workspace.onCheckpoint,
onPhase: input.workspace.onPhase,
});
}));
} else {
const legacyWarmLifecycle =
executionTarget?.kind === "remote" &&
@@ -1047,7 +1051,7 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele
providerResourceDispositionForRun = "keep_running";
onProviderResourceDisposition(providerResourceDispositionForRun);
}
await db
await measureRunPhase("runtime_selection.run_write", () => db
.update(heartbeatRuns)
.set({
runtimeMode: "legacy",
@@ -1065,7 +1069,7 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele
else '{}'::jsonb end) || ${JSON.stringify(providerTraceRequested ? { providerTrace: { mode: "raw", traceId: providerTraceCapture?.metadata.id ?? null, maxBytes: PROVIDER_TRACE_MAX_BYTES } } : {})}::jsonb`,
updatedAt: new Date(),
})
.where(eq(heartbeatRuns.id, run.id));
.where(eq(heartbeatRuns.id, run.id)));
}
return { selected: true, nativeExecution, nativeRunnerInstanceId, getNativeFreshSessionHandoff, nativeWorkspaceSync };
}
+11 -9
View File
@@ -1,3 +1,5 @@
import { measureRunTransaction } from "./execution-transaction-timing.js";
import { measureRunPhase } from "./execution-preparation-timing.js";
import { automaticReadRecoveryCleanupCondition, touchedByUserCondition } from "./issue-participation.js";
import { HUMAN_TASK_ORIGIN_KINDS } from "@paperclipai/shared";
import { resolveProjectDefaultWorkspaceId } from "./project-default-workspace.js";
@@ -12439,16 +12441,16 @@ export function issueService(db: Db) {
): Promise<IssueComment> {
if (dbOrTx === db && (actor.runId || actor.userId)) {
const append = () =>
db.transaction(async (tx) => {
measureRunTransaction(db, "response", async (tx) => {
// Serialize run-authored comments on the issue so a provider retry
// cannot publish the same visible result twice. This needs no schema
// change: the issue row is the transaction fence, and the recursive
// call below performs the lookup and insert while holding it.
await tx
await measureRunPhase("response.row_lock", () => tx
.select({ id: issues.id })
.from(issues)
.where(eq(issues.id, issueId))
.for("update");
.for("update"));
return addComment(issueId, body, actor, options, tx);
});
return options?.authorizationReason ===
@@ -12710,7 +12712,7 @@ export function issueService(db: Db) {
}
}
if (!comment) {
[comment] = await dbOrTx
[comment] = await measureRunPhase("response.write", () => dbOrTx
.insert(issueComments)
.values({
companyId: issue.companyId,
@@ -12730,7 +12732,7 @@ export function issueService(db: Db) {
? { updatedAt: currentInsertTimestamp }
: {}),
})
.returning();
.returning());
}
if (!comment) throw new Error("Failed to create issue comment");
if (options?.completionReply && actor.agentId && createdByRunId) await acknowledgeChatCompletionReply(dbOrTx, createdByRunId, comment.id);
@@ -12983,7 +12985,7 @@ export function issueService(db: Db) {
);
const publicationCreatedAt = new Date();
for (const binding of bindings) {
await dbOrTx.transaction(async (tx: DbTransaction) => {
await measureRunPhase("response.publish", () => dbOrTx.transaction(async (tx: DbTransaction) => {
await notifyChatPublicationWork(tx);
return tx
.insert(chatPublications)
@@ -13004,12 +13006,12 @@ export function issueService(db: Db) {
updatedAt: publicationCreatedAt,
})
.onConflictDoNothing();
});
}));
for (const [index, attachment] of boundAttachments.entries()) {
const attachmentCreatedAt = new Date(
publicationCreatedAt.getTime() + index + 1,
);
await dbOrTx.transaction(async (tx: DbTransaction) => {
await measureRunPhase("response.publish", () => dbOrTx.transaction(async (tx: DbTransaction) => {
await notifyChatPublicationWork(tx);
return tx
.insert(chatPublications)
@@ -13031,7 +13033,7 @@ export function issueService(db: Db) {
updatedAt: attachmentCreatedAt,
})
.onConflictDoNothing();
});
}));
}
}
}
@@ -1,9 +1,12 @@
import { context, trace, TraceFlags } from "@opentelemetry/api";
import { AsyncLocalStorageContextManager } from "@opentelemetry/context-async-hooks";
import { AlwaysOffSampler, ParentBasedSampler, TracerProvider, InMemorySpanExporter, SimpleSpanProcessor } from "@opentelemetry/sdk-trace";
import { expect, it, vi } from "vitest";
import { afterEach, beforeEach, expect, it, vi } from "vitest";
import { createNativeRunTrace } from "./native-run-trace.js";
beforeEach(() => vi.stubEnv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://collector.test"));
afterEach(() => vi.unstubAllEnvs());
it.each([TraceFlags.NONE, TraceFlags.SAMPLED])("exports a complete native run independently of trigger flags %i", async traceFlags => {
const manager = new AsyncLocalStorageContextManager().enable();
const exporter = new InMemorySpanExporter();
@@ -444,3 +444,29 @@ describe("native runner performance trace", () => {
).toBe(true);
});
});
it.each(["start", "context", "attribute", "status", "end"])("keeps execution and cleanup independent of a throwing %s tracer method", async broken => {
const failure = new Error("original private execution failure");
const spans: Array<{ end: ReturnType<typeof vi.fn> }> = [];
const traceContext: StartupTraceContextHandle = {
tracer: { startSpan() {
if (broken === "start") throw new Error("tracer failed");
const span = {
setAttribute() { if (broken === "attribute") throw new Error("attribute failed"); },
setStatus() { if (broken === "status") throw new Error("status failed"); },
end: vi.fn(() => { if (broken === "end") throw new Error("end failed"); }),
};
spans.push(span);
return span;
} },
contextWithSpan() { if (broken === "context") throw new Error("context failed"); return undefined; },
};
const native = createNativeRunTrace({ runId: "throwing-tracer", traceContext });
const result = {}, work = vi.fn(async () => result), failedWork = vi.fn(async () => { throw failure; });
await expect(native.measure("runner.turn.submit", work)).resolves.toBe(result);
await expect(native.measure("agent.turn", failedWork)).rejects.toBe(failure);
await expect(native.finish("failed")).resolves.toBeUndefined();
expect(work).toHaveBeenCalledOnce();
expect(failedWork).toHaveBeenCalledOnce();
for (const span of spans) expect(span.end).toHaveBeenCalledOnce();
});
@@ -1,3 +1,4 @@
import { getRunExecutionTraceContext } from "../execution-preparation-timing.js";
import { AsyncLocalStorage } from "node:async_hooks";
import { createHash } from "node:crypto";
import { isSpanContextValid, ROOT_CONTEXT, trace } from "@opentelemetry/api";
@@ -211,7 +212,10 @@ export function createNativeRunTrace(input: {
const runIdHash = hashedId(input.runId);
let rootSpan: SpanHandle;
try {
const trigger = trace.getActiveSpan()?.spanContext();
const owned = getRunExecutionTraceContext(input.runId);
const trigger = process.env.OTEL_EXPORTER_OTLP_ENDPOINT?.trim()
? (owned ? trace.getSpan(owned) : trace.getActiveSpan())?.spanContext()
: undefined;
rootSpan = tracing.tracer.startSpan("task.provider_session", {
startTime: traceStartedAtMs,
attributes: {
@@ -341,7 +345,7 @@ export function createNativeRunTrace(input: {
),
},
},
parent.context,
parent.context ?? ROOT_CONTEXT,
);
} catch {
span = NOOP_SPAN;
@@ -407,12 +411,8 @@ export function createNativeRunTrace(input: {
duration_ms: durationMs,
outcome,
});
try {
if (outcome === "failed") state.span.setStatus({ code: 2 });
state.span.end(endedAtMs);
} catch {
// Tracing is diagnostic-only and must never change runner control flow.
}
if (outcome === "failed") { try { state.span.setStatus({ code: 2 }); } catch {} }
try { state.span.end(endedAtMs); } catch {}
await emitCompletedSpanEvent({
name: state.name,
parentName: state.parentName,
@@ -497,17 +497,10 @@ export function createNativeRunTrace(input: {
for (const scope of [...openScopes].reverse()) {
await end(scope, { endedAtMs, outcome });
}
try {
rootSpan.setAttribute(
"paperclip.task.run.wall_ms",
endedAtMs - traceStartedAtMs,
);
rootSpan.setAttribute("paperclip.task.run.outcome", outcome);
if (outcome === "failed") rootSpan.setStatus({ code: 2 });
rootSpan.end(endedAtMs);
} catch {
// Tracing is diagnostic-only and must never change runner control flow.
}
try { rootSpan.setAttribute("paperclip.task.run.wall_ms", endedAtMs - traceStartedAtMs); } catch {}
try { rootSpan.setAttribute("paperclip.task.run.outcome", outcome); } catch {}
if (outcome === "failed") { try { rootSpan.setStatus({ code: 2 }); } catch {} }
try { rootSpan.end(endedAtMs); } catch {}
// Preserve the historical run-log record without emitting a duplicate
// full-width child span into the OpenTelemetry waterfall.
await emitCompletedSpanEvent({
@@ -11747,13 +11747,14 @@ async function createRunnerdBackendWithinSessionClaim(
stagedRemoteProviderPackRoot
) {
const packSource = await prepareVerifiedRemoteProviderPack({
trace: input.trace,
verifyStaged: () => measureNativeRunnerSpan(
input.trace,
"provider_pack.verify",
() => verifyRemoteProviderPack(stagedRemoteProviderPackRoot),
),
usePreinstalled: async () => {
let preinstalledProviderPack = await discoverPreinstalledProviderPack();
let preinstalledProviderPack = await measureNativeRunnerSpan(input.trace, "provider_pack.discover", discoverPreinstalledProviderPack);
if (preinstalledProviderPack) {
try {
await measureNativeRunnerSpan(
@@ -11772,7 +11773,7 @@ async function createRunnerdBackendWithinSessionClaim(
const escapedParent = posix
.dirname(stagedRemoteProviderPackRoot)
.replaceAll("'", "'\\''");
const linked = await remoteCommandRunner.execute({
const linked = await measureNativeRunnerSpan(input.trace, "provider_pack.link_preinstalled", () => remoteCommandRunner.execute({
command: "sh",
args: [
"-c",
@@ -11781,7 +11782,7 @@ async function createRunnerdBackendWithinSessionClaim(
cwd: remoteTarget.remoteCwd,
bypassSession: true,
timeoutMs: 10_000,
});
}));
if (linked.exitCode !== 0 || linked.timedOut) {
throw new Error(
"runner_remote_provider_artifact_incompatible: preinstalled provider pack could not be linked",
@@ -11811,26 +11812,26 @@ async function createRunnerdBackendWithinSessionClaim(
"'",
"'\\''",
);
const cleared = await remoteCommandRunner.execute({
const cleared = await measureNativeRunnerSpan(input.trace, "provider_pack.clear_stale", () => remoteCommandRunner.execute({
command: "sh",
args: ["-c", `rm -rf '${escapedPackRoot}'`],
cwd: remoteTarget.remoteCwd,
bypassSession: true,
timeoutMs: 10_000,
});
}));
if (cleared.exitCode !== 0 || cleared.timedOut) {
throw new Error(
"runner_remote_provider_artifact_incompatible: stale provider pack could not be replaced",
);
}
await stageRemoteRunnerDirectory({
await measureNativeRunnerSpan(input.trace, "provider_pack.upload", () => stageRemoteRunnerDirectory({
target: remoteTarget,
runner: remoteCommandRunner,
sourcePath: configuredProviderPackRoot,
targetPath: stagedRemoteProviderPackRoot,
mode: 0o700,
});
await measureNativeRunnerSpan(input.trace, "provider_pack.verify", () =>
}));
await measureNativeRunnerSpan(input.trace, "provider_pack.verify_uploaded", () =>
verifyRemoteProviderPack(stagedRemoteProviderPackRoot),
);
activeRemoteProviderPackRoot = stagedRemoteProviderPackRoot;
@@ -12096,7 +12097,7 @@ async function createRunnerdBackendWithinSessionClaim(
},
},
);
const restored = await inspectRemoteHarnessState();
const restored = await measureNativeRunnerSpan(input.trace, "harness_state.inspect", () => inspectRemoteHarnessState());
if (
!restored.complete ||
!restored.runnerState ||
@@ -12283,7 +12284,7 @@ async function createRunnerdBackendWithinSessionClaim(
);
} else {
const reuseStartedAtMs = Date.now();
const state = await inspectRemoteHarnessState();
const state = await measureNativeRunnerSpan(input.trace, "harness_state.inspect", () => inspectRemoteHarnessState());
if (
state.complete &&
state.runnerState &&
@@ -12312,7 +12313,7 @@ async function createRunnerdBackendWithinSessionClaim(
if (
remoteDirectory &&
existsSync(localDirectory) &&
!(await remoteRunnerPathExists({
!(await measureNativeRunnerSpan(input.trace, "harness_state.path_check", () => remoteRunnerPathExists({
runner: remoteCommandRunner,
path:
directory.name === "runner"
@@ -12320,7 +12321,7 @@ async function createRunnerdBackendWithinSessionClaim(
: remoteDirectory,
kind:
directory.name === "runner" ? "file" : "directory",
}))
})))
) {
await stageRemoteRunnerDirectory({
target: remoteTarget,
@@ -12352,7 +12353,7 @@ async function createRunnerdBackendWithinSessionClaim(
// this invocation's isolated Codex auth/config. The later launch must
// still stage those fresh files even when history was already restored.
// Launch material is never recovered from a failover backup.
await materializeRemoteHarnessLaunchState();
await measureNativeRunnerSpan(input.trace, "harness_state.launch_material", () => materializeRemoteHarnessLaunchState());
if (
remoteTarget &&
remoteCommandRunner &&
@@ -12440,14 +12441,14 @@ async function createRunnerdBackendWithinSessionClaim(
// `runner_harness_state_mismatch`. Only checkpoint a harness that runnerd
// has proved complete. A malformed or identity-conflicting state still
// throws from inspectRemoteHarnessState and therefore fails closed.
let checkpointable = await inspectRemoteHarnessState();
let checkpointable = await measureNativeRunnerSpan(input.trace, "harness_state.inspect", () => inspectRemoteHarnessState());
// The remote runner writes its suspended lifecycle and provider state
// before the outer process-owner RPC necessarily observes completion.
// Allow a very small bounded visibility window without ever accepting an
// active, incomplete, malformed, or identity-conflicting checkpoint.
for (let attempt = 1; !checkpointable.complete && attempt < 3; attempt++) {
await new Promise<void>((resolve) => setTimeout(resolve, 100));
checkpointable = await inspectRemoteHarnessState();
checkpointable = await measureNativeRunnerSpan(input.trace, "harness_state.inspect", () => inspectRemoteHarnessState());
}
if (!checkpointable.complete) {
const incompleteFailure = remoteCheckpointIncompleteFailure(
@@ -12477,7 +12478,7 @@ async function createRunnerdBackendWithinSessionClaim(
input.trace,
"harness_state.backup.persist",
async () => {
const verified = await inspectRemoteHarnessState();
const verified = await measureNativeRunnerSpan(input.trace, "harness_state.inspect", () => inspectRemoteHarnessState());
if (
!verified.complete ||
!verified.runnerState ||
@@ -268,9 +268,9 @@ it("wires exact-session recovery and guarded selected identity into heartbeat pe
new URL("../heartbeat/runtime-selection.ts", import.meta.url),
"utf8",
);
expect(source).toContain("await findNativeSessionResumeRun(db,");
expect(source).toContain("findNativeSessionResumeRun(db,");
expect(source).toContain(
"await prepareNativeSessionBootstrapPersistence(tx,",
"prepareNativeSessionBootstrapPersistence(tx,",
);
expect(source).toContain("nativeSessionId: persistedNativeSessionId,");
expect(source).toContain("lockedProfile.sessionCheckpoint != null");
@@ -1,3 +1,4 @@
import { beginRunPhase } from "../execution-preparation-timing.js";
import { logActivity } from "../activity-log.js";
import { issueRecoveryActionService } from "../issue-recovery-actions.js";
import { nativeSha256 } from "./canonical.js";
@@ -70,6 +71,7 @@ export async function withNativeWorkspaceFinalizationOwnership<T>(
const owned = and(scope, sql`${heartbeatRuns.runnerProfileJson}->${OWNER_KEY}->>'token' = ${owner.token}`);
let claimed = false;
let actionPromise: Promise<T> | undefined;
const finishAcquisition = beginRunPhase("execution.workspace_ownership_wait");
try {
return await withDedicatedDbConnection(input.db, (dedicated) => dedicated.transaction(async (lock) => {
const rows = await lock.execute(sql`select pg_try_advisory_xact_lock(hashtextextended(${`native-workspace-finalization:${input.companyId}:${input.runId}`}, 0)) as acquired,
@@ -213,13 +215,18 @@ export async function withNativeWorkspaceFinalizationOwnership<T>(
eq(issueRecoveryActions.fingerprint, nativeSha256({ runId: input.runId, owner: rawPrior })),
inArray(issueRecoveryActions.status, ["active", "escalated"]),
));
finishAcquisition();
actionPromise = action(ownership);
const value = await actionPromise;
await ownership.assertHeld();
return { acquired: true, value } as const;
});
}));
} catch (error) {
finishAcquisition(true);
throw error;
} finally {
finishAcquisition();
// postgres.js may reject the transaction when the socket dies before the
// callback settles. Join that callback before releasing its durable fence.
await actionPromise?.catch(() => undefined);
@@ -1,3 +1,4 @@
import { measureRunPhase } from "../execution-preparation-timing.js";
import type { WorkspaceCheckpointMetrics } from "@paperclipai/adapter-utils/workspace-checkpoint";
import type { RuntimeSpanRunner } from "@paperclipai/adapter-utils/acpx-engine/startup-timing";
import fs from "node:fs/promises";
@@ -826,38 +827,40 @@ async function finalizePreparedRuntime(input: {
}): Promise<NativeWorkspaceSyncReference> {
if (input.onPhase) await input.onPhase("workspace.restore", () => input.runtime.restoreWorkspace());
else await input.runtime.restoreWorkspace();
await input.assertOwnership?.();
const finalSnapshot =
await import("@paperclipai/adapter-utils/workspace-restore-merge").then(
({ captureDirectorySnapshotOffThread: captureDirectorySnapshot }) =>
captureDirectorySnapshot(input.descriptor.binding.localCwd, {
exclude: input.runtime.workspaceSyncSnapshot?.baseline.exclude ?? [],
ignoredPaths: input.runtime.workspaceSyncSnapshot?.baseline.ignoredPaths,
diskBacked: true,
}),
);
const finalHostSha256 = directorySnapshotSha256(finalSnapshot);
await disposeDirectorySnapshot(finalSnapshot);
const stamp = finalizedWorkspaceStamp({
descriptor: input.descriptor,
hostSha256: finalHostSha256,
return measureRunPhase("execution.workspace_commit", async () => {
await input.assertOwnership?.();
const finalSnapshot =
await import("@paperclipai/adapter-utils/workspace-restore-merge").then(
({ captureDirectorySnapshotOffThread: captureDirectorySnapshot }) =>
captureDirectorySnapshot(input.descriptor.binding.localCwd, {
exclude: input.runtime.workspaceSyncSnapshot?.baseline.exclude ?? [],
ignoredPaths: input.runtime.workspaceSyncSnapshot?.baseline.ignoredPaths,
diskBacked: true,
}),
);
const finalHostSha256 = directorySnapshotSha256(finalSnapshot);
await disposeDirectorySnapshot(finalSnapshot);
const stamp = finalizedWorkspaceStamp({
descriptor: input.descriptor,
hostSha256: finalHostSha256,
});
await writeRemoteStamp({ target: input.target, stamp });
const finalizedDescriptor: NativeWorkspaceSyncDescriptor = {
...input.descriptor,
state: "finalized",
finalizedAt: new Date().toISOString(),
finalHostSha256,
};
await input.assertOwnership?.();
const reference = await writeDescriptor(finalizedDescriptor);
await persistRunReference(input.db, input.runId, reference);
await persistLeaseStamp({
db: input.db,
leaseId: input.descriptor.binding.leaseId,
stamp,
});
return reference;
});
await writeRemoteStamp({ target: input.target, stamp });
const finalizedDescriptor: NativeWorkspaceSyncDescriptor = {
...input.descriptor,
state: "finalized",
finalizedAt: new Date().toISOString(),
finalHostSha256,
};
await input.assertOwnership?.();
const reference = await writeDescriptor(finalizedDescriptor);
await persistRunReference(input.db, input.runId, reference);
await persistLeaseStamp({
db: input.db,
leaseId: input.descriptor.binding.leaseId,
stamp,
});
return reference;
}
export async function prepareNativeWorkspaceSync(input: {
@@ -1,16 +1,32 @@
import type { NativeRunTrace } from "./native-run-trace.js";
/** Reuse only a pack that passes the same full verification as a new upload. */
export async function prepareVerifiedRemoteProviderPack(input: {
trace?: NativeRunTrace;
verifyStaged: () => Promise<void>;
usePreinstalled: () => Promise<boolean>;
stageAndVerify: () => Promise<void>;
}): Promise<"staged" | "preinstalled" | "uploaded"> {
const scope = input.trace?.start("provider_pack.select");
let failed = false;
const select = async () => {
try {
await input.verifyStaged();
return "staged" as const;
} catch {}
if (await input.usePreinstalled()) return "preinstalled" as const;
await input.stageAndVerify();
return "uploaded" as const;
};
try {
await input.verifyStaged();
return "staged";
} catch {
// Missing, stale, or modified packs are never executable cache hits.
const source = scope ? await input.trace!.run(scope, select) : await select();
if (scope) input.trace!.annotate(scope, { stateSource: source,
reason: source === "uploaded" ? "cached_packs_unverified" : "verified_pack_reused" });
return source;
} catch (error) {
failed = true;
throw error;
} finally {
if (scope) await input.trace!.end(scope, { outcome: failed ? "failed" : "ok" });
}
if (await input.usePreinstalled()) return "preinstalled";
await input.stageAndVerify();
return "uploaded";
}
@@ -37,6 +37,10 @@ vi.mock("../tool-access.js", () => ({
}));
import { createHash } from "node:crypto";
import { context as otelContext, trace } from "@opentelemetry/api";
import { AsyncLocalStorageContextManager } from "@opentelemetry/context-async-hooks";
import { InMemorySpanExporter, SimpleSpanProcessor, TracerProvider } from "@opentelemetry/sdk-trace";
import { withRunExecutionTiming } from "../execution-preparation-timing.js";
import { buildNativeRuntimeContext, resolveNativeRuntimeMcpSnapshot } from "./runtime-context.js";
const temporaryRoots: string[] = [];
@@ -420,3 +424,53 @@ it("pins a voice tool only when the current requester can call through that assi
expect((await resolveNativeRuntimeMcpSnapshot(input)).bindingId).toBe("native-mcp:run-1");
expect(serviceMocks.spekoToolsForSession).toHaveBeenLastCalledWith(db, { companyId: "company-1", agentId: "agent-1", runId: "run-1", issueId: "task-1", identityContextId: "identity-1" });
});
it("keeps instructions, selected skills, and MCP concurrent and distinguishes cold materialization from warm verification", async () => {
vi.stubEnv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://collector.test");
const exporter = new InMemorySpanExporter();
const provider = new TracerProvider({ spanProcessors: [new SimpleSpanProcessor({ exporter })] });
const manager = new AsyncLocalStorageContextManager().enable();
expect(otelContext.setGlobalContextManager(manager)).toBe(true);
expect(trace.setGlobalTracerProvider(provider)).toBe(true);
let release!: () => void;
const gate = new Promise<void>(resolve => { release = resolve; });
serviceMocks.exportFiles.mockImplementation(async () => {
await gate;
return { entryFile: "AGENTS.md", files: { "AGENTS.md": "Private fixture instructions" } };
});
const source = path.join(temporaryRoots.at(-1)!, "selected-skill");
await mkdir(source);
await writeFile(path.join(source, "SKILL.md"), "Private skill fixture");
const input = {
db: {} as Db, agent: { id: "agent-1", companyId: "company-1", name: "Reviewer", adapterConfig: {} },
runId: "concurrent-context", runtimeConfig: { paperclipSkillSync: { desiredSkills: ["selected"] } },
runtimeSkillEntries: [{ key: "selected", runtimeName: "selected", source }],
};
const pending = withRunExecutionTiming(input.runId, () => buildNativeRuntimeContext(input));
try {
await vi.waitFor(() => {
expect(exporter.getFinishedSpans().map(span => span.name)).toEqual(expect.arrayContaining(["runtime_context.skills", "runtime_context.mcp"]));
});
expect(exporter.getFinishedSpans().some(span => span.name === "runtime_context.instructions")).toBe(false);
release();
const cold = await pending;
expect(cold.skills).toHaveLength(1);
expect(exporter.getFinishedSpans().some(span => span.name === "runtime_context.asset_materialize")).toBe(true);
exporter.reset();
const warm = await withRunExecutionTiming(input.runId, () => buildNativeRuntimeContext(input));
expect(warm.aggregateDigest).toBe(cold.aggregateDigest);
const spans = exporter.getFinishedSpans();
expect(spans.some(span => span.name === "runtime_context.asset_verify")).toBe(true);
expect(spans.some(span => span.name === "runtime_context.asset_materialize")).toBe(false);
expect(JSON.stringify(spans.map(span => span.attributes))).not.toContain("Private");
const siblings = spans.filter(span => ["runtime_context.instructions", "runtime_context.skills", "runtime_context.mcp"].includes(span.name));
expect(siblings).toHaveLength(3);
expect(new Set(siblings.map(span => span.parentSpanContext?.spanId)).size).toBe(1);
} finally {
release();
await pending.catch(() => undefined);
trace.disable(); otelContext.disable(); manager.disable();
await provider.shutdown();
vi.unstubAllEnvs();
}
});
@@ -1,3 +1,4 @@
import { measureRunPhase } from "../execution-preparation-timing.js";
import { spekoToolsForSession } from "../voice/speko-agent-tools.js";
import { githubBotConnectionIdsForRun } from "../chat-github-tools.js";
import { isBrowserUseConnection } from "../browser-use-client.js";
@@ -80,7 +81,7 @@ async function verifyMaterializedAsset(
rootPath: string,
manifestFiles: Array<{ path: string; sha256: string; mode: number; size: number }>,
): Promise<void> {
const actual = await collectDirectoryFiles(rootPath);
const actual = await measureRunPhase("runtime_context.asset_read", () => collectDirectoryFiles(rootPath));
const actualByPath = new Map(actual.map((file) => [file.path, file]));
if (actualByPath.size !== manifestFiles.length) throw new Error("runtime context asset file count mismatch");
for (const expected of manifestFiles) {
@@ -92,59 +93,64 @@ async function verifyMaterializedAsset(
}
export async function materializeAsset(files: AssetFile[]): Promise<NativeRuntimeAssetReference> {
const sorted = [...files].sort((a, b) => a.path.localeCompare(b.path));
const manifestFiles = sorted.map((file) => ({ path: safeRelativePath(file.path, "runtime context path"), sha256: sha256(file.content), mode: file.mode & 0o555, size: file.content.byteLength }));
const totalBytes = manifestFiles.reduce((sum, file) => sum + file.size, 0);
if (!manifestFiles.length || manifestFiles.length > MAX_ASSET_FILES || totalBytes > MAX_ASSET_BYTES) throw new Error("runtime context bundle is empty or exceeds its bound");
const assetDigest = sha256(JSON.stringify(manifestFiles));
const manifestText = `${JSON.stringify({ schema: "paperclip.runtime-asset-manifest.v1", digest: assetDigest, fileCount: manifestFiles.length, totalBytes, files: manifestFiles })}\n`;
const manifestDigest = sha256(manifestText);
const { sorted, manifestFiles, totalBytes, assetDigest, manifestText, manifestDigest } = await measureRunPhase("runtime_context.asset_manifest", () => {
const sorted = [...files].sort((a, b) => a.path.localeCompare(b.path));
const manifestFiles = sorted.map((file) => ({ path: safeRelativePath(file.path, "runtime context path"), sha256: sha256(file.content), mode: file.mode & 0o555, size: file.content.byteLength }));
const totalBytes = manifestFiles.reduce((sum, file) => sum + file.size, 0);
if (!manifestFiles.length || manifestFiles.length > MAX_ASSET_FILES || totalBytes > MAX_ASSET_BYTES) throw new Error("runtime context bundle is empty or exceeds its bound");
const assetDigest = sha256(JSON.stringify(manifestFiles));
const manifestText = `${JSON.stringify({ schema: "paperclip.runtime-asset-manifest.v1", digest: assetDigest, fileCount: manifestFiles.length, totalBytes, files: manifestFiles })}\n`;
const manifestDigest = sha256(manifestText);
return { sorted, manifestFiles, totalBytes, assetDigest, manifestText, manifestDigest };
});
const assetsRoot = path.join(resolvePaperclipInstanceRoot(), "runtime-context-assets");
const rootPath = path.join(assetsRoot, "bundles", assetDigest);
const manifestPath = path.join(assetsRoot, "manifests", `${assetDigest}.json`);
const existing = await fs.readFile(manifestPath, "utf8").catch(() => null);
const existing = await measureRunPhase("runtime_context.asset_cache", () => fs.readFile(manifestPath, "utf8").catch(() => null));
if (existing !== null) {
if (sha256(existing) !== manifestDigest || !(await fs.stat(rootPath).catch(() => null))?.isDirectory()) throw new Error(`runtime context asset verification failed: ${assetDigest}`);
await verifyMaterializedAsset(rootPath, manifestFiles);
await measureRunPhase("runtime_context.asset_verify", () => verifyMaterializedAsset(rootPath, manifestFiles), { fileCount: manifestFiles.length, bytes: totalBytes });
return { schema: NATIVE_RUNTIME_ASSET_SCHEMA, digest: assetDigest, manifestDigest, rootPath, fileCount: manifestFiles.length, totalBytes };
}
const stagingRoot = path.join(assetsRoot, ".staging", randomUUID());
await fs.mkdir(stagingRoot, { recursive: true, mode: 0o700 });
try {
for (const file of sorted) {
const target = path.join(stagingRoot, file.path);
await fs.mkdir(path.dirname(target), { recursive: true, mode: 0o700 });
await fs.writeFile(target, file.content, { mode: file.mode & 0o555 });
await fs.chmod(target, file.mode & 0o555);
await measureRunPhase("runtime_context.asset_materialize", async () => {
const stagingRoot = path.join(assetsRoot, ".staging", randomUUID());
await fs.mkdir(stagingRoot, { recursive: true, mode: 0o700 });
try {
for (const file of sorted) {
const target = path.join(stagingRoot, file.path);
await fs.mkdir(path.dirname(target), { recursive: true, mode: 0o700 });
await fs.writeFile(target, file.content, { mode: file.mode & 0o555 });
await fs.chmod(target, file.mode & 0o555);
}
await fs.mkdir(path.dirname(rootPath), { recursive: true, mode: 0o700 });
let publishedRoot = false;
await fs.rename(stagingRoot, rootPath).then(() => {
publishedRoot = true;
}).catch(async (error: NodeJS.ErrnoException) => {
// Concurrent publishers can surface EACCES on macOS when the winning
// destination has already been made read-only. Treat that as a race only
// when the content-addressed destination now exists; a genuine permission
// failure with no published destination must still fail loudly.
const publishedByPeer = (await fs.stat(rootPath).catch(() => null))?.isDirectory() === true;
if (error.code !== "EEXIST" && error.code !== "ENOTEMPTY" && !publishedByPeer) throw error;
await fs.rm(stagingRoot, { recursive: true, force: true });
});
// Publish the writable staging directory first. Renaming a read-only
// directory can fail with EACCES on macOS even when both parents are owned
// and writable. Files are already immutable; directories become immutable
// immediately after this process wins publication.
if (publishedRoot) await makeDirectoriesReadOnly(rootPath);
await fs.mkdir(path.dirname(manifestPath), { recursive: true, mode: 0o700 });
await fs.writeFile(manifestPath, manifestText, { flag: "wx", mode: 0o400 }).catch(async (error: NodeJS.ErrnoException) => {
if (error.code !== "EEXIST") throw error;
if (sha256(await measureRunPhase("runtime_context.asset_cache", () => fs.readFile(manifestPath, "utf8"))) !== manifestDigest) throw new Error(`runtime context manifest race mismatch: ${assetDigest}`);
});
await measureRunPhase("runtime_context.asset_verify", () => verifyMaterializedAsset(rootPath, manifestFiles), { fileCount: manifestFiles.length, bytes: totalBytes });
} catch (error) {
await fs.rm(stagingRoot, { recursive: true, force: true }).catch(() => undefined);
throw error;
}
await fs.mkdir(path.dirname(rootPath), { recursive: true, mode: 0o700 });
let publishedRoot = false;
await fs.rename(stagingRoot, rootPath).then(() => {
publishedRoot = true;
}).catch(async (error: NodeJS.ErrnoException) => {
// Concurrent publishers can surface EACCES on macOS when the winning
// destination has already been made read-only. Treat that as a race only
// when the content-addressed destination now exists; a genuine permission
// failure with no published destination must still fail loudly.
const publishedByPeer = (await fs.stat(rootPath).catch(() => null))?.isDirectory() === true;
if (error.code !== "EEXIST" && error.code !== "ENOTEMPTY" && !publishedByPeer) throw error;
await fs.rm(stagingRoot, { recursive: true, force: true });
});
// Publish the writable staging directory first. Renaming a read-only
// directory can fail with EACCES on macOS even when both parents are owned
// and writable. Files are already immutable; directories become immutable
// immediately after this process wins publication.
if (publishedRoot) await makeDirectoriesReadOnly(rootPath);
await fs.mkdir(path.dirname(manifestPath), { recursive: true, mode: 0o700 });
await fs.writeFile(manifestPath, manifestText, { flag: "wx", mode: 0o400 }).catch(async (error: NodeJS.ErrnoException) => {
if (error.code !== "EEXIST") throw error;
if (sha256(await fs.readFile(manifestPath, "utf8")) !== manifestDigest) throw new Error(`runtime context manifest race mismatch: ${assetDigest}`);
});
await verifyMaterializedAsset(rootPath, manifestFiles);
} catch (error) {
await fs.rm(stagingRoot, { recursive: true, force: true }).catch(() => undefined);
throw error;
}
}, { fileCount: manifestFiles.length, bytes: totalBytes });
return { schema: NATIVE_RUNTIME_ASSET_SCHEMA, digest: assetDigest, manifestDigest, rootPath, fileCount: manifestFiles.length, totalBytes };
}
@@ -181,7 +187,7 @@ async function materializeSelectedSkills(runtimeConfig: Record<string, unknown>,
const entry = byKey.get(key);
if (!entry) throw new Error(`assigned runtime skill is missing from the company library: ${key}`);
if (entry.sourceStatus === "missing") throw new Error(entry.missingDetail ?? `assigned runtime skill source is missing: ${key}`);
return { key: entry.key, runtimeName: safeRelativePath(entry.runtimeName, `runtime name for ${key}`), versionId: entry.versionId ?? entry.currentVersionId ?? null, bundle: await materializeAsset(await collectDirectoryFiles(entry.source)) };
return { key: entry.key, runtimeName: safeRelativePath(entry.runtimeName, `runtime name for ${key}`), versionId: entry.versionId ?? entry.currentVersionId ?? null, bundle: await materializeAsset(await measureRunPhase("runtime_context.asset_read", () => collectDirectoryFiles(entry.source))) };
}));
}
@@ -246,9 +252,9 @@ export async function resolveNativeRuntimeMcpSnapshot(input: { db: Db; agent: Pi
export async function buildNativeRuntimeContext(input: { db: Db; agent: RuntimeAgent; runId: string; runtimeConfig: Record<string, unknown>; runtimeSkillEntries: PaperclipSkillEntry[]; instructionWorkingCopy?: { rootPath: string; entryPath: string; kind?: "agent_files" } }): Promise<NativeRuntimeContextSnapshot> {
const [instructions, skills, mcp] = await Promise.all([
materializeInstructionBundle(input.db, input.agent, input.instructionWorkingCopy?.kind === "agent_files"),
materializeSelectedSkills(input.runtimeConfig, input.runtimeSkillEntries, input.agent.adapterType === "paperclip_runner"),
resolveNativeRuntimeMcpSnapshot({ db: input.db, agent: input.agent, runId: input.runId }),
measureRunPhase("runtime_context.instructions", () => materializeInstructionBundle(input.db, input.agent, input.instructionWorkingCopy?.kind === "agent_files")),
measureRunPhase("runtime_context.skills", () => materializeSelectedSkills(input.runtimeConfig, input.runtimeSkillEntries, input.agent.adapterType === "paperclip_runner")),
measureRunPhase("runtime_context.mcp", () => resolveNativeRuntimeMcpSnapshot({ db: input.db, agent: input.agent, runId: input.runId })),
]);
const snapshot = {
prompt: { revision: PAPERCLIP_EXECUTION_PROMPT_REVISION, text: PAPERCLIP_EXECUTION_PROMPT, digest: nativeRuntimePromptDigest() },
@@ -1,3 +1,4 @@
import { measureRunPhase } from "./execution-preparation-timing.js";
import { executionWorkspaces, issues, type Db } from "@paperclipai/db";
import { and, eq, inArray, or, sql } from "drizzle-orm";
import { forbidden } from "../errors.js";
@@ -12,23 +13,23 @@ export async function assertTaskWorkspaceSourceProjectAccess(
db: Db | DbTransaction, actor: AuthorizationActor, companyId: string, projectId: string | null, options: WorkspaceAccessContext = {},
) {
if (!projectId) return;
const decision = await accessService(db as Db).decide({
const decision = await measureRunPhase("workspace.access.authorization", () => accessService(db as Db).decide({
actor, action: "project:read", resource: { type: "project", companyId, projectId },
});
}));
if (!decision.allowed) throw forbidden(decision.explanation);
if (options.write) {
const [task] = actor.type === "board" && options.issueId
? await db.select({ assigneeAgentId: issues.assigneeAgentId, assigneeUserId: issues.assigneeUserId }).from(issues)
.where(and(eq(issues.id, options.issueId), eq(issues.companyId, companyId)))
? await measureRunPhase("workspace.access.inventory", async () => db.select({ assigneeAgentId: issues.assigneeAgentId, assigneeUserId: issues.assigneeUserId }).from(issues)
.where(and(eq(issues.id, options.issueId!), eq(issues.companyId, companyId))))
: [];
const assigneeAgentId = actor.type === "agent" ? actor.agentId ?? null : task?.assigneeAgentId ?? options.assigneeAgentId ?? null;
const assigneeUserId = actor.type === "agent" ? null : task?.assigneeUserId ?? options.assigneeUserId ?? null;
const assignment = await accessService(db as Db).decide({
const assignment = await measureRunPhase("workspace.access.authorization", () => accessService(db as Db).decide({
actor, action: "project:write_workspace",
resource: { type: "issue", companyId, projectId, assigneeAgentId, assigneeUserId,
issueId: options.issueId, parentIssueId: options.parentIssueId },
scope: { projectId, assigneeAgentId, assigneeUserId },
});
}));
if (!assignment.allowed) throw forbidden(assignment.explanation);
}
}
@@ -36,28 +37,28 @@ export async function assertTaskWorkspaceSourceProjectAccess(
export async function assertTaskWorkspaceAccess(
db: Db | DbTransaction, actor: AuthorizationActor, companyId: string, workspaceId: string, options: WorkspaceAccessContext = {},
) {
const [workspace] = await db.select().from(executionWorkspaces).where(and(
const [workspace] = await measureRunPhase("workspace.access.inventory", async () => db.select().from(executionWorkspaces).where(and(
eq(executionWorkspaces.id, workspaceId), eq(executionWorkspaces.companyId, companyId),
await executionWorkspaceReadSqlCondition(db, actor),
));
await measureRunPhase("workspace.access.privacy", () => executionWorkspaceReadSqlCondition(db, actor)),
)));
if (!workspace) throw forbidden("Task workspace access is no longer available");
await assertTaskWorkspaceSourceProjectAccess(db, actor, companyId, workspace.projectId, { ...options, write: options.write && workspace.strategyType !== "git_worktree" });
const retainedIssueIds = Object.keys(workspace.metadata?._issuePrivacySources ?? {});
if (workspace.sourceIssueId) retainedIssueIds.push(workspace.sourceIssueId);
const sources = await db.select({ id: issues.id }).from(issues).where(and(
const sources = await measureRunPhase("workspace.access.inventory", async () => db.select({ id: issues.id }).from(issues).where(and(
eq(issues.companyId, companyId),
or(eq(issues.executionWorkspaceId, workspaceId), retainedIssueIds.length > 0
? inArray(sql`${issues.id}`, retainedIssueIds.map(id => canonicalExecutionUuid(sql`${id}::text`))) : undefined),
));
)));
const sourceIds = new Set(sources.map(source => source.id));
if (retainedIssueIds.some(id => !sourceIds.has(id))) throw forbidden("Task workspace source is no longer accessible");
const access = accessService(db as Db);
if (sources.length === 0 && !workspace.projectId) {
const decision = await access.decide({ actor, action: "company_scope:read", resource: { type: "company", companyId } });
const decision = await measureRunPhase("workspace.access.authorization", () => access.decide({ actor, action: "company_scope:read", resource: { type: "company", companyId } }));
if (!decision.allowed) throw forbidden("An unscoped workspace is outside this actor's authorization boundary");
}
for (const source of sources) {
const decision = await access.decide({ actor, action: "issue:read", resource: { type: "issue", companyId, issueId: source.id } });
const decision = await measureRunPhase("workspace.access.authorization", () => access.decide({ actor, action: "issue:read", resource: { type: "issue", companyId, issueId: source.id } }));
if (!decision.allowed) throw forbidden("Task workspace retains files outside this actor's authorization boundary");
}
}
@@ -71,51 +72,53 @@ export async function assertTaskWorkspacesReadAccess(
export async function assertTaskWorkspacesReadAccessForActors(
db: Db | DbTransaction, actors: AuthorizationActor[], companyId: string, workspaceIds: Iterable<string>,
) {
const ids = [...new Set(workspaceIds)];
if (!ids.length || !actors.length) return;
const privacyActors = [...new Map(actors.map(actor => [JSON.stringify({ ...actor, runId: undefined }), actor])).values()];
const privacy = and(...await Promise.all(privacyActors.map(actor => executionWorkspaceReadSqlCondition(db, actor))));
const projectIds = new Set<string>(), sourceIds = new Set<string>();
let needsCompanyRead = false;
for (let offset = 0; offset < ids.length; offset += 500) {
const batch = ids.slice(offset, offset + 500);
const workspaces = await db.select({ id: executionWorkspaces.id, projectId: executionWorkspaces.projectId,
sourceIssueId: executionWorkspaces.sourceIssueId, metadata: executionWorkspaces.metadata,
}).from(executionWorkspaces).where(and(eq(executionWorkspaces.companyId, companyId), inArray(executionWorkspaces.id, batch), privacy));
if (workspaces.length !== batch.length) throw forbidden("Task workspace access is no longer available");
const retained = new Set(workspaces.flatMap(workspace => [
...Object.keys(workspace.metadata?._issuePrivacySources ?? {}), ...(workspace.sourceIssueId ? [workspace.sourceIssueId] : []),
]));
const attached = await db.select({ id: issues.id, workspaceId: issues.executionWorkspaceId }).from(issues)
.where(and(eq(issues.companyId, companyId), inArray(issues.executionWorkspaceId, batch)));
const attachedWorkspaceIds = new Set(attached.map(source => source.workspaceId));
for (const source of attached) sourceIds.add(source.id);
const retainedIds = [...retained];
for (let start = 0; start < retainedIds.length; start += 500) {
const retainedBatch = retainedIds.slice(start, start + 500);
const sources = await db.select({ id: issues.id }).from(issues).where(and(eq(issues.companyId, companyId),
inArray(sql`${issues.id}`, retainedBatch.map(id => canonicalExecutionUuid(sql`${id}::text`)))));
const found = new Set(sources.map(source => source.id));
if (retainedBatch.some(id => !found.has(id))) throw forbidden("Task workspace source is no longer accessible");
for (const source of sources) sourceIds.add(source.id);
return measureRunPhase("workspace.access", async () => {
const ids = [...new Set(workspaceIds)];
if (!ids.length || !actors.length) return;
const privacyActors = [...new Map(actors.map(actor => [JSON.stringify({ ...actor, runId: undefined }), actor])).values()];
const privacy = and(...await Promise.all(privacyActors.map(actor => executionWorkspaceReadSqlCondition(db, actor))));
const projectIds = new Set<string>(), sourceIds = new Set<string>();
let needsCompanyRead = false;
for (let offset = 0; offset < ids.length; offset += 500) {
const batch = ids.slice(offset, offset + 500);
const workspaces = await measureRunPhase("workspace.access.inventory", async () => db.select({ id: executionWorkspaces.id, projectId: executionWorkspaces.projectId,
sourceIssueId: executionWorkspaces.sourceIssueId, metadata: executionWorkspaces.metadata,
}).from(executionWorkspaces).where(and(eq(executionWorkspaces.companyId, companyId), inArray(executionWorkspaces.id, batch), privacy)));
if (workspaces.length !== batch.length) throw forbidden("Task workspace access is no longer available");
const retained = new Set(workspaces.flatMap(workspace => [
...Object.keys(workspace.metadata?._issuePrivacySources ?? {}), ...(workspace.sourceIssueId ? [workspace.sourceIssueId] : []),
]));
const attached = await measureRunPhase("workspace.access.inventory", async () => db.select({ id: issues.id, workspaceId: issues.executionWorkspaceId }).from(issues)
.where(and(eq(issues.companyId, companyId), inArray(issues.executionWorkspaceId, batch))));
const attachedWorkspaceIds = new Set(attached.map(source => source.workspaceId));
for (const source of attached) sourceIds.add(source.id);
const retainedIds = [...retained];
for (let start = 0; start < retainedIds.length; start += 500) {
const retainedBatch = retainedIds.slice(start, start + 500);
const sources = await measureRunPhase("workspace.access.inventory", async () => db.select({ id: issues.id }).from(issues).where(and(eq(issues.companyId, companyId),
inArray(sql`${issues.id}`, retainedBatch.map(id => canonicalExecutionUuid(sql`${id}::text`))))));
const found = new Set(sources.map(source => source.id));
if (retainedBatch.some(id => !found.has(id))) throw forbidden("Task workspace source is no longer accessible");
for (const source of sources) sourceIds.add(source.id);
}
for (const workspace of workspaces) {
if (workspace.projectId) projectIds.add(workspace.projectId);
else if (!workspace.sourceIssueId && !Object.keys(workspace.metadata?._issuePrivacySources ?? {}).length
&& !attachedWorkspaceIds.has(workspace.id)) needsCompanyRead = true;
}
}
for (const workspace of workspaces) {
if (workspace.projectId) projectIds.add(workspace.projectId);
else if (!workspace.sourceIssueId && !Object.keys(workspace.metadata?._issuePrivacySources ?? {}).length
&& !attachedWorkspaceIds.has(workspace.id)) needsCompanyRead = true;
const resources = [
...[...projectIds].map(projectId => ({ type: "project" as const, companyId, projectId })),
...[...sourceIds].map(issueId => ({ type: "issue" as const, companyId, issueId })),
...(needsCompanyRead ? [{ type: "company" as const, companyId }] : []),
];
const batches = await measureRunPhase("workspace.access.authorization", () => authorizationService(db).decideReadsForActors(actors, resources));
for (const decisions of batches) for (let index = 0; index < decisions.length; index++) {
if (decisions[index].allowed) continue;
throw forbidden(resources[index].type === "issue"
? "Task workspace retains files outside this actor's authorization boundary"
: resources[index].type === "company"
? "An unscoped workspace is outside this actor's authorization boundary" : decisions[index].explanation);
}
}
const resources = [
...[...projectIds].map(projectId => ({ type: "project" as const, companyId, projectId })),
...[...sourceIds].map(issueId => ({ type: "issue" as const, companyId, issueId })),
...(needsCompanyRead ? [{ type: "company" as const, companyId }] : []),
];
const batches = await authorizationService(db).decideReadsForActors(actors, resources);
for (const decisions of batches) for (let index = 0; index < decisions.length; index++) {
if (decisions[index].allowed) continue;
throw forbidden(resources[index].type === "issue"
? "Task workspace retains files outside this actor's authorization boundary"
: resources[index].type === "company"
? "An unscoped workspace is outside this actor's authorization boundary" : decisions[index].explanation);
}
});
}
+1 -1
View File
@@ -31,7 +31,7 @@ describe("Honeycomb run links", () => {
expect(query.filter_combination).toBe("AND");
expect(query.filters).toEqual([
{ column: "service.name", op: "=", value: "paperclip" },
{ column: "name", op: "in", value: ["task.run", "task.provider_session"] },
{ column: "name", op: "in", value: ["task.run", "task.provider_session", "task.execution_attempt"] },
{
column: HONEYCOMB_RUN_HASH_ATTRIBUTE,
op: "=",
+1 -1
View File
@@ -26,7 +26,7 @@ export function buildHoneycombRunQueryUrl(runIdHash: string): string {
breakdowns: ["trace.trace_id"],
filters: [
{ column: "service.name", op: "=", value: "paperclip" },
{ column: "name", op: "in", value: ["task.run", "task.provider_session"] },
{ column: "name", op: "in", value: ["task.run", "task.provider_session", "task.execution_attempt"] },
{ column: HONEYCOMB_RUN_HASH_ATTRIBUTE, op: "=", value: runIdHash },
],
filter_combination: "AND",