Merge origin/master and rebase feature migration sequence

Preserve upstream usage attribution and sandbox restore retry semantics alongside task workspace persistence. Regenerate task and physical-root migrations after upstream 0333 with replay-safe guards.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip committed 2026-10-12 07:17:58 -05:00
commit c205ab39f0
148 files changed
+66073 -776

No files matched your search

+8
View File
@@ -94,6 +94,14 @@ git push -u origin <descriptive-name>
git push origin --delete <old-name>
```
### Keep Screenshots Out of the Repo
Screenshots, screen recordings, and other review evidence belong in the PR description or a PR comment, not in the repository. Do **not** commit them (for example under `docs/pr-screenshots/`). Committed review images land on `master` permanently and make every clone bigger, even though nobody needs them after the PR merges.
To show a visual change, drag the image into the PR description or a comment on GitHub, which hosts the file for you. If your tooling cannot upload to GitHub, leave the images out and describe what you checked under **Verification**.
Images that are part of the product documentation itself, such as a screenshot embedded in a guide under `doc/`, are fine to commit.
### Model Used (Required)
Every PR must include a **Model Used** section specifying which AI model produced or assisted with the change. Include the provider, exact model ID/version, context window size, and any relevant capability details (e.g., reasoning mode, tool use). If no AI was used, write "None — human-authored". This applies to all contributors — human and AI alike.
+2 -2
View File
@@ -747,10 +747,10 @@ transaction boundary.
### Task workspace records (2026-10-09)
Migration [0333_pretty_infant_terrible.sql](../packages/db/src/migrations/0333_pretty_infant_terrible.sql) makes `execution_workspaces.project_id` nullable and changes its project reference to `ON DELETE RESTRICT`: source-project policy cannot disappear while a workspace still depends on it. `issues.execution_workspace_id` remains the sole active task binding. Versioned selection/pending-selection JSON and a binding revision record intent and protect transitions; they are not additional active bindings.
Migration [0334_right_hannibal_king.sql](../packages/db/src/migrations/0334_right_hannibal_king.sql) makes `execution_workspaces.project_id` nullable and changes its project reference to `ON DELETE RESTRICT`: source-project policy cannot disappear while a workspace still depends on it. `issues.execution_workspace_id` remains the sole active task binding. Versioned selection/pending-selection JSON and a binding revision record intent and protect transitions; they are not additional active bindings.
`execution_workspace_repositories` is company-scoped inventory beneath an execution workspace. It records repository identity, contained relative path, requested ref, pinned commit, preparation state and idempotency receipts. Credentials are resolved at use and are never stored in this inventory. Connection/destination `execution_defaults` JSON contains optional typed project/workspace choices, never arbitrary host paths or commands.
Migration [0334_stiff_ted_forrester.sql](../packages/db/src/migrations/0334_stiff_ted_forrester.sql) adds internal `execution_workspace_repository_roots` records. A company and hash of its canonical physical path uniquely identify a sticky `anonymous_only` or `managed` acquisition reservation. Short transactions serialize reservations with repository requests and credential promotion, preventing private publication after low-trust admission checks. Records have no workspace foreign key: mutable workspace metadata, relocation, archival or deletion cannot release a physical folder still accessible to an admitted run. There is no automatic mode change or reservation deletion.
Migration [0335_oval_screwball.sql](../packages/db/src/migrations/0335_oval_screwball.sql) adds internal `execution_workspace_repository_roots` records. A company and hash of its canonical physical path uniquely identify a sticky `anonymous_only` or `managed` acquisition reservation. Short transactions serialize reservations with repository requests and credential promotion, preventing private publication after low-trust admission checks. Records have no workspace foreign key: mutable workspace metadata, relocation, archival or deletion cannot release a physical folder still accessible to an admitted run. There is no automatic mode change or reservation deletion.
No migration scans or copies existing agent home directories. Keep admitted native descriptors and recovery references until compatible controllers finish their persistence obligations. See [the architecture rollout contract](plans/2026-10-09-task-workspaces.md).
+42
View File
@@ -531,6 +531,35 @@ verifying Vite/HMR behavior.
For normal issue work, start with the smallest targeted check that proves the change. Reserve repo-wide typecheck/build/test runs for PR-ready handoff or changes broad enough that narrow checks do not cover the risk.
#### Embedded Postgres template cache
Suites that call `startEmbeddedPostgresTestDatabase` (from `@paperclipai/db`)
get their own embedded Postgres cluster. The first such suite on a host builds
one fully migrated cluster and publishes its data directory under
`$TMPDIR/paperclip-embedded-postgres-templates-<uid>/<key>`. Every later suite
copies that template and starts Postgres on the copy instead of running
`initdb` and all migrations again, and the `getEmbeddedPostgresTestSupport`
probe reuses it instead of starting a throwaway cluster. The key hashes the
embedded-postgres version, the initdb flags, and every migration file, so a
changed migration builds a new template. Each suite still runs
`applyPendingMigrations` on its copy, which is a no-op when the template is
current. Templates for other keys are removed after seven days without use.
A template is a Postgres data directory, and starting it executes its
configuration, so the cache is private to the current OS user: the root is
created with mode `0700` and is only used when it is a real directory (not a
symlink) owned by the current user with no group or world write bit, and each
template is checked the same way before it is copied. A root that fails those
checks (for example one pre-created by another user in a shared `/tmp`) is left
alone and every suite in that process uses a fresh cluster.
Set `PAPERCLIP_TEST_POSTGRES_TEMPLATE=0` to bypass the cache and give every
suite a fresh `initdb`, for example when debugging cluster startup itself. A
failure on the template path (root checks, build, copy, start, or publish)
falls back to the fresh path automatically, so the cache can make a suite
slower but never fail it or skip it: a migration that cannot apply still fails
the suite on the fresh path.
### Task search evaluation
The task search relevance rubric and regression corpus are documented in
@@ -1428,6 +1457,19 @@ Claude, which can reject unavailable models. Package/version verification is
independent of model selection. Environment tests verify runtime installation;
a successful provider run additionally verifies credentials and model access.
The server must install and bundle `@agentclientprotocol/claude-agent-acp`
at the qualified version, including the patched bridge and pinned SDK. The
vendor dependency check covers this dynamic lookup as well as ordinary imports.
Source builds bind the vendored runner to the workspace's pnpm store while
keeping the server manifest as its provider authority. Published npm bundles
retain the pinned SDK JavaScript and declare its native packages as server
optional dependencies, so npm installs the executable for the consumer's
platform. Runtime admission still checks the exact version and executable digest.
A missing provider package is reported as **Runtime missing**, with automatic
retries stopped. Reinstall or rebuild Paperclip with its provider dependencies,
restart the server, and run the agent's runtime test before retrying the task.
Changing the model or signing in again cannot repair a missing runtime package.
ACPX Claude supports Linux x64 and macOS ARM64/x64 with pinned SDK executables.
On macOS the launcher uses private verified module/executable snapshots instead
of Linux `/proc` descriptors. Dependency isolation, process ownership, and
+20
View File
@@ -1601,6 +1601,26 @@ Validation:
Read-time aggregate queries are acceptable for V1.
Materialized rollups can be added later if query latency exceeds targets.
The Costs overview distinguishes unknown billing categories, missing model/price
data, and terminal runs awaiting final receipts. Bounded, company-scoped details
load on demand at `GET /companies/:companyId/costs/data-quality`. Entries link to
the run and, where applicable, agent configuration for future usage. Normal
credits and fixed billing are not unknown billing warnings. Unavailable historic
prices require no cleanup and must not be presented as zero-cost receipts.
`POST /companies/:companyId/costs/retry-accounting` accepts a run ID, requires a
board cost reader with a non-viewer membership, and uses the existing audited,
idempotent accounting service. It never executes an agent. The UI offers retry
only when the saved run is eligible to record costs. Missing final usage changes
its display label after fifteen minutes; elapsed time never settles the debt or
discards a late receipt. Raw provider errors and credentials are excluded from
cost details. ACP usage retains the runtime-selected model, including defaults;
conflicting before/after model IDs do not authorize a single-model estimate.
Remote OpenCode billing uses only credentials forwarded to the execution target.
Cost detail reads use company-scoped partial indexes matching their filters and
sort order. Provider and biller cards retain estimate and missing-price labels;
an accounting retry confirmation survives removal of the resolved cost gap.
## 14. UI Requirements (Board App)
V1 UI routes:
+40
View File
@@ -1136,8 +1136,25 @@ An interrupted conversation does not permanently block its task. For local conve
Shutdown, process loss, and provider failure use the existing durable failure retry counter and delay. Ordinary failure recovery permits at most two automatic retries in a failure chain. Accepted-interaction infrastructure recovery retains its existing bounded policy. Repeated scheduler visits reuse the same successor; restarting the server does not reset the counter. After exhaustion, automatic attempts stop. A new explicit user message can start a fresh run and failure budget. Productive max-turn continuation and confirmed workspace waits keep their separate existing semantics.
Graceful shutdown interrupts an owned legacy adapter through its cancellation
control, then waits within one shared 30-second window for provider stop and
adapter settlement across all affected runs. Before interruption becomes visible,
it commits a settlement fence on the exact sandbox lease. Cleanup on this server
or another server defers while that fence is pending. If the controller dies,
cleanup records exact stop-only retention and a repair hold after the deadline,
without claiming a successful restore. The shipped Compose services allow 90
seconds for settlement and subsequent shutdown cleanup.
The sandbox lease remains available during workspace export and instruction
collection. Lease release and recovery classification follow settlement and use
the current execution evidence. If settlement exceeds the deadline with an
active sandbox source, shutdown records a restore failure and pins the exact
source for stop and retention before cleanup. A later run must pass the normal
workspace repair gate. Native runner ownership keeps its existing restart path.
Real gates still apply: company and task ownership, active provider ownership, budget limits, agent availability, dependencies, pending approval/review paths, and explicit pause holds. Native runner reattachment and finalization retain their existing ownership protocol. Process, HTTP, and gateway adapters retain their recovery rules because invoking those adapters can itself repeat an external action rather than start a conversation turn.
A source that already stopped and was released remains fenced until its export settles; restart reconciliation scans these released sources too. Both provider reuse and destruction check the physical allocation for a shutdown fence or file-repair pin, including historical lease rows.
An operator Stop waits for provider termination. Remote sandbox providers may return a stopped/deleted receipt after their control-plane operation completes. Paperclip binds that receipt to the company, run, and exact lease; successful file cleanup, a terminal run row, or an in-sandbox shutdown event is not sufficient. Legacy conversational runs receive their cancellation acknowledgement after all remote leases have confirmed termination. Stop alone never creates a continuation. A user message queued during remote cleanup is reconsidered when the provider confirms termination; it still passes normal admission and adopts pending comment IDs in order. Once stopped, the next explicit wake uses the same queue. A compatible saved ACP session can resume, and an unavailable or incompatible session can start fresh with the full task context. Run credentials and scratch paths remain scoped to the new run. A subtree pause requires Resume; a message does not bypass it.
Cancelled runs and runs with a recorded pending stop lose run-scoped API write authority, while reads remain available for diagnostics. This applies to ordinary tasks as well as conversations. Task and interaction mutations recheck that authority in the write transaction, so a cancelled run cannot overwrite a recovery disposition with a late Done or resolve an interaction after revocation. A task handoff that intentionally stops its own run can commit only with a server receipt tied to that exact request. Direct sandbox CLI adapters such as Grok register cancellation before preparation and keep ownership until host-owned termination is verified, including a sandbox acquired before adapter registration. A failed stop does not acknowledge cancellation or abandon an outstanding remote command. Interrupted workspace restore failures remain recorded for recovery; a stop receipt proves termination, not successful file restoration.
@@ -1602,6 +1619,29 @@ question does not start setup. The real connection card keeps user identity,
access grants, the decision, and continuation together. This guidance does not
approve a connection or bypass its normal user decision.
An ordinary tool connection request becomes obsolete when an already authorized
connection is usable by its requesting agent with the addressed user's identity.
Startup and periodic connection delivery reconcile that condition, retire the
card as expired, and record a system decision without a fabricated human approval.
The saved continuation is delivered once after rechecking the same access. The
scheduled retry gate also verifies this condition before the next sweep, so the
obsolete card cannot cancel a valid retry. Missing, unhealthy, disabled, ambiguous,
or unauthorized identities remain blocked. Requests for additional tool access,
AI authentication, inbox setup, or an upstream aggregator service keep their normal
approval paths. Task ownership, real questions, approvals, budgets, pauses, and
workspace recovery checks still govern admission.
New requests notify the connection worker so this reconciliation also starts after an idle startup scan. Readiness uses the gateway's runtime tool
listing policy with issue, project, and run context. System-retired requests are
rechecked against the next run's responsible user and current access, and saved
continuations defer while another question or linked approval remains pending.
Those checks run again during queued and final dispatch admission. An unstarted
system continuation cancelled for a new response wait rearms its delivery in the
same transaction. Verified absence of execution keeps that wait out of workspace
repair incidents. Historical retirements do not bind all future retries to an
old account; an already authorized replacement can refresh the system observation
with its own audit record before delivery. Human decisions remain unchanged.
## Responses submitted during an active run
A confirmation, checkbox confirmation, or question answer is new conversation
+49 -4
View File
@@ -35,6 +35,32 @@ signed bootstrap writes. Only the task-drain read skips request tracking.
Its authentication finishes under a separate work token before entering the
report, so it counts concurrent writes without blocking on its own scan.
## Retry history
Retry promotion keeps `scheduled_retry_at` on a run for inspection. That date
is consumed history when the run is terminal, has a finish time at or after the
retry date, and has no accounting work pending. The idle report uses the same
proof for failed ephemeral leases, coalesced wakes, and acknowledged question
responses. A live retry child still blocks through the instance-wide run check.
Unfinished runs, retry dates after the finish time, and pending accounting stay
awake. The inspection does not clear dates or change retry scheduling.
## Retained local workspaces
Local directories for primary projects and Git worktrees do not require an
awake process by themselves. The report permits their retained rows only in
known modes and states, with no runtime configuration, service state, commands,
pending reopen, or cleanup already eligible. Saved environment references alone
are passive. Shared workspaces with a project workspace remain blockers because
they can inherit runtime configuration. Unfamiliar providers, strategies,
configuration and malformed metadata also remain blockers.
This does not delete files, alter retention, or clear workspace records. Future
cleanup eligibility is an earliest allowed time; eligible cleanup can wait for
the next wake. Due cleanup, unfinished operations, runtime leases and runtime
services still keep the instance awake through their separate checks. The host
must keep workspace storage attached across sleep and preserve its backup policy.
## Accepted work and cleanup
HTTP admission is installed before body parsing, authentication, webhooks and
@@ -113,17 +139,36 @@ operator reconciliation; this API does not discard it to save compute.
The report checks all companies in one bounded read-only database transaction.
Queued and orphaned runs, timers, retries, accounting debt, unfinished issues,
cleanup, active routines, integrations, external API credentials, and plugin
cleanup, active routines, integrations, external ingress credentials, and plugin
work block sleep. Completed ordinary run history does not. Less common work
sources conservatively block on any retained records. Active issue watchdogs
block sleep even when their watched issue is complete and no review has started.
Missing migrations, malformed configuration and database errors produce unknown
without exposing SQL or tenant data.
Active `runtime_auth` connections with purpose `ai` are saved credentials, not
independent workers. Their saved configuration does not block sleep. Login
sessions and runs still pass the existing durable checks. Detached subscription
observations and quota reads remain counted through their actual completion,
including token refresh after a quota response times out. Credential-recovery
journals, partial writes and probes block sleep across restart. Empty company
directories left after successful recovery do not; invalid paths and read
failures remain unknown. Credentials and connection records are retained.
MCP, chat and other connections still block until they have a wake contract.
A failed ephemeral lease is history only with successful cleanup, a release
timestamp, and an exact provider termination receipt bound to its company, run,
lease and resource. Its run must be finished without retry or accounting debt.
Pending cleanup claims and export/stop-and-retain intents still block. Failure
labels or cleanup success without that proof do not authorize sleep. Lease
history is retained; this check performs no provider teardown or data updates.
Finished login-session history permits sleep only after its promotion claim is
cleared and any retained provider reference has a matching released or expired
lease with successful cleanup and a release timestamp in the same company and
environment. Missing cleanup evidence, `cleanup_pending`, and live login flows
cleared and any retained provider reference has a matching released, expired,
or verified failed ephemeral lease with successful cleanup and a release
timestamp in the same company and environment. The failed lease must satisfy
the same termination and run checks above. Missing cleanup evidence,
`cleanup_pending`, and live login flows
remain blockers. A finished coalesced wake is history only when its exact
company/agent's linked run has finished without retry or accounting debt.
Image-setup history still blocks: promotion can succeed before teardown is
+23
View File
@@ -599,6 +599,16 @@ still describes the earlier failure. The overall error message is unchanged.
Elapsed time is bounded to seven days, and the budget to one day. Unknown or
malformed optional values are omitted without invalidating cleanup ownership.
When the deadline expires while Daytona creation is still pending, the plugin
makes one read-only lookup (bounded to five seconds, inside the host's 30-second
RPC margin) of its provisional creation name and reports
`environmentAcquisitionProviderState`. The value is one of `absent`, `creating`,
`pending_build`, `building_snapshot`, `pulling_snapshot`, `starting`, `started`,
`error`, `build_failed`, `other`, or `unavailable` (lookup failed, timed out, or
ownership labels did not match). It records where the provider left the
allocation, not why. The lookup does not delete, retry, or change the cleanup
envelope.
These fields cross the existing creation-cleanup RPC envelope only for lease
acquisition. The host accepts them only from an actual RPC error with matching
company, environment, and run ownership. A private receipt binds the report to
@@ -647,6 +657,19 @@ resolved policy describes the configured limit, not proof that its timer fired;
stderr remains in the instance run log and is not used as the timeout exception
message. An earlier provider timeout does not establish its upstream cause.
Legacy OpenCode requests Daytona's persistent process session for
its main CLI command. Daytona targets with the effective persistent-session capability
use asynchronous dispatch and log observation instead of holding one synchronous
command HTTP request open for the whole run. The command's resolved timeout is
forwarded separately from individual status/log observation timeouts. One timer
bounds dispatch and observation, fences late log callbacks, and clears at settlement.
A timeout does not prove command termination; the existing stop/lease cleanup
path still owns that evidence. Existing session callers without this deadline
keep their previous lifetime. Probes, preparation and other providers
keep their existing paths. Local and SSH execution do not use this option.
The resolved execution budget and lease cleanup remain unchanged; the session
request does not prove that a previously observed gateway timeout had this cause.
When settlement records a workspace restore failure, `run_execution` also
includes `workspaceRestoreFailure` with one of the shared, path-free codes:
`restore_permission_denied`, `restore_lock_timeout`, `restore_unsafe_archive`,
+9
View File
@@ -57,6 +57,15 @@ that an earlier failed collection discarded.
## Required remote restore failures
The copy-back first repairs itself. While the run still holds its merge
baseline, a workspace copy-back that fails before writing any host file is
retried up to three times in total, with a short backoff. This covers dropped
transfers, provider errors, and a merge lock held by another restore. An unsafe
archive, a permission failure, or any failure after the merge started writing
host files is never retried. Only a copy-back that still fails reaches the
repair hold below. Adapter environment Tests restore into a throwaway temp
workspace, so they opt out of these retries and report a failure at once.
When a legacy sandbox run cannot copy required workspace files back, the server
keeps the original allocation for repair. It writes the failed run, the exact
lease retention intent, and a board-owned repair hold in one transaction. The
+2
View File
@@ -16,6 +16,8 @@ services:
# backstop if something leaks anyway — the container hits its own limit and
# dies visibly rather than starving the host of pids.
pids_limit: 2048
# Includes the shared 30s adapter-settlement window and final cleanup.
stop_grace_period: 90s
ports:
- "${PAPERCLIP_PORT:-3100}:3100"
environment:
+2
View File
@@ -28,6 +28,8 @@ services:
# this container hits its own ceiling and dies visibly rather than starving
# the whole host of pids.
pids_limit: 2048
# Includes the shared 30s adapter-settlement window and final cleanup.
stop_grace_period: 90s
ports:
- "3100:3100"
environment:
@@ -17,6 +17,51 @@ Cost reporting happens automatically through adapters. When an agent heartbeat c
The server records this as a cost event for budget tracking.
Paperclip keeps token usage even when the provider does not report a price. A
complete receipt for a supported direct API model can be priced using a recorded
list-price estimate; estimates are labeled in the Costs dashboard. Missing costs
are never treated as proof of free usage.
## Understanding incomplete cost data
The Costs overview shows a compact line such as **12 run costs unavailable** when
usage is missing a price or final receipt. Expand it with the arrow to see the
affected entries, grouped alphabetically by agent with links to
each run. Unknown billing types use their own label rather than counting as
unavailable costs. Estimates remain labeled on spend cards and tables.
Select an agent's name to open its configuration.
Provider and biller cards also label estimates and unavailable prices. A total
with only unpriced entries shows a dash instead of $0.00. Successful recording
shows a confirmation even when it resolves the last gap.
- **Billing category unknown:** select an AI connection in the agent's
configuration for future runs. Recorded charges remain in the inference ledger.
- **Model not reported:** select an explicit model for future runs. ACP adapters
also retain the provider-selected model when the runtime reports it, including
its default model. A model change during a turn is not priced as one model.
- **Cost unavailable:** the past entry has no reliable price. No cleanup is
required; totals include known charges only.
- **Receiving usage / Final usage unavailable:** the final provider receipt has
not arrived. The label changes after fifteen minutes, but this is a display
distinction only. Paperclip still accepts late receipts and preserves the
accounting debt. Rerunning an agent does not recover its previous receipt.
- **Ready to record:** Paperclip can retry saved accounting data. **Retry cost
recording** records costs only and never starts agent work. Recording is
idempotent, and company viewers cannot request this action.
Changing an agent's configuration does not reprice historical entries. Normal
credits and fixed billing are listed as other billing usage, without an unknown
billing warning. OpenCode API billing can be identified from a nonempty API key
for its selected provider; the provider name alone is insufficient evidence.
Remote OpenCode runs use only keys forwarded to the remote process. Keys present
on the Paperclip server do not establish a remote run's billing category.
Board cost readers can inspect bounded details at
`GET /api/companies/{companyId}/costs/data-quality`, with the same date bounds as
other cost reports. Details contain no provider errors or credentials and load
only when expanded in the UI. Authorized board members can request recording at
`POST /api/companies/{companyId}/costs/retry-accounting` with `{ "runId": "uuid" }`.
## Cost Events API
Cost events can also be reported directly:
+1 -1
View File
@@ -61,7 +61,7 @@
"smoke:posthog-live": "node scripts/smoke/posthog-live.mjs",
"smoke:pipelines-tutorial": "./scripts/smoke/pipelines-tutorial-smoke.sh",
"smoke:terminal-bench-loop-skill": "node scripts/smoke/terminal-bench-loop-skill-smoke.mjs",
"test:release-registry": "node --test scripts/__tests__/canary-startup-playwright.test.mjs scripts/__tests__/cursor-public-install-sandbox.test.mjs scripts/__tests__/grok-public-install-sandbox.test.mjs scripts/__tests__/prefetch-child.test.mjs scripts/__tests__/retain-runner-qualification-packages.test.mjs scripts/verify-release-registry-state.test.mjs scripts/release-package-map.test.mjs scripts/check-release-package-bootstrap.test.mjs scripts/check-no-git-push.test.mjs scripts/release-lib.test.mjs scripts/release-registry-versions.test.mjs scripts/link-plugin-dev-sdk.test.js scripts/acpx-patch-packaging.test.mjs scripts/service-onboard-smoke.test.mjs scripts/docker-onboard-smoke.test.mjs scripts/preview-artifacts.test.mjs scripts/cloud-migrator-artifacts.test.mjs",
"test:release-registry": "node --test scripts/__tests__/canary-startup-playwright.test.mjs scripts/__tests__/cursor-public-install-sandbox.test.mjs scripts/__tests__/grok-public-install-sandbox.test.mjs scripts/__tests__/prefetch-child.test.mjs scripts/__tests__/stage-bundled-packages.test.mjs scripts/__tests__/pack-staged-packages.test.mjs scripts/__tests__/retain-runner-qualification-packages.test.mjs scripts/verify-release-registry-state.test.mjs scripts/release-package-map.test.mjs scripts/check-release-package-bootstrap.test.mjs scripts/check-no-git-push.test.mjs scripts/release-lib.test.mjs scripts/release-registry-versions.test.mjs scripts/link-plugin-dev-sdk.test.js scripts/acpx-patch-packaging.test.mjs scripts/service-onboard-smoke.test.mjs scripts/docker-onboard-smoke.test.mjs scripts/preview-artifacts.test.mjs scripts/cloud-migrator-artifacts.test.mjs",
"storybook-visual:baseline": "node scripts/storybook-visual-baseline.mjs",
"test:storybook-visual": "node scripts/storybook-visual-baseline.mjs download && node scripts/storybook-visual-baseline.mjs verify && pnpm build-storybook && npx playwright test --config tests/storybook-visual/playwright.config.ts",
"test:storybook-visual:update": "node scripts/storybook-visual-baseline.mjs download && pnpm build-storybook && npx playwright test --config tests/storybook-visual/playwright.config.ts --update-snapshots && node scripts/storybook-visual-baseline.mjs pack",
@@ -0,0 +1,18 @@
import { describe, expect, it } from "vitest";
import { acpxAccountingModel } from "./accounting-model.js";
const selected = (id: string) => ({ models: { currentModelId: id, availableModelIds: [id] } });
describe("ACP accounting model", () => {
it("retains the runtime's default and effective model instead of the requested label", () => {
expect(acpxAccountingModel("", selected("gpt-6.1-sol"), selected("gpt-6.1-sol"))).toBe("gpt-6.1-sol");
expect(acpxAccountingModel("default", null, selected("gpt-6-astra"))).toBe("gpt-6-astra");
expect(acpxAccountingModel("", selected("gpt-6-astra"), null)).toBe("gpt-6-astra");
});
it("uses explicit configuration only when no selected model was reported", () => {
expect(acpxAccountingModel("gpt-6.1-sol", null, null)).toBe("gpt-6.1-sol");
expect(acpxAccountingModel("", { models: { availableModelIds: ["gpt-6.1-sol"] } }, null)).toBeNull();
});
it("does not price a turn that changed models as one model", () => {
expect(acpxAccountingModel("gpt-6.1-sol", selected("gpt-6.1-sol"), selected("gpt-6-astra"))).toBeNull();
});
});
@@ -0,0 +1,10 @@
import type { AcpRuntimeStatus } from "acpx/runtime";
/** Use provider-selected IDs, including its default. A mid-turn model change
* cannot be priced as a single-model receipt; never guess from a display name. */
export function acpxAccountingModel(requested: string, before: AcpRuntimeStatus | null, after: AcpRuntimeStatus | null): string | null {
const pre = before?.models?.currentModelId?.trim();
const post = after?.models?.currentModelId?.trim();
if (pre && post && pre !== post) return null;
return post || pre || requested.trim() || null;
}
@@ -1403,6 +1403,7 @@ describe("shared ACPX engine runtime behavior", () => {
const stateDir = path.join(root, "state");
const logs: Array<{ stream: string; text: string }> = [];
let statusCalls = 0;
const onUsage = vi.fn();
const execute = createAcpxEngineExecutor({
createRuntime: () => ({
ensureSession: async () => ({
@@ -1413,8 +1414,9 @@ describe("shared ACPX engine runtime behavior", () => {
getStatus: async () => {
statusCalls += 1;
return statusCalls === 1
? { usage: { cost: { amount: 0.4, currency: "USD" } } }
? { models: { currentModelId: "claude-observed-default", availableModelIds: [] }, usage: { cost: { amount: 0.4, currency: "USD" } } }
: {
models: { currentModelId: "claude-observed-default", availableModelIds: [] },
usage: {
cumulative: {
inputTokens: 120,
@@ -1448,6 +1450,7 @@ describe("shared ACPX engine runtime behavior", () => {
const result = await execute({
runId: "run-usage-capture",
onUsage,
agent: {
id: "agent-1",
companyId: "company-1",
@@ -1471,6 +1474,8 @@ describe("shared ACPX engine runtime behavior", () => {
expect(result.provider).toBe("anthropic");
expect(result.biller).toBe("anthropic");
expect(result.billingType).toBe("api");
expect(result.model).toBe("claude-observed-default");
expect(onUsage).toHaveBeenLastCalledWith(expect.objectContaining({ model: "claude-observed-default", complete: true }));
expect((result.resultJson as Record<string, unknown>)?.cumulativeCostUsd).toBeCloseTo(1.15);
expect((result.resultJson as Record<string, unknown>)?.usage).toEqual({
inputTokens: 120,
@@ -129,6 +129,7 @@ import type {
TurnCompletion,
} from "./run-contracts.js";
import { createRunResourceLedger } from "./run-resource-ledger.js";
import { acpxAccountingModel } from "./accounting-model.js";
import { settleAcpRun, type SettlementSteps } from "./settlement-sequence.js";
import {
runAttempt,
@@ -4765,6 +4766,7 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) {
// scoped hoisted local, so the settlement `endSession` step can cancel it.
let runPrompt = "";
let preTurnStatus: AcpRuntimeStatus | null = null;
let accountingModel: string | null = prepared.requestedModel || null;
// Phase-timing markers for the prepare_turn and turn phases. The prepare
// phase covers the prompt build and the pre-turn usage snapshot; the turn
// phase covers the started turn and the event relay.
@@ -4836,6 +4838,7 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) {
// Snapshot pre-turn usage so cumulative agent-reported cost can be
// attributed to this run alone.
preTurnStatus = await readRuntimeStatus(runtime, sessionHandle);
accountingModel = acpxAccountingModel(prepared.requestedModel, preTurnStatus, null);
// The prepare phase (prompt build + usage snapshot) finished; the turn
// phase starts next.
await emitPhase("prepare_turn", preparePhaseStart, "ok");
@@ -4965,7 +4968,7 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) {
const checkpoint = summarizeAcpxTurnUsage({ preStatus: preTurnStatus, postStatus: null, eventBreakdown, eventCostUsd });
observedUsage = { usage: checkpoint.usage ?? undefined, costUsd: checkpoint.costUsd, usageBasis: "per_run", usageComplete: false };
await ctx.onUsage?.({ ...billingFields, usage: checkpoint.usage ?? undefined, costUsd: checkpoint.costUsd,
model: prepared.requestedModel, usageBasis: "per_run", complete: false });
model: accountingModel, usageBasis: "per_run", complete: false });
}
await emitRuntimeEvent(ctx, event, toolTitles, prepared.coalescePlaceholderToolUpdates);
}
@@ -5046,6 +5049,7 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) {
const turnSucceeded = terminal.status === "completed" && !timedOut && !channelLost;
// Read usage before the settlement can discard runtime state.
const postTurnStatus = await readRuntimeStatus(runtime, sessionHandle);
accountingModel = acpxAccountingModel(prepared.requestedModel, preTurnStatus, postTurnStatus);
const turnUsage = summarizeAcpxTurnUsage({
preStatus: preTurnStatus,
postStatus: postTurnStatus,
@@ -5057,7 +5061,7 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) {
const usageComplete = !channelLost && (turnSucceeded || postTurnStatus?.usage != null);
observedUsage = { usage: turnUsage.usage ?? undefined, costUsd: turnUsage.costUsd, usageBasis: "per_run", usageComplete };
await ctx.onUsage?.({ ...billingFields, usage: turnUsage.usage ?? undefined, costUsd: turnUsage.costUsd,
model: prepared.requestedModel, usageBasis: "per_run", complete: usageComplete });
model: accountingModel, usageBasis: "per_run", complete: usageComplete });
const failedTurn = terminal.status === "failed" || terminal.status === "cancelled" || timedOut;
// ACPX can defer session/load until runTurn. Forget an unavailable
// session so the next bounded turn receives the full task conversation.
@@ -5140,7 +5144,7 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) {
sessionDisplayId: sessionHandle.agentSessionId ?? sessionHandle.backendSessionId ?? sessionHandle.runtimeSessionName,
...billingFields,
...referencedProjectStagingFailuresField,
model: prepared.requestedModel || null,
model: accountingModel,
...(turnUsage.usage ? { usage: turnUsage.usage, usageBasis: "per_run" as const } : {}),
costUsd: turnUsage.costUsd,
usageComplete,
@@ -5281,7 +5285,7 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) {
...(!turnDispatchAttempted ? { executionRecovery: { kind: "bootstrap" as const, providerWorkStarted: false as const } } : {}),
...billingFields,
...referencedProjectStagingFailuresField,
model: prepared.requestedModel || null,
model: accountingModel,
clearSession: clearSession || timedOut,
resultJson: { phase, ...activityDiagnostics },
summary: message,
@@ -0,0 +1,20 @@
import { describe, expect, it } from "vitest";
import { apiKeyBillingType } from "./api-billing-type.js";
describe("verified API billing", () => {
it.each([
["openrouter", "OPENROUTER_API_KEY"], ["openai", "OPENAI_API_KEY"],
["anthropic", "ANTHROPIC_API_KEY"], ["deepseek", "DEEPSEEK_API_KEY"],
])("classifies %s with its matching credential", (provider, key) => {
expect(apiKeyBillingType(provider, { [key]: "fixture-key" })).toBe("metered_api");
});
it("recognizes Codex's key alias only for the Codex runtime", () => {
const env = { CODEX_API_KEY: "fixture" };
expect(apiKeyBillingType("openai", env, "codex")).toBe("metered_api");
expect(apiKeyBillingType("openai", env)).toBe("unknown");
expect(apiKeyBillingType("openrouter", env, "codex")).toBe("unknown");
});
it.each(["openrouter", "openai", "constructor", "toString", null, "unsupported"]) ("does not infer billing for %s", provider => {
expect(apiKeyBillingType(provider, { ANTHROPIC_API_KEY: "unrelated-key", OPENAI_API_KEY: " " })).toBe("unknown");
});
});
@@ -0,0 +1,17 @@
import type { AdapterBillingType } from "./types.js";
const API_KEY_NAMES: Readonly<Record<string, readonly string[]>> = {
openai: ["OPENAI_API_KEY"],
openrouter: ["OPENROUTER_API_KEY"],
anthropic: ["ANTHROPIC_API_KEY"],
deepseek: ["DEEPSEEK_API_KEY"],
};
/** A model/provider name alone does not prove billing. Only a matching API
* credential establishes API use; ambient auth files remain unclassified. */
export function apiKeyBillingType(provider: string | null, env: Record<string, string | undefined>, runtime?: "codex"): AdapterBillingType {
// Codex's credential alias is not accepted by other OpenAI clients.
if (provider === "openai" && runtime === "codex" && env.CODEX_API_KEY?.trim()) return "metered_api";
const keys = provider && Object.hasOwn(API_KEY_NAMES, provider) ? API_KEY_NAMES[provider] : undefined;
return keys?.some(key => Boolean(env[key]?.trim())) ? "metered_api" : "unknown";
}
@@ -82,6 +82,9 @@ export interface CommandManagedRuntimeRunner {
env?: Record<string, string>;
stdin?: string;
timeoutMs?: number;
/** Optional execution deadline for a session command, separate from an
* individual provider observation timeout. Zero means no command deadline. */
commandTimeoutMs?: number;
onLog?: (stream: "stdout" | "stderr", chunk: string) => Promise<void>;
onSpawn?: (meta: { pid: number; startedAt: string }) => Promise<void>;
/**
@@ -569,6 +572,8 @@ export async function prepareCommandManagedRuntime(input: {
// prepareSandboxManagedRuntime so the host pack time rides one `pack` span
// under the `stage.sync` step. The default is a no-op.
runtimeSpan?: RuntimeSpanRunner;
/** Copy-back retry delays; pass `[]` for throwaway workspaces that must not wait on retries. */
workspaceRestoreRetryDelaysMs?: readonly number[];
}): Promise<PreparedSandboxManagedRuntime> {
const timeoutMs = input.spec.timeoutMs && input.spec.timeoutMs > 0 ? input.spec.timeoutMs : 300_000;
const workspaceRemoteDir = input.workspaceRemoteDir ?? input.spec.remoteCwd;
@@ -632,6 +637,7 @@ export async function prepareCommandManagedRuntime(input: {
onProgress: input.onProgress,
onRuntimeProgress: input.onRuntimeProgress,
runtimeSpan: input.runtimeSpan,
workspaceRestoreRetryDelaysMs: input.workspaceRestoreRetryDelaysMs,
});
}
}
@@ -682,5 +688,6 @@ export async function prepareCommandManagedRuntime(input: {
onProgress: input.onProgress,
onRuntimeProgress: input.onRuntimeProgress,
runtimeSpan: input.runtimeSpan,
workspaceRestoreRetryDelaysMs: input.workspaceRestoreRetryDelaysMs,
});
}
@@ -227,6 +227,35 @@ describe("runAdapterExecutionTargetProcess", () => {
vi.unstubAllEnvs();
});
it.each([true, false, undefined])("preserves the sandbox session choice %s and command outcome", async (useSession) => {
const stopped = vi.fn();
const onLog = vi.fn(async (_stream: "stdout" | "stderr", _chunk: string) => {});
const completed = {
exitCode: 7, signal: null, timedOut: false, stdout: "partial output",
stderr: "fixture failure", pid: null, startedAt: "2020-01-01T00:00:00.000Z",
};
const execute = vi.fn(async (input: { onLog?: (stream: "stdout" | "stderr", chunk: string) => Promise<void> }) => {
await input.onLog?.("stdout", completed.stdout);
return completed;
});
const result = await runAdapterExecutionTargetProcess("run-fixture", {
kind: "remote", transport: "sandbox", remoteCwd: "/workspace", runner: { execute },
}, "opencode", ["run"], {
cwd: "/local", env: {}, timeoutSec: 14400, graceSec: 20,
onLog, onProcessStopped: stopped, ...(useSession === undefined ? {} : { useSession }),
});
const call = execute.mock.calls[0]![0];
expect(call).toMatchObject({ command: "opencode", args: ["run"], cwd: "/workspace", timeoutMs: 14400000 });
if (useSession === undefined) expect(call).not.toHaveProperty("useSession");
else expect(call).toHaveProperty("useSession", useSession);
if (useSession === true) expect(call).toHaveProperty("commandTimeoutMs", 14400000);
else expect(call).not.toHaveProperty("commandTimeoutMs");
expect(result).toEqual(completed);
expect(execute).toHaveBeenCalledTimes(1);
expect(onLog.mock.calls).toEqual([["stdout", "partial output"]]);
expect(stopped).toHaveBeenCalledTimes(1);
});
it("sanitizes inherited host env before SSH process execution", async () => {
vi.stubEnv("PATH", "/host/bin:/usr/bin");
vi.stubEnv("HOME", "/Users/local");
@@ -269,6 +298,7 @@ describe("runAdapterExecutionTargetProcess", () => {
},
timeoutSec: 5,
graceSec: 1,
useSession: true,
onLog: async () => {},
},
);
@@ -283,6 +313,7 @@ describe("runAdapterExecutionTargetProcess", () => {
},
}),
);
expect(runChildProcessSpy.mock.calls[0]![3]).not.toHaveProperty("useSession");
});
});
@@ -280,6 +280,9 @@ export interface AdapterExecutionTargetProcessOptions {
stdin?: string;
timeoutSec: number;
graceSec: number;
/** Sandbox-only: request the provider's persistent process session for a
* long-lived command. The host still gates this on provider capabilities. */
useSession?: boolean;
onLog: (stream: "stdout" | "stderr", chunk: string) => Promise<void>;
onRuntimeProgress?: RuntimeStatusSink;
onSpawn?: (meta: { pid: number; processGroupId: number | null; startedAt: string }) => Promise<void>;
@@ -883,6 +886,8 @@ export async function runAdapterExecutionTargetProcess(
env,
stdin: options.stdin,
timeoutMs: options.timeoutSec > 0 ? options.timeoutSec * 1000 : target.timeoutMs ?? undefined,
...(options.useSession !== undefined ? { useSession: options.useSession } : {}),
...(options.useSession === true ? { commandTimeoutMs: Math.max(0, options.timeoutSec) * 1000 } : {}),
// The tail loop already streams incremental chunks; suppress the
// runner's end-of-run batched onLog to avoid duplicate log bytes.
onLog: runLogTail ? undefined : options.onLog,
@@ -1464,6 +1469,11 @@ export async function prepareAdapterExecutionTargetRuntime(input: {
// so the host pack time rides one `pack` span under the `stage.sync` step. The
// SSH and local lanes ignore it. The default is a no-op.
runtimeSpan?: RuntimeSpanRunner;
/**
* Sandbox copy-back retry delays. Environment probes restore into a throwaway
* temp workspace and pass `[]` so a failed restore does not stall the Test.
*/
workspaceRestoreRetryDelaysMs?: readonly number[];
}): Promise<PreparedAdapterExecutionTargetRuntime> {
const target = input.target ?? { kind: "local" as const };
if (target.kind === "local") {
@@ -1543,6 +1553,7 @@ export async function prepareAdapterExecutionTargetRuntime(input: {
onProgress: input.onProgress,
onRuntimeProgress: input.onRuntimeProgress,
runtimeSpan: input.runtimeSpan,
workspaceRestoreRetryDelaysMs: input.workspaceRestoreRetryDelaysMs,
});
return {
target,
@@ -4461,12 +4461,14 @@ describe("sandbox git-bundle export transport", () => {
const client = makeTransportClient(step !== "workspace_extract", { syncOutOperations: [], readFilePaths: [] });
const prepared = await prepareSandboxManagedRuntime({
spec: gitSpec(remoteWorkspaceDir), adapterKey: "test-adapter", client, workspaceLocalDir: localWorkspaceDir,
workspaceRestoreRetryDelaysMs: [0, 0],
});
await commitInSandbox(remoteWorkspaceDir);
const failure = Object.assign(new Error("private-restore-command /private-restore-path"), { code: 7, stderr: "private-restore-stderr" });
// Failures before the first host write are retried, so they must persist.
if (step === "git_export") client.run = async () => { throw failure; };
if (step === "git_import") vi.spyOn(gitSync, "fetchGitBundleIntoLocalRef").mockRejectedValueOnce(failure);
if (step === "directory_merge") vi.spyOn(restoreMerge, "mergeDirectoryWithBaseline").mockRejectedValueOnce(failure);
if (step === "git_import") vi.spyOn(gitSync, "fetchGitBundleIntoLocalRef").mockRejectedValue(failure);
if (step === "directory_merge") vi.spyOn(restoreMerge, "mergeDirectoryWithBaseline").mockRejectedValue(failure);
if (step === "git_integration") vi.spyOn(gitSync, "integrateImportedGitHead").mockRejectedValueOnce(failure);
if (step === "index_reset") vi.spyOn(gitSync, "resetLocalGitIndexToHead").mockRejectedValueOnce(failure);
if (step === "git_ref_cleanup") vi.spyOn(gitSync, "deleteLocalGitRef").mockRejectedValueOnce(failure);
@@ -4495,6 +4497,111 @@ describe("sandbox git-bundle export transport", () => {
} finally { vi.restoreAllMocks(); }
});
it.each([
["a dropped transfer", Object.assign(new Error("transfer reset"), { code: "ECONNRESET" })],
["a busy merge lock", Object.assign(new Error("lock busy"), { code: restoreMerge.WORKSPACE_RESTORE_LOCK_TIMEOUT_CODE })],
])("retries the copy-back after %s before any host write", async (_label, failure) => {
const { localWorkspaceDir, remoteWorkspaceDir } = await setupGitBackedWorkspace("paperclip-restore-retry-");
const client = makeTransportClient(true, { syncOutOperations: [], readFilePaths: [] });
const prepared = await prepareSandboxManagedRuntime({
spec: gitSpec(remoteWorkspaceDir), adapterKey: "test-adapter", client, workspaceLocalDir: localWorkspaceDir,
workspaceRestoreRetryDelaysMs: [0, 0],
});
await commitInSandbox(remoteWorkspaceDir);
let directoryTransfers = 0;
const syncOut = client.syncOut!;
client.syncOut = async (operations) => {
const directory = operations.some((operation) => operation.files.some((file) => file.kind === "directory"));
if (directory && ++directoryTransfers === 1 && failure.code === "ECONNRESET") throw failure;
return syncOut(operations);
};
if (failure.code !== "ECONNRESET") vi.spyOn(restoreMerge, "mergeDirectoryWithBaseline").mockRejectedValueOnce(failure);
try {
await prepared.restoreWorkspace();
expect(directoryTransfers).toBe(2);
expect(await git(localWorkspaceDir, ["log", "-1", "--pretty=%s"])).toBe("sandbox update");
await expect(readFile(path.join(localWorkspaceDir, "remote-only.txt"), "utf8")).resolves.toBe("from sandbox\n");
} finally { vi.restoreAllMocks(); }
});
it("gives up after three copy-back attempts", async () => {
const { localWorkspaceDir, remoteWorkspaceDir } = await setupGitBackedWorkspace("paperclip-restore-retry-limit-");
const client = makeTransportClient(true, { syncOutOperations: [], readFilePaths: [] });
const prepared = await prepareSandboxManagedRuntime({
spec: gitSpec(remoteWorkspaceDir), adapterKey: "test-adapter", client, workspaceLocalDir: localWorkspaceDir,
workspaceRestoreRetryDelaysMs: [0, 0],
});
await commitInSandbox(remoteWorkspaceDir);
const failure = new Error("sandbox unreachable");
let directoryTransfers = 0;
const syncOut = client.syncOut!;
client.syncOut = async (operations) => {
if (!operations.some((operation) => operation.files.some((file) => file.kind === "directory"))) return syncOut(operations);
directoryTransfers++;
throw failure;
};
await expect(prepared.restoreWorkspace()).rejects.toBe(failure);
expect(directoryTransfers).toBe(3);
});
it.each([
["an unsafe archive", Object.assign(new Error("unsafe"), { code: "WORKSPACE_RESTORE_UNSAFE_ARCHIVE" }), "directory_merge"],
["a permission failure", Object.assign(new Error("denied"), { code: "EACCES" }), "directory_merge"],
["a failure after the merge started writing", new Error("integration failed"), "git_integration"],
] as const)("does not retry %s", async (_label, failure, step) => {
const { localWorkspaceDir, remoteWorkspaceDir } = await setupGitBackedWorkspace("paperclip-restore-no-retry-");
const client = makeTransportClient(true, { syncOutOperations: [], readFilePaths: [] });
const prepared = await prepareSandboxManagedRuntime({
spec: gitSpec(remoteWorkspaceDir), adapterKey: "test-adapter", client, workspaceLocalDir: localWorkspaceDir,
workspaceRestoreRetryDelaysMs: [0, 0],
});
await commitInSandbox(remoteWorkspaceDir);
const spy = step === "directory_merge"
? vi.spyOn(restoreMerge, "mergeDirectoryWithBaseline").mockRejectedValue(failure)
: vi.spyOn(gitSync, "integrateImportedGitHead").mockRejectedValue(failure);
try {
await expect(prepared.restoreWorkspace()).rejects.toBe(failure);
expect(spy).toHaveBeenCalledTimes(1);
} finally { vi.restoreAllMocks(); }
});
it("does not retry a non-Git copy-back that failed after writing a host file", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-restore-partial-"));
cleanupDirs.push(rootDir);
const localWorkspaceDir = path.join(rootDir, "local-workspace");
const remoteWorkspaceDir = path.join(rootDir, "remote-workspace");
await mkdir(localWorkspaceDir, { recursive: true });
await writeFile(path.join(localWorkspaceDir, "base.txt"), "base\n", "utf8");
const client = makeTransportClient(true, { syncOutOperations: [], readFilePaths: [] });
const prepared = await prepareSandboxManagedRuntime({
spec: gitSpec(remoteWorkspaceDir), adapterKey: "test-adapter", client, workspaceLocalDir: localWorkspaceDir,
workspaceRestoreRetryDelaysMs: [0, 0],
});
await writeFile(path.join(remoteWorkspaceDir, "a-first.txt"), "first\n", "utf8");
await writeFile(path.join(remoteWorkspaceDir, "z-later.txt"), "later\n", "utf8");
let directoryTransfers = 0;
const syncOut = client.syncOut!;
client.syncOut = async (operations) => {
if (operations.some((operation) => operation.files.some((file) => file.kind === "directory"))) directoryTransfers++;
return syncOut(operations);
};
// The merge copies entries in path order, so the first file lands before the
// later copy fails with an otherwise retryable I/O error.
const failure = Object.assign(new Error("i/o error"), { code: "EIO" });
const copyFile = fsPromises.copyFile.bind(fsPromises);
vi.spyOn(fsPromises, "copyFile").mockImplementation(async (source, target, mode) => {
if (path.basename(String(source)) === "z-later.txt") throw failure;
return copyFile(source, target, mode);
});
const merge = vi.spyOn(restoreMerge, "mergeDirectoryWithBaseline");
try {
await expect(prepared.restoreWorkspace()).rejects.toBe(failure);
await expect(readFile(path.join(localWorkspaceDir, "a-first.txt"), "utf8")).resolves.toBe("first\n");
expect(directoryTransfers).toBe(1);
expect(merge).toHaveBeenCalledTimes(1);
} finally { vi.restoreAllMocks(); }
});
it("moves the bundle through one native syncOut file mapping, never through readFile", async () => {
const capture: TransportCapture = { syncOutOperations: [], readFilePaths: [] };
const { localWorkspaceDir, remoteWorkspaceDir } = await setupGitBackedWorkspace("paperclip-bundle-native-");
@@ -35,6 +35,7 @@ import {
import {
captureDirectorySnapshot,
directorySnapshotSha256,
classifyWorkspaceRestoreFailure,
disposeDirectorySnapshot,
selectDirectorySnapshot,
mergeDirectoryWithBaseline,
@@ -320,6 +321,37 @@ async function delay(ms: number): Promise<void> {
await new Promise<void>((resolve) => setTimeout(resolve, ms));
}
/**
* Backoff before each automatic retry of a workspace copy-back: three total
* attempts. A failed copy-back otherwise retains the sandbox for board repair,
* so a dropped transfer or a busy merge lock must not end the run's restore.
*/
const WORKSPACE_RESTORE_RETRY_DELAYS_MS = [2_000, 5_000] as const;
/**
* Retry a copy-back attempt only while it has written nothing into the host
* workspace. The baseline stays valid until then, so a fresh attempt is the
* same merge. Never retry an unsafe archive or a permission failure, and never
* retry once the merge has started writing host files.
*/
async function retryWorkspaceRestoreBeforeHostWrite(
delaysMs: readonly number[],
attempt: (markHostWriteStarted: () => void) => Promise<void>,
): Promise<void> {
for (let retry = 0; ; retry++) {
let hostWriteStarted = false;
try {
await attempt(() => { hostWriteStarted = true; });
return;
} catch (error) {
const code = classifyWorkspaceRestoreFailure(error);
const retryable = code === "restore_failed" || code === "restore_lock_timeout";
if (hostWriteStarted || !retryable || retry >= delaysMs.length) throw error;
await delay(delaysMs[retry]!);
}
}
}
/**
* True only when `error` carries the workspace Git scan scheduler's typed
* saturation code on its `code` property. Matches the code alone, never
@@ -1143,6 +1175,8 @@ export async function prepareSandboxManagedRuntime(input: {
// no-op that keeps the current behavior and control flow. A throwing runner
// never changes control flow (see `createRuntimeSpanRunner`).
runtimeSpan?: RuntimeSpanRunner;
/** Backoff before each automatic workspace copy-back retry. Tests shorten it. */
workspaceRestoreRetryDelaysMs?: readonly number[];
}): Promise<PreparedSandboxManagedRuntime> {
const workspaceRemoteDir = input.workspaceRemoteDir ?? input.spec.remoteCwd;
const runtimeRootDir = input.runtimeRootDir ?? path.posix.join(workspaceRemoteDir, ".paperclip-runtime", input.adapterKey);
@@ -1865,10 +1899,13 @@ export async function prepareSandboxManagedRuntime(input: {
ignoredPaths: input.workspaceFileMode === "all" ? undefined : repository.snapshot.ignoredPaths,
}),
onRuntimeProgress: input.onRuntimeProgress,
workspaceRestoreRetryDelaysMs: input.workspaceRestoreRetryDelaysMs,
});
await nested.restoreWorkspace(restoreSink);
}
await withTempDir("paperclip-sandbox-restore-", async (tempDir) => {
await retryWorkspaceRestoreBeforeHostWrite(
input.workspaceRestoreRetryDelaysMs ?? WORKSPACE_RESTORE_RETRY_DELAYS_MS,
(markHostWriteStarted) => withTempDir("paperclip-sandbox-restore-", async (tempDir) => {
let importedRef: string | null = null;
let importedHead: string | null = null;
let remoteWorkspaceStatus = "dirty";
@@ -2081,15 +2118,16 @@ export async function prepareSandboxManagedRuntime(input: {
sourceDir: extractedDir,
...(checkpointSnapshot ? { snapshots: { source: checkpointSnapshot } } : {}),
targetDir: input.workspaceLocalDir,
beforeApply: gitHeadToIntegrate
? async () => {
await withWorkspaceRestoreStep("git_integration", () => integrateImportedGitHead({
localDir: input.workspaceLocalDir,
importedHead: gitHeadToIntegrate,
baseline: gitSnapshot ?? undefined,
}));
}
: undefined,
// Runs under the merge lock, before the first host write.
beforeApply: async () => {
markHostWriteStarted();
if (!gitHeadToIntegrate) return;
await withWorkspaceRestoreStep("git_integration", () => integrateImportedGitHead({
localDir: input.workspaceLocalDir,
importedHead: gitHeadToIntegrate,
baseline: gitSnapshot ?? undefined,
}));
},
afterApply: gitSnapshot
? async () => {
await withWorkspaceRestoreStep("index_reset", () => resetLocalGitIndexToHead({
@@ -2108,7 +2146,7 @@ export async function prepareSandboxManagedRuntime(input: {
await withWorkspaceRestoreStep("git_ref_cleanup", () => deleteLocalGitRef({ localDir: input.workspaceLocalDir, ref: importedRef! }));
}
}
});
}));
}), restoreSink, (diagnostic) => { outboundDiagnostics[0] = diagnostic; }),
);
}
@@ -314,6 +314,8 @@ export async function prepareSandboxClaudeProbeRuntime(input: {
workspaceLocalDir: tempWorkspaceDir,
workspaceRemoteDir: managedRemoteCwd,
timeoutSec: Math.max(1, input.helloProbeTimeoutSec),
// The probe workspace is a throwaway temp dir; never wait on copy-back retries.
workspaceRestoreRetryDelaysMs: [],
assets: [
{
key: "config-seed",
@@ -196,6 +196,8 @@ async function prepareCodexHelloProbe(input: {
workspaceRemoteDir: input.cwd,
installCommand: SANDBOX_INSTALL_COMMAND,
detectCommand: input.command,
// The probe workspace is a throwaway temp dir; never wait on copy-back retries.
workspaceRestoreRetryDelaysMs: [],
assets: [
{
key: "home",
@@ -171,6 +171,8 @@ export async function testEnvironment(
runId, target, adapterKey: "grok",
workspaceLocalDir: runtimeWorkspaceLocalDir,
workspaceRemoteDir: cwd,
// The probe workspace is a throwaway temp dir; never wait on copy-back retries.
workspaceRestoreRetryDelaysMs: [],
assets: [{ key: "home", localDir: stagedHome, followSymlinks: true,
restore: async ({ assetDir, readFile }) => { await copyBackGrokAuth({
readSandboxAuth: () => readFile(path.posix.join(assetDir, "auth.json")),
@@ -13,11 +13,11 @@ const {
syncDirectoryToSsh,
startAdapterExecutionTargetPaperclipBridge,
} = vi.hoisted(() => ({
runChildProcess: vi.fn(async (_runId: string, _command: string, args: string[]) => {
runChildProcess: vi.fn(async (_runId: string, _command: string, args: string[], _options?: { onLog?: (stream: "stdout" | "stderr", chunk: string) => Promise<void> }) => {
if (args.includes("models")) {
return {
exitCode: 0,
signal: null,
signal: null as string | null,
timedOut: false,
stdout: "opencode/gpt-5-nano\nopenai/gpt-4.1\n",
stderr: "",
@@ -27,7 +27,7 @@ const {
}
return {
exitCode: 0,
signal: null,
signal: null as string | null,
timedOut: false,
stdout: [
JSON.stringify({ type: "step_start", sessionID: "session_123" }),
@@ -126,6 +126,37 @@ describe("opencode remote execution", () => {
}
});
it.each([[false, false], [false, true], [true, false], [true, true]])(
"uses only forwarded credentials for remote billing (forwarded=%s, timeout=%s)", async (forwarded, timedOut) => {
vi.stubEnv("OPENROUTER_API_KEY", "host-only-api-key");
const workspace = await mkdtemp(path.join(os.tmpdir(), "opencode-remote-billing-"));
cleanupDirs.push(workspace);
const usageLine = JSON.stringify({ type: "step_finish", sessionID: "session-billing",
part: { cost: 0.001, tokens: { input: 10, output: 2, cache: { read: 1, write: 0 } } } });
runChildProcess.mockImplementationOnce(async (_runId, _command, _args, options) => {
await options?.onLog?.("stdout", `${usageLine}\n`);
return { stdout: usageLine, stderr: "", exitCode: timedOut ? 143 : 0,
signal: timedOut ? "SIGTERM" : null, timedOut, pid: 123, startedAt: new Date().toISOString() };
});
const onUsage = vi.fn();
const result = await execute({
runId: "remote-billing", agent: { id: "agent", companyId: "company", name: "OpenCode", adapterType: "opencode_local", adapterConfig: {} },
runtime: { sessionId: null, sessionParams: null, sessionDisplayId: null, taskKey: null },
config: { command: "opencode", cwd: workspace, model: "openrouter/anthropic/claude-sonnet-4.5",
env: { OPENCODE_ALLOW_ALL_MODELS: "1", ...(forwarded ? { OPENROUTER_API_KEY: "forwarded-api-key" } : {}) } },
context: {}, executionTransport: { remoteExecution: { host: "127.0.0.1", port: 2222, username: "fixture", remoteCwd: "/remote/workspace", remoteWorkspacePath: "/remote/workspace" } },
onLog: async () => {}, onUsage,
});
const expected = forwarded ? "metered_api" : "unknown";
expect(result.billingType).toBe(expected);
expect(result.timedOut).toBe(timedOut);
expect(onUsage).toHaveBeenCalledWith(expect.objectContaining({ billingType: expected, complete: false }));
expect(onUsage).toHaveBeenLastCalledWith(expect.objectContaining({ billingType: expected, complete: !timedOut }));
const runCall = runChildProcess.mock.calls.find(call => call[2].includes("run"))!;
expect((runCall[3] as { env: Record<string, string> }).env.OPENROUTER_API_KEY).toBe(forwarded ? "forwarded-api-key" : undefined);
},
);
it.each([false, true])("prepares the workspace, syncs OpenCode skills, and restores workspace changes for remote SSH execution (managed=%s)", async (managed) => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-opencode-remote-"));
cleanupDirs.push(rootDir);
@@ -177,7 +177,7 @@ describe("OpenCode local skill injection", () => {
}
});
it("passes an OpenRouter key and complete model to OpenCode without logging the key", async () => {
it.each([false, true])("preserves API billing in streamed usage and final results without leaking keys (timedOut=%s)", async (timedOut) => {
const root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-opencode-openrouter-"));
const workspace = path.join(root, "workspace");
const commandPath = path.join(root, "opencode");
@@ -187,15 +187,15 @@ describe("OpenCode local skill injection", () => {
await fs.writeFile(commandPath, "#!/bin/sh\nexit 0\n", "utf8");
await fs.chmod(commandPath, 0o755);
runProcessMock.mockReset();
runProcessMock.mockResolvedValueOnce(probeResult({
stdout: JSON.stringify({
type: "text",
sessionID: "session-openrouter",
part: { text: "done" },
}),
}));
const usageLine = JSON.stringify({ type: "step_finish", sessionID: "session-openrouter",
part: { cost: 0.0025, tokens: { input: 10, output: 2, cache: { read: 100, write: 0 } } } });
runProcessMock.mockImplementationOnce(async (...args) => {
await args[4].onLog("stdout", `${usageLine}\n`);
return probeResult({ stdout: usageLine, timedOut, exitCode: timedOut ? 143 : 0, signal: timedOut ? "SIGTERM" : null });
});
const logs: string[] = [];
const metadata: unknown[] = [];
const onUsage = vi.fn();
try {
const result = await execute({
@@ -225,6 +225,7 @@ describe("OpenCode local skill injection", () => {
},
context: {},
authToken: "run-jwt-token",
onUsage,
onLog: async (_stream, chunk) => {
logs.push(chunk);
},
@@ -233,8 +234,13 @@ describe("OpenCode local skill injection", () => {
},
});
expect(result.exitCode).toBe(0);
expect(result.exitCode).toBe(timedOut ? 143 : 0);
expect(result.usageComplete).toBe(!timedOut);
expect(result.model).toBe(model);
expect(result.billingType).toBe("metered_api");
expect(result.biller).toBe("openrouter");
expect(onUsage).toHaveBeenCalledWith(expect.objectContaining({ billingType: "metered_api", biller: "openrouter", model, complete: false }));
expect(onUsage).toHaveBeenLastCalledWith(expect.objectContaining({ billingType: "metered_api", biller: "openrouter", model, complete: !timedOut }));
const executionCall = runProcessMock.mock.calls.at(-1)!;
expect(executionCall[3]).toContain("--model");
expect(executionCall[3]).toContain(model);
@@ -49,7 +49,7 @@ describe("OpenCode timeout reporting", () => {
config: { command: "opencode", model: "openai/fixture", cwd: dir, timeoutSec,
env: { OPENCODE_ALLOW_ALL_MODELS: "1" } },
context: {}, onLog: async () => {},
...(remote ? { executionTarget: { kind: "remote", transport: "sandbox", remoteCwd: "/workspace" } } : {}),
...(remote ? { executionTarget: { kind: "remote", transport: "sandbox", providerKey: "daytona", remoteCwd: "/workspace" } } : {}),
});
it.each([
@@ -80,6 +80,40 @@ describe("OpenCode timeout reporting", () => {
expect(mocks.stop).toHaveBeenCalledTimes(1);
});
it("runs the sandbox CLI through the actual process seam with session dispatch and incremental output", async () => {
const actual = await vi.importActual<typeof import("@paperclipai/adapter-utils/execution-target")>("@paperclipai/adapter-utils/execution-target");
mocks.process.mockImplementation(actual.runAdapterExecutionTargetProcess);
const logs = vi.fn(async () => {});
const runner = {
execute: vi.fn(async (input: { useSession?: boolean; onLog?: (stream: "stdout" | "stderr", chunk: string) => Promise<void> }) => {
// A synchronous command can lose its HTTP response before the run's
// configured deadline. Session dispatch must keep that request off the
// main CLI path; it must not replay the agent to recover the response.
if (!input.useSession) return failed;
await input.onLog?.("stdout", "streamed fixture output");
return { ...failed, timedOut: false, exitCode: 0, stdout: "streamed fixture output", stderr: "" };
}),
};
const ctx = context(0);
ctx.onLog = logs;
ctx.executionTarget = { kind: "remote", transport: "sandbox", providerKey: "daytona", remoteCwd: "/workspace", runner };
const result = await execute(ctx);
expect(result).toMatchObject({ timedOut: false, exitCode: 0, errorMessage: null,
resultJson: { stdout: "streamed fixture output", adapterExecutionTimeout: { timeoutSec: 14400, source: "sandbox_default" } } });
expect(runner.execute).toHaveBeenCalledTimes(1);
expect(runner.execute.mock.calls[0]![0]).toMatchObject({ command: "opencode", useSession: true, timeoutMs: 14400000, commandTimeoutMs: 14400000 });
expect(logs).toHaveBeenCalledWith("stdout", "streamed fixture output");
expect(mocks.restore).toHaveBeenCalledTimes(1);
expect(mocks.stop).toHaveBeenCalledTimes(1);
});
it("keeps other sandbox providers on their existing command path", async () => {
const ctx = context(12);
ctx.executionTarget = { kind: "remote", transport: "sandbox", providerKey: "fixture-provider", remoteCwd: "/workspace" };
await execute(ctx);
expect(mocks.process.mock.calls[0]![4]).not.toHaveProperty("useSession");
});
it("retains real local deadline and termination behavior without claiming provider origin", async () => {
const actual = await vi.importActual<typeof import("@paperclipai/adapter-utils/execution-target")>("@paperclipai/adapter-utils/execution-target");
mocks.process.mockImplementation(actual.runAdapterExecutionTargetProcess);
@@ -61,6 +61,7 @@ import {
requireOpenCodeModelId,
} from "./models.js";
import { removeMaintainerOnlySkillSymlinks } from "@paperclipai/adapter-utils/server-utils";
import { apiKeyBillingType } from "@paperclipai/adapter-utils/api-billing-type";
import { prepareOpenCodeRuntimeConfig, prepareManagedOpenCodeRemoteHomes } from "./runtime-config.js";
import { SANDBOX_INSTALL_COMMAND } from "../index.js";
import { resolveOpenCodeSkillsHome } from "./skills.js";
@@ -632,6 +633,8 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
return args;
};
// Remote targets receive only the prepared environment, never host keys.
const accountingEnv = executionTargetIsRemote ? preparedRuntimeConfig.env : runtimeEnv;
const runAttempt = async (resumeSessionId: string | null) => {
await hydrateFreshSessionHandoff(ctx, { resumedSession: Boolean(resumeSessionId) });
const { basePrompt, promptMetrics } = buildPrompt(Boolean(resumeSessionId));
@@ -662,10 +665,14 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
const provider = parseModelProvider(model || null);
return { usage: parsed.usageReported ? parsed.usage : undefined, costUsd: parsed.costUsd,
costStatus: parsed.usageComplete || parsed.costUsd != null ? undefined : "unpriced",
usageBasis: "per_run", provider, biller: resolveOpenCodeBiller(runtimeEnv, provider), billingType: "unknown", model, complete: false };
usageBasis: "per_run", provider, biller: resolveOpenCodeBiller(accountingEnv, provider), billingType: apiKeyBillingType(provider, accountingEnv), model, complete: false };
});
const proc = await runAdapterExecutionTargetProcess(runId, runtimeExecutionTarget, command, args, {
onProcessStopped: providerStop.beginInvocation(),
// Keep a long sandbox run off the synchronous command HTTP request.
// Probes and preparation keep their existing execution paths.
...(runtimeExecutionTarget?.kind === "remote" && runtimeExecutionTarget.transport === "sandbox"
&& runtimeExecutionTarget.providerKey === "daytona" ? { useSession: true } : {}),
cwd,
env: preparedRuntimeConfig.env,
stdin: prompt,
@@ -713,9 +720,9 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
usage: attempt.parsed.usageReported ? attempt.parsed.usage : undefined,
usageBasis: "per_run",
provider: parseModelProvider(model || null),
biller: resolveOpenCodeBiller(runtimeEnv, parseModelProvider(model || null)),
biller: resolveOpenCodeBiller(accountingEnv, parseModelProvider(model || null)),
model,
billingType: "unknown",
billingType: apiKeyBillingType(parseModelProvider(model || null), accountingEnv),
costUsd: attempt.parsed.costUsd,
costStatus: attempt.parsed.usageComplete || attempt.parsed.costUsd != null ? undefined : "unpriced",
// A provider or output-observation timeout can arrive before the
@@ -772,9 +779,9 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
sessionParams: resolvedSessionParams,
sessionDisplayId: resolvedSessionId,
provider: parseModelProvider(modelId),
biller: resolveOpenCodeBiller(runtimeEnv, parseModelProvider(modelId)),
biller: resolveOpenCodeBiller(accountingEnv, parseModelProvider(modelId)),
model: modelId,
billingType: "unknown",
billingType: apiKeyBillingType(parseModelProvider(modelId), accountingEnv),
costUsd: attempt.parsed.costUsd,
costStatus: attempt.parsed.usageComplete || attempt.parsed.costUsd != null ? undefined : "unpriced",
resultJson: {
@@ -161,6 +161,8 @@ export async function testEnvironment(
workspaceRemoteDir: cwd,
installCommand: SANDBOX_INSTALL_COMMAND,
detectCommand: command,
// The probe workspace is a throwaway temp dir; never wait on copy-back retries.
workspaceRestoreRetryDelaysMs: [],
assets: localRuntimeConfigHome
? [{
key: "xdgConfig",
@@ -0,0 +1,2 @@
CREATE INDEX "cost_events_data_quality_idx" ON "cost_events" USING btree ("company_id","occurred_at" DESC NULLS LAST,"id") WHERE "cost_events"."billing_type" = 'unknown' or ("cost_events"."cost_status" = 'unpriced' and "cost_events"."billing_type" <> 'subscription_included');--> statement-breakpoint
CREATE INDEX "heartbeat_runs_cost_data_quality_idx" ON "heartbeat_runs" USING btree ("company_id",coalesce("finished_at", "created_at") desc,"id") WHERE "heartbeat_runs"."cost_accounting_pending" = true and "heartbeat_runs"."status" in ('succeeded','failed','timed_out','cancelled','interrupted');
@@ -1,12 +0,0 @@
CREATE TABLE "execution_workspace_repository_roots" (
"id" uuid PRIMARY KEY DEFAULT gen_random_uuid() NOT NULL,
"company_id" uuid NOT NULL,
"canonical_cwd" text NOT NULL,
"root_key" text NOT NULL,
"mode" text NOT NULL,
"created_at" timestamp with time zone DEFAULT now() NOT NULL,
"updated_at" timestamp with time zone DEFAULT now() NOT NULL
);
--> statement-breakpoint
ALTER TABLE "execution_workspace_repository_roots" ADD CONSTRAINT "execution_workspace_repository_roots_company_id_companies_id_fk" FOREIGN KEY ("company_id") REFERENCES "public"."companies"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint
CREATE UNIQUE INDEX "execution_workspace_repository_roots_company_root_idx" ON "execution_workspace_repository_roots" USING btree ("company_id","root_key");
@@ -0,0 +1,17 @@
-- Reapplication preserves root reservations from earlier development migration numbers.
CREATE TABLE IF NOT EXISTS "execution_workspace_repository_roots" (
"id" uuid PRIMARY KEY DEFAULT gen_random_uuid() NOT NULL,
"company_id" uuid NOT NULL,
"canonical_cwd" text NOT NULL,
"root_key" text NOT NULL,
"mode" text NOT NULL,
"created_at" timestamp with time zone DEFAULT now() NOT NULL,
"updated_at" timestamp with time zone DEFAULT now() NOT NULL
);
--> statement-breakpoint
DO $migration$ BEGIN
IF NOT EXISTS (SELECT 1 FROM pg_constraint WHERE conrelid = 'execution_workspace_repository_roots'::regclass AND conname = 'execution_workspace_repository_roots_company_id_companies_id_fk'::name) THEN
ALTER TABLE "execution_workspace_repository_roots" ADD CONSTRAINT "execution_workspace_repository_roots_company_id_companies_id_fk" FOREIGN KEY ("company_id") REFERENCES "public"."companies"("id") ON DELETE cascade ON UPDATE no action;
END IF;
END $migration$;--> statement-breakpoint
CREATE UNIQUE INDEX IF NOT EXISTS "execution_workspace_repository_roots_company_root_idx" ON "execution_workspace_repository_roots" USING btree ("company_id","root_key");
+59 -280
View File
@@ -1,5 +1,5 @@
{
"id": "64c77972-2c34-4fb7-ba72-f48dadcf794d",
"id": "fb1a65b0-7eff-4e5a-a023-fecec51f938c",
"prevId": "8dadd21e-8c02-4ada-b99a-55ba133e7f70",
"version": "7",
"dialect": "postgresql",
@@ -9858,12 +9858,6 @@
"notNull": true,
"default": false
},
"execution_defaults": {
"name": "execution_defaults",
"type": "jsonb",
"primaryKey": false,
"notNull": false
},
"metadata": {
"name": "metadata",
"type": "jsonb",
@@ -10104,12 +10098,6 @@
"notNull": true,
"default": "''"
},
"execution_defaults": {
"name": "execution_defaults",
"type": "jsonb",
"primaryKey": false,
"notNull": false
},
"allow_direct_messages": {
"name": "allow_direct_messages",
"type": "boolean",
@@ -19637,6 +19625,34 @@
"method": "btree",
"with": {}
},
"cost_events_data_quality_idx": {
"name": "cost_events_data_quality_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "occurred_at",
"isExpression": false,
"asc": false,
"nulls": "last"
},
{
"expression": "id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"where": "\"cost_events\".\"billing_type\" = 'unknown' or (\"cost_events\".\"cost_status\" = 'unpriced' and \"cost_events\".\"billing_type\" <> 'subscription_included')",
"concurrently": false,
"method": "btree",
"with": {}
},
"cost_events_company_occurred_idx": {
"name": "cost_events_company_occurred_idx",
"columns": [
@@ -26242,252 +26258,6 @@
"checkConstraints": {},
"isRLSEnabled": false
},
"public.execution_workspace_repositories": {
"name": "execution_workspace_repositories",
"schema": "",
"columns": {
"id": {
"name": "id",
"type": "uuid",
"primaryKey": true,
"notNull": true,
"default": "gen_random_uuid()"
},
"company_id": {
"name": "company_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"execution_workspace_id": {
"name": "execution_workspace_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"requested_by_issue_id": {
"name": "requested_by_issue_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"repository_identity": {
"name": "repository_identity",
"type": "text",
"primaryKey": false,
"notNull": true
},
"catalog_repository_id": {
"name": "catalog_repository_id",
"type": "text",
"primaryKey": false,
"notNull": false
},
"repo_url": {
"name": "repo_url",
"type": "text",
"primaryKey": false,
"notNull": true
},
"relative_path": {
"name": "relative_path",
"type": "text",
"primaryKey": false,
"notNull": true
},
"requested_ref": {
"name": "requested_ref",
"type": "text",
"primaryKey": false,
"notNull": true,
"default": "'HEAD'"
},
"pinned_commit": {
"name": "pinned_commit",
"type": "text",
"primaryKey": false,
"notNull": false
},
"branch_name": {
"name": "branch_name",
"type": "text",
"primaryKey": false,
"notNull": false
},
"request_key": {
"name": "request_key",
"type": "text",
"primaryKey": false,
"notNull": true
},
"request_keys": {
"name": "request_keys",
"type": "jsonb",
"primaryKey": false,
"notNull": true,
"default": "'[]'::jsonb"
},
"state": {
"name": "state",
"type": "text",
"primaryKey": false,
"notNull": true,
"default": "'pending'"
},
"failure_code": {
"name": "failure_code",
"type": "text",
"primaryKey": false,
"notNull": false
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
},
"updated_at": {
"name": "updated_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {
"execution_workspace_repositories_identity_idx": {
"name": "execution_workspace_repositories_identity_idx",
"columns": [
{
"expression": "execution_workspace_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "repository_identity",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": true,
"concurrently": false,
"method": "btree",
"with": {}
},
"execution_workspace_repositories_path_idx": {
"name": "execution_workspace_repositories_path_idx",
"columns": [
{
"expression": "execution_workspace_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "relative_path",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": true,
"concurrently": false,
"method": "btree",
"with": {}
},
"execution_workspace_repositories_request_idx": {
"name": "execution_workspace_repositories_request_idx",
"columns": [
{
"expression": "execution_workspace_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "request_key",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": true,
"concurrently": false,
"method": "btree",
"with": {}
},
"execution_workspace_repositories_company_workspace_idx": {
"name": "execution_workspace_repositories_company_workspace_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "execution_workspace_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {
"execution_workspace_repositories_company_id_companies_id_fk": {
"name": "execution_workspace_repositories_company_id_companies_id_fk",
"tableFrom": "execution_workspace_repositories",
"tableTo": "companies",
"columnsFrom": [
"company_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"execution_workspace_repositories_execution_workspace_id_execution_workspaces_id_fk": {
"name": "execution_workspace_repositories_execution_workspace_id_execution_workspaces_id_fk",
"tableFrom": "execution_workspace_repositories",
"tableTo": "execution_workspaces",
"columnsFrom": [
"execution_workspace_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"execution_workspace_repositories_requested_by_issue_id_issues_id_fk": {
"name": "execution_workspace_repositories_requested_by_issue_id_issues_id_fk",
"tableFrom": "execution_workspace_repositories",
"tableTo": "issues",
"columnsFrom": [
"requested_by_issue_id"
],
"columnsTo": [
"id"
],
"onDelete": "set null",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.execution_workspace_runtime_leases": {
"name": "execution_workspace_runtime_leases",
"schema": "",
@@ -26743,7 +26513,7 @@
"name": "project_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
"notNull": true
},
"project_workspace_id": {
"name": "project_workspace_id",
@@ -27021,7 +26791,7 @@
"columnsTo": [
"id"
],
"onDelete": "restrict",
"onDelete": "cascade",
"onUpdate": "no action"
},
"execution_workspaces_project_workspace_id_project_workspaces_id_fk": {
@@ -30617,6 +30387,34 @@
"method": "btree",
"with": {}
},
"heartbeat_runs_cost_data_quality_idx": {
"name": "heartbeat_runs_cost_data_quality_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "coalesce(\"finished_at\", \"created_at\") desc",
"asc": true,
"isExpression": true,
"nulls": "last"
},
{
"expression": "id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"where": "\"heartbeat_runs\".\"cost_accounting_pending\" = true and \"heartbeat_runs\".\"status\" in ('succeeded','failed','timed_out','cancelled','interrupted')",
"concurrently": false,
"method": "btree",
"with": {}
},
"heartbeat_runs_execution_status_delivery_idx": {
"name": "heartbeat_runs_execution_status_delivery_idx",
"columns": [
@@ -37781,25 +37579,6 @@
"primaryKey": false,
"notNull": false
},
"workspace_binding_revision": {
"name": "workspace_binding_revision",
"type": "integer",
"primaryKey": false,
"notNull": true,
"default": 0
},
"workspace_selection": {
"name": "workspace_selection",
"type": "jsonb",
"primaryKey": false,
"notNull": false
},
"workspace_pending_selection": {
"name": "workspace_pending_selection",
"type": "jsonb",
"primaryKey": false,
"notNull": false
},
"execution_workspace_preference": {
"name": "execution_workspace_preference",
"type": "text",
+58 -96
View File
@@ -1,6 +1,6 @@
{
"id": "77110fc0-2a50-45eb-913d-e507ba74439a",
"prevId": "64c77972-2c34-4fb7-ba72-f48dadcf794d",
"id": "a534b058-84e4-42be-a795-143fdd738b37",
"prevId": "fb1a65b0-7eff-4e5a-a023-fecec51f938c",
"version": "7",
"dialect": "postgresql",
"tables": {
@@ -19637,6 +19637,34 @@
"method": "btree",
"with": {}
},
"cost_events_data_quality_idx": {
"name": "cost_events_data_quality_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "occurred_at",
"isExpression": false,
"asc": false,
"nulls": "last"
},
{
"expression": "id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"where": "\"cost_events\".\"billing_type\" = 'unknown' or (\"cost_events\".\"cost_status\" = 'unpriced' and \"cost_events\".\"billing_type\" <> 'subscription_included')",
"concurrently": false,
"method": "btree",
"with": {}
},
"cost_events_company_occurred_idx": {
"name": "cost_events_company_occurred_idx",
"columns": [
@@ -26488,100 +26516,6 @@
"checkConstraints": {},
"isRLSEnabled": false
},
"public.execution_workspace_repository_roots": {
"name": "execution_workspace_repository_roots",
"schema": "",
"columns": {
"id": {
"name": "id",
"type": "uuid",
"primaryKey": true,
"notNull": true,
"default": "gen_random_uuid()"
},
"company_id": {
"name": "company_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"canonical_cwd": {
"name": "canonical_cwd",
"type": "text",
"primaryKey": false,
"notNull": true
},
"root_key": {
"name": "root_key",
"type": "text",
"primaryKey": false,
"notNull": true
},
"mode": {
"name": "mode",
"type": "text",
"primaryKey": false,
"notNull": true
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
},
"updated_at": {
"name": "updated_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {
"execution_workspace_repository_roots_company_root_idx": {
"name": "execution_workspace_repository_roots_company_root_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "root_key",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": true,
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {
"execution_workspace_repository_roots_company_id_companies_id_fk": {
"name": "execution_workspace_repository_roots_company_id_companies_id_fk",
"tableFrom": "execution_workspace_repository_roots",
"tableTo": "companies",
"columnsFrom": [
"company_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.execution_workspace_runtime_leases": {
"name": "execution_workspace_runtime_leases",
"schema": "",
@@ -30711,6 +30645,34 @@
"method": "btree",
"with": {}
},
"heartbeat_runs_cost_data_quality_idx": {
"name": "heartbeat_runs_cost_data_quality_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "coalesce(\"finished_at\", \"created_at\") desc",
"asc": true,
"isExpression": true,
"nulls": "last"
},
{
"expression": "id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"where": "\"heartbeat_runs\".\"cost_accounting_pending\" = true and \"heartbeat_runs\".\"status\" in ('succeeded','failed','timed_out','cancelled','interrupted')",
"concurrently": false,
"method": "btree",
"with": {}
},
"heartbeat_runs_execution_status_delivery_idx": {
"name": "heartbeat_runs_execution_status_delivery_idx",
"columns": [
File diff suppressed because it is too large. Load diff
+11 -4
View File
@@ -2322,15 +2322,22 @@
{
"idx": 333,
"version": "7",
"when": 1791717218544,
"tag": "0333_pretty_infant_terrible",
"when": 1791781882729,
"tag": "0333_normal_wild_pack",
"breakpoints": true
},
{
"idx": 334,
"version": "7",
"when": 1791719734041,
"tag": "0334_stiff_ted_forrester",
"when": 1791807201073,
"tag": "0334_right_hannibal_king",
"breakpoints": true
},
{
"idx": 335,
"version": "7",
"when": 1791807232294,
"tag": "0335_oval_screwball",
"breakpoints": true
}
]
+2
View File
@@ -46,6 +46,8 @@ export const costEvents = pgTable(
nonnegativeAmounts: check("cost_events_nonnegative_amounts", sql`${table.costCents} >= 0 and ${table.inputTokens} >= 0 and ${table.cachedInputTokens} >= 0 and ${table.outputTokens} >= 0`),
companyProjectOccurredIdx: index("cost_events_company_project_occurred_idx").on(table.companyId, table.projectId, table.occurredAt),
unpricedIdx: index("cost_events_unpriced_idx").on(table.companyId, table.occurredAt, table.id).where(sql`${table.costStatus} = 'unpriced' and ${table.billingType} <> 'subscription_included'`),
dataQualityIdx: index("cost_events_data_quality_idx").on(table.companyId, table.occurredAt.desc(), table.id)
.where(sql`${table.billingType} = 'unknown' or (${table.costStatus} = 'unpriced' and ${table.billingType} <> 'subscription_included')`),
companyOccurredIdx: index("cost_events_company_occurred_idx").on(table.companyId, table.occurredAt),
companySubscriptionOccurredIdx: index("cost_events_company_subscription_occurred_idx").on(table.companyId, table.subscriptionId, table.occurredAt),
companyAgentOccurredIdx: index("cost_events_company_agent_occurred_idx").on(
+3
View File
@@ -117,6 +117,9 @@ export const heartbeatRuns = pgTable(
OR ${table.scopeKind} = 'issue'`,
),
costAccountingPendingIdx: index("heartbeat_runs_cost_accounting_pending_idx").on(table.updatedAt, table.id).where(sql`${table.costAccountingPending} = true`),
costDataQualityIdx: index("heartbeat_runs_cost_data_quality_idx")
.on(table.companyId, sql`coalesce(${table.finishedAt}, ${table.createdAt}) desc`, table.id)
.where(sql`${table.costAccountingPending} = true and ${table.status} in ('succeeded','failed','timed_out','cancelled','interrupted')`),
executionStatusDeliveryIdx: index("heartbeat_runs_execution_status_delivery_idx")
.on(table.executionStatusDeliveryId).where(sql`${table.executionStatusDeliveryId} is not null`),
executionControlDeadlineIdx: index("heartbeat_runs_execution_control_deadline_idx")
@@ -1,9 +1,21 @@
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
import postgres from "postgres";
import { afterEach, describe, expect, it } from "vitest";
import { inspectMigrations } from "./client.js";
import {
__embeddedPostgresStartMaxAttemptsForTests as MAX_ATTEMPTS,
__embeddedPostgresTemplateRootDirForTests as templateRootDir,
__isUsableEmbeddedPostgresTemplateForTests as isUsableTemplate,
__lastEmbeddedPostgresStartModeForTests as lastStartMode,
__resetEmbeddedPostgresTemplateStateForTests as resetTemplateState,
__setEmbeddedPostgresCtorProviderForTests,
__startEmbeddedPostgresWithRetryForTests as startWithRetry,
computeEmbeddedPostgresTemplateKey,
EMBEDDED_POSTGRES_TEST_TIMEOUT_MS,
getEmbeddedPostgresTestSupport,
startEmbeddedPostgresTestDatabase,
} from "./test-embedded-postgres.js";
// A fake embedded-postgres constructor. It records every constructed instance so
@@ -111,3 +123,371 @@ describe("startEmbeddedPostgresWithRetry", () => {
expect((error as Error).message).toContain("Address already in use");
});
});
describe("computeEmbeddedPostgresTemplateKey", () => {
function writeMigrations(dir: string, files: Record<string, string>) {
fs.mkdirSync(path.join(dir, "meta"), { recursive: true });
fs.writeFileSync(path.join(dir, "meta", "_journal.json"), JSON.stringify({ entries: Object.keys(files) }));
for (const [name, content] of Object.entries(files)) {
fs.writeFileSync(path.join(dir, name), content);
}
}
it("is stable for identical inputs and changes with any migration, flag, or version", () => {
const root = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-template-key-"));
try {
const a = path.join(root, "a");
const b = path.join(root, "b");
writeMigrations(a, { "0000_one.sql": "create table one();", "0001_two.sql": "create table two();" });
writeMigrations(b, { "0000_one.sql": "create table one();", "0001_two.sql": "create table two();" });
const fixed = { packageVersion: "18.1.0", initdbFlags: ["--locale=C"], user: "paperclip" };
const keyA = computeEmbeddedPostgresTemplateKey(a, fixed);
expect(keyA).toMatch(/^[0-9a-f]{24}$/);
expect(computeEmbeddedPostgresTemplateKey(a, fixed)).toBe(keyA);
expect(computeEmbeddedPostgresTemplateKey(b, fixed)).toBe(keyA);
// Content change, new file, flags, version, and user each change the key.
fs.writeFileSync(path.join(b, "0001_two.sql"), "create table two(id int);");
const contentChanged = computeEmbeddedPostgresTemplateKey(b, fixed);
expect(contentChanged).not.toBe(keyA);
fs.writeFileSync(path.join(b, "0002_three.sql"), "create table three();");
expect(computeEmbeddedPostgresTemplateKey(b, fixed)).not.toBe(contentChanged);
expect(computeEmbeddedPostgresTemplateKey(a, { ...fixed, initdbFlags: ["--locale=C", "--encoding=UTF8"] })).not.toBe(keyA);
expect(computeEmbeddedPostgresTemplateKey(a, { ...fixed, packageVersion: "18.2.0" })).not.toBe(keyA);
expect(computeEmbeddedPostgresTemplateKey(a, { ...fixed, user: "other" })).not.toBe(keyA);
} finally {
fs.rmSync(root, { recursive: true, force: true });
}
});
it("hashes the real migrations directory of this package", () => {
const key = computeEmbeddedPostgresTemplateKey();
expect(key).toMatch(/^[0-9a-f]{24}$/);
expect(computeEmbeddedPostgresTemplateKey()).toBe(key);
});
});
const describeOnPosix = process.platform === "win32" ? describe.skip : describe;
describe("isUsableTemplate", () => {
let root: string;
function writeCluster(dir: string): void {
fs.mkdirSync(dir, { recursive: true, mode: 0o700 });
fs.writeFileSync(path.join(dir, "PG_VERSION"), "18\n");
}
afterEach(() => {
fs.rmSync(root, { recursive: true, force: true });
});
it("accepts a private directory that holds a cleanly stopped cluster", () => {
root = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-template-usable-"));
const template = path.join(root, "template");
writeCluster(template);
expect(isUsableTemplate(template)).toBe(true);
// A running or crashed cluster leaves `postmaster.pid` behind.
fs.writeFileSync(path.join(template, "postmaster.pid"), "1\n");
expect(isUsableTemplate(template)).toBe(false);
fs.rmSync(path.join(template, "postmaster.pid"));
expect(isUsableTemplate(template)).toBe(true);
// A directory without a cluster, or no directory at all.
fs.rmSync(path.join(template, "PG_VERSION"));
expect(isUsableTemplate(template)).toBe(false);
expect(isUsableTemplate(path.join(root, "missing"))).toBe(false);
});
it("rejects a symbolic link even when it points at a usable cluster", () => {
root = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-template-usable-"));
const template = path.join(root, "template");
writeCluster(template);
const link = path.join(root, "link");
fs.symlinkSync(template, link, "dir");
expect(isUsableTemplate(template)).toBe(true);
expect(isUsableTemplate(link)).toBe(false);
});
describeOnPosix("on POSIX", () => {
it("rejects a cluster directory that group or others can write", () => {
root = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-template-usable-"));
const template = path.join(root, "template");
writeCluster(template);
for (const mode of [0o720, 0o702, 0o777]) {
fs.chmodSync(template, mode);
expect(isUsableTemplate(template), `mode ${mode.toString(8)}`).toBe(false);
}
// Read bits for others are not a write path; Postgres narrows the copy
// to 0700 before it starts anyway.
fs.chmodSync(template, 0o755);
expect(isUsableTemplate(template)).toBe(true);
fs.chmodSync(template, 0o700);
expect(isUsableTemplate(template)).toBe(true);
});
});
});
// Real-cluster coverage of the template path. Skipped on hosts that cannot run
// embedded Postgres, like every other suite that uses the fixture.
const support = await getEmbeddedPostgresTestSupport();
const describeWithPostgres = support.supported ? describe : describe.skip;
// Points `os.tmpdir()` at a private scratch directory so a test can drive the
// cache root from an empty state. `os.tmpdir()` reads the environment on every
// call: TMPDIR on POSIX, TEMP and then TMP on Windows, so all three are set.
// Returns the scratch directory; `restore` puts the environment and the
// fixture's cached state back.
const TMPDIR_VARIABLES = ["TMPDIR", "TEMP", "TMP"] as const;
function useScratchTmpdir(prefix: string): { scratch: string; restore(): void } {
const scratch = fs.mkdtempSync(path.join(os.tmpdir(), prefix));
const previous = new Map(TMPDIR_VARIABLES.map((name) => [name, process.env[name]] as const));
for (const name of TMPDIR_VARIABLES) process.env[name] = scratch;
return {
scratch,
restore() {
for (const [name, value] of previous) {
if (value === undefined) delete process.env[name];
else process.env[name] = value;
}
resetTemplateState();
fs.rmSync(scratch, { recursive: true, force: true });
},
};
}
describeWithPostgres("startEmbeddedPostgresTestDatabase template path", () => {
const originalTemplateSetting = process.env.PAPERCLIP_TEST_POSTGRES_TEMPLATE;
afterEach(() => {
if (originalTemplateSetting === undefined) delete process.env.PAPERCLIP_TEST_POSTGRES_TEMPLATE;
else process.env.PAPERCLIP_TEST_POSTGRES_TEMPLATE = originalTemplateSetting;
});
it("publishes a complete template and starts suites from a copy of it", { timeout: EMBEDDED_POSTGRES_TEST_TIMEOUT_MS }, async () => {
// This test covers the template path, so it enables the cache even when
// the suite runs under PAPERCLIP_TEST_POSTGRES_TEMPLATE=0. The first
// start then builds the template if the support probe did not.
delete process.env.PAPERCLIP_TEST_POSTGRES_TEMPLATE;
const database = await startEmbeddedPostgresTestDatabase("paperclip-template-copy-");
try {
expect(lastStartMode()).toBe("template");
// The copy is fully migrated and independent of the template.
const state = await inspectMigrations(database.connectionString);
expect(state.status).toBe("upToDate");
const sql = postgres(database.connectionString, { max: 1 });
try {
await sql`create table paperclip_template_copy_probe (id int)`;
} finally {
await sql.end();
}
} finally {
await database.cleanup();
}
const rootDir = templateRootDir();
const templateDir = path.join(rootDir, computeEmbeddedPostgresTemplateKey());
expect(fs.existsSync(path.join(templateDir, "PG_VERSION"))).toBe(true);
expect(fs.existsSync(path.join(templateDir, "postmaster.pid"))).toBe(false);
expect(isUsableTemplate(templateDir)).toBe(true);
if (process.platform !== "win32") {
// The root is private to this user, and the template is a data
// directory Postgres accepts: both are 0700.
expect(path.basename(rootDir)).toBe(`paperclip-embedded-postgres-templates-${process.getuid?.()}`);
expect(fs.statSync(rootDir).mode & 0o777).toBe(0o700);
expect(fs.statSync(templateDir).mode & 0o777).toBe(0o700);
}
// Writing to the copy never touches the template, so a second copy does
// not see the table.
const second = await startEmbeddedPostgresTestDatabase("paperclip-template-copy-");
try {
expect(lastStartMode()).toBe("template");
const sql = postgres(second.connectionString, { max: 1 });
try {
const rows = await sql`select to_regclass('paperclip_template_copy_probe') as name`;
expect(rows[0]?.name).toBeNull();
} finally {
await sql.end();
}
} finally {
await second.cleanup();
}
});
it("refreshes the template's timestamp on every reuse so idle time drives pruning", { timeout: EMBEDDED_POSTGRES_TEST_TIMEOUT_MS }, async () => {
delete process.env.PAPERCLIP_TEST_POSTGRES_TEMPLATE;
// Make sure a template exists before ageing it: the support probe does
// not publish one under PAPERCLIP_TEST_POSTGRES_TEMPLATE=0, and a name
// filter can run this test on its own.
resetTemplateState();
const first = await startEmbeddedPostgresTestDatabase("paperclip-template-touch-");
await first.cleanup();
expect(lastStartMode()).toBe("template");
const templateDir = path.join(templateRootDir(), computeEmbeddedPostgresTemplateKey());
expect(isUsableTemplate(templateDir)).toBe(true);
// Age the template as if it had not been used for ten days.
const old = new Date(Date.now() - 10 * 24 * 60 * 60 * 1000);
fs.utimesSync(templateDir, old, old);
expect(fs.statSync(templateDir).mtimeMs).toBeLessThan(Date.now() - 9 * 24 * 60 * 60 * 1000);
resetTemplateState();
const database = await startEmbeddedPostgresTestDatabase("paperclip-template-touch-");
try {
expect(lastStartMode()).toBe("template");
} finally {
await database.cleanup();
}
expect(fs.statSync(templateDir).mtimeMs).toBeGreaterThan(Date.now() - 60 * 1000);
});
it("rebuilds the template when it was pruned after this process resolved it", { timeout: EMBEDDED_POSTGRES_TEST_TIMEOUT_MS }, async () => {
delete process.env.PAPERCLIP_TEST_POSTGRES_TEMPLATE;
const scratch = useScratchTmpdir("paperclip-template-pruned-");
try {
resetTemplateState();
const first = await startEmbeddedPostgresTestDatabase("paperclip-template-pruned-copy-");
await first.cleanup();
expect(lastStartMode()).toBe("template");
const templateDir = path.join(templateRootDir(), computeEmbeddedPostgresTemplateKey());
expect(isUsableTemplate(templateDir)).toBe(true);
// Another process prunes the template while this one still holds its path.
fs.rmSync(templateDir, { recursive: true, force: true });
const second = await startEmbeddedPostgresTestDatabase("paperclip-template-pruned-copy-");
await second.cleanup();
// This start fell back without retrying the missing path; the fallback
// forgot the stale path, so the next start rebuilt the template.
expect(lastStartMode()).toBe("fresh");
const third = await startEmbeddedPostgresTestDatabase("paperclip-template-pruned-copy-");
await third.cleanup();
expect(lastStartMode()).toBe("template");
expect(isUsableTemplate(templateDir)).toBe(true);
} finally {
scratch.restore();
}
});
it("runs the fresh initdb path when PAPERCLIP_TEST_POSTGRES_TEMPLATE=0", { timeout: EMBEDDED_POSTGRES_TEST_TIMEOUT_MS }, async () => {
process.env.PAPERCLIP_TEST_POSTGRES_TEMPLATE = "0";
const database = await startEmbeddedPostgresTestDatabase("paperclip-template-bypass-");
try {
expect(lastStartMode()).toBe("fresh");
const state = await inspectMigrations(database.connectionString);
expect(state.status).toBe("upToDate");
} finally {
await database.cleanup();
}
});
describeOnPosix("cache root safety", () => {
it("uses fresh clusters when the cache root is not a private directory of this user", { timeout: EMBEDDED_POSTGRES_TEST_TIMEOUT_MS }, async () => {
delete process.env.PAPERCLIP_TEST_POSTGRES_TEMPLATE;
const scratch = useScratchTmpdir("paperclip-template-root-");
const warnings: string[] = [];
const warn = console.warn;
console.warn = (...args: unknown[]) => {
warnings.push(args.map(String).join(" "));
};
try {
// Someone pre-created the predictable root path, world-writable, the
// way a foreign user on a shared host would.
const rootDir = templateRootDir();
expect(rootDir.startsWith(scratch.scratch)).toBe(true);
fs.mkdirSync(rootDir);
fs.chmodSync(rootDir, 0o777);
resetTemplateState();
const database = await startEmbeddedPostgresTestDatabase("paperclip-template-root-copy-");
try {
expect(lastStartMode()).toBe("fresh");
} finally {
await database.cleanup();
}
// Nothing was published into the rejected root, and the reason was logged once.
expect(fs.readdirSync(rootDir)).toEqual([]);
expect(warnings.filter((line) => line.includes("template cache root"))).toHaveLength(1);
expect(warnings[0]).toContain("writable by group or others");
// The probe still reports support, from the throwaway cluster.
resetTemplateState();
expect(await getEmbeddedPostgresTestSupport()).toEqual({ supported: true });
} finally {
console.warn = warn;
scratch.restore();
}
});
it("uses fresh clusters when the cache root path is a symbolic link", { timeout: EMBEDDED_POSTGRES_TEST_TIMEOUT_MS }, async () => {
delete process.env.PAPERCLIP_TEST_POSTGRES_TEMPLATE;
const scratch = useScratchTmpdir("paperclip-template-root-");
const warn = console.warn;
console.warn = () => {};
try {
const target = path.join(scratch.scratch, "elsewhere");
fs.mkdirSync(target, { mode: 0o700 });
fs.symlinkSync(target, templateRootDir(), "dir");
resetTemplateState();
const database = await startEmbeddedPostgresTestDatabase("paperclip-template-root-copy-");
try {
expect(lastStartMode()).toBe("fresh");
} finally {
await database.cleanup();
}
expect(fs.readdirSync(target)).toEqual([]);
} finally {
console.warn = warn;
scratch.restore();
}
});
});
});
describe("support probe when the template cannot be built", () => {
afterEach(() => {
__setEmbeddedPostgresCtorProviderForTests(null);
resetTemplateState();
});
it("falls through to the fresh probe instead of reporting the host as unsupported", { timeout: EMBEDDED_POSTGRES_TEST_TIMEOUT_MS }, async () => {
const originalTemplateSetting = process.env.PAPERCLIP_TEST_POSTGRES_TEMPLATE;
delete process.env.PAPERCLIP_TEST_POSTGRES_TEMPLATE;
const scratch = useScratchTmpdir("paperclip-template-build-fail-");
const warnings: string[] = [];
const warn = console.warn;
console.warn = (...args: unknown[]) => {
warnings.push(args.map(String).join(" "));
};
try {
// This test runs on every platform, so check the redirect took effect
// before asserting on paths under it.
expect(os.tmpdir()).toBe(scratch.scratch);
// The fake cluster "starts" without listening, so the template build
// fails after the start, at the migration step, the way a broken
// migration would. A real broken migration then fails the suite on
// the fresh path instead of being hidden by `describe.skip`.
const { ctor, constructed } = makeFakeCtor(0);
__setEmbeddedPostgresCtorProviderForTests(async () => ctor);
resetTemplateState();
expect(await getEmbeddedPostgresTestSupport()).toEqual({ supported: true });
// One build attempt, then the throwaway probe, both under the scratch root.
expect(constructed).toHaveLength(2);
for (const options of constructed) expect(options.databaseDir.startsWith(scratch.scratch)).toBe(true);
expect(constructed[0]?.databaseDir).toContain("paperclip-embedded-postgres-template-build-");
expect(constructed[1]?.databaseDir).toContain("paperclip-embedded-postgres-probe-");
expect(warnings.filter((line) => line.includes("template build failed"))).toHaveLength(1);
// Nothing was published.
const rootDir = templateRootDir();
expect(fs.existsSync(rootDir) ? fs.readdirSync(rootDir) : []).toEqual([]);
} finally {
console.warn = warn;
scratch.restore();
if (originalTemplateSetting === undefined) delete process.env.PAPERCLIP_TEST_POSTGRES_TEMPLATE;
else process.env.PAPERCLIP_TEST_POSTGRES_TEMPLATE = originalTemplateSetting;
}
});
});
+440 -9
View File
@@ -1,7 +1,10 @@
import { createHash } from "node:crypto";
import fs from "node:fs";
import { createRequire } from "node:module";
import net from "node:net";
import os from "node:os";
import path from "node:path";
import { fileURLToPath } from "node:url";
import { applyPendingMigrations, closeRegisteredClients, ensurePostgresDatabase } from "./client.js";
import {
createEmbeddedPostgresLogBuffer,
@@ -113,8 +116,19 @@ async function getAvailablePort(): Promise<number> {
);
}
const EMBEDDED_POSTGRES_USER = "paperclip";
const EMBEDDED_POSTGRES_PASSWORD = "paperclip";
const EMBEDDED_POSTGRES_INITDB_FLAGS = ["--encoding=UTF8", "--locale=C", "--lc-messages=C"];
async function createEmbeddedPostgresTestInstance(tempDirPrefix: string) {
const dataDir = fs.mkdtempSync(path.join(os.tmpdir(), tempDirPrefix));
return await createEmbeddedPostgresTestInstanceAt(dataDir);
}
// Wraps an embedded-postgres instance around `dataDir`. The caller decides
// whether the directory still needs `initialise()` (a fresh cluster) or already
// holds a copied cluster that only needs `start()`.
async function createEmbeddedPostgresTestInstanceAt(dataDir: string) {
const port = await getAvailablePort();
const EmbeddedPostgres = await getEmbeddedPostgresCtor();
// Postgres writes the true reason for a failed start to its output, for
@@ -124,11 +138,11 @@ async function createEmbeddedPostgresTestInstance(tempDirPrefix: string) {
const logBuffer = createEmbeddedPostgresLogBuffer();
const instance = new EmbeddedPostgres({
databaseDir: dataDir,
user: "paperclip",
password: "paperclip",
user: EMBEDDED_POSTGRES_USER,
password: EMBEDDED_POSTGRES_PASSWORD,
port,
persistent: true,
initdbFlags: ["--encoding=UTF8", "--locale=C", "--lc-messages=C"],
initdbFlags: EMBEDDED_POSTGRES_INITDB_FLAGS,
onLog: (message) => logBuffer.append(message),
onError: (message) => logBuffer.append(message),
});
@@ -136,6 +150,14 @@ async function createEmbeddedPostgresTestInstance(tempDirPrefix: string) {
return { dataDir, port, instance, getRecentLogs: () => logBuffer.getRecentLogs() };
}
function adminConnectionStringFor(port: number): string {
return `postgres://${EMBEDDED_POSTGRES_USER}:${EMBEDDED_POSTGRES_PASSWORD}@127.0.0.1:${port}/postgres`;
}
function databaseConnectionStringFor(port: number): string {
return `postgres://${EMBEDDED_POSTGRES_USER}:${EMBEDDED_POSTGRES_PASSWORD}@127.0.0.1:${port}/paperclip`;
}
function cleanupEmbeddedPostgresTestDirs(dataDir: string) {
fs.rmSync(dataDir, { recursive: true, force: true });
}
@@ -237,13 +259,391 @@ async function startEmbeddedPostgresWithRetry(tempDirPrefix: string): Promise<{
);
}
// ---------------------------------------------------------------------------
// Template cluster cache
//
// Every suite that calls `startEmbeddedPostgresTestDatabase` used to pay for
// `initdb`, a cluster start, and all pending migrations (331 files at the time
// of writing) before its first test ran, and the `getEmbeddedPostgresTestSupport`
// probe started and stopped a second throwaway cluster per test file. On the
// serial CI shards that was 5-6s per suite plus 2.5-3s per probe; measured
// locally it is 3.2s per suite plus 0.75s per probe.
//
// A cluster data directory that was shut down cleanly can be copied and started
// again as an independent cluster. So the first caller on a host builds one
// fully migrated cluster, stops it, and publishes the data directory as a
// template under the OS temp directory. Every later caller copies the template
// (about 90 MB, ~0.2s) and starts Postgres on the copy (~0.05s), then still runs
// `ensurePostgresDatabase` and `applyPendingMigrations` so the returned database
// is exactly what the fresh path produces; both are no-ops on a template copy.
//
// The template key hashes everything that determines the cluster's contents:
// the embedded-postgres package version, the initdb flags, the user, and the
// name and content of every migration file plus the journal. A changed
// migration therefore builds a new template instead of reusing a stale one.
// Publishing is a rename, which is atomic on the same filesystem, so a
// concurrent builder either wins the rename or discards its copy and uses the
// winner's; a template that exists is always complete.
//
// A template is a Postgres data directory, and starting one executes its
// configuration (for example `shared_preload_libraries`). The cache root is
// therefore private to the current OS user: its name carries the user id, it
// is created with mode 0700, and it is only used when it is a real directory
// (not a symlink) owned by this user with no group or world write bit. Each
// template is checked the same way before it is copied. On a host with a
// shared temp directory another user can pre-create the predictable path, and
// those checks reject it. Windows has no uid; its temp directory is already
// per user, so the ownership and mode checks are skipped there.
//
// Set PAPERCLIP_TEST_POSTGRES_TEMPLATE=0 to bypass the cache and run the fresh
// path for every suite. Any failure on the template path (root checks, build,
// copy, start, or publish) falls back to the fresh path, so a broken cache can
// make a suite slower but never fail it or skip it: the fresh path still
// reports a migration that cannot apply or a cluster that cannot start.
// ---------------------------------------------------------------------------
const TEMPLATE_ROOT_DIR_NAME = "paperclip-embedded-postgres-templates";
// Templates whose key no longer matches are pruned once they have not been
// used for this long. Every reuse refreshes the template's timestamp.
const TEMPLATE_PRUNE_AGE_MS = 7 * 24 * 60 * 60 * 1000;
// Publishing a template requires the builder cluster to have exited for real,
// so its data files are quiescent before the rename. Give that stop more room
// than the per-suite bounded stop; a template build happens once per host.
const TEMPLATE_BUILD_STOP_TIMEOUT_MS = 30_000;
export type EmbeddedPostgresStartMode = "template" | "fresh";
function isTemplateCacheDisabled(): boolean {
return process.env.PAPERCLIP_TEST_POSTGRES_TEMPLATE === "0";
}
function migrationsDirectory(): string {
return fileURLToPath(new URL("./migrations/", import.meta.url));
}
function embeddedPostgresPackageVersion(): string {
try {
const require = createRequire(import.meta.url);
const manifest = require("embedded-postgres/package.json") as { version?: unknown };
return typeof manifest.version === "string" ? manifest.version : "unknown";
} catch {
return "unknown";
}
}
// Hash of everything that determines the contents of a migrated cluster. Reads
// every migration file, which costs a few milliseconds per process.
export function computeEmbeddedPostgresTemplateKey(
migrationsDir: string = migrationsDirectory(),
input: { packageVersion?: string; initdbFlags?: string[]; user?: string } = {},
): string {
const hash = createHash("sha256");
hash.update(`embedded-postgres=${input.packageVersion ?? embeddedPostgresPackageVersion()}\n`);
hash.update(`initdb=${(input.initdbFlags ?? EMBEDDED_POSTGRES_INITDB_FLAGS).join(" ")}\n`);
hash.update(`user=${input.user ?? EMBEDDED_POSTGRES_USER}\n`);
const journalPath = path.join(migrationsDir, "meta", "_journal.json");
if (fs.existsSync(journalPath)) {
hash.update(`journal\n`);
hash.update(fs.readFileSync(journalPath));
hash.update(`\n`);
}
const migrationFiles = fs
.readdirSync(migrationsDir)
.filter((name) => name.endsWith(".sql"))
.sort((a, b) => a.localeCompare(b));
for (const name of migrationFiles) {
hash.update(`${name}\n`);
hash.update(fs.readFileSync(path.join(migrationsDir, name)));
hash.update(`\n`);
}
return hash.digest("hex").slice(0, 24);
}
// The uid on POSIX, else the user name. Part of the cache root's name so two
// users on one host never share a root.
function templateOwnerTag(): string {
const uid = process.getuid?.();
if (uid !== undefined) return String(uid);
let username = "user";
try {
username = os.userInfo().username;
} catch {
// Fall through to the generic tag.
}
return username.replace(/[^A-Za-z0-9_.-]/g, "_");
}
function templateRootDir(): string {
return path.join(os.tmpdir(), `${TEMPLATE_ROOT_DIR_NAME}-${templateOwnerTag()}`);
}
// Why `dir` cannot be trusted as a private directory of the current user, or
// null when it can. Uses `lstat` so a symlink planted at the path is rejected
// rather than followed. On Windows (no uid) only the directory check applies.
function describePrivateDirectoryProblem(dir: string): string | null {
let stat: fs.Stats;
try {
stat = fs.lstatSync(dir);
} catch (error) {
return `cannot be inspected (${(error as NodeJS.ErrnoException).code ?? "unknown error"})`;
}
if (stat.isSymbolicLink()) return "is a symbolic link";
if (!stat.isDirectory()) return "is not a directory";
const uid = process.getuid?.();
if (uid === undefined) return null;
if (stat.uid !== uid) return `is owned by uid ${stat.uid}, not the current user (uid ${uid})`;
if ((stat.mode & 0o022) !== 0) {
return `is writable by group or others (mode ${(stat.mode & 0o777).toString(8)})`;
}
return null;
}
// Creates the cache root with mode 0700 and verifies it is private to the
// current user. Returns null, after one warning, when it is not: the cache is
// then unused for this process and every suite takes the fresh path.
function prepareTemplateRootDir(): string | null {
const rootDir = templateRootDir();
try {
fs.mkdirSync(rootDir, { recursive: true, mode: 0o700 });
const problem = describePrivateDirectoryProblem(rootDir);
if (problem !== null) {
warnTemplateOnce("root", `template cache root ${rootDir} ${problem}; using fresh clusters`);
return null;
}
// A root created by an earlier version may be wider than 0700. It is
// ours (checked above), so narrow it.
if (process.getuid !== undefined && (fs.lstatSync(rootDir).mode & 0o777) !== 0o700) {
fs.chmodSync(rootDir, 0o700);
}
return rootDir;
} catch (error) {
warnTemplateOnce(
"root",
`template cache root ${rootDir} could not be prepared: ${(error as Error).message}; using fresh clusters`,
);
return null;
}
}
// A template is usable when it is a private directory of the current user
// that holds a cleanly stopped cluster: `PG_VERSION` present, no
// `postmaster.pid`. The ownership check repeats the root's check so a
// template is never copied from a directory this user does not own.
function isUsableTemplate(dir: string): boolean {
if (describePrivateDirectoryProblem(dir) !== null) return false;
return fs.existsSync(path.join(dir, "PG_VERSION")) && !fs.existsSync(path.join(dir, "postmaster.pid"));
}
// Records a use of the template so `pruneStaleTemplates` measures idle time,
// not age. Best effort.
function touchTemplate(dir: string): void {
try {
const now = new Date();
fs.utimesSync(dir, now, now);
} catch {
// The template still works without the refreshed timestamp.
}
}
const warnedTemplateTopics = new Set<string>();
function warnTemplateOnce(topic: string, message: string): void {
if (warnedTemplateTopics.has(topic)) return;
warnedTemplateTopics.add(topic);
console.warn(`[embedded-postgres] ${message}`);
}
// Best-effort removal of templates for other keys that have not been used for
// a week, plus abandoned build directories. Every reuse refreshes a template's
// timestamp, so a template that any process resolved in the last week stays.
// A suite never depends on the template after its copy is made, so pruning
// can only affect a process that resolved the path and has not copied yet;
// that process then rebuilds the template instead of failing. Never throws.
function pruneStaleTemplates(rootDir: string, keepDir: string): void {
let entries: fs.Dirent[];
try {
entries = fs.readdirSync(rootDir, { withFileTypes: true });
} catch {
return;
}
const now = Date.now();
for (const entry of entries) {
const entryPath = path.join(rootDir, entry.name);
if (entryPath === keepDir || !entry.isDirectory()) continue;
try {
const age = now - fs.statSync(entryPath).mtimeMs;
if (age > TEMPLATE_PRUNE_AGE_MS) fs.rmSync(entryPath, { recursive: true, force: true });
} catch {
// Another process may be using or removing it; leave it alone.
}
}
}
// Waits for the raw `stop()` so the builder cluster has really exited before
// its data directory is published. Returns false if the stop overran.
async function stopForTemplatePublish(instance: EmbeddedPostgresInstance): Promise<boolean> {
let timer: ReturnType<typeof setTimeout> | undefined;
try {
return await Promise.race([
instance.stop().then(() => true, () => false),
new Promise<boolean>((resolve) => {
timer = setTimeout(() => resolve(false), TEMPLATE_BUILD_STOP_TIMEOUT_MS);
timer.unref?.();
}),
]);
} finally {
if (timer) clearTimeout(timer);
}
}
// Builds a migrated cluster and publishes its data directory at `templateDir`.
// Throws when the cluster cannot start or a migration cannot apply. Returns
// false when the cluster started but the template could not be published;
// callers then use the fresh path for this process.
async function buildEmbeddedPostgresTemplate(templateDir: string): Promise<boolean> {
const rootDir = path.dirname(templateDir);
const { port, dataDir, instance } = await startEmbeddedPostgresWithRetry(
"paperclip-embedded-postgres-template-build-",
);
let published = false;
try {
await ensurePostgresDatabase(adminConnectionStringFor(port), "paperclip");
await applyPendingMigrations(databaseConnectionStringFor(port));
const stopped = await stopForTemplatePublish(instance);
if (!stopped || !isUsableTemplate(dataDir)) return false;
try {
fs.renameSync(dataDir, templateDir);
published = true;
} catch (error) {
// Another process published the same key first. Use theirs.
const code = (error as NodeJS.ErrnoException).code;
if (code === "EEXIST" || code === "ENOTEMPTY" || code === "EPERM") {
return isUsableTemplate(templateDir);
}
throw error;
}
pruneStaleTemplates(rootDir, templateDir);
return true;
} finally {
if (!published) {
await stopEmbeddedPostgresBounded(instance, () => cleanupEmbeddedPostgresTestDirs(dataDir));
}
}
}
let templateDirPromise: Promise<string | null> | null = null;
// Resolves the template data directory for this host, building it on first
// use. Resolves null when the cache is disabled, the cache root is not private
// to this user, or the template could not be built or published; every caller
// then takes the fresh path, which reports the real error itself (a cluster
// that cannot start, a migration that cannot apply). Never rejects, so a
// broken cache can never turn into `supported: false` and skip a suite.
async function getEmbeddedPostgresTemplateDir(): Promise<string | null> {
if (isTemplateCacheDisabled()) return null;
if (!templateDirPromise) {
templateDirPromise = (async () => {
const rootDir = prepareTemplateRootDir();
if (rootDir === null) return null;
const templateDir = path.join(rootDir, computeEmbeddedPostgresTemplateKey());
if (isUsableTemplate(templateDir)) {
touchTemplate(templateDir);
return templateDir;
}
try {
const built = await buildEmbeddedPostgresTemplate(templateDir);
return built && isUsableTemplate(templateDir) ? templateDir : null;
} catch (error) {
warnTemplateOnce(
"build",
`template build failed, using fresh clusters: ${(error as Error).message}`,
);
return null;
}
})();
}
return await templateDirPromise;
}
// Forgets the resolved template so the next caller in this process checks the
// cache again. Used after a template start failed, for example because the
// template was pruned by another process after this one resolved its path.
function resetTemplateDirCache(): void {
templateDirPromise = null;
}
// Copies the template and starts a cluster on the copy. Each attempt uses a
// fresh copy and a fresh port, mirroring `startEmbeddedPostgresWithRetry`. A
// template that is no longer usable (pruned, or no longer private) is not
// retried: the caller falls back and the next caller rebuilds it.
async function startEmbeddedPostgresFromTemplateWithRetry(
templateDir: string,
tempDirPrefix: string,
): Promise<{ port: number; dataDir: string; instance: EmbeddedPostgresInstance }> {
let lastError = new Error("embedded Postgres startup from template failed");
for (let attempt = 1; attempt <= EMBEDDED_POSTGRES_START_MAX_ATTEMPTS; attempt += 1) {
if (!isUsableTemplate(templateDir)) {
throw new Error(`embedded Postgres template ${templateDir} is no longer usable`);
}
const rootDir = fs.mkdtempSync(path.join(os.tmpdir(), tempDirPrefix));
const dataDir = path.join(rootDir, "data");
let created: Awaited<ReturnType<typeof createEmbeddedPostgresTestInstanceAt>> | null = null;
try {
fs.cpSync(templateDir, dataDir, { recursive: true });
// Postgres refuses a data directory that is group or world accessible.
fs.chmodSync(dataDir, 0o700);
created = await createEmbeddedPostgresTestInstanceAt(dataDir);
await created.instance.start();
return { port: created.port, dataDir: rootDir, instance: created.instance };
} catch (error) {
lastError = formatEmbeddedPostgresError(error, {
fallbackMessage: "embedded Postgres startup from template failed",
recentLogs: created?.getRecentLogs(),
});
await stopEmbeddedPostgresBounded(created?.instance ?? null, () =>
cleanupEmbeddedPostgresTestDirs(rootDir),
);
}
}
throw new Error(
`Failed to start embedded PostgreSQL from template after ${EMBEDDED_POSTGRES_START_MAX_ATTEMPTS} attempts: ${lastError.message}`,
);
}
let templateFallbackWarned = false;
let lastStartMode: EmbeddedPostgresStartMode | null = null;
// Test-only accessors. Production callers use `startEmbeddedPostgresTestDatabase`
// or `getEmbeddedPostgresTestSupport`. A test drives the bounded retry directly
// so it does not need a real Postgres connection.
export const __startEmbeddedPostgresWithRetryForTests = startEmbeddedPostgresWithRetry;
export const __embeddedPostgresStartMaxAttemptsForTests = EMBEDDED_POSTGRES_START_MAX_ATTEMPTS;
export const __embeddedPostgresTemplateRootDirForTests = templateRootDir;
export const __isUsableEmbeddedPostgresTemplateForTests = isUsableTemplate;
export function __lastEmbeddedPostgresStartModeForTests(): EmbeddedPostgresStartMode | null {
return lastStartMode;
}
// Forgets the support probe result, the resolved template, and the one-time
// warnings so a test can drive the probe and the cache again in one process.
export function __resetEmbeddedPostgresTemplateStateForTests(): void {
embeddedPostgresSupportPromise = null;
templateDirPromise = null;
templateFallbackWarned = false;
warnedTemplateTopics.clear();
lastStartMode = null;
}
async function probeEmbeddedPostgresSupport(): Promise<EmbeddedPostgresTestSupport> {
// A published template proves this host can run the cluster: it was built
// here. Reusing it makes the probe free for every file after the first.
// Building it costs one cluster start, the same as the throwaway probe did.
// When the template cannot be built or used, the throwaway probe below
// decides, so only a cluster that cannot start reports `supported: false`;
// a migration that cannot apply fails the suite loudly on the fresh path.
if ((await getEmbeddedPostgresTemplateDir()) !== null) return { supported: true };
let started: { dataDir: string; instance: EmbeddedPostgresInstance } | null = null;
try {
@@ -271,17 +671,48 @@ export async function getEmbeddedPostgresTestSupport(): Promise<EmbeddedPostgres
return await embeddedPostgresSupportPromise;
}
// Starts the cluster for one suite: from the host template when one is
// available, otherwise a fresh `initdb`. Template failures fall back to the
// fresh path so the cache can only make a suite faster, never fail it.
async function startEmbeddedPostgresClusterForSuite(
tempDirPrefix: string,
): Promise<{ port: number; dataDir: string; instance: EmbeddedPostgresInstance; mode: EmbeddedPostgresStartMode }> {
const templateDir = await getEmbeddedPostgresTemplateDir();
if (templateDir !== null) {
try {
const started = await startEmbeddedPostgresFromTemplateWithRetry(templateDir, tempDirPrefix);
return { ...started, mode: "template" };
} catch (error) {
// The next caller in this process re-resolves the template: if it was
// pruned it is rebuilt, instead of retrying a path that is gone.
resetTemplateDirCache();
if (!templateFallbackWarned) {
templateFallbackWarned = true;
console.warn(
`[embedded-postgres] template start failed, using a fresh cluster: ${(error as Error).message}`,
);
}
}
}
// The bounded retry hardens the cluster start against the port race. It throws
// with the real Postgres output if every attempt fails.
const started = await startEmbeddedPostgresWithRetry(tempDirPrefix);
return { ...started, mode: "fresh" };
}
export async function startEmbeddedPostgresTestDatabase(
tempDirPrefix: string,
): Promise<EmbeddedPostgresTestDatabase> {
// The bounded retry hardens the cluster start against the port race. It throws
// with the real Postgres output if every attempt fails.
const { port, dataDir, instance } = await startEmbeddedPostgresWithRetry(tempDirPrefix);
const { port, dataDir, instance, mode } = await startEmbeddedPostgresClusterForSuite(tempDirPrefix);
lastStartMode = mode;
try {
const adminConnectionString = `postgres://paperclip:paperclip@127.0.0.1:${port}/postgres`;
await ensurePostgresDatabase(adminConnectionString, "paperclip");
const connectionString = `postgres://paperclip:paperclip@127.0.0.1:${port}/paperclip`;
// Both calls are no-ops on a template copy and keep the returned database
// identical to the fresh path even if the template were ever stale.
await ensurePostgresDatabase(adminConnectionStringFor(port), "paperclip");
const connectionString = databaseConnectionStringFor(port);
await applyPendingMigrations(connectionString);
return {
@@ -844,6 +844,7 @@ fn stderr_diagnostic_categories(value: &str) -> BTreeSet<&'static str> {
fn response_error_classification(error: &ResponseError) -> &'static str {
match error.code.as_str() {
"ACPX_RUNTIME_PACKAGE_MISSING" => return "runtime_package_missing",
"ACP_MODEL_UNSUPPORTED" => return "requested_model_unsupported",
"AGENT_STARTUP_FAILED" => return "agent_startup_failed",
"AGENT_STARTUP_FAILED.UNVERIFIED_MODULE" => return "agent_startup_unverified_module",
@@ -1038,6 +1039,7 @@ mod tests {
#[test]
fn candidate_auth_diagnostics_use_only_closed_codes_and_never_provider_text() {
for (code, expected) in [
("ACPX_RUNTIME_PACKAGE_MISSING", "runtime_package_missing"),
("ACPX_TOOL_CALL_STALE", "provider_tool_call_retired"),
("ACPX_TOOL_CALL_STALE_EXTRA", "unclassified"),
("COPILOT_AUTH_REQUIRED", "authentication_required"),
@@ -186,3 +186,9 @@ describe("ACPX sidecar input sequencing", () => {
);
});
});
it("preserves a missing runtime package category across wrapped bootstrap errors", () => {
const error = Object.assign(new Error("private resolver stack"), { code: "ACPX_RUNTIME_PACKAGE_MISSING" });
expect(acpxSidecarErrorCode(new Error("bootstrap failed", { cause: error }))).toBe("ACPX_RUNTIME_PACKAGE_MISSING");
expect(acpxSidecarErrorCode(acpxBootstrapBlockedError(error, "session.read")!)).toBe("ACPX_RUNTIME_PACKAGE_MISSING");
});
@@ -25,6 +25,7 @@ export function acpxBootstrapBlockedError(
return failure
? new Error(
`ACPX provider bootstrap failed before ${command}: ${failure.message}`,
{ cause: failure },
)
: null;
}
@@ -130,6 +131,7 @@ const GENERIC_ACPX_OUTPUT_CODES = new Set([
]);
const STABLE_ACPX_SIDECAR_CODES = new Set([
"ACPX_RUNTIME_PACKAGE_MISSING",
"COPILOT_POLICY_VIOLATION",
"COPILOT_DETACHED_WORK_UNSUPPORTED",
"ACP_MODEL_UNSUPPORTED",
@@ -54,6 +54,77 @@ afterEach(async () => {
});
describe("ACPX installation integrity", () => {
it.each([["linux", "x64"], ["darwin", "arm64"], ["darwin", "x64"]] as const)(
"retains Claude executable admission for a published bundle on %s %s", async (os, cpu) => {
const parent = await realpath(await mkdtemp(join(tmpdir(), "paperclip-claude-npm-admission-")));
temporaryDirectories.push(parent);
const root = join(parent, "node_modules/@paperclipai/server");
const bridge = join(root, "node_modules/@agentclientprotocol/claude-agent-acp");
const runtime = join(bridge, "node_modules/@anthropic-ai/claude-agent-sdk");
const name = `@anthropic-ai/claude-agent-sdk-${os}-${cpu}`;
const platformRoot = join(parent, "node_modules", name);
const optionalDependencies = { [name]: "0.3.286" };
const runtimeMetadata = { name: "@anthropic-ai/claude-agent-sdk", version: "0.3.286" };
const platformMetadata = { name, version: "0.3.286", os: [os], cpu: [cpu] };
await Promise.all([mkdir(runtime, { recursive: true }), mkdir(platformRoot, { recursive: true })]);
const command = 'process.stdout.write("unexecuted fixture");';
const reset = () => Promise.all([
writeFile(join(root, "package.json"), JSON.stringify({ name: "@paperclipai/server", optionalDependencies })),
writeFile(join(runtime, "package.json"), JSON.stringify(runtimeMetadata)),
writeFile(join(platformRoot, "package.json"), JSON.stringify(platformMetadata)),
]);
await reset();
await Promise.all([
writeFile(join(bridge, "package.json"), JSON.stringify({ name: "@agentclientprotocol/claude-agent-acp", version: "0.73.0", bin: "server.js" })),
writeFile(join(bridge, "server.js"), command),
writeFile(join(platformRoot, "claude"), "unqualified native bytes", { mode: 0o755 }),
]);
const profile = { ...resolveQualifiedAcpxProfile("claude", "claude-opus-5-5"), commandDigest: `sha256:${createHash("sha256").update(command).digest("hex")}` };
const resolver = createAcpxPackageJsonResolver(root);
expect(resolver(name, join(runtime, "package.json"))).toBe(join(platformRoot, "package.json"));
expect(() => resolver(name)).toThrow("outside the selected provider root");
for (const [file, metadata] of [
[join(root, "package.json"), { name: "@paperclipai/server", optionalDependencies: { [name]: "latest" } }],
[join(runtime, "package.json"), { ...runtimeMetadata, version: "0.3.285" }],
[join(runtime, "package.json"), { ...runtimeMetadata, optionalDependencies: {} }],
[join(platformRoot, "package.json"), { ...platformMetadata, name: "unqualified-provider" }],
[join(platformRoot, "package.json"), { ...platformMetadata, version: "0.3.285" }],
[join(platformRoot, "package.json"), { ...platformMetadata, cpu: ["ia32"] }],
] as const) {
await writeFile(file, JSON.stringify(metadata));
expect(() => resolver(name, join(runtime, "package.json"))).toThrow("outside the selected provider root");
await reset();
}
const platformSpy = vi.spyOn(process, "platform", "get").mockReturnValue(os);
const archSpy = vi.spyOn(process, "arch", "get").mockReturnValue(cpu);
try {
await expect(verifyQualifiedAcpxInstallation(profile, (...args) => resolver(...args))).rejects.toThrow("runtime omitted its verified platform executable package");
await expect(verifyQualifiedAcpxInstallation(profile, resolver)).rejects.toThrow(/digest mismatch/);
vi.stubEnv("PAPERCLIP_ACPX_PROVIDER_PACKAGE_ROOT", root);
vi.stubEnv("PAPERCLIP_ACPX_PROVIDER_PACKAGE_MANIFEST", undefined);
await expect(verifyQualifiedAcpxInstallation(profile)).rejects.toThrow(/digest mismatch/);
} finally {
vi.unstubAllEnvs();
platformSpy.mockRestore();
archSpy.mockRestore();
}
},
);
it("explains a missing Claude ACP installation without leaking the resolver stack", async () => {
const missingPackage = Object.assign(new Error(
"Cannot find module '@agentclientprotocol/claude-agent-acp/package.json'\nRequire stack: /srv/private/server.js",
), { code: "MODULE_NOT_FOUND" });
const error = await verifyQualifiedAcpxInstallation(
resolveQualifiedAcpxProfile("claude", "claude-opus-5-5"),
() => { throw missingPackage; },
).catch(error => error);
expect(error).toMatchObject({ code: "ACPX_RUNTIME_PACKAGE_MISSING" });
expect(error.message).toContain("@agentclientprotocol/claude-agent-acp");
expect(error.message).toContain("Reinstall or rebuild");
expect(error.message).not.toContain("/srv/private");
expect(error.message).not.toContain("Require stack");
});
it.each([false, true])("admits only the declared qualified Codex platform in its npm-hoisted slot (published platform declaration: %s)", async (publishedDeclaration) => {
const parent = await realpath(await mkdtemp(join(tmpdir(), "paperclip-codex-npm-hoist-")));
temporaryDirectories.push(parent);
@@ -354,7 +354,7 @@ export function createAcpxPackageJsonResolver(
resolvePackageJsonFromIssuer(packageName, canonicalIssuer),
);
if (!pathIsInside(canonicalNodeModules, packageJsonPath)
&& !isDeclaredHoistedCodexPlatform(canonicalRoot, canonicalManifest, canonicalIssuer, packageName, packageJsonPath)) {
&& !isDeclaredHoistedProviderPlatform(canonicalRoot, canonicalManifest, canonicalIssuer, packageName, packageJsonPath)) {
throw new Error(
`ACPX provider package ${packageName} resolves outside the selected provider root`,
);
@@ -365,25 +365,35 @@ export function createAcpxPackageJsonResolver(
return resolver;
}
function hasPublishedCodexPlatformDeclaration(resolver: AcpxPackageJsonResolver, runtimeManifest: string): boolean {
function hasPublishedProviderPlatformDeclaration(
resolver: AcpxPackageJsonResolver,
runtimeManifest: string,
qualified: { runtimePackageName: string; runtimePackageVersion: string; packageName: string; dependencyDeclaration: string },
): boolean {
const authority = rootedPackageResolvers.get(resolver);
if (!authority || !pathIsInside(authority.root, runtimeManifest)) return false;
const selected = readResolverPackageJson(authority.manifest);
const runtime = readResolverPackageJson(runtimeManifest);
const declarations = selected.optionalDependencies as Record<string, unknown> | undefined;
return selected.name === "@paperclipai/server"
&& runtime.name === QUALIFIED_CODEX_LINUX_X64_RUNTIME.runtimePackageName
&& runtime.version === QUALIFIED_CODEX_LINUX_X64_RUNTIME.runtimePackageVersion
&& runtime.name === qualified.runtimePackageName
&& runtime.version === qualified.runtimePackageVersion
&& runtime.optionalDependencies === undefined
&& declarations?.[QUALIFIED_CODEX_LINUX_X64_RUNTIME.packageName] === QUALIFIED_CODEX_LINUX_X64_RUNTIME.dependencyDeclaration;
&& declarations?.[qualified.packageName] === qualified.dependencyDeclaration;
}
// npm installs an optional platform alias outside a bundled server graph.
// Admit only the qualified executable package through its declared npm slot;
// all executable bytes still pass the existing digest/no-follow admission.
function isDeclaredHoistedCodexPlatform(root: string, manifest: string, issuer: string, packageName: string, packageJsonPath: string): boolean {
const qualified = QUALIFIED_CODEX_LINUX_X64_RUNTIME;
if (packageName !== qualified.packageName || issuer === manifest) return false;
function isDeclaredHoistedProviderPlatform(root: string, manifest: string, issuer: string, packageName: string, packageJsonPath: string): boolean {
const platformQualification = [
{ qualified: QUALIFIED_CODEX_LINUX_X64_RUNTIME, os: "linux", cpu: "x64", alias: true },
{ qualified: QUALIFIED_CLAUDE_LINUX_X64_RUNTIME, os: "linux", cpu: "x64", alias: false },
{ qualified: QUALIFIED_CLAUDE_DARWIN_RUNTIMES.arm64, os: "darwin", cpu: "arm64", alias: false },
{ qualified: QUALIFIED_CLAUDE_DARWIN_RUNTIMES.x64, os: "darwin", cpu: "x64", alias: false },
].find(value => value.qualified.packageName === packageName);
if (!platformQualification || issuer === manifest) return false;
const { qualified, os, cpu, alias } = platformQualification;
try {
const selected = readResolverPackageJson(manifest);
const runtime = readResolverPackageJson(issuer);
@@ -404,10 +414,10 @@ function isDeclaredHoistedCodexPlatform(root: string, manifest: string, issuer:
});
if (!declaredSlot) return false;
const platform = readResolverPackageJson(packageJsonPath);
return (platform.name === qualified.packageName || platform.name === qualified.runtimePackageName)
return (platform.name === qualified.packageName || (alias && platform.name === qualified.runtimePackageName))
&& platform.version === qualified.packageVersion
&& Array.isArray(platform.os) && platform.os.includes("linux")
&& Array.isArray(platform.cpu) && platform.cpu.includes("x64");
&& Array.isArray(platform.os) && platform.os.includes(os)
&& Array.isArray(platform.cpu) && platform.cpu.includes(cpu);
} catch { return false; }
}
@@ -633,9 +643,9 @@ export async function verifyQualifiedAcpxInstallation(
profile: QualifiedAcpxProfile,
resolvePackageJson: AcpxPackageJsonResolver = defaultPackageJsonResolver,
): Promise<VerifiedAcpxInstallation> {
// Keep the selected package authority available when its bundled Codex
// Keep the selected package authority available when its bundled provider
// wrapper delegates platform installation to the published server manifest.
if (profile.agent === "codex" && resolvePackageJson === defaultPackageJsonResolver && process.env.PAPERCLIP_ACPX_PROVIDER_PACKAGE_ROOT !== undefined) {
if ((profile.agent === "codex" || profile.agent === "claude") && resolvePackageJson === defaultPackageJsonResolver && process.env.PAPERCLIP_ACPX_PROVIDER_PACKAGE_ROOT !== undefined) {
resolvePackageJson = createAcpxPackageJsonResolver(
process.env.PAPERCLIP_ACPX_PROVIDER_PACKAGE_ROOT,
process.env.PAPERCLIP_ACPX_PROVIDER_PACKAGE_MANIFEST,
@@ -645,7 +655,22 @@ export async function verifyQualifiedAcpxInstallation(
if (builtin && (profile.agentServerPackage !== "builtin:grok-acp" || profile.agentServerVersion !== "1" || profile.agentRuntimePackage !== "native:grok" || profile.agentRuntimeVersion !== "1.0.13")) {
throw new Error("Grok builtin profile identity mismatch");
}
const serverPackageJsonPath = builtin ? null : await realpath(resolvePackageJson(profile.agentServerPackage));
let serverPackageJsonPath: string | null = null;
if (!builtin) {
try {
serverPackageJsonPath = await realpath(resolvePackageJson(profile.agentServerPackage));
} catch (error) {
if ((profile.agent === "claude" || profile.agent === "codex") &&
error instanceof Error && "code" in error &&
(error.code === "MODULE_NOT_FOUND" || error.code === "ERR_MODULE_NOT_FOUND")) {
throw Object.assign(new Error(
`The ACPX ${profile.agent} runtime is incomplete: required package ${profile.agentServerPackage} is missing. ` +
"Reinstall or rebuild Paperclip with its bundled provider dependencies, restart the server, then test the agent's runtime before retrying.",
), { code: "ACPX_RUNTIME_PACKAGE_MISSING" });
}
throw error;
}
}
const serverPackage: AcpxPackageMetadata = builtin
? { version: "1", bin: "launcher.cjs", type: "commonjs" }
: await readPackageJson(serverPackageJsonPath!, profile.agentServerPackage);
@@ -1019,8 +1044,8 @@ async function verifyQualifiedRuntimeExecutable(input: {
const ownDeclaration = typeof optionalDependencies === "object"
&& optionalDependencies !== null && !Array.isArray(optionalDependencies)
&& (optionalDependencies as Record<string, unknown>)[qualification.packageName] === qualification.dependencyDeclaration;
const publishedDeclaration = input.profile.agent === "codex" && optionalDependencies === undefined
&& hasPublishedCodexPlatformDeclaration(input.resolvePackageJson, input.runtimePackageJsonPath);
const publishedDeclaration = optionalDependencies === undefined
&& hasPublishedProviderPlatformDeclaration(input.resolvePackageJson, input.runtimePackageJsonPath, qualification);
if (!ownDeclaration && !publishedDeclaration) {
throw new Error(
`ACPX ${input.profile.agent} runtime omitted its verified platform executable package`,
@@ -1633,6 +1633,23 @@ it("uses the public server npm package as the authority for vendored sidecars",
});
});
it.each([false, true])("binds a vendored source server to its pnpm dependency root (self-rooted: %s)", async selfRooted => {
const workspace = await mkdtemp(join(tmpdir(), "paperclip-vendored-source-"));
const server = join(workspace, "server");
await mkdir(join(workspace, "node_modules", ".pnpm"), { recursive: true });
await mkdir(join(server, "node_modules", ...(selfRooted ? [".pnpm"] : [])), { recursive: true });
await writeFile(join(workspace, "package.json"), JSON.stringify({ name: "paperclip", private: true }));
await writeFile(join(workspace, "pnpm-workspace.yaml"), "packages:\n - server\n");
await writeFile(join(server, "package.json"), JSON.stringify({ name: "@paperclipai/server" }));
try {
expect(runnerdLaunchProfileInternals.acpxProviderPackageAuthority(
join(server, "dist/vendor/paperclip-runner/cli/acpx-runtime-sidecar.cjs"),
)).toEqual({ root: selfRooted ? server : workspace, manifest: join(server, "package.json") });
} finally {
await rm(workspace, { recursive: true, force: true });
}
});
it("keeps a self-rooted pnpm deployment inside its dependency authority", async () => {
const deploymentRoot = await mkdtemp(
join(tmpdir(), "paperclip-deployed-provider-root-"),
@@ -3030,6 +3030,20 @@ function acpxProviderPackageAuthority(
basename(resolve(cliDirectory, "../..")) === "vendor" &&
basename(resolve(cliDirectory, "../../..")) === "dist") {
const serverRoot = resolve(cliDirectory, "../../../..");
const workspaceRoot = dirname(serverRoot);
// Source/Docker builds share the workspace's pnpm store. Published npm
// bundles and self-rooted pnpm deployments retain their server authority.
if (basename(serverRoot) === "server" &&
!existsSync(resolve(serverRoot, "node_modules/.pnpm")) &&
existsSync(resolve(workspaceRoot, "node_modules/.pnpm")) &&
existsSync(resolve(workspaceRoot, "pnpm-workspace.yaml")) &&
existsSync(resolve(workspaceRoot, "package.json"))) {
const workspace = JSON.parse(readFileSync(resolve(workspaceRoot, "package.json"), "utf8"));
const server = JSON.parse(readFileSync(resolve(serverRoot, "package.json"), "utf8"));
if (workspace.name === "paperclip" && workspace.private === true && server.name === "@paperclipai/server") {
return { root: workspaceRoot, manifest: resolve(serverRoot, "package.json") };
}
}
return { root: serverRoot, manifest: resolve(serverRoot, "package.json") };
}
if (
@@ -569,9 +569,83 @@ describe("Daytona sandbox provider plugin", () => {
}).catch(error => error);
await vi.advanceTimersByTimeAsync(2_000);
const error = await pending;
expect(readEnvironmentAcquisitionDiagnostic(error)).toEqual({ phase, elapsedMs: 2_000, budgetMs: 2_000 });
// The default lookup mock returns no sandbox, so a pending create reports `unavailable`.
expect(readEnvironmentAcquisitionDiagnostic(error)).toEqual({ phase, elapsedMs: 2_000, budgetMs: 2_000,
...(phase === "create" ? { providerState: "unavailable" } : {}) });
expect(error.message).toBe("Daytona lease acquisition timed out; allocation cleanup is pending");
expect(sandbox.delete).not.toHaveBeenCalled();
if (phase !== "create") expect(mockGet).not.toHaveBeenCalled();
});
function ownedCreation(state: string) {
const requested = mockCreate.mock.calls.at(-1)![0];
return { ...createMockSandbox({ state }), name: requested.name, labels: { ...requested.labels } };
}
it.each([
["pulling_snapshot", "pulling_snapshot"], ["starting", "starting"], ["pending_build", "pending_build"],
["error", "error"], ["stopped", "other"], ["11184809", "other"],
])("records provider state %s as %s when creation is still pending at the deadline", async (state, expected) => {
mockCreate.mockImplementation(() => new Promise(() => {}));
let found: ReturnType<typeof ownedCreation> | undefined;
mockGet.mockImplementation(async () => (found = ownedCreation(state)));
const pending = plugin.definition.onEnvironmentAcquireLease!({ ...params, config: { ...params.config, timeoutMs: 2_000 } }).catch(error => error);
await vi.advanceTimersByTimeAsync(2_000);
const error = await pending;
expect(readEnvironmentAcquisitionDiagnostic(error)).toEqual({ phase: "create", elapsedMs: 2_000, budgetMs: 2_000, providerState: expected });
expect(mockGet).toHaveBeenCalledOnce();
expect(mockGet).toHaveBeenCalledWith(mockCreate.mock.calls[0][0].name);
// Read-only: no deletion, and the host-owned cleanup envelope is unchanged.
expect(found!.delete).not.toHaveBeenCalled();
expect(readEnvironmentCreationCleanupError(error)?.observedProviderLeaseId).toBeUndefined();
expect(JSON.stringify(environmentCreationCleanupErrorData(error, true))).not.toContain(found!.id);
});
it.each([
["absent", () => mockGet.mockRejectedValue(new MockDaytonaNotFoundError("private-not-found"))],
["unavailable", () => mockGet.mockRejectedValue(new Error("private-provider-failure"))],
["unavailable", () => mockGet.mockImplementation(async () => {
const foreign = ownedCreation("started"); foreign.labels["paperclip-run-id"] = "foreign-run"; return foreign;
})],
])("records %s when the deadline lookup cannot attribute an allocation", async (expected, arrange) => {
mockCreate.mockImplementation(() => new Promise(() => {}));
arrange();
const pending = plugin.definition.onEnvironmentAcquireLease!({ ...params, config: { ...params.config, timeoutMs: 2_000 } }).catch(error => error);
await vi.advanceTimersByTimeAsync(2_000);
const error = await pending;
expect(readEnvironmentAcquisitionDiagnostic(error)?.providerState).toBe(expected);
expect(JSON.stringify(environmentCreationCleanupErrorData(error, true))).not.toContain("private-");
});
it("bounds a hung deadline lookup well inside the host's RPC margin", async () => {
mockCreate.mockImplementation(() => new Promise(() => {}));
mockGet.mockImplementation(() => new Promise(() => {}));
let settled = false;
const pending = plugin.definition.onEnvironmentAcquireLease!(params).catch(error => { settled = true; return error; });
await vi.advanceTimersByTimeAsync(304_999);
expect(settled).toBe(false);
await vi.advanceTimersByTimeAsync(1);
const error = await pending;
expect(readEnvironmentAcquisitionDiagnostic(error)).toEqual({ phase: "create", elapsedMs: 300_000, budgetMs: 300_000, providerState: "unavailable" });
expect(error.message).toBe("Daytona lease acquisition timed out; allocation cleanup is pending");
});
it("keeps the timeout result when a late create settles during the deadline lookup", async () => {
const sandbox = createMockSandbox();
mockCreate.mockImplementation(() => new Promise(resolve => setTimeout(() => resolve(sandbox), 2_500)));
mockGet.mockImplementation(() => new Promise(resolve => setTimeout(() => resolve(ownedCreation("starting")), 4_000)));
let settled = false;
const pending = plugin.definition.onEnvironmentAcquireLease!({ ...params, config: { ...params.config, timeoutMs: 2_000 } })
.catch(error => { settled = true; return error; });
// The host serializes the error on settlement; it must wait for the lookup.
await vi.advanceTimersByTimeAsync(3_000);
expect(settled).toBe(false);
await vi.advanceTimersByTimeAsync(3_000);
const error = await pending;
expect(readEnvironmentAcquisitionDiagnostic(error)).toEqual({ phase: "create", elapsedMs: 2_000, budgetMs: 2_000, providerState: "starting" });
expect(readEnvironmentCreationCleanupError(error)?.providerLeaseId).toBe(mockCreate.mock.calls[0][0].name);
expect(sandbox.getWorkDir).not.toHaveBeenCalled();
expect(sandbox.delete).not.toHaveBeenCalled();
});
it.each(["create", "workspace"] as const)("keeps the original %s observation when cleanup consumes the remaining deadline", async phase => {
@@ -598,7 +672,7 @@ describe("Daytona sandbox provider plugin", () => {
await vi.advanceTimersByTimeAsync(100);
const second = plugin.definition.onEnvironmentAcquireLease!({ ...params, config: { ...params.config, timeoutMs: 3_000 } }).catch(error => error);
await vi.advanceTimersByTimeAsync(3_000);
expect(readEnvironmentAcquisitionDiagnostic(await first)).toEqual({ phase: "create", elapsedMs: 2_000, budgetMs: 2_000 });
expect(readEnvironmentAcquisitionDiagnostic(await first)).toEqual({ phase: "create", elapsedMs: 2_000, budgetMs: 2_000, providerState: "unavailable" });
expect(readEnvironmentAcquisitionDiagnostic(await second)).toEqual({ phase: "workspace", elapsedMs: 3_000, budgetMs: 3_000 });
mockCreate.mockResolvedValue(createMockSandbox());
expect(await plugin.definition.onEnvironmentAcquireLease!(params)).not.toHaveProperty("acquisitionDiagnostic");
@@ -2512,6 +2586,143 @@ describe("Daytona sandbox provider plugin", () => {
finally { clearInterval(timer); }
}
it("completes a session command beyond a synchronous gateway deadline without redispatch", async () => {
process.env.DAYTONA_API_KEY = "host-key";
const sandbox = createMockSandbox();
sandbox.process.executeCommand.mockRejectedValue(new MockDaytonaTimeoutError("opaque gateway timeout"));
sandbox.process.getSessionCommand.mockResolvedValue({ exitCode: 0 });
sandbox.process.getSessionCommandLogs.mockImplementation(
async (_sid: string, _cmdId: string, onStdout?: (chunk: string) => void, onStderr?: (chunk: string) => void) => {
if (!onStdout) return { stdout: "first;last;", stderr: "" };
onStdout("first;");
await streamFor(3_600_001, onStderr);
onStdout("last;");
},
);
mockGet.mockResolvedValue(sandbox);
vi.useFakeTimers();
try {
const params = streamExecParams({ timeoutMs: 14_400_000, commandTimeoutMs: 14_400_000 });
// Characterize the one-shot transport failure with the same command
// and budget, then exercise the existing asynchronous session path.
expect(await plugin.definition.onEnvironmentExecute?.({ ...params, bypassSession: true }))
.toMatchObject({ timedOut: true, exitCode: null });
sandbox.process.executeCommand.mockClear();
const pending = plugin.definition.onEnvironmentExecute?.(params);
await vi.advanceTimersByTimeAsync(3_600_002);
expect(await pending).toMatchObject({ timedOut: false, exitCode: 0, stdout: "first;last;" });
expect(sandbox.process.executeCommand).not.toHaveBeenCalled();
expect(sandbox.process.executeSessionCommand).toHaveBeenCalledTimes(1);
expect(sandbox.process.executeSessionCommand.mock.calls[0]![1]).toMatchObject({ runAsync: true });
expect(sandbox.process.executeSessionCommand.mock.calls[0]![2]).toBe(14400);
await plugin.definition.onEnvironmentReleaseLease?.({
driverKey: "daytona", companyId: "company-1", environmentId: "env-1",
providerLeaseId: "sandbox-123", config: params.config,
});
expect(sandbox.process.deleteSession).toHaveBeenCalledTimes(1);
} finally { vi.useRealTimers(); }
});
it.each([undefined, 0])("preserves the existing healthy session lifetime with command timeout %s", async (commandTimeoutMs) => {
process.env.DAYTONA_API_KEY = "host-key";
const sandbox = createMockSandbox();
sandbox.process.getSessionCommand.mockResolvedValue({ exitCode: 0 });
sandbox.process.getSessionCommandLogs.mockImplementation(async (_s: string, _c: string, stdout?: (chunk: string) => void, stderr?: (chunk: string) => void) => {
if (!stdout) return { stdout: "done;", stderr: "" };
await streamFor(60_000, stderr);
stdout("done;");
});
mockGet.mockResolvedValue(sandbox);
vi.useFakeTimers();
try {
const pending = plugin.definition.onEnvironmentExecute?.(streamExecParams({ timeoutMs: 1000, commandTimeoutMs }));
await vi.advanceTimersByTimeAsync(60_000);
expect(await pending).toMatchObject({ timedOut: false, exitCode: 0, stdout: "done;" });
expect(vi.getTimerCount()).toBe(0);
} finally { vi.useRealTimers(); }
});
it("removes deadline listeners from completed reads during a quiet polling run", async () => {
process.env.DAYTONA_API_KEY = "host-key";
const sandbox = createMockSandbox();
sandbox.process.getSessionCommand.mockResolvedValue({ exitCode: undefined });
sandbox.process.getSessionCommandLogs.mockImplementation(async (_s: string, _c: string, stdout?: (chunk: string) => void) => {
if (stdout) { stdout("partial;"); return; }
return { stdout: "partial;", stderr: "" };
});
mockGet.mockResolvedValue(sandbox);
const added = vi.spyOn(AbortSignal.prototype, "addEventListener");
const removed = vi.spyOn(AbortSignal.prototype, "removeEventListener");
vi.useFakeTimers();
try {
const pending = plugin.definition.onEnvironmentExecute?.(streamExecParams({ timeoutMs: 60_000, commandTimeoutMs: 60_000 }));
await vi.advanceTimersByTimeAsync(30_000);
const signal = added.mock.contexts.find((value, index) => added.mock.calls[index]?.[0] === "abort");
expect(signal).toBeDefined();
const count = (spy: typeof added | typeof removed) => spy.mock.calls.filter((call, index) => call[0] === "abort" && spy.mock.contexts[index] === signal).length;
expect(count(added)).toBeGreaterThan(50);
expect(count(added) - count(removed)).toBeLessThanOrEqual(1);
await vi.advanceTimersByTimeAsync(30_000);
expect(await pending).toMatchObject({ timedOut: true, exitCode: null, stdout: "partial;" });
expect(count(removed)).toBe(count(added));
expect(sandbox.process.executeSessionCommand).toHaveBeenCalledTimes(1);
expect(vi.getTimerCount()).toBe(0);
} finally { added.mockRestore(); removed.mockRestore(); vi.useRealTimers(); }
});
it.each(["stream", "status", "snapshot", "dispatch", "final snapshot"])("preserves a command deadline during a healthy or stalled %s observation", async (phase) => {
process.env.DAYTONA_API_KEY = "host-key";
const executionLog = vi.fn();
const restore = __setDaytonaPluginContextForTest({ execution: { log: executionLog } } as unknown as PluginContext);
const sandbox = createMockSandbox();
let lateOutput: ((chunk: string) => void) | undefined;
let rejectStream: ((error: Error) => void) | undefined;
let keepAlive: ReturnType<typeof setInterval> | undefined;
sandbox.process.getSessionCommand.mockResolvedValue({ exitCode: phase === "final snapshot" ? 0 : undefined });
sandbox.process.getSessionCommandLogs.mockImplementation(async (_s: string, _c: string, stdout?: (chunk: string) => void) => {
if (!stdout) {
if (phase === "snapshot" || phase === "final snapshot") return new Promise(() => {});
return { stdout: "first;", stderr: "" };
}
stdout("first;");
if (phase === "stream") {
lateOutput = stdout;
keepAlive = setInterval(() => stdout("."), 10000);
return new Promise((_resolve, reject) => { rejectStream = reject; });
}
});
if (phase === "status") sandbox.process.getSessionCommand.mockImplementation(async () => new Promise(() => {}));
if (phase === "dispatch") sandbox.process.executeSessionCommand.mockImplementation(async () => new Promise(() => {}));
mockGet.mockResolvedValue(sandbox);
vi.useFakeTimers();
try {
const pending = plugin.definition.onEnvironmentExecute?.(streamExecParams({ timeoutMs: 14_400_000, commandTimeoutMs: 60000 }));
let settled = false;
void pending?.then(() => { settled = true; });
await vi.advanceTimersByTimeAsync(59999);
expect(settled).toBe(false);
await vi.advanceTimersByTimeAsync(1);
const result = await pending;
expect(result).toMatchObject({ timedOut: true, exitCode: null });
if (phase === "final snapshot") {
expect(result?.metadata).toMatchObject({ timeoutScope: "session_log_observation", commandExitConfirmed: true, observedExitCode: 0 });
expect(result?.stderr).toContain("Session log observation timed out after 60 s; final output may be incomplete.");
} else expect(result?.stderr).toContain("Command timed out after 60 s.");
expect(sandbox.process.executeSessionCommand).toHaveBeenCalledTimes(1);
expect(sandbox.process.executeCommand).not.toHaveBeenCalled();
const logged = executionLog.mock.calls.length;
lateOutput?.("late private output");
rejectStream?.(new Error("late transport rejection"));
await vi.advanceTimersByTimeAsync(1);
expect(executionLog.mock.calls.length).toBe(logged);
expect(JSON.stringify(result)).not.toContain("late private");
clearInterval(keepAlive);
await plugin.definition.onEnvironmentReleaseLease?.({ driverKey: "daytona", companyId: "company-1", environmentId: "env-1", providerLeaseId: "sandbox-123", config: streamExecParams().config });
expect(sandbox.process.deleteSession).toHaveBeenCalledTimes(1);
expect(vi.getTimerCount()).toBe(0);
} finally { clearInterval(keepAlive); restore(); vi.useRealTimers(); }
});
it.each(["", "prefix;🙂;"])("recovers unseen output and cancellation from a stuck socket after prefix %j", async (prefix) => {
process.env.DAYTONA_API_KEY = "host-key";
const executionLog = vi.fn();
@@ -14,6 +14,7 @@ import type {
PluginContext,
PluginEnvironmentCreationCleanup,
PluginEnvironmentAcquisitionDiagnostic,
PluginEnvironmentAcquisitionProviderState,
PluginTracer,
PluginEnvironmentAcquireLeaseParams,
PluginEnvironmentCancelInteractiveSetupParams,
@@ -1045,6 +1046,28 @@ async function destroyFailedCreation(
await withLivenessTimeout("sandbox.failedCreateDelete", 15_000, () => orphan.delete(10, true));
}
// Daytona states reported as-is after a create deadline; any other is `other`.
const REPORTED_CREATE_STATES = new Set<string>([
"creating", "pending_build", "building_snapshot", "pulling_snapshot", "starting", "started", "error", "build_failed",
] satisfies PluginEnvironmentAcquisitionProviderState[]);
// One read-only lookup of the provisional creation name, so a create deadline
// records where the provider left the allocation. It neither deletes nor
// records an observed ID: the host-owned cleanup envelope is unchanged.
async function observeCreateState(
config: DaytonaDriverConfig, cleanup: PluginEnvironmentCreationCleanup,
): Promise<PluginEnvironmentAcquisitionProviderState> {
try {
const sandbox = await withLivenessTimeout("sandbox.createStateLookup", 5_000,
() => createDaytonaClient(config).get(cleanup.providerLeaseId));
if (!sandbox || sandbox.name !== cleanup.providerLeaseId || typeof sandbox.state !== "string" ||
Object.entries(cleanup.labels).some(([key, value]) => sandbox.labels?.[key] !== value)) return "unavailable";
return REPORTED_CREATE_STATES.has(sandbox.state) ? sandbox.state as PluginEnvironmentAcquisitionProviderState : "other";
} catch (error) {
return error instanceof DaytonaNotFoundError ? "absent" : "unavailable";
}
}
// ─── Per-lease started-sandbox handle cache ──────────────────────────────────
// Memoize the started Daytona `Sandbox` handle so repeated exec/sync/resume/
// teardown calls on one lease skip the per-call `client.get(sandboxId)` REST
@@ -1927,6 +1950,53 @@ type SessionLogStreamResult = {
class SessionCommandDeadlineError extends Error {}
class SessionObservationTimeoutError extends Error {}
function createCommandDeadline(timeoutMs: number | undefined) {
if (timeoutMs === undefined || timeoutMs === 0) return undefined;
if (!Number.isFinite(timeoutMs) || timeoutMs < 0 || timeoutMs > 2_147_483_647) {
throw new Error("Invalid session command timeout");
}
let expired = false;
const dueAt = Date.now() + timeoutMs;
const controller = new AbortController();
const timer = setTimeout(() => {
expired = true;
controller.abort();
}, timeoutMs);
return {
expired: () => expired,
remainingMs: () => Math.max(0, dueAt - Date.now()),
race: async <T>(action: () => Promise<T>): Promise<T> => {
if (expired) throw new SessionCommandDeadlineError();
let rejectDeadline!: (error: SessionCommandDeadlineError) => void;
const deadline = new Promise<never>((_resolve, reject) => { rejectDeadline = reject; });
const onAbort = () => rejectDeadline(new SessionCommandDeadlineError());
controller.signal.addEventListener("abort", onAbort, { once: true });
try {
return await Promise.race([action(), deadline]);
} finally {
// Retain only active observations. A shared pending promise would keep
// every finished poll and its result until a long command ends.
controller.signal.removeEventListener("abort", onAbort);
}
},
clear: () => clearTimeout(timer),
};
}
type CommandDeadline = ReturnType<typeof createCommandDeadline>;
function withinCommandDeadline<T>(deadline: CommandDeadline, action: () => Promise<T>): Promise<T> {
return deadline ? deadline.race(action) : action();
}
async function sleepWithinCommandDeadline(deadline: CommandDeadline, delayMs: number): Promise<void> {
if (!deadline) return await sleep(delayMs);
let timer: ReturnType<typeof setTimeout> | undefined;
try {
await deadline.race(() => new Promise<void>((resolve) => { timer = setTimeout(resolve, delayMs); }));
} finally {
if (timer !== undefined) clearTimeout(timer);
}
}
async function beforeSessionDeadline<T>(deadlineMs: number, action: () => Promise<T>): Promise<T> {
const remainingMs = deadlineMs - Date.now();
if (remainingMs <= 0) throw new SessionCommandDeadlineError();
@@ -1943,12 +2013,18 @@ async function beforeSessionDeadline<T>(deadlineMs: number, action: () => Promis
}
}
// A provider operation timeout bounds one observation, not a healthy stream's
// lifetime. The caller's RPC/run guard and session teardown own that lifetime.
async function observeSessionCommand<T>(timeoutMs: number, action: () => Promise<T>): Promise<T> {
// A provider operation timeout bounds one observation. An explicit command
// deadline also bounds the stream; existing callers retain their run/Stop guard.
async function observeSessionCommand<T>(timeoutMs: number, action: () => Promise<T>, commandDeadline?: CommandDeadline): Promise<T> {
try {
// Use the earlier shared command timer instead of leaving a longer
// observation timer alive after the command has already settled.
if (commandDeadline && commandDeadline.remainingMs() <= timeoutMs) {
return await commandDeadline.race(action);
}
return await beforeSessionDeadline(Date.now() + timeoutMs, action);
} catch (error) {
if (commandDeadline?.expired()) throw new SessionCommandDeadlineError();
if (error instanceof SessionCommandDeadlineError || error instanceof DaytonaTimeoutError) {
throw new SessionObservationTimeoutError();
}
@@ -1966,6 +2042,7 @@ async function runSessionLogStream(
commandId: string,
timeoutMs: number,
buffer: ReturnType<typeof createSessionStreamBuffer>,
commandDeadline?: CommandDeadline,
): Promise<SessionLogStreamResult> {
let reconnects = 0;
let pollUntilExit = false;
@@ -1980,14 +2057,14 @@ async function runSessionLogStream(
idleTimer = setTimeout(signalIdle, SESSION_LOG_STREAM_IDLE_MS);
};
const receive = (stream: "stdout" | "stderr", chunk: string) => {
if (!acceptingOutput || !chunk) return;
if (!acceptingOutput || commandDeadline?.expired() || !chunk) return;
armIdleTimer();
if (stream === "stdout") buffer.onStdout(chunk);
else buffer.onStderr(chunk);
};
try {
armIdleTimer();
const outcome = await Promise.race([
const outcome = await withinCommandDeadline(commandDeadline, () => Promise.race([
sandbox.process.getSessionCommandLogs(
sessionId,
commandId,
@@ -1995,7 +2072,7 @@ async function runSessionLogStream(
(chunk) => receive("stderr", chunk),
).then(() => "closed" as const),
idle,
]);
]));
if (outcome === "idle") {
// Do not wait for the abandoned socket or open another one. The SDK
// exposes no cancellation handle. Fence its callbacks before the
@@ -2004,7 +2081,8 @@ async function runSessionLogStream(
}
streamClosed = true;
pollUntilExit = true;
} catch {
} catch (error) {
if (error instanceof SessionCommandDeadlineError) throw error;
// The command can still be running after a log transport failure.
} finally {
// The SDK exposes no cancellation handle for this socket. Session
@@ -2013,7 +2091,7 @@ async function runSessionLogStream(
clearTimeout(idleTimer);
}
if (streamClosed) {
const exitCode = await readSessionExitCode(sandbox, sessionId, commandId, timeoutMs);
const exitCode = await readSessionExitCode(sandbox, sessionId, commandId, timeoutMs, commandDeadline);
if (exitCode !== null) return { exitCode, pollUntilExit, buffer };
}
if (reconnects >= MAX_SESSION_STREAM_RECONNECTS) {
@@ -2033,14 +2111,15 @@ async function readSessionExitCode(
sessionId: string,
commandId: string,
timeoutMs: number,
commandDeadline?: CommandDeadline,
): Promise<number | null> {
const first = await observeSessionCommand(timeoutMs, () => sandbox.process.getSessionCommand(sessionId, commandId));
const first = await observeSessionCommand(timeoutMs, () => sandbox.process.getSessionCommand(sessionId, commandId), commandDeadline);
if (typeof first.exitCode === "number") {
return first.exitCode;
}
for (const delayMs of SESSION_EXIT_CODE_RETRY_DELAYS_MS) {
await sleep(delayMs);
const status = await observeSessionCommand(timeoutMs, () => sandbox.process.getSessionCommand(sessionId, commandId));
await sleepWithinCommandDeadline(commandDeadline, delayMs);
const status = await observeSessionCommand(timeoutMs, () => sandbox.process.getSessionCommand(sessionId, commandId), commandDeadline);
if (typeof status.exitCode === "number") {
return status.exitCode;
}
@@ -2083,6 +2162,7 @@ async function executeInSession(
// attributed to the provider boundary.
let execStart: number | null = null;
let confirmedExitCode: number | null = null;
let commandDeadline: CommandDeadline;
const streamBuffer = createSessionStreamBuffer((stream, text) => pluginContext?.execution.log(stream, text));
try {
@@ -2102,11 +2182,12 @@ async function executeInSession(
const command = `( ${loginScript} )`;
execStart = timingNow();
const dispatched = await sandbox.process.executeSessionCommand(
commandDeadline = createCommandDeadline(params.commandTimeoutMs);
const dispatched = await withinCommandDeadline(commandDeadline, () => sandbox.process.executeSessionCommand(
sessionId,
{ command, runAsync: true },
timeoutSeconds,
);
));
const commandId = dispatched.cmdId;
// Log-stream path. A session command always tries the stream first: it
@@ -2124,12 +2205,13 @@ async function executeInSession(
commandId,
effectiveTimeoutMs,
streamBuffer,
commandDeadline,
);
if (streamResult.exitCode !== null) {
confirmedExitCode = streamResult.exitCode;
// A log socket can close before the process emits its final bytes. Even
// a now-visible exit code does not prove the streamed prefix is complete.
const logs = await observeSessionCommand(effectiveTimeoutMs, () => sandbox.process.getSessionCommandLogs(sessionId, commandId));
const logs = await observeSessionCommand(effectiveTimeoutMs, () => sandbox.process.getSessionCommandLogs(sessionId, commandId), commandDeadline);
streamBuffer.resetConnectionCursors();
streamBuffer.onStdout(logs?.stdout ?? "");
streamBuffer.onStderr(logs?.stderr ?? "");
@@ -2155,8 +2237,8 @@ async function executeInSession(
const observe = async <T>(action: () => Promise<T>): Promise<T> => {
try {
return await (streamResult.pollUntilExit
? observeSessionCommand(effectiveTimeoutMs, action)
: beforeSessionDeadline(deadlineMs, action));
? observeSessionCommand(effectiveTimeoutMs, action, commandDeadline)
: commandDeadline ? commandDeadline.race(action) : beforeSessionDeadline(deadlineMs, action));
} catch (error) {
if (error instanceof DaytonaTimeoutError) throw new SessionObservationTimeoutError();
throw error;
@@ -2183,11 +2265,12 @@ async function executeInSession(
// Full log snapshots are heavier than a status read. Limit fallback
// traffic to one snapshot per second. Failed-stream fallback retains
// its original deadline, including the time spent waiting between reads.
await sleep(streamResult.pollUntilExit ? SESSION_LOG_POLL_INTERVAL_MS
await sleepWithinCommandDeadline(commandDeadline, streamResult.pollUntilExit ? SESSION_LOG_POLL_INTERVAL_MS
: Math.max(0, Math.min(SESSION_LOG_POLL_INTERVAL_MS, deadlineMs - Date.now())));
}
} catch (error) {
if (error instanceof SessionObservationTimeoutError) {
if (error instanceof SessionObservationTimeoutError ||
(error instanceof SessionCommandDeadlineError && confirmedExitCode !== null)) {
// Preserve undecided decoder suffixes in the partial result, without
// injecting a possibly incomplete character into the live protocol.
streamBuffer.finish(false);
@@ -2195,7 +2278,7 @@ async function executeInSession(
exitCode: null,
timedOut: true,
stdout: streamBuffer.stdout,
stderr: `${streamBuffer.stderr}Session log observation timed out after ${Math.round(effectiveTimeoutMs / 1000)} s; ${confirmedExitCode === null ? "command exit remains unconfirmed" : "final output may be incomplete"}.\n`,
stderr: `${streamBuffer.stderr}Session log observation timed out after ${Math.round((commandDeadline?.expired() ? params.commandTimeoutMs! : effectiveTimeoutMs) / 1000)} s; ${confirmedExitCode === null ? "command exit remains unconfirmed" : "final output may be incomplete"}.\n`,
metadata: {
...(execStart != null ? { durationMs: timingNow() - execStart } : {}),
timeoutScope: "session_log_observation",
@@ -2209,7 +2292,7 @@ async function executeInSession(
const timeoutMessage = gitNet
? `Git network operation timed out after ${Math.round(effectiveTimeoutMs / 1000)} s — the remote may be unreachable or noninteractive credentials are not configured.`
: error instanceof SessionCommandDeadlineError
? `Command timed out after ${Math.round(effectiveTimeoutMs / 1000)} s.`
? `Command timed out after ${Math.round((commandDeadline?.expired() ? params.commandTimeoutMs! : effectiveTimeoutMs) / 1000)} s.`
: error.message.trim();
const durationMs = execStart != null ? timingNow() - execStart : undefined;
return {
@@ -2222,6 +2305,7 @@ async function executeInSession(
}
throw error;
} finally {
commandDeadline?.clear();
if (stdinPath) {
await sandbox.fs.deleteFile(stdinPath).catch(() => undefined);
}
@@ -2439,6 +2523,15 @@ const plugin = definePlugin({
const assertActive = () => {
if (expired || Date.now() >= deadline) throw timeoutFailure();
};
const deadlineFailure = async () => {
// Only a create that has neither resolved nor rejected is still pending.
const createPending = phase === "create" && firstFailure === undefined ? cleanup : undefined;
const diagnostic = observeFailure();
// Stop late setup before the lookup, so a late create cannot be admitted.
expired = true;
if (createPending) diagnostic.providerState = await observeCreateState(config, createPending);
return timeoutFailure();
};
const acquire = async (): Promise<PluginEnvironmentLease> => {
const sandbox = await createSandbox(params, config, {
onCreateAttempt: (attempt) => { cleanup = attempt; },
@@ -2525,11 +2618,14 @@ const plugin = definePlugin({
}
};
let timer: ReturnType<typeof setTimeout> | undefined;
let timedOut = false;
try {
return await Promise.race([
acquire(),
// After the deadline only the timer settles the call: a late create
// rejection cannot replace the timeout's ownership and observation.
acquire().catch((error: unknown) => timedOut ? new Promise<never>(() => undefined) : Promise.reject(error)),
new Promise<never>((_resolve, reject) => {
timer = setTimeout(() => reject(timeoutFailure()), budgetMs);
timer = setTimeout(() => { timedOut = true; void deadlineFailure().then(reject); }, budgetMs);
}),
]);
} finally {
@@ -19,8 +19,19 @@ export interface PluginEnvironmentAcquisitionDiagnostic {
/** Milliseconds from acquisition start to the first failure, before cleanup. */
elapsedMs: number;
budgetMs: number;
/**
* Provider state of the provisional allocation, observed once after a create
* deadline. Only a fixed set of states is reported; it is not the cause.
*/
providerState?: PluginEnvironmentAcquisitionProviderState;
}
export const ENVIRONMENT_ACQUISITION_PROVIDER_STATES = [
"absent", "creating", "pending_build", "building_snapshot", "pulling_snapshot",
"starting", "started", "error", "build_failed", "other", "unavailable",
] as const;
export type PluginEnvironmentAcquisitionProviderState = typeof ENVIRONMENT_ACQUISITION_PROVIDER_STATES[number];
export function readEnvironmentAcquisitionDiagnostic(error: unknown): PluginEnvironmentAcquisitionDiagnostic | null {
try {
// Diagnostics never make an invalid ownership envelope usable for cleanup.
@@ -33,10 +44,14 @@ export function readEnvironmentAcquisitionDiagnostic(error: unknown): PluginEnvi
const phase = fields.phase?.value;
const elapsedMs = fields.elapsedMs?.value;
const budgetMs = fields.budgetMs?.value;
const providerState = fields.providerState?.value;
if (!["create", "workspace", "shell", "expiry", "sentinel"].includes(phase as string) ||
typeof elapsedMs !== "number" || !Number.isSafeInteger(elapsedMs) || elapsedMs < 0 || elapsedMs > 604_800_000 ||
typeof budgetMs !== "number" || !Number.isSafeInteger(budgetMs) || budgetMs < 1 || budgetMs > 86_400_000) return null;
return { phase: phase as PluginEnvironmentAcquisitionDiagnostic["phase"], elapsedMs, budgetMs };
// An unknown optional state is omitted; it never invalidates the observation.
const knownState = ENVIRONMENT_ACQUISITION_PROVIDER_STATES.find(value => value === providerState);
return { phase: phase as PluginEnvironmentAcquisitionDiagnostic["phase"], elapsedMs, budgetMs,
...(knownState ? { providerState: knownState } : {}) };
} catch { return null; }
}
+2 -2
View File
@@ -450,8 +450,8 @@ export {
PRINCIPAL_TYPES,
} from "@paperclipai/shared";
export { PluginEnvironmentCreationCleanupError, environmentCreationCleanupErrorData, readEnvironmentCreationCleanupError, readEnvironmentAcquisitionDiagnostic } from "./environment-creation-cleanup.js";
export type { PluginEnvironmentCreationCleanup, PluginEnvironmentAcquisitionDiagnostic } from "./environment-creation-cleanup.js";
export { PluginEnvironmentCreationCleanupError, environmentCreationCleanupErrorData, readEnvironmentCreationCleanupError, readEnvironmentAcquisitionDiagnostic, ENVIRONMENT_ACQUISITION_PROVIDER_STATES } from "./environment-creation-cleanup.js";
export type { PluginEnvironmentCreationCleanup, PluginEnvironmentAcquisitionDiagnostic, PluginEnvironmentAcquisitionProviderState } from "./environment-creation-cleanup.js";
export { preserveEnvironmentSyncErrorDiagnostic, environmentSyncErrorData, readEnvironmentSyncErrorDiagnostic, withEnvironmentSyncErrorCapture, withEnvironmentSyncTransferStep, recordEnvironmentSyncError } from "./environment-sync-error.js";
export type { PluginEnvironmentSyncErrorDiagnostic } from "./environment-sync-error.js";
+4
View File
@@ -713,6 +713,10 @@ export interface PluginEnvironmentExecuteParams extends PluginEnvironmentDriverB
env?: Record<string, string>;
stdin?: string;
timeoutMs?: number;
/** Optional total command execution budget, in milliseconds. A session
* provider must keep this distinct from individual observation timeouts.
* Zero disables this deadline. Absence keeps the existing session lifetime. */
commandTimeoutMs?: number;
/**
* Run this command outside the lease's persistent session.
*
@@ -92,6 +92,20 @@ describe("acquisition failure observations", () => {
expect(readEnvironmentAcquisitionDiagnostic({ data: { schema, cleanup: ownership } })).toBeNull();
});
it("carries a fixed-enum provider state and omits unknown or private states", async () => {
const withState = { ...diagnostic, phase: "create", providerState: "pulling_snapshot" } as const;
const response = await invoke("environmentAcquireLease", new PluginEnvironmentCreationCleanupError([], "Cleanup required", ownership, withState));
if (!("error" in response) || !response.error) throw new Error("Expected RPC error");
expect(readEnvironmentAcquisitionDiagnostic(new JsonRpcCallError(response.error))).toEqual(withState);
for (const providerState of ["private-state", "STARTED", 3, null]) {
const error = new PluginEnvironmentCreationCleanupError([], "Cleanup required", ownership,
{ ...withState, providerState } as unknown as typeof withState);
// An unrecognized optional state never invalidates the other observations.
expect(readEnvironmentAcquisitionDiagnostic(error)).toEqual({ ...diagnostic, phase: "create" });
expect(JSON.stringify(environmentCreationCleanupErrorData(error, true))).not.toContain("private-");
}
});
it.each(["environmentDestroyLease", "environmentProbe"])("omits acquisition diagnostics from %s", async method => {
const response = await invoke(method, new PluginEnvironmentCreationCleanupError([], "Cleanup required", ownership, diagnostic));
expect("error" in response && response.error).not.toHaveProperty("data.acquisitionDiagnostic");
+3
View File
@@ -1268,6 +1268,9 @@ export type {
BudgetIncidentResolutionInput,
CostEvent,
CostSummary,
CostDataQualityItem,
CostDataQualityReport,
CostDataQualityReason,
IssueCostSummary,
CostByAgent,
CostByUser,
+32
View File
@@ -33,6 +33,8 @@ export interface CostSummary {
pendingRunCount: number;
unpricedEventCount: number;
estimatedEventCount?: number;
/** Receipts whose API/subscription billing identity was not established. */
unknownBillingEventCount?: number;
pricingComplete: boolean;
companyId: string;
spendCents: number;
@@ -41,6 +43,30 @@ export interface CostSummary {
utilizationPercent: number;
}
export type CostDataQualityReason = "unknown_billing" | "missing_model" | "missing_price"
| "waiting_for_receipt" | "receipt_unavailable" | "retryable";
export interface CostDataQualityItem {
id: string;
runId: string | null;
agentId: string | null;
agentName: string | null;
agentAppearance?: AgentAppearance | null;
occurredAt: string;
provider: string | null;
model: string | null;
reason: CostDataQualityReason;
}
export interface CostDataQualityReport {
canRetry: boolean;
/** Each category is bounded independently so one cannot hide another. */
events: CostDataQualityItem[];
runs: CostDataQualityItem[];
hasMoreEvents: boolean;
hasMoreRuns: boolean;
}
export interface IssueCostSummary {
issueId: string;
issueCount: number;
@@ -110,6 +136,9 @@ export interface CostByProviderModel {
model: string;
costCents: number;
costCentsExact?: string;
eventCount?: number;
estimatedEventCount?: number;
unpricedEventCount?: number;
inputTokens: number;
cachedInputTokens: number;
outputTokens: number;
@@ -124,6 +153,9 @@ export interface CostByBiller {
biller: string;
costCents: number;
costCentsExact?: string;
eventCount?: number;
estimatedEventCount?: number;
unpricedEventCount?: number;
inputTokens: number;
cachedInputTokens: number;
outputTokens: number;
+1 -1
View File
@@ -900,7 +900,7 @@ export type {
RoutineExecutionIssueOrigin,
RoutineListItem,
} from "./routine.js";
export type { CostEvent, CostSummary, IssueCostSummary, CostByUser, CostByUserReport, CostByAgent, CostByProviderModel, CostByBiller, CostByAgentModel, CostWindowSpendRow, CostByProject } from "./cost.js";
export type { CostEvent, CostSummary, CostDataQualityItem, CostDataQualityReport, CostDataQualityReason, IssueCostSummary, CostByUser, CostByUserReport, CostByAgent, CostByProviderModel, CostByBiller, CostByAgentModel, CostWindowSpendRow, CostByProject } from "./cost.js";
export type { FinanceEvent, FinanceSummary, FinanceByBiller, FinanceByKind } from "./finance.js";
export type {
AgentWakeupResponse,
+34
View File
@@ -884,6 +884,9 @@ importers:
server:
dependencies:
'@agentclientprotocol/claude-agent-acp':
specifier: 0.73.0
version: 0.73.0(patch_hash=riqgguivxjxydoydnzpedjy5au)(@anthropic-ai/sdk@0.129.0(zod@4.4.3))(@modelcontextprotocol/sdk@1.31.0(zod@4.4.3))
'@agentclientprotocol/codex-acp':
specifier: 1.6.2
version: 1.6.2(patch_hash=ilkykmzaax6atpgtip6bxikv6i)
@@ -9167,6 +9170,15 @@ snapshots:
'@adobe/css-tools@4.5.0': {}
'@agentclientprotocol/claude-agent-acp@0.73.0(patch_hash=riqgguivxjxydoydnzpedjy5au)(@anthropic-ai/sdk@0.129.0(zod@4.4.3))(@modelcontextprotocol/sdk@1.31.0(zod@4.4.3))':
dependencies:
'@agentclientprotocol/sdk': 1.4.0(zod@4.4.3)
'@anthropic-ai/claude-agent-sdk': 0.3.286(@anthropic-ai/sdk@0.129.0(zod@4.4.3))(@modelcontextprotocol/sdk@1.31.0(zod@4.4.3))(zod@4.4.3)
zod: 4.4.3
transitivePeerDependencies:
- '@anthropic-ai/sdk'
- '@modelcontextprotocol/sdk'
'@agentclientprotocol/claude-agent-acp@0.73.0(patch_hash=riqgguivxjxydoydnzpedjy5au)(@anthropic-ai/sdk@0.129.0(zod@4.6.5))(@modelcontextprotocol/sdk@1.31.0(zod@4.6.5))':
dependencies:
'@agentclientprotocol/sdk': 1.4.0(zod@4.4.3)
@@ -9314,6 +9326,21 @@ snapshots:
'@anthropic-ai/claude-agent-sdk-win32-x64@0.3.286':
optional: true
'@anthropic-ai/claude-agent-sdk@0.3.286(@anthropic-ai/sdk@0.129.0(zod@4.4.3))(@modelcontextprotocol/sdk@1.31.0(zod@4.4.3))(zod@4.4.3)':
dependencies:
'@anthropic-ai/sdk': 0.129.0(zod@4.4.3)
'@modelcontextprotocol/sdk': 1.31.0(zod@4.4.3)
zod: 4.4.3
optionalDependencies:
'@anthropic-ai/claude-agent-sdk-darwin-arm64': 0.3.286
'@anthropic-ai/claude-agent-sdk-darwin-x64': 0.3.286
'@anthropic-ai/claude-agent-sdk-linux-arm64': 0.3.286
'@anthropic-ai/claude-agent-sdk-linux-arm64-musl': 0.3.286
'@anthropic-ai/claude-agent-sdk-linux-x64': 0.3.286
'@anthropic-ai/claude-agent-sdk-linux-x64-musl': 0.3.286
'@anthropic-ai/claude-agent-sdk-win32-arm64': 0.3.286
'@anthropic-ai/claude-agent-sdk-win32-x64': 0.3.286
'@anthropic-ai/claude-agent-sdk@0.3.286(@anthropic-ai/sdk@0.129.0(zod@4.6.5))(@modelcontextprotocol/sdk@1.31.0(zod@4.6.5))(zod@4.4.3)':
dependencies:
'@anthropic-ai/sdk': 0.129.0(zod@4.6.5)
@@ -9329,6 +9356,13 @@ snapshots:
'@anthropic-ai/claude-agent-sdk-win32-arm64': 0.3.286
'@anthropic-ai/claude-agent-sdk-win32-x64': 0.3.286
'@anthropic-ai/sdk@0.129.0(zod@4.4.3)':
dependencies:
json-schema-to-ts: 3.1.1
standardwebhooks: 1.1.1
optionalDependencies:
zod: 4.4.3
'@anthropic-ai/sdk@0.129.0(zod@4.6.5)':
dependencies:
json-schema-to-ts: 3.1.1
@@ -4,6 +4,8 @@ import { spawnSync } from 'node:child_process';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import test from 'node:test';
import { fileURLToPath } from 'node:url';
import { isWorkspacePackage, parseWorkspaceEntries } from '../build-standalone-public-packages.mjs';
import { GROK_PUBLIC_INSTALL_IMAGE, GROK_PUBLIC_INSTALL_LIFECYCLE, assertInstalledCodexPlatformProvenance, assertNoBundledCodexPayloads, grokConsumerDockerArgs, installedCodexProbeSource } from '../grok-public-install-sandbox.mjs';
const paths = { assets: '/private/staging/assets', consumer: '/private/staging/consumer', cache: '/private/staging/cache', uid: 1001, gid: 1001 };
@@ -110,6 +112,87 @@ test('the image and Grok downloads start before staging and join without weakeni
assert.equal(source.match(/rmSync\(root/g).length, 1);
});
test('the bundled-dependency installs run together while the other packages are copied, and pack in order', () => {
const source = readFileSync(new URL('../verify-grok-npm-install.mjs', import.meta.url), 'utf8');
const position = fragment => { const index = source.indexOf(fragment); assert.notEqual(index, -1, `verifier must contain ${fragment}`); return index; };
// The installs are children of the script's own CLI entry through the helper
// covered by stage-bundled-packages.test.mjs, never in-process calls.
assert.doesNotMatch(source, /prepareBundledPackage/);
assert.ok(source.includes("import { bundledInstallTimings, startBundledInstalls } from './stage-bundled-packages.mjs'"));
// Only the bundled packages' sources are staged before the installs start;
// the installs are registered for cleanup at once, before anything else can
// fail; the other packages are copied while the installs run; the join comes
// before the packs start; and the packs run through the helper covered by
// pack-staged-packages.test.mjs, bundled packages first, each registered for
// cleanup as it starts, with the results put back into dependency order
// before every tarball is checked.
const bundledSources = position('for (const entry of entries.filter(entry => entry.bundled)) stageSource(entry)');
const start = position("startBundledInstalls(entries.filter(entry => entry.bundled), { logDir: prefetched, cwd: repo })");
const registered = position('prefetches.push(...installs.map(({ handle }) => handle))');
const otherSources = position('for (const entry of entries.filter(entry => !entry.bundled))');
const join = position("await awaited('bundled', Promise.all(installs.map(({ handle }) => handle.done)))");
assert.ok(source.includes("import { packConcurrency, packStagedPackages, packTimings } from './pack-staged-packages.mjs'"));
const packs = position("await packStagedPackages([...entries].sort((a, b) => Number(b.bundled) - Number(a.bundled)), { logDir: prefetched, env, concurrency, register: handle => prefetches.push(handle) })");
const reordered = position('packs.sort((a, b) => a.index - b.index)');
const checks = position("for (const tarball of tarballs) assertNoBundledCodexPayloads(run('tar', ['-tzf', tarball])");
const copied = position('for (const tarball of tarballs) {');
assert.ok(bundledSources < start && start < registered && registered < otherSources && otherSources < join && join < packs && packs < reordered && reordered < checks && checks < copied);
assert.equal(source.match(/run\('npm', \['pack'/g).length, 1, 'only the lifecycle sentinel is packed in-process');
assert.ok(source.includes("destination: join(root, `pack-${index}`)"), 'each pack writes into its own directory');
// The timing line reports the split staging phase, each install, and the
// pack step with what the packs would have cost one after another.
for (const key of ['stage_ui_dist_ms', 'stage_sources_ms', 'stage_bundled_ms', 'stage_pack_ms', 'stage_pack_check_ms']) assert.ok(source.includes(`timing.${key} =`));
assert.ok(source.includes('...bundledInstallTimings(installs)'));
assert.ok(source.includes('Object.assign(timing, packTimings(packs, concurrency))'));
});
test('the verifier stages only workspace packages, so it skips the standalone build release.sh needs', () => {
const source = readFileSync(new URL('../verify-grok-npm-install.mjs', import.meta.url), 'utf8');
const position = fragment => { const index = source.indexOf(fragment); assert.notEqual(index, -1, `verifier must contain ${fragment}`); return index; };
// The standalone public packages are the ones outside the pnpm workspace
// (the sandbox-provider plugins). release.sh builds them because it publishes
// them; the verifier packs only the closure of the server and the CLI under
// `workspace:` specs, and a `workspace:` spec can only name a workspace
// package, so no tarball here ever contains or depends on a standalone build.
// The one reader of that build is qualification-package retention, which
// packs the Daytona plugin from its `dist`, so the build runs exactly when
// retention is requested: the single reference sits inside that gate, and
// the same flag is what later decides whether retention runs at all.
const standaloneBuild = "run(process.execPath, [join(repo, 'scripts/build-standalone-public-packages.mjs')], repo)";
assert.equal(source.split('build-standalone-public-packages').length - 1, 1, 'the standalone build is referenced once, inside the retention gate');
const gate = position("const retainQualificationPackages = Boolean(process.env.PAPERCLIP_RUNNER_QUALIFICATION_PACKAGES_DIR);\n if (retainQualificationPackages) {\n const standaloneStarted = performance.now();\n " + standaloneBuild);
assert.ok(source.includes('timing.stage_standalone_build_ms ='), 'the gated build is timed under its own key');
assert.ok(source.includes('if (retainQualificationPackages) {\n const retained = retainRunnerQualificationPackages({'), 'retention is gated by the same flag as the build');
assert.doesNotMatch(source, /if \(process\.env\.PAPERCLIP_RUNNER_QUALIFICATION_PACKAGES_DIR\)/, 'no second reading of the retention flag can diverge from the build gate');
const retention = readFileSync(new URL('../retain-runner-qualification-packages.mjs', import.meta.url), 'utf8');
assert.ok(retention.includes("const pluginRelative = 'packages/plugins/sandbox-providers/daytona';") && retention.includes("cpSync(join(repo, pluginRelative, 'dist'), join(plugin, 'dist'), { recursive: true })"), 'retention still packs the Daytona plugin from its dist, which is why the gated build exists');
const repo = new URL('../../', import.meta.url);
const listing = spawnSync(process.execPath, [fileURLToPath(new URL('scripts/release-package-map.mjs', repo)), 'list'], { cwd: fileURLToPath(repo), encoding: 'utf8' });
assert.equal(listing.status, 0, listing.stderr);
const packages = new Map(listing.stdout.trim().split('\n').map(line => line.split('\t')).map(([dir, name]) => [name, { dir, manifest: JSON.parse(readFileSync(new URL(`${dir}/package.json`, repo), 'utf8')) }]));
const needed = new Set();
const visit = name => {
if (needed.has(name)) return;
const entry = packages.get(name); assert.ok(entry, `missing public workspace dependency ${name}`); needed.add(name);
for (const [dep, spec] of Object.entries({ ...entry.manifest.dependencies, ...entry.manifest.optionalDependencies })) if (spec.startsWith('workspace:')) visit(dep);
};
assert.ok(source.includes("visit('@paperclipai/server');\n visit('paperclipai');"));
visit('@paperclipai/server'); visit('paperclipai');
const workspace = parseWorkspaceEntries(readFileSync(new URL('pnpm-workspace.yaml', repo), 'utf8'));
const standalone = [...packages.values()].filter(({ dir }) => !isWorkspacePackage(dir, workspace)).map(({ dir }) => dir);
assert.ok(standalone.length > 0, 'the public package map must still carry standalone packages for this check to mean anything');
assert.ok(standalone.includes('packages/plugins/sandbox-providers/daytona'), 'the Daytona plugin retention packs is a standalone package, so pnpm build alone cannot produce its dist');
for (const name of needed) assert.ok(isWorkspacePackage(packages.get(name).dir, workspace), `${name} is staged by the verifier but is not a workspace package`);
// The server publishes `ui-dist`, so that copy stays, before any package is
// staged, reusing the UI that `pnpm build` produced, and it is timed. The
// gated standalone build keeps release.sh's order: after the closure is
// known and before the UI copy and any staged source.
assert.deepEqual(JSON.parse(readFileSync(new URL('server/package.json', repo), 'utf8')).files, ['dist', 'ui-dist', 'skills']);
assert.ok(source.includes("PAPERCLIP_RELEASE_REUSE_UI_DIST: '1'"));
const uiDist = position("run('bash', [join(repo, 'scripts/prepare-server-ui-dist.sh')], repo)");
assert.ok(position("visit('paperclipai')") < gate && gate < uiDist && uiDist < position('timing.stage_ui_dist_ms =') && uiDist < position('const stageSource = '));
});
test('the installed Codex probe exercises the public export and rejects incomplete or mismatched packages', () => {
const root = realpathSync(mkdtempSync(join(tmpdir(), 'installed-codex-probe-')));
try {
@@ -0,0 +1,107 @@
import assert from 'node:assert/strict';
import { existsSync, mkdirSync, mkdtempSync, readFileSync, readdirSync, rmSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import test from 'node:test';
import { stopPrefetches } from '../prefetch-child.mjs';
import { PACK_ARGS, packConcurrency, packStagedPackages, packTimings } from '../pack-staged-packages.mjs';
// A small local script stands in for npm; no test here needs npm or a network.
// It is run as `node <script> pack --ignore-scripts --pack-destination <dir>`,
// the same argument list the real npm receives.
const withScratch = async body => {
const root = mkdtempSync(join(tmpdir(), 'pack-staged-packages-'));
try { await body(root); } finally { rmSync(root, { recursive: true, force: true }); }
};
const entriesFor = (root, names) => names.map((name, index) => {
const target = join(root, `package-${index}`);
mkdirSync(target);
return { name, index, target, destination: join(root, `pack-${index}`) };
});
const fakeNpm = (root, body) => { const script = join(root, 'fake-npm.mjs'); writeFileSync(script, body); return [process.execPath, script]; };
const settle = ms => new Promise(resolve => setTimeout(resolve, ms));
test('the concurrency defaults to one child per core', () => {
assert.ok(Number.isInteger(packConcurrency()) && packConcurrency() >= 1);
assert.deepEqual(PACK_ARGS, ['pack', '--ignore-scripts', '--pack-destination']);
});
test('packs run together up to the concurrency, in the order given, and the tarballs come back in entry order', async () => withScratch(async root => {
// Each stand-in records when it started and finished, writes one tarball
// named after its package, and prints the tarball name as npm does. The
// first two packs are slow and the last two fast, so with two children at
// a time the slow pair runs together and the fast pair starts only once one
// of them has finished; the fast packs then finish out of entry order.
const command = fakeNpm(root, `
import { basename, join } from 'node:path';
import { writeFileSync } from 'node:fs';
const args = process.argv.slice(2);
const destination = args[args.length - 1];
const name = basename(process.cwd());
const started = Date.now();
const delay = name === 'package-0' || name === 'package-1' ? 400 : 50;
Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, delay);
writeFileSync(join(destination, name + '-1.0.0.tgz'), 'tarball');
writeFileSync(join(destination, '..', 'span-' + name), started + ' ' + Date.now());
console.log(args.join(' '));
console.log(name + '-1.0.0.tgz');
`);
const entries = entriesFor(root, ['@paperclipai/server', '@paperclipai/db', '@paperclipai/shared', 'paperclipai']);
const registered = [];
const packs = await packStagedPackages(entries, { logDir: root, concurrency: 2, register: handle => registered.push(handle), command });
assert.deepEqual(packs.map(({ name }) => name), entries.map(({ name }) => name));
assert.deepEqual(packs.map(({ index }) => index), [0, 1, 2, 3]);
for (const [index, pack] of packs.entries()) {
assert.equal(pack.tarball, join(root, `pack-${index}`, `package-${index}-1.0.0.tgz`));
assert.equal(readFileSync(pack.tarball, 'utf8'), 'tarball');
assert.ok(Number.isInteger(pack.durationMs) && pack.durationMs >= 0);
const log = readFileSync(join(root, `pack-${index}.log`), 'utf8');
assert.equal(log.split('\n')[0], `pack --ignore-scripts --pack-destination ${join(root, `pack-${index}`)}`);
}
assert.equal(registered.length, 4, 'every pack is registered for cleanup as it starts');
for (const handle of registered) assert.equal(handle.child.exitCode, 0);
const span = name => readFileSync(join(root, `span-${name}`), 'utf8').split(' ').map(Number);
const [slowA, slowB, fastA, fastB] = ['package-0', 'package-1', 'package-2', 'package-3'].map(span);
assert.ok(slowA[0] < slowB[1] && slowB[0] < slowA[1], 'the two slow packs ran at the same time');
const firstSlowEnd = Math.min(slowA[1], slowB[1]);
assert.ok(fastA[0] >= firstSlowEnd - 50 && fastB[0] >= firstSlowEnd - 50, 'a third pack starts only once a slot is free');
const timings = packTimings(packs, 2);
assert.equal(timings.pack_concurrency, 2);
assert.equal(timings.pack_serial_ms, packs.reduce((total, { durationMs }) => total + durationMs, 0));
assert.equal(timings.pack_longest_ms, Math.max(...packs.map(({ durationMs }) => durationMs)));
}));
test('a failed pack rejects with the tail of its log, launches nothing more, and leaves the running pack to be stopped', async () => withScratch(async root => {
const command = fakeNpm(root, `
import { basename } from 'node:path';
if (basename(process.cwd()) === 'package-1') { console.error('pack exploded'); process.exit(2); }
setTimeout(() => {}, 30_000);
`);
const entries = entriesFor(root, ['@paperclipai/server', '@paperclipai/db', '@paperclipai/shared', 'paperclipai']);
const registered = [];
await assert.rejects(packStagedPackages(entries, { logDir: root, concurrency: 2, register: handle => registered.push(handle), command }), /pack-1 prefetch failed \(exit 2\): pack exploded/);
await settle(100);
assert.equal(registered.length, 2, 'the two queued packs never start after the failure');
assert.equal(registered[0].child.exitCode, null, 'the other pack is still running when the failure surfaces');
assert.ok(!existsSync(join(root, 'pack-2')) && !existsSync(join(root, 'pack-3')));
await stopPrefetches(registered);
assert.notEqual(registered[0].child.signalCode ?? registered[0].child.exitCode, null);
}));
test('a pack that exits without producing exactly one tarball is a failure', async () => withScratch(async root => {
const command = fakeNpm(root, `
import { basename, join } from 'node:path';
import { writeFileSync } from 'node:fs';
const destination = process.argv[process.argv.length - 1];
const name = basename(process.cwd());
if (name === 'package-0') { writeFileSync(join(destination, 'a.tgz'), ''); writeFileSync(join(destination, 'b.tgz'), ''); }
`);
const entries = entriesFor(root, ['@paperclipai/server', '@paperclipai/db']);
await assert.rejects(packStagedPackages(entries, { logDir: root, concurrency: 1, command }), /@paperclipai\/server pack produced 2 tarballs/);
assert.deepEqual(readdirSync(root).filter(name => name.startsWith('pack-') && !name.endsWith('.log')), ['pack-0'], 'the second pack never started');
await assert.rejects(packStagedPackages([entries[1]], { logDir: root, concurrency: 1, command }), /@paperclipai\/db pack produced 0 tarballs/);
}));
test('no entries resolve at once', async () => withScratch(async root => {
assert.deepEqual(await packStagedPackages([], { logDir: root, concurrency: 2, command: [process.execPath, '-e', ''] }), []);
}));
+95 -2
View File
@@ -1,5 +1,5 @@
import assert from 'node:assert/strict';
import { spawnSync } from 'node:child_process';
import { spawn, spawnSync } from 'node:child_process';
import { mkdtempSync, readFileSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
@@ -25,6 +25,8 @@ const untilLogHas = async (path, text) => {
test('a successful prefetch resolves, keeps its output, and is left alone by stop', async () => withScratch(async root => {
const handle = start(root, 'ok', "setTimeout(() => process.stdout.write('fetched\\n'), 150)");
// The parent keeps its default, immediate termination: no handler of ours.
for (const signal of ['SIGINT', 'SIGTERM', 'SIGHUP']) assert.equal(process.listenerCount(signal), 0, `${signal} handler installed in the parent`);
await handle.done;
assert.equal(handle.child.exitCode, 0);
assert.equal(readFileSync(join(root, 'ok.log'), 'utf8'), 'fetched\n');
@@ -66,7 +68,7 @@ test('stopping a running prefetch waits for it to exit so its directory can be r
await assert.rejects(handle.done, /slow prefetch failed \(SIGTERM\): started/);
}));
test('a prefetch that ignores SIGTERM is killed after the grace period', async () => withScratch(async root => {
test('a prefetch that ignores SIGTERM is killed after the grace period', { skip: process.platform === 'win32' && 'SIGTERM cannot be ignored on Windows' }, async () => withScratch(async root => {
const handle = start(root, 'stubborn', "process.on('SIGTERM', () => {}); process.stdout.write('ignoring\\n'); setInterval(() => {}, 1000)");
await untilLogHas(join(root, 'stubborn.log'), 'ignoring');
const stopping = Date.now();
@@ -75,6 +77,97 @@ test('a prefetch that ignores SIGTERM is killed after the grace period', async (
assert.equal(handle.child.signalCode, 'SIGKILL');
}));
// A process is gone once the kernel no longer knows it, or once it is a zombie
// that only waits to be reaped: either way it can no longer write anywhere.
const gone = pid => {
try { process.kill(pid, 0); } catch (error) { return error.code === 'ESRCH'; }
try { return /^\d+ \(.*\) Z /.test(readFileSync(`/proc/${pid}/stat`, 'utf8')); } catch { return false; }
};
const untilGone = async pid => {
for (let attempt = 0; attempt < 200; attempt++) {
if (gone(pid)) return true;
await new Promise(resolve => setTimeout(resolve, 25));
}
return false;
};
// The child starts a process of its own that announces its pid and then runs
// on, the way the bundled-dependency installs block in `npm install`.
const GRANDCHILD_SCRIPT = `
const { spawn } = require('node:child_process');
const grandchild = spawn(process.execPath, ['-e', "process.stdout.write('grandchild ' + process.pid + '\\\\n'); setInterval(() => {}, 1000)"], { stdio: 'inherit' });
grandchild.on('exit', () => process.exit(0));
`;
const grandchildPid = async log => { await untilLogHas(log, 'grandchild '); return Number(readFileSync(log, 'utf8').match(/grandchild (\d+)/)[1]); };
test('stopping a prefetch also stops the processes it started', { skip: process.platform === 'win32' && 'process groups are POSIX only' }, async () => withScratch(async root => {
const handle = start(root, 'tree', GRANDCHILD_SCRIPT);
const pid = await grandchildPid(join(root, 'tree.log'));
assert.ok(!gone(pid), 'the grandchild runs before the stop');
await stopPrefetches([handle]);
assert.ok(await untilGone(pid), `grandchild ${pid} outlived stopPrefetches`);
assert.notEqual(handle.child.signalCode ?? handle.child.exitCode, null);
}));
test('a stop request written to the pipe ends the command and the processes it started, with no signal to the supervisor', { skip: process.platform === 'win32' && 'the grandchild check reads /proc' }, async () => withScratch(async root => {
// This is the stop path on a platform without process groups: there `kill`
// would end the supervisor outright and leave its command running, so
// `stopPrefetches` writes a request instead. The supervisor answers it
// the way it answers the parent's exit, and its own exit still reports the
// command's fate.
const handle = start(root, 'tree', GRANDCHILD_SCRIPT);
const pid = await grandchildPid(join(root, 'tree.log'));
handle.child.stdin.end('stop\n');
await assert.rejects(handle.done, /tree prefetch failed \(SIGTERM\)/);
assert.ok(await untilGone(pid), `grandchild ${pid} outlived the stop request`);
assert.match(readFileSync(join(root, 'tree.log'), 'utf8'), /stop requested; stopping/);
assert.doesNotMatch(readFileSync(join(root, 'tree.log'), 'utf8'), /parent exited/);
await stopPrefetches([handle]);
}));
// A separate parent process stands in for the verifier: it starts a prefetch
// whose child starts a grandchild, then does `work` (a module-scope statement).
const parentScript = (root, work) => `
import { execFileSync } from 'node:child_process';
import { startPrefetch } from ${JSON.stringify(new URL('../prefetch-child.mjs', import.meta.url).href)};
startPrefetch({ name: 'tree', cmd: process.execPath, args: ['-e', ${JSON.stringify(GRANDCHILD_SCRIPT)}], cwd: ${JSON.stringify(root)}, env: process.env, logPath: ${JSON.stringify(join(root, 'tree.log'))} });
${work}
`;
for (const signal of ['SIGINT', 'SIGKILL']) {
test(`${signal} to the parent ends the processes a prefetch started, and the parent dies of it at once`, { skip: process.platform === 'win32' && 'process groups are POSIX only' }, async () => withScratch(async root => {
// The parent runs no handler, so it dies of the signal immediately, and the
// prefetch tree stops because the parent is gone, not because it was told:
// SIGKILL leaves the parent no chance to tell anyone anything.
const parent = spawn(process.execPath, ['--input-type=module', '-e', parentScript(root, 'setInterval(() => {}, 1000);')], { cwd: root, stdio: 'ignore' });
const exited = new Promise(resolve => parent.once('exit', (code, signal) => resolve({ code, signal })));
const pid = await grandchildPid(join(root, 'tree.log'));
parent.kill(signal);
assert.deepEqual(await exited, { code: null, signal });
assert.ok(await untilGone(pid), `grandchild ${pid} outlived the parent's ${signal}`);
assert.match(readFileSync(join(root, 'tree.log'), 'utf8'), /parent exited; stopping/);
}));
}
test('Ctrl-C while the parent is blocked in synchronous work stops it at once, and the prefetch tree with it', { skip: process.platform === 'win32' && 'process groups are POSIX only' }, async () => withScratch(async root => {
// The verifier runs `execFileSync` for the standalone builds while prefetches
// run, and a JavaScript signal handler cannot run until that returns. The
// parent gets its own group here so the signal can be sent the way a
// terminal sends Ctrl-C: to the foreground group, which is the parent and
// its synchronous command, and not the detached prefetch tree.
const parent = spawn(process.execPath, ['--input-type=module', '-e', parentScript(root, "execFileSync(process.execPath, ['-e', 'setTimeout(() => {}, 20000)'], { stdio: 'ignore' });")], { cwd: root, stdio: 'ignore', detached: true });
try {
const exited = new Promise(resolve => parent.once('exit', (code, signal) => resolve({ code, signal })));
const pid = await grandchildPid(join(root, 'tree.log'));
const signalled = Date.now();
process.kill(-parent.pid, 'SIGINT');
assert.deepEqual(await exited, { code: null, signal: 'SIGINT' });
assert.ok(Date.now() - signalled < 2000, `the parent took ${Date.now() - signalled}ms to die: it waited for the synchronous command`);
assert.ok(await untilGone(pid), `grandchild ${pid} outlived the parent's Ctrl-C`);
} finally {
try { process.kill(-parent.pid, 'SIGKILL'); } catch {}
}
}));
test('an unawaited failure never becomes an unhandled rejection', () => {
const script = `
import { startPrefetch } from ${JSON.stringify(new URL('../prefetch-child.mjs', import.meta.url).href)};
@@ -0,0 +1,104 @@
import assert from 'node:assert/strict';
import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import test from 'node:test';
import { stopPrefetches } from '../prefetch-child.mjs';
import { PREPARE_BUNDLED_PACKAGE_SCRIPT, bundledInstallTimings, bundledPackageSlug, startBundledInstalls } from '../stage-bundled-packages.mjs';
// A small local script stands in for `prepare-bundled-package.mjs`; no test
// here needs npm or a network.
const withScratch = async body => {
const root = mkdtempSync(join(tmpdir(), 'stage-bundled-packages-'));
try { await body(root); } finally { rmSync(root, { recursive: true, force: true }); }
};
const entriesFor = (root, names) => names.map((name, index) => ({ name, stagedSource: join(root, `source-${index}`), target: join(root, `package-${index}`) }));
const fakeScript = (root, body) => { const script = join(root, 'fake-prepare.mjs'); writeFileSync(script, body); return script; };
test('package names map to stable log and timing slugs', () => {
assert.equal(bundledPackageSlug('@paperclipai/server'), 'server');
assert.equal(bundledPackageSlug('@paperclipai/adapter-utils'), 'adapter_utils');
assert.equal(bundledPackageSlug('@paperclipai/db'), 'db');
});
test('the children run the CLI entry release.sh uses, never an in-process call', () => {
assert.ok(existsSync(PREPARE_BUNDLED_PACKAGE_SCRIPT));
const source = readFileSync(PREPARE_BUNDLED_PACKAGE_SCRIPT, 'utf8');
assert.ok(source.includes('if (process.argv[1] === fileURLToPath(import.meta.url))'));
assert.ok(source.includes('prepareBundledPackage(resolve(sourceDir), resolve(destinationDir))'));
const release = readFileSync(new URL('../release.sh', import.meta.url), 'utf8');
assert.match(release, /node "\$REPO_ROOT\/scripts\/prepare-bundled-package\.mjs" "\$REPO_ROOT\/\$pkg_dir" "\$publish_dir"/);
});
test('the installs run at the same time, each with its own source and target', async () => withScratch(async root => {
// Each child records its arguments, announces itself, and exits only once
// the other child has announced itself too. Serial children would deadlock
// here and the five-second guard would turn that into a failure.
const script = fakeScript(root, `
import { existsSync, writeFileSync } from 'node:fs';
import { basename, join } from 'node:path';
const [source, target] = process.argv.slice(2);
const root = join(target, '..');
writeFileSync(join(root, 'started-' + basename(target)), '');
console.log('install ' + source + ' -> ' + target);
const other = basename(target) === 'package-0' ? 'package-1' : 'package-0';
const deadline = Date.now() + 5000;
while (!existsSync(join(root, 'started-' + other))) {
if (Date.now() > deadline) { console.error('the other install never started'); process.exit(3); }
Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, 20);
}
writeFileSync(join(target, 'package.json'), '{}');
`);
const entries = entriesFor(root, ['@paperclipai/server', '@paperclipai/db']);
for (const { target } of entries) mkdirSync(target);
const installs = startBundledInstalls(entries, { logDir: root, cwd: root, script });
assert.deepEqual(installs.map(({ slug }) => slug), ['server', 'db']);
await Promise.all(installs.map(({ handle }) => handle.done));
for (const [index, { stagedSource, target, slug, handle }] of installs.entries()) {
assert.equal(handle.child.exitCode, 0);
assert.equal(readFileSync(join(root, `bundled-${slug}.log`), 'utf8'), `install ${stagedSource} -> ${target}\n`);
assert.equal(target, join(root, `package-${index}`));
assert.ok(existsSync(join(target, 'package.json')));
}
const timings = bundledInstallTimings(installs);
assert.deepEqual(Object.keys(timings), ['stage_bundled_server_ms', 'stage_bundled_db_ms']);
for (const value of Object.values(timings)) assert.ok(Number.isInteger(value) && value >= 0);
}));
test('stopping the installs after one fails also stops the npm process the other started', { skip: process.platform === 'win32' && 'process groups are POSIX only' }, async () => withScratch(async root => {
// The real wrapper blocks in `execFileSync('npm', ['install', ...])`; here
// the stand-in npm announces its pid and runs on, and the wrapper blocks on
// it the same way. Stopping only the wrapper would leave it writing.
const script = fakeScript(root, `
import { execFileSync } from 'node:child_process';
const [, target] = process.argv.slice(2);
if (target.endsWith('package-1')) { console.error('bundled install exploded'); process.exit(2); }
execFileSync(process.execPath, ['-e', "process.stdout.write('npm ' + process.pid + '\\\\n'); setInterval(() => {}, 1000)"], { stdio: 'inherit' });
`);
const installs = startBundledInstalls(entriesFor(root, ['@paperclipai/server', '@paperclipai/db']), { logDir: root, cwd: root, script });
await assert.rejects(Promise.all(installs.map(({ handle }) => handle.done)), /bundled-db prefetch failed \(exit 2\)/);
const log = join(root, 'bundled-server.log');
const logHasPid = () => existsSync(log) && /npm \d+/.test(readFileSync(log, 'utf8'));
for (let attempt = 0; attempt < 200 && !logHasPid(); attempt++) await new Promise(resolve => setTimeout(resolve, 25));
const npmPid = Number(readFileSync(log, 'utf8').match(/npm (\d+)/)[1]);
const alive = pid => { try { process.kill(pid, 0); return !/^\d+ \(.*\) Z /.test(readFileSync(`/proc/${pid}/stat`, 'utf8')); } catch (error) { return error.code !== 'ESRCH'; } };
assert.ok(alive(npmPid), 'the stand-in npm runs before the stop');
await stopPrefetches(installs.map(({ handle }) => handle));
let stopped = false;
for (let attempt = 0; attempt < 200 && !(stopped = !alive(npmPid)); attempt++) await new Promise(resolve => setTimeout(resolve, 25));
assert.ok(stopped, `stand-in npm ${npmPid} outlived stopPrefetches`);
assert.notEqual(installs[0].handle.child.signalCode ?? installs[0].handle.child.exitCode, null);
}));
test('a failed install rejects with the tail of its own log and leaves the others to be stopped', async () => withScratch(async root => {
const script = fakeScript(root, `
const [, target] = process.argv.slice(2);
if (target.endsWith('package-1')) { console.error('bundled install exploded'); process.exit(2); }
setTimeout(() => {}, 30_000);
`);
const installs = startBundledInstalls(entriesFor(root, ['@paperclipai/server', '@paperclipai/adapter-utils']), { logDir: root, cwd: root, script });
await assert.rejects(Promise.all(installs.map(({ handle }) => handle.done)), /bundled-adapter_utils prefetch failed \(exit 2\): bundled install exploded/);
assert.equal(installs[0].handle.child.exitCode, null, 'the other install is still running when the failure surfaces');
await stopPrefetches(installs.map(({ handle }) => handle));
assert.notEqual(installs[0].handle.child.signalCode ?? installs[0].handle.child.exitCode, null);
}));
+96 -3
View File
@@ -2,6 +2,7 @@ import assert from "node:assert/strict";
import { execFileSync } from "node:child_process";
import {
existsSync,
cpSync,
lstatSync,
mkdirSync,
mkdtempSync,
@@ -56,6 +57,33 @@ function esbuildFixture(t) {
}
const codexTargets = ["linux-x64", "linux-arm64", "darwin-x64", "darwin-arm64", "win32-x64", "win32-arm64"];
const codexOptional = Object.fromEntries(codexTargets.map(target => [`@openai/codex-${target}`, `npm:@openai/codex@0.160.0-${target}`]));
const claudeTargets = [...codexTargets, "linux-x64-musl", "linux-arm64-musl"];
const claudeOptional = Object.fromEntries(claudeTargets.map(target => [`@anthropic-ai/claude-agent-sdk-${target}`, "0.3.286"]));
function claudeFixture(t) {
const directory = realpathSync(mkdtempSync(join(tmpdir(), "paperclip-bundled-claude-")));
t.after(() => rmSync(directory, { recursive: true, force: true }));
const graph = join(directory, "node_modules");
const writePackage = (location, metadata) => {
mkdirSync(location, { recursive: true });
writeFileSync(join(location, "package.json"), JSON.stringify(metadata));
return location;
};
const bridge = writePackage(join(graph, "@agentclientprotocol/claude-agent-acp"), {
name: "@agentclientprotocol/claude-agent-acp", version: "0.73.0", dependencies: { "@anthropic-ai/claude-agent-sdk": "0.3.286" },
});
writeFileSync(join(bridge, "index.js"), "export {};\n");
const runtime = writePackage(join(graph, "@anthropic-ai/claude-agent-sdk"), {
name: "@anthropic-ai/claude-agent-sdk", version: "0.3.286", exports: { ".": "./sdk.mjs" }, optionalDependencies: claudeOptional,
});
writeFileSync(join(runtime, "sdk.mjs"), "// official JavaScript fixture, never executed\n");
const producer = writePackage(join(graph, "@anthropic-ai/claude-agent-sdk-linux-x64"), {
name: "@anthropic-ai/claude-agent-sdk-linux-x64", version: "0.3.286", os: ["linux"], cpu: ["x64"],
});
writeFileSync(join(producer, "claude"), "producer native fixture, never executed");
const profiles = [{ agent: "claude", ...profileData.profiles.claude }];
const publish = { name: "paperclip-claude-pack-fixture", version: "1.0.0", dependencies: { "@agentclientprotocol/claude-agent-acp": "0.73.0" }, bundleDependencies: ["@agentclientprotocol/claude-agent-acp"] };
return { directory, graph, bridge, runtime, producer, profiles, publish, writePackage };
}
function codexFixture(t) {
const directory = realpathSync(mkdtempSync(join(tmpdir(), "paperclip-bundled-codex-")));
t.after(() => rmSync(directory, { recursive: true, force: true }));
@@ -176,13 +204,26 @@ test("published packages preserve the patched ACPX runtime", () => {
assert.equal(cliEsbuildConfig.external.includes("acpx"), false);
});
test("published Claude bridge pins its qualified SDK and provider dependency closure", () => {
const published = materializePublishManifest(serverPackage);
const { installManifest } = configureBundledProviderOverrides(
createBundledInstallManifest(published, serverPackage.bundleDependencies),
serverPackage.bundleDependencies, rootPackage, profileData,
);
assert.deepEqual(installManifest.overrides["@agentclientprotocol/claude-agent-acp@0.73.0"], {
"@anthropic-ai/claude-agent-sdk": "0.3.286",
"@agentclientprotocol/sdk": "1.4.0",
zod: "4.4.3",
});
});
test("published Codex bridge uses its qualified producer closure without consumer overrides", () => {
const published = materializePublishManifest(serverPackage);
const { installManifest, profiles } = configureBundledProviderOverrides(
createBundledInstallManifest(published, serverPackage.bundleDependencies),
serverPackage.bundleDependencies, rootPackage, profileData,
);
assert.equal(profiles.length, 1);
assert.deepEqual(profiles.map(profile => profile.agent), ["codex", "claude"]);
for (const profile of profiles) {
assert.equal(published.dependencies[profile.agentServerPackage], profile.agentServerVersion);
assert.ok(published.bundleDependencies.includes(profile.agentServerPackage));
@@ -201,6 +242,58 @@ test("published Codex bridge uses its qualified producer closure without consume
assert.throws(() => configureBundledProviderOverrides(conflict, serverPackage.bundleDependencies, rootPackage, profileData), /conflicting producer override/);
});
test("Linux-produced Claude bundles retain the SDK and install native packages for the consumer platform", (t) => {
const { directory, bridge, runtime, producer, profiles, publish } = claudeFixture(t);
const bridgeBefore = readFileSync(join(bridge, "package.json"), "utf8");
const staged = stageBundledProviderOptionalDependencies(directory, publish, profiles);
assert.deepEqual(staged.optionalDependencies, claudeOptional);
assert.equal(existsSync(producer), false);
assert.equal(JSON.parse(readFileSync(join(runtime, "package.json"), "utf8")).optionalDependencies, undefined);
assert.equal(readFileSync(join(bridge, "package.json"), "utf8"), bridgeBefore);
assert.deepEqual(stageBundledProviderOptionalDependencies(directory, staged, profiles), staged);
writeFileSync(join(directory, "package.json"), JSON.stringify(staged));
const packs = JSON.parse(execFileSync("npm", ["pack", "--ignore-scripts", "--offline", "--json"], {
cwd: directory, encoding: "utf8", timeout: 30_000,
env: { ...process.env, HOME: directory, npm_config_cache: join(directory, "cache"), npm_config_update_notifier: "false" },
}));
const paths = packs[0].files.map(file => file.path);
assert.ok(paths.some(path => path.endsWith("@anthropic-ai/claude-agent-sdk/sdk.mjs")));
assert.equal(paths.some(path => /@anthropic-ai\/claude-agent-sdk-(?:linux|darwin|win32)-/.test(path)), false);
});
test("Claude bundle validation rejects unqualified declarations before removing native payloads", (t) => {
const { directory, runtime, producer, profiles, publish } = claudeFixture(t);
const file = join(runtime, "package.json"), original = JSON.parse(readFileSync(file, "utf8"));
for (const metadata of [
{ ...original, version: "0.3.285" },
{ ...original, optionalDependencies: { ...claudeOptional, "@anthropic-ai/claude-agent-sdk-darwin-arm64": "latest" } },
{ ...original, optionalDependencies: {} },
]) {
writeFileSync(file, JSON.stringify(metadata));
assert.throws(() => stageBundledProviderOptionalDependencies(directory, publish, profiles), /version mismatch|not qualified/);
assert.equal(existsSync(producer), true);
}
});
test("mixed provider bundles validate both runtimes before removing either producer executable", (t) => {
const codex = codexFixture(t), claude = claudeFixture(t);
for (const name of ["@agentclientprotocol/claude-agent-acp", "@anthropic-ai/claude-agent-sdk", "@anthropic-ai/claude-agent-sdk-linux-x64"]) {
cpSync(join(claude.graph, name), join(codex.graph, name), { recursive: true });
}
const profiles = [...codex.profiles, ...claude.profiles];
const publish = { ...codex.publish, dependencies: { ...codex.publish.dependencies, ...claude.publish.dependencies },
bundleDependencies: [...codex.publish.bundleDependencies, ...claude.publish.bundleDependencies] };
const file = join(codex.graph, "@anthropic-ai/claude-agent-sdk/package.json");
const metadata = JSON.parse(readFileSync(file, "utf8"));
writeFileSync(file, JSON.stringify({ ...metadata, version: "0.3.285" }));
assert.throws(() => stageBundledProviderOptionalDependencies(codex.directory, publish, profiles), /Claude runtime version mismatch/);
assert.equal(existsSync(codex.producer), true);
assert.deepEqual(JSON.parse(readFileSync(join(codex.runtime, "package.json"), "utf8")).optionalDependencies, codexOptional);
writeFileSync(file, JSON.stringify(metadata));
const staged = stageBundledProviderOptionalDependencies(codex.directory, publish, profiles);
assert.deepEqual(staged.optionalDependencies, { ...codexOptional, ...claudeOptional });
});
test("Codex bundles retain pinned JavaScript and delegate official consumer platform aliases without native payloads", (t) => {
const { directory, graph, bridge, runtime, producer, nested, profiles, publish } = codexFixture(t);
const before = [bridge, runtime].map(path => readFileSync(join(path, "package.json"), "utf8"));
@@ -530,7 +623,7 @@ mkdir -p "$destination/node_modules/.pnpm"
set -euo pipefail
printf 'npm %s\\n' "$*" >> "$FAKE_CALL_LOG"
[ "$*" = "install --omit=dev --ignore-scripts --no-audit --no-fund" ]
node -e 'const fs = require("node:fs"); const pkg = require("./package.json"); if ("devDependencies" in pkg) process.exit(1); for (const [name, version] of Object.entries(pkg.dependencies)) { const dir = "node_modules/" + name; fs.mkdirSync(dir + "/dist", { recursive: true }); fs.writeFileSync(dir + "/package.json", JSON.stringify({ name, version })); } for (const [bridge, runtimes] of Object.entries(pkg.overrides ?? {})) { for (const [name, version] of Object.entries(runtimes)) { const dir = "node_modules/" + name; fs.mkdirSync(dir, { recursive: true }); const native = name === "@openai/codex"; const optionalDependencies = native ? Object.fromEntries(["linux-x64","linux-arm64","darwin-x64","darwin-arm64","win32-x64","win32-arm64"].map(target => [name+"-"+target,"npm:"+name+"@"+version+"-"+target])) : undefined; fs.writeFileSync(dir + "/package.json", JSON.stringify({ name, version, optionalDependencies })); } }'
node -e 'const fs = require("node:fs"); const pkg = require("./package.json"); if ("devDependencies" in pkg) process.exit(1); for (const [name, version] of Object.entries(pkg.dependencies)) { const dir = "node_modules/" + name; fs.mkdirSync(dir + "/dist", { recursive: true }); fs.writeFileSync(dir + "/package.json", JSON.stringify({ name, version })); } for (const [bridge, runtimes] of Object.entries(pkg.overrides ?? {})) { for (const [name, version] of Object.entries(runtimes)) { const dir = "node_modules/" + name; fs.mkdirSync(dir, { recursive: true }); const native = name === "@openai/codex"; const targets = ["linux-x64","linux-arm64","darwin-x64","darwin-arm64","win32-x64","win32-arm64"]; const optionalDependencies = native ? Object.fromEntries(targets.map(target => [name+"-"+target,"npm:"+name+"@"+version+"-"+target])) : name === "@anthropic-ai/claude-agent-sdk" ? Object.fromEntries([...targets,"linux-x64-musl","linux-arm64-musl"].map(target => [name+"-"+target,version])) : undefined; fs.writeFileSync(dir + "/package.json", JSON.stringify({ name, version, optionalDependencies })); } }'
node -e 'const fs=require("node:fs");fs.cpSync(process.env.FAKE_NATIVE_ARTIFACTS+"/@openai/codex-linux-x64","node_modules/@openai/codex-linux-x64",{recursive:true});fs.mkdirSync("node_modules/@openai/codex/vendor",{recursive:true});fs.writeFileSync("node_modules/@openai/codex/vendor/codex","producer fallback fixture");'
mkdir -p node_modules/acpx/dist
node -e 'const fs=require("node:fs");fs.mkdirSync("node_modules/esbuild",{recursive:true});fs.writeFileSync("node_modules/esbuild/package.json",JSON.stringify({name:"esbuild",version:"0.28.2",scripts:{postinstall:"node install.js"},optionalDependencies:Object.fromEntries(["linux-x64","darwin-arm64","darwin-x64"].map(target=>["@esbuild/"+target,"0.28.2"]))}));fs.mkdirSync("node_modules/@esbuild/linux-x64",{recursive:true});fs.writeFileSync("node_modules/@esbuild/linux-x64/package.json",JSON.stringify({name:"@esbuild/linux-x64",version:"0.28.2",os:["linux"],cpu:["x64"]}));'
@@ -590,7 +683,7 @@ printf 'patched spawnEnvironment runtime\\n' > "$target/dist/runtime.js"
assert.equal(lstatSync(stagedAcpxDir).isSymbolicLink(), false);
assert.equal(existsSync(join(destinationDir, "node_modules/.pnpm")), false);
const stagedManifest = JSON.parse(readFileSync(join(destinationDir, "package.json"), "utf8"));
for (const [name, version] of Object.entries(codexOptional)) {
for (const [name, version] of Object.entries({ ...codexOptional, ...claudeOptional })) {
assert.equal(stagedManifest.optionalDependencies[name], version);
assert.equal(existsSync(join(destinationDir, "node_modules", name)), false);
assert.equal(stagedManifest.bundleDependencies.includes(name), false);
+2 -2
View File
@@ -17,7 +17,7 @@ const repoRoot = path.resolve(scriptDir, "..");
const workspacePath = path.join(repoRoot, "pnpm-workspace.yaml");
const releasePackageMapPath = path.join(repoRoot, "scripts", "release-package-map.mjs");
function parseWorkspaceEntries(workspaceText) {
export function parseWorkspaceEntries(workspaceText) {
// Keep this aligned with the repo's block-sequence `packages:` format in
// pnpm-workspace.yaml. If that file moves to a more complex YAML shape,
// switch this parser to a real YAML parser instead of line matching.
@@ -61,7 +61,7 @@ function globToRegExp(pattern) {
return new RegExp(`^${regex}$`);
}
function isWorkspacePackage(pkgDir, workspaceEntries) {
export function isWorkspacePackage(pkgDir, workspaceEntries) {
let included = false;
for (const entry of workspaceEntries) {
+64
View File
@@ -0,0 +1,64 @@
// The public-install verifier packs every staged public package with
// `npm pack`. Each pack reads only its own `package-<index>` directory and
// writes one tarball, and the work is a Node start-up plus CPU-bound gzip per
// package, so the packs run as concurrent children through the prefetch
// helper, at most one per core, instead of one after another. Each child
// packs into its own destination directory, so the tarball a package produced
// is read from there rather than guessed from a shared directory, and the
// results come back in the order the entries were given.
import { mkdirSync, readdirSync } from 'node:fs';
import { availableParallelism } from 'node:os';
import { join } from 'node:path';
import { startPrefetch } from './prefetch-child.mjs';
export const PACK_ARGS = ['pack', '--ignore-scripts', '--pack-destination'];
export const packConcurrency = () => Math.max(1, availableParallelism());
// Packs `entries` (`{ name, target, destination }`) with at most `concurrency`
// children at once, launched in the order given, and resolves with each entry
// extended by its `tarball` path and `durationMs`, in the same order. Every
// child is passed to `register` as it starts, so the caller can stop the ones
// still running when something fails. The first failure rejects with the tail
// of that child's log, and nothing further is launched after it.
export function packStagedPackages(entries, { logDir, env = process.env, concurrency = packConcurrency(), register = () => {}, command = ['npm'] }) {
const [cmd, ...commandArgs] = command;
const results = new Array(entries.length);
const queue = entries.map((entry, index) => ({ entry, index }));
let active = 0;
let failed = false;
return new Promise((resolve, reject) => {
const fail = error => { failed = true; reject(error); };
const launch = () => {
if (!failed && active === 0 && queue.length === 0) return resolve(results);
while (!failed && active < concurrency && queue.length > 0) {
const { entry, index } = queue.shift();
active += 1;
let handle;
try {
mkdirSync(entry.destination, { recursive: true });
handle = startPrefetch({ name: `pack-${index}`, cmd, args: [...commandArgs, ...PACK_ARGS, entry.destination], cwd: entry.target, env, logPath: join(logDir, `pack-${index}.log`) });
register(handle);
} catch (error) {
return fail(error);
}
handle.done.then(() => {
const packed = readdirSync(entry.destination).filter(file => file.endsWith('.tgz'));
if (packed.length !== 1) throw new Error(`${entry.name} pack produced ${packed.length} tarballs in ${entry.destination}`);
results[index] = { ...entry, tarball: join(entry.destination, packed[0]), durationMs: handle.durationMs };
active -= 1;
launch();
}).catch(fail);
}
};
launch();
});
}
// Timing-line keys: the concurrency used, the sum of the per-pack times (what
// the packs would have cost one after another) and the longest single pack.
export const packTimings = (packs, concurrency) => ({
pack_concurrency: concurrency,
pack_serial_ms: packs.reduce((total, { durationMs }) => total + durationMs, 0),
pack_longest_ms: packs.reduce((longest, { durationMs }) => Math.max(longest, durationMs), 0),
});
+68 -10
View File
@@ -2,23 +2,72 @@
// output in a log file, join it where the result is needed, and stop it before
// the scratch directory it writes into is removed. No shell, no inherited
// terminal, and a failure is reported where the caller awaits it.
//
// A child may start processes of its own: the bundled-dependency installs are
// a Node wrapper blocked in `npm install`. Signalling only the wrapper would
// leave npm writing into the scratch root while it is removed, so on POSIX
// each command runs under `prefetch-supervisor.mjs` as the leader of its own
// process group, and stopping signals and waits for the whole group. A group
// of its own no longer receives the terminal's Ctrl-C, and the parent cannot
// forward it: the verifier spends most of its time blocked in `execFileSync`,
// where no JavaScript signal handler can run, and installing one would also
// replace the parent's immediate default termination. Instead the supervisor
// holds a pipe from the parent and stops its group when the pipe closes, which
// the kernel does when the parent exits for any reason. The parent installs
// no signal handlers.
//
// Windows has no process groups and `kill` there ends a process outright, so a
// signalled supervisor could pass nothing on and its command would run on
// after cleanup returned. There a deliberate stop goes through the same pipe
// as a written request, which the supervisor answers by ending its command,
// and only the SIGKILL escalation targets the supervisor itself.
import { spawn } from 'node:child_process';
import { closeSync, openSync, readFileSync, statSync } from 'node:fs';
import { fileURLToPath } from 'node:url';
export const PREFETCH_SUPERVISOR_SCRIPT = fileURLToPath(new URL('./prefetch-supervisor.mjs', import.meta.url));
const GROUPS = process.platform !== 'win32';
const running = child => child.pid !== undefined && child.exitCode === null && child.signalCode === null;
const sleep = ms => new Promise(resolve => setTimeout(resolve, ms));
// Returns { name, child, done, durationMs }. `done` resolves on exit code 0
// and otherwise rejects with the tail of the log; the rejection is observed
// here so an unawaited failure never surfaces as an unhandled rejection.
// Signals the child's whole process group (POSIX). Without groups the
// supervisor is asked to stop its command through the pipe it watches, since
// `kill` would end the supervisor at once and leave the command running; only
// SIGKILL is sent to the supervisor itself, after it has had its chance.
const signalTree = (child, signal) => {
if (child.pid === undefined) return;
if (!GROUPS) {
if (!running(child)) return;
if (signal === 'SIGKILL') child.kill(signal);
else if (child.stdin?.writable) child.stdin.end('stop\n');
return;
}
try { process.kill(-child.pid, signal); } catch (error) { if (error.code !== 'ESRCH') throw error; }
};
// True while any member of the child's group exists. On a platform without
// groups only the child itself is known.
const treeAlive = child => {
if (child.pid === undefined) return false;
if (!GROUPS) return running(child);
try { process.kill(-child.pid, 0); return true; } catch (error) { return error.code !== 'ESRCH'; }
};
// Returns { name, child, done, durationMs }. `child` is the supervisor, whose
// exit code or signal is the command's. `done` resolves on exit code 0 and
// otherwise rejects with the tail of the log; the rejection is observed here
// so an unawaited failure never surfaces as an unhandled rejection.
export function startPrefetch({ name, cmd, args, cwd, env, logPath }) {
const startedAt = Date.now();
const log = openSync(logPath, 'w');
let child;
try {
child = spawn(cmd, args, { cwd, env, stdio: ['ignore', log, log] });
// stdin is the pipe the supervisor watches; nothing is ever written to it.
child = spawn(process.execPath, [PREFETCH_SUPERVISOR_SCRIPT, cmd, ...args], { cwd, env, stdio: ['pipe', log, log], detached: GROUPS });
} finally {
closeSync(log); // the child holds its own descriptor
}
child.stdin?.on('error', () => {});
const handle = { name, child, done: null, durationMs: null };
let settled = false;
handle.done = new Promise((settle, reject) => {
@@ -48,13 +97,22 @@ export function startPrefetch({ name, cmd, args, cwd, env, logPath }) {
return handle;
}
// Stops every prefetch that is still running and resolves only once all of
// them have exited, so the caller can remove their working directory without
// racing a write. A child that ignores SIGTERM is killed after `graceMs`.
// Stops every prefetch that is still running, together with any process it
// started, and resolves only once all of them have exited, so the caller can
// remove their working directory without racing a write. A tree that ignores
// SIGTERM is killed after `graceMs`; one that still exists `graceMs` after
// that is given up on so cleanup cannot hang.
export async function stopPrefetches(prefetches, { graceMs = 5000 } = {}) {
await Promise.all(prefetches.map(async ({ child, done }) => {
if (running(child)) child.kill('SIGTERM');
const escalate = setTimeout(() => { if (running(child)) child.kill('SIGKILL'); }, graceMs);
try { await done; } catch {} finally { clearTimeout(escalate); }
const started = Date.now();
let termed = false, killed = false;
while (treeAlive(child)) {
const elapsed = Date.now() - started;
if (!termed) { signalTree(child, 'SIGTERM'); termed = true; }
else if (!killed && elapsed >= graceMs) { signalTree(child, 'SIGKILL'); killed = true; }
else if (killed && elapsed >= 2 * graceMs) break;
await sleep(25);
}
try { await done; } catch {}
}));
}
+78
View File
@@ -0,0 +1,78 @@
// Child side of prefetch-child.mjs: runs one prefetch command as the leader of
// its own process group and stops the whole group when the parent that started
// it is gone.
//
// The parent holds the write end of this process's stdin. The kernel closes
// that end when the parent exits for any reason: a Ctrl-C delivered while the
// parent is blocked in synchronous work, a plain `kill`, or SIGKILL. Reading
// EOF here is therefore the only parent-death signal needed, and the parent
// runs no signal handler of its own, so its default termination stays
// immediate. The parent writes to the pipe for one reason only: to ask for a
// deliberate stop where it cannot signal the group, which is Windows, where
// `kill` ends a process outright and a signalled supervisor could not pass
// anything on. Group-wide signals the parent sends through `stopPrefetches`
// on POSIX reach this process too; it passes them on to the command and stays
// alive until the command exits, so its own exit reports the command's
// outcome (exit code, or the same signal).
import { spawn } from 'node:child_process';
import { fstatSync } from 'node:fs';
import { constants } from 'node:os';
const GROUPS = process.platform !== 'win32';
const GRACE_MS = Number(process.env.PAPERCLIP_PREFETCH_GRACE_MS) || 5000;
const [cmd, ...args] = process.argv.slice(2);
if (!cmd) {
process.stderr.write('prefetch supervisor: usage: prefetch-supervisor.mjs <command> [args...]\n');
process.exit(64);
}
const child = spawn(cmd, args, { stdio: ['ignore', 'inherit', 'inherit'] });
const signalChild = signal => { try { child.kill(signal); } catch {} };
// On POSIX this process leads the group (the parent starts it detached), so
// the group id is its own pid. Without groups only the command is known.
const signalGroup = signal => {
if (!GROUPS) return signalChild(signal);
try { process.kill(-process.pid, signal); } catch (error) { if (error.code === 'ESRCH') signalChild(signal); }
};
// A signal to the group arrives here as well as at the command; passing it on
// is harmless when the command already has it, and covers a signal sent to
// this process alone. The parent owns the SIGKILL escalation in that case.
for (const signal of ['SIGINT', 'SIGTERM', 'SIGHUP']) process.on(signal, () => signalChild(signal));
let stopping = false;
const stopGroup = reason => {
if (stopping) return;
stopping = true;
process.stderr.write(`prefetch supervisor: ${reason}; stopping ${cmd}\n`);
signalGroup('SIGTERM');
setTimeout(() => signalGroup('SIGKILL'), GRACE_MS).unref();
};
// Only a pipe can carry a stop request or report the parent's exit (Node
// creates its stdio pipes as socket pairs). Run by hand with a terminal or
// /dev/null on stdin, there is no parent to watch and nothing is read.
let stdinIsPipe = false;
try { const stdin = fstatSync(0); stdinIsPipe = stdin.isFIFO() || stdin.isSocket(); } catch {}
if (stdinIsPipe) {
process.stdin.on('data', () => stopGroup('stop requested'));
const parentGone = () => stopGroup('parent exited');
process.stdin.on('end', parentGone);
process.stdin.on('close', parentGone);
process.stdin.on('error', parentGone);
process.stdin.resume();
}
child.on('error', error => {
process.stderr.write(`prefetch supervisor: ${error.message}\n`);
process.exit(127);
});
child.on('exit', (code, signal) => {
if (signal !== null) {
// Die of the same signal, so the parent sees the command's fate on this
// process. Dropping the listener restores the default disposition first.
process.removeAllListeners(signal);
try { process.kill(process.pid, signal); } catch {}
process.exit(128 + (constants.signals[signal] ?? 1));
}
process.exit(code ?? 1);
});
+99 -79
View File
@@ -51,13 +51,14 @@ export function createBundledInstallManifest(publishManifest, bundledDependencie
}
// npm consumers cannot inherit the workspace's pnpm overrides or patches.
// Materialize the already-qualified Codex bridge closure in the tarball,
// Materialize the already-qualified provider bridge closures in the tarball,
// with overrides confined to the temporary, scripts-disabled producer graph.
export function configureBundledProviderOverrides(installManifest, bundledDependencies, rootPackage, profileData) {
const result = structuredClone(installManifest);
const selected = [];
for (const [agent, serverPackage, runtimePackage] of [
["codex", "@agentclientprotocol/codex-acp", "@openai/codex"],
["claude", "@agentclientprotocol/claude-agent-acp", "@anthropic-ai/claude-agent-sdk"],
]) {
if (!bundledDependencies.includes(serverPackage)) continue;
const profile = profileData?.profiles?.[agent];
@@ -75,7 +76,9 @@ export function configureBundledProviderOverrides(installManifest, bundledDepend
if (result.overrides?.[npmSelector] !== undefined) {
throw new Error(`Bundled ${agent} runtime has a conflicting producer override`);
}
const overrides = { [runtimePackage]: profile.agentRuntimeVersion };
const overrides = { [runtimePackage]: profile.agentRuntimeVersion,
...(agent === "claude" ? { "@agentclientprotocol/sdk": "1.4.0", zod: "4.4.3" } : {}),
};
result.overrides = { ...result.overrides, [npmSelector]: overrides };
selected.push({ agent, ...profile });
}
@@ -87,88 +90,105 @@ const inside = (root, candidate) => {
return value !== "" && value !== ".." && !value.startsWith(`..${sep}`) && !isAbsolute(value);
};
// Bundle the pinned JavaScript closure, never the producer's Codex executable.
// Bundle the pinned JavaScript closure, never the producer's executable.
// npm installs the official optional package for the consumer's own platform.
export function stageBundledProviderOptionalDependencies(destinationDir, publishManifest, profiles) {
const graphRoot = resolve(destinationDir, "node_modules");
if (realpathSync(graphRoot) !== graphRoot) throw new Error("Bundled Codex graph must be a canonical owned directory");
const result = structuredClone(publishManifest), remove = new Set(), optional = {}, normalize = new Map();
const profile = profiles.find(value => value.agent === "codex");
if (profiles.length !== 1 || !profile) throw new Error("Bundled provider graph must contain its qualified Codex profile");
const targets = ["linux-x64", "linux-arm64", "darwin-x64", "darwin-arm64", "win32-x64", "win32-arm64"];
const declarations = Object.fromEntries(targets.map(target => [`${profile.agentRuntimePackage}-${target}`,
`npm:${profile.agentRuntimePackage}@${profile.agentRuntimeVersion}-${target}`]));
const ownedDirectory = directory => {
const stat = lstatSync(directory);
if (!stat.isDirectory() || stat.isSymbolicLink() || !inside(graphRoot, realpathSync(directory))) {
throw new Error("Bundled Codex dependency escapes its producer graph");
}
};
const manifestAt = directory => {
const path = resolve(directory, "package.json"), stat = lstatSync(path);
if (!stat.isFile() || stat.isSymbolicLink() || stat.nlink !== 1 || stat.size > 256 * 1024) {
throw new Error("Bundled Codex dependency manifest must be a bounded regular file");
}
return JSON.parse(readFileSync(path, "utf8"));
};
const bridgeDirectory = resolve(graphRoot, profile.agentServerPackage);
ownedDirectory(bridgeDirectory);
const bridge = manifestAt(bridgeDirectory);
if (bridge.name !== profile.agentServerPackage || bridge.version !== profile.agentServerVersion) throw new Error("Bundled Codex bridge version mismatch");
const issuer = createRequire(resolve(bridgeDirectory, "package.json"));
const runtimeManifest = realpathSync(issuer.resolve(`${profile.agentRuntimePackage}/package.json`));
if (!inside(graphRoot, runtimeManifest)) throw new Error("Bundled Codex runtime escapes its producer graph");
let runtimeCount = 0, packageCount = 0;
const pending = [{ directory: graphRoot, depth: 0 }];
while (pending.length) {
const { directory, depth } = pending.pop();
if (depth > 64) throw new Error("Bundled Codex graph exceeds its depth limit");
const packages = [];
for (const entry of readdirSync(directory)) {
if (entry.startsWith(".")) continue;
const candidate = resolve(directory, entry); ownedDirectory(candidate);
if (entry.startsWith("@")) {
for (const child of readdirSync(candidate)) {
const scoped = resolve(candidate, child); ownedDirectory(scoped); packages.push(scoped);
}
} else packages.push(candidate);
}
for (const packageDirectory of packages) {
if (++packageCount > 20_000) throw new Error("Bundled Codex graph exceeds its package limit");
const metadata = manifestAt(packageDirectory);
const binding = basename(dirname(packageDirectory)) === "@openai" ? `@openai/${basename(packageDirectory)}` : null;
if (binding?.startsWith("@openai/codex-")) {
const target = binding.slice("@openai/codex-".length), [os, cpu] = target.split("-");
if (!Object.hasOwn(declarations, binding) || ![binding, profile.agentRuntimePackage].includes(metadata.name)
|| metadata.version !== `${profile.agentRuntimeVersion}-${target}` || !metadata.os?.includes(os) || !metadata.cpu?.includes(cpu)) {
throw new Error("Bundled Codex installed platform identity mismatch");
}
remove.add(packageDirectory);
continue;
}
const nested = resolve(packageDirectory, "node_modules");
if (lstatExists(nested)) { ownedDirectory(nested); pending.push({ directory: nested, depth: depth + 1 }); }
if (binding !== profile.agentRuntimePackage) continue;
if (metadata.name !== profile.agentRuntimePackage || metadata.version !== profile.agentRuntimeVersion) throw new Error("Bundled Codex runtime version mismatch");
runtimeCount++;
const delegated = metadata.optionalDependencies === undefined
&& Object.entries(declarations).every(([name, specifier]) => result.optionalDependencies?.[name] === specifier);
if (!delegated && (Object.keys(metadata.optionalDependencies ?? {}).length !== targets.length
|| targets.some(target => metadata.optionalDependencies?.[`${profile.agentRuntimePackage}-${target}`] !== declarations[`${profile.agentRuntimePackage}-${target}`]))) {
throw new Error("Bundled Codex platform declaration is not qualified");
}
if (!delegated) {
const normalized = { ...metadata }; delete normalized.optionalDependencies;
normalize.set(resolve(packageDirectory, "package.json"), normalized);
}
const vendor = resolve(packageDirectory, "vendor");
if (lstatExists(vendor)) { ownedDirectory(vendor); remove.add(vendor); }
}
if (!profiles.length || new Set(profiles.map(profile => profile.agent)).size !== profiles.length) {
throw new Error("Bundled provider graph must contain distinct qualified profiles");
}
if (!runtimeCount) throw new Error("Bundled Codex runtime omitted its platform declarations");
for (const [name, specifier] of Object.entries(declarations)) {
if (result.optionalDependencies?.[name] !== undefined && result.optionalDependencies[name] !== specifier) throw new Error(`Bundled Codex platform conflicts with published dependency: ${name}`);
optional[name] = specifier;
for (const profile of profiles) {
const expected = profile.agent === "codex" ? ["@agentclientprotocol/codex-acp", "@openai/codex"]
: profile.agent === "claude" ? ["@agentclientprotocol/claude-agent-acp", "@anthropic-ai/claude-agent-sdk"] : null;
if (!expected || profile.agentServerPackage !== expected[0] || profile.agentRuntimePackage !== expected[1]) {
throw new Error("Bundled provider graph contains an unknown provider profile");
}
const label = profile.agent === "codex" ? "Codex" : "Claude";
const targets = ["linux-x64", "linux-arm64", "darwin-x64", "darwin-arm64", "win32-x64", "win32-arm64",
...(profile.agent === "claude" ? ["linux-x64-musl", "linux-arm64-musl"] : [])];
const declarations = Object.fromEntries(targets.map(target => [`${profile.agentRuntimePackage}-${target}`,
profile.agent === "codex" ? `npm:${profile.agentRuntimePackage}@${profile.agentRuntimeVersion}-${target}` : profile.agentRuntimeVersion]));
const ownedDirectory = directory => {
const stat = lstatSync(directory);
if (!stat.isDirectory() || stat.isSymbolicLink() || !inside(graphRoot, realpathSync(directory))) {
throw new Error(`Bundled ${label} dependency escapes its producer graph`);
}
};
const manifestAt = directory => {
const path = resolve(directory, "package.json"), stat = lstatSync(path);
if (!stat.isFile() || stat.isSymbolicLink() || stat.nlink !== 1 || stat.size > 256 * 1024) {
throw new Error(`Bundled ${label} dependency manifest must be a bounded regular file`);
}
return JSON.parse(readFileSync(path, "utf8"));
};
const bridgeDirectory = resolve(graphRoot, profile.agentServerPackage);
ownedDirectory(bridgeDirectory);
const bridge = manifestAt(bridgeDirectory);
if (bridge.name !== profile.agentServerPackage || bridge.version !== profile.agentServerVersion) throw new Error(`Bundled ${label} bridge version mismatch`);
const issuer = createRequire(resolve(bridgeDirectory, "package.json"));
let runtimeManifest;
try { runtimeManifest = issuer.resolve(`${profile.agentRuntimePackage}/package.json`); }
catch (error) {
if (error.code !== "ERR_PACKAGE_PATH_NOT_EXPORTED") throw error;
runtimeManifest = resolve(dirname(issuer.resolve(profile.agentRuntimePackage)), "package.json");
}
runtimeManifest = realpathSync(runtimeManifest);
if (!inside(graphRoot, runtimeManifest)) throw new Error(`Bundled ${label} runtime escapes its producer graph`);
let runtimeCount = 0, packageCount = 0;
const pending = [{ directory: graphRoot, depth: 0 }];
while (pending.length) {
const { directory, depth } = pending.pop();
if (depth > 64) throw new Error(`Bundled ${label} graph exceeds its depth limit`);
const packages = [];
for (const entry of readdirSync(directory)) {
if (entry.startsWith(".")) continue;
const candidate = resolve(directory, entry); ownedDirectory(candidate);
if (entry.startsWith("@")) {
for (const child of readdirSync(candidate)) {
const scoped = resolve(candidate, child); ownedDirectory(scoped); packages.push(scoped);
}
} else packages.push(candidate);
}
for (const packageDirectory of packages) {
if (++packageCount > 20_000) throw new Error(`Bundled ${label} graph exceeds its package limit`);
const metadata = manifestAt(packageDirectory);
const scope = basename(dirname(packageDirectory));
const binding = scope.startsWith("@") ? `${scope}/${basename(packageDirectory)}` : basename(packageDirectory);
if (binding.startsWith(`${profile.agentRuntimePackage}-`)) {
const target = binding.slice(profile.agentRuntimePackage.length + 1), [os, cpu] = target.split("-");
if (!Object.hasOwn(declarations, binding) || !(metadata.name === binding || (profile.agent === "codex" && metadata.name === profile.agentRuntimePackage))
|| metadata.version !== (profile.agent === "codex" ? `${profile.agentRuntimeVersion}-${target}` : profile.agentRuntimeVersion) || !metadata.os?.includes(os) || !metadata.cpu?.includes(cpu)) {
throw new Error(`Bundled ${label} installed platform identity mismatch`);
}
remove.add(packageDirectory);
continue;
}
const nested = resolve(packageDirectory, "node_modules");
if (lstatExists(nested)) { ownedDirectory(nested); pending.push({ directory: nested, depth: depth + 1 }); }
if (binding !== profile.agentRuntimePackage) continue;
if (metadata.name !== profile.agentRuntimePackage || metadata.version !== profile.agentRuntimeVersion) throw new Error(`Bundled ${label} runtime version mismatch`);
runtimeCount++;
const delegated = metadata.optionalDependencies === undefined
&& Object.entries(declarations).every(([name, specifier]) => result.optionalDependencies?.[name] === specifier);
if (!delegated && (Object.keys(metadata.optionalDependencies ?? {}).length !== targets.length
|| targets.some(target => metadata.optionalDependencies?.[`${profile.agentRuntimePackage}-${target}`] !== declarations[`${profile.agentRuntimePackage}-${target}`]))) {
throw new Error(`Bundled ${label} platform declaration is not qualified`);
}
if (!delegated) {
const normalized = { ...metadata }; delete normalized.optionalDependencies;
normalize.set(resolve(packageDirectory, "package.json"), normalized);
}
const vendor = resolve(packageDirectory, "vendor");
if (profile.agent === "codex" && lstatExists(vendor)) { ownedDirectory(vendor); remove.add(vendor); }
}
}
if (!runtimeCount) throw new Error(`Bundled ${label} runtime omitted its platform declarations`);
for (const [name, specifier] of Object.entries(declarations)) {
if (result.optionalDependencies?.[name] !== undefined && result.optionalDependencies[name] !== specifier) throw new Error(`Bundled ${label} platform conflicts with published dependency: ${name}`);
optional[name] = specifier;
}
}
// A bundled wrapper's optional edges make npm treat missing platform slots
// as bundled too. Delegate those exact declarations to the unbundled server
@@ -178,7 +198,7 @@ export function stageBundledProviderOptionalDependencies(destinationDir, publish
for (const directory of remove) rmSync(directory, { recursive: true, force: true });
result.optionalDependencies = { ...result.optionalDependencies, ...optional };
for (const field of ["bundleDependencies", "bundledDependencies"]) {
if (Array.isArray(result[field])) result[field] = result[field].filter(name => !Object.hasOwn(declarations, name));
if (Array.isArray(result[field])) result[field] = result[field].filter(name => !Object.hasOwn(optional, name));
}
delete result.paperclipProviderArtifacts;
return result;
+32
View File
@@ -0,0 +1,32 @@
// The public-install verifier stages three packages that bundle dependencies
// (`@paperclipai/server`, `@paperclipai/adapter-utils`, `@paperclipai/db`).
// Each one is a host-side `npm install` of that package's bundled closure into
// its own pack directory, and none reads what another produces, so they run as
// concurrent children instead of one after another inside the staging loop.
// Each child is the CLI entry of `prepare-bundled-package.mjs`, the same path
// release.sh runs, with its output in a log file that a failure reports.
import { join } from 'node:path';
import { fileURLToPath } from 'node:url';
import { startPrefetch } from './prefetch-child.mjs';
export const PREPARE_BUNDLED_PACKAGE_SCRIPT = fileURLToPath(new URL('./prepare-bundled-package.mjs', import.meta.url));
// `@paperclipai/adapter-utils` -> `adapter_utils`, for log names and timing keys.
export const bundledPackageSlug = name => name.replace(/^@paperclipai\//, '').replace(/[^a-z0-9]+/gi, '_');
// Starts one child per entry (`{ name, stagedSource, target }`) at once and
// returns the entries with their `slug` and prefetch `handle`. The caller joins
// `handle.done` where the packed output is needed and registers the handles
// for cleanup, so an earlier failure stops every install still running.
export function startBundledInstalls(entries, { logDir, cwd, env = process.env, script = PREPARE_BUNDLED_PACKAGE_SCRIPT }) {
return entries.map(entry => {
const slug = bundledPackageSlug(entry.name);
const handle = startPrefetch({ name: `bundled-${slug}`, cmd: process.execPath, args: [script, entry.stagedSource, entry.target], cwd, env, logPath: join(logDir, `bundled-${slug}.log`) });
return { ...entry, slug, handle };
});
}
// Per-package wall time for the timing line. The helper measures from the
// start to the child's last output; the parent is idle while it waits for the
// installs, so the exit is observed as it happens.
export const bundledInstallTimings = installs => Object.fromEntries(installs.map(({ slug, handle }) => [`stage_bundled_${slug}_ms`, handle.durationMs]));
+83 -22
View File
@@ -8,10 +8,12 @@ import { chmodSync, cpSync, existsSync, mkdirSync, mkdtempSync, readFileSync, re
import { tmpdir } from 'node:os';
import { basename, dirname, join, resolve } from 'node:path';
import { fileURLToPath } from 'node:url';
import { materializePublishManifest, prepareBundledPackage } from './prepare-bundled-package.mjs';
import { packConcurrency, packStagedPackages, packTimings } from './pack-staged-packages.mjs';
import { materializePublishManifest } from './prepare-bundled-package.mjs';
import { GROK_PUBLIC_INSTALL_IMAGE, GROK_PUBLIC_INSTALL_LIFECYCLE, assertNoBundledCodexPayloads, grokConsumerDockerArgs, installedCodexProbeSource } from './grok-public-install-sandbox.mjs';
import { startPrefetch, stopPrefetches } from './prefetch-child.mjs';
import { retainRunnerQualificationPackages } from './retain-runner-qualification-packages.mjs';
import { bundledInstallTimings, startBundledInstalls } from './stage-bundled-packages.mjs';
const repo = resolve(dirname(fileURLToPath(import.meta.url)), '..');
assert.equal(process.platform, 'linux', 'Run this verification on disposable EC2 Linux, not a developer host');
const root = mkdtempSync(join(tmpdir(), 'paperclip-grok-public-install-'));
@@ -28,13 +30,18 @@ const releaseVersion = `0.0.0-grok-verify.${sourceRevision.slice(0, 12)}`;
// `native/grok` path that the post-install assertion inspects.
const prefetched = join(root, 'prefetch'); mkdirSync(prefetched);
const stagedGrok = join(prefetched, 'grok');
// Every background child, downloads and bundled-dependency installs alike, is
// stopped in `finally`; `downloads` is the subset the timing line reports as
// prefetches.
const prefetches = [];
const downloads = [];
const timing = {};
// A prefetch failure surfaces where its result is awaited, never as an
// unhandled rejection that would mask an earlier staging error.
const prefetch = (name, cmd, args) => {
const handle = startPrefetch({ name, cmd, args, cwd: root, env, logPath: join(prefetched, `${name}.log`) });
prefetches.push(handle);
downloads.push(handle);
return handle.done;
};
let phaseStarted = performance.now();
@@ -57,15 +64,47 @@ try {
}
visit('@paperclipai/server');
visit('paperclipai');
// release.sh also builds the standalone public packages (the sandbox-provider
// plugins outside the pnpm workspace) before it stages anything, because it
// publishes them. Nothing staged here reaches them: `needed` is closed under
// `workspace:` specs, so every package staged below is a workspace package
// that `pnpm build` already built, and that build would only cost one
// network install and compile per plugin whose output no tarball reads.
// Canary Dry Run still runs it through release.sh. The one path that does
// read a standalone build is qualification-package retention at the end:
// it packs the Daytona plugin from its `dist`, which only this build
// produces, so the build runs when retention is requested and nowhere else.
const retainQualificationPackages = Boolean(process.env.PAPERCLIP_RUNNER_QUALIFICATION_PACKAGES_DIR);
if (retainQualificationPackages) {
const standaloneStarted = performance.now();
run(process.execPath, [join(repo, 'scripts/build-standalone-public-packages.mjs')], repo);
timing.stage_standalone_build_ms = Math.round(performance.now() - standaloneStarted);
}
// The server's `ui-dist` payload is prepared here; with the reuse flag set
// above, that copies the `ui/dist` that `pnpm build` produced instead of
// building the UI again.
const uiDistStarted = performance.now();
run('bash', [join(repo, 'scripts/prepare-server-ui-dist.sh')], repo);
timing.stage_ui_dist_ms = Math.round(performance.now() - uiDistStarted);
// Match release.sh's unified versioning in temporary staging directories.
// Source manifests remain untouched, including independently versioned SDKs.
run(process.execPath, [join(repo, 'scripts/build-standalone-public-packages.mjs')], repo);
run('bash', [join(repo, 'scripts/prepare-server-ui-dist.sh')], repo);
const tarballs = [];
for (const [index, name] of [...needed].entries()) {
// Each package's published files are staged into `source-<index>` and its
// pack input into `package-<index>`, with <index> in dependency order so the
// tarball list keeps matching `needed`. The packages that bundle dependencies
// each need a host-side `npm install` of their bundled closure; those installs
// write only their own `package-<index>` and read nothing another package
// produces, so they start first and run together while the other packages
// are copied, instead of one after another inside one loop. Packing is a
// Node start-up plus CPU-bound gzip per package that reads only the package's
// own directory, so the packs also run together, each into its own
// `pack-<index>`, and the tarballs are checked in order once all are made.
const entries = [...needed].map((name, index) => {
const { dir, manifest } = packages.get(name);
const target = join(root, `package-${index}`); mkdirSync(target);
const stagedSource = join(root, `source-${index}`); mkdirSync(stagedSource);
const bundled = (manifest.bundleDependencies ?? manifest.bundledDependencies ?? []).length > 0;
return { name, index, dir, manifest, bundled, target: join(root, `package-${index}`), stagedSource: join(root, `source-${index}`), destination: join(root, `pack-${index}`) };
});
const stageSource = ({ dir, manifest, stagedSource }) => {
mkdirSync(stagedSource);
for (const file of manifest.files ?? ['dist']) {
// release.sh stages runtime skills into these public packages before pack.
const releaseSkills = file === 'skills' && ['server', 'packages/adapters/claude-local', 'packages/adapters/codex-local'].includes(dir);
@@ -76,19 +115,37 @@ try {
}
const releaseManifest = { ...manifest, version: releaseVersion };
writeFileSync(join(stagedSource, 'package.json'), JSON.stringify(releaseManifest));
if ((manifest.bundleDependencies ?? manifest.bundledDependencies ?? []).length) {
prepareBundledPackage(stagedSource, target);
} else {
cpSync(stagedSource, target, { recursive: true });
writeFileSync(join(target, 'package.json'), JSON.stringify(materializePublishManifest(releaseManifest)));
}
run('npm', ['pack', '--ignore-scripts', '--pack-destination', root], target);
const packed = readdirSync(root).filter(f => f.endsWith('.tgz') && !tarballs.includes(join(root, f)));
assert.equal(packed.length, 1);
const tarball = join(root, packed[0]);
assertNoBundledCodexPayloads(run('tar', ['-tzf', tarball]).toString().trim().split('\n'));
tarballs.push(tarball);
return releaseManifest;
};
const sourcesStarted = performance.now();
for (const entry of entries.filter(entry => entry.bundled)) stageSource(entry);
const installsStarted = performance.now();
const installs = startBundledInstalls(entries.filter(entry => entry.bundled), { logDir: prefetched, cwd: repo });
prefetches.push(...installs.map(({ handle }) => handle));
for (const entry of entries.filter(entry => !entry.bundled)) {
const releaseManifest = stageSource(entry);
mkdirSync(entry.target);
cpSync(entry.stagedSource, entry.target, { recursive: true });
writeFileSync(join(entry.target, 'package.json'), JSON.stringify(materializePublishManifest(releaseManifest)));
}
timing.stage_sources_ms = Math.round(performance.now() - sourcesStarted);
await awaited('bundled', Promise.all(installs.map(({ handle }) => handle.done)));
timing.stage_bundled_ms = Math.round(performance.now() - installsStarted);
const packStarted = performance.now();
// The bundled packages make the largest tarballs, so they are launched first
// and the short packs fill the remaining slots behind them. Each pack is
// registered for cleanup as it starts, so a failure stops the ones still
// running before the scratch root is removed. The results are put back into
// dependency order before the tarballs are checked and copied.
const concurrency = packConcurrency();
const packs = await packStagedPackages([...entries].sort((a, b) => Number(b.bundled) - Number(a.bundled)), { logDir: prefetched, env, concurrency, register: handle => prefetches.push(handle) });
packs.sort((a, b) => a.index - b.index);
const checksStarted = performance.now();
const tarballs = packs.map(({ tarball }) => tarball);
for (const tarball of tarballs) assertNoBundledCodexPayloads(run('tar', ['-tzf', tarball]).toString().trim().split('\n'));
timing.stage_pack_check_ms = Math.round(performance.now() - checksStarted);
timing.stage_pack_ms = Math.round(performance.now() - packStarted);
Object.assign(timing, packTimings(packs, concurrency));
const assets = join(root, 'assets'); mkdirSync(assets, { mode: 0o755 });
const consumer = join(root, 'consumer'); mkdirSync(consumer);
const cache = join(root, 'cache'); mkdirSync(cache);
@@ -184,9 +241,13 @@ try {
// without re-running it. `*_prefetch_ms` runs from a download's start to its
// last output, and `*_wait_ms` is how long the main flow waited for a
// prefetch that had not finished yet; zero means the overlap hid it entirely.
const downloads = Object.fromEntries(prefetches.map(({ name, durationMs }) => [`${name}_prefetch_ms`, durationMs]));
console.log(JSON.stringify({ schema: 'paperclip.public-npm-install.timing.v1', ...timing, ...downloads }));
if (process.env.PAPERCLIP_RUNNER_QUALIFICATION_PACKAGES_DIR) {
// `stage_ui_dist_ms`, `stage_sources_ms`, `stage_bundled_ms` and
// `stage_pack_ms` split the staging phase, and `stage_bundled_<package>_ms`
// is each concurrent install. `stage_standalone_build_ms` appears only when
// qualification-package retention made the standalone build necessary.
const downloaded = Object.fromEntries(downloads.map(({ name, durationMs }) => [`${name}_prefetch_ms`, durationMs]));
console.log(JSON.stringify({ schema: 'paperclip.public-npm-install.timing.v1', ...timing, ...downloaded, ...bundledInstallTimings(installs) }));
if (retainQualificationPackages) {
const retained = retainRunnerQualificationPackages({ repo, output: process.env.PAPERCLIP_RUNNER_QUALIFICATION_PACKAGES_DIR, sourceRevision, releaseVersion, env,
publicArchives: [...needed].map((name, index) => ({ name, file: tarballs[index] })),
});
+2
View File
@@ -44,6 +44,7 @@
"typecheck": "pnpm run prepare:runner-vendor && pnpm --filter @paperclipai/plugin-sdk ensure-build-deps && tsc --noEmit"
},
"dependencies": {
"@agentclientprotocol/claude-agent-acp": "0.73.0",
"@agentclientprotocol/codex-acp": "1.6.2",
"@ai-sdk/amazon-bedrock": "5.0.113",
"@ai-sdk/anthropic": "4.0.78",
@@ -113,6 +114,7 @@
"zod": "^4.4.3"
},
"bundleDependencies": [
"@agentclientprotocol/claude-agent-acp",
"@agentclientprotocol/codex-acp",
"@chat-adapter/discord",
"@chat-adapter/github",
@@ -100,6 +100,21 @@ export async function findRunnerExternalPackages(entryPoints) {
return externalPackageNames;
}
// Provider entrypoints are resolved with createRequire from profile metadata,
// so esbuild cannot see them. These two bridges ship in the server package;
// other ACP agents use separately provisioned provider packs.
export async function findRunnerRuntimePackages(entryPoints, profileData) {
const packages = await findRunnerExternalPackages(entryPoints);
for (const agent of ["claude", "codex"]) {
const profile = profileData?.profiles?.[agent];
if (profileData?.schema !== "paperclip.acpx-profiles.v1" || !profile?.agentServerPackage) {
throw new Error(`paperclip-runner vendor check: missing ${agent} provider profile.`);
}
packages.add(profile.agentServerPackage);
}
return packages;
}
function readDependencyNames(packageJsonPath) {
const packageJson = JSON.parse(readFileSync(packageJsonPath, "utf8"));
return new Map(Object.entries(packageJson.dependencies ?? {}));
@@ -125,7 +140,8 @@ function explainMissingDependencies(missing, runnerDependencyNames) {
async function main() {
const entryPoints = ENTRY_POINT_NAMES.map((name) => resolve(runnerDist, name));
const externalPackageNames = await findRunnerExternalPackages(entryPoints);
const profileData = JSON.parse(readFileSync(resolve(runnerRoot, "acpx-profiles.json"), "utf8"));
const externalPackageNames = await findRunnerRuntimePackages(entryPoints, profileData);
const serverDependencyNames = readDependencyNames(resolve(serverRoot, "package.json"));
const missing = findMissingVendorDependencies(
@@ -1,4 +1,4 @@
import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
@@ -7,6 +7,7 @@ import { afterEach, describe, expect, it } from "vitest";
import {
findMissingVendorDependencies,
findRunnerExternalPackages,
findRunnerRuntimePackages,
} from "./verify-runner-vendor-dependencies.mjs";
describe("findMissingVendorDependencies", () => {
@@ -87,6 +88,15 @@ describe("findRunnerExternalPackages", () => {
expect(externalPackageNames).toEqual(new Set(["smol-toml"]));
});
it("checks dynamically resolved bundled providers even without a JavaScript import", async () => {
const entryPoints = writeFixture();
const profiles = JSON.parse(readFileSync(new URL("../../packages/paperclip-runner/acpx-profiles.json", import.meta.url), "utf8"));
const packages = await findRunnerRuntimePackages(entryPoints, profiles);
expect(findMissingVendorDependencies(packages, new Set(["smol-toml", "@agentclientprotocol/codex-acp"])))
.toEqual(["@agentclientprotocol/claude-agent-acp"]);
expect(packages.has("pi-acp")).toBe(false);
});
it("throws a clear, actionable error when an entry point is missing", async () => {
fixtureDir = mkdtempSync(join(tmpdir(), "paperclip-runner-vendor-fixture-"));
const missingEntryPoint = join(fixtureDir, "index.js");
@@ -96,3 +106,12 @@ describe("findRunnerExternalPackages", () => {
);
});
});
// Claude is resolved through createRequire at runtime, outside esbuild's graph.
it("ships the qualified Claude ACP server with the vendored runner", () => {
const server = JSON.parse(readFileSync(new URL("../package.json", import.meta.url), "utf8"));
const profiles = JSON.parse(readFileSync(new URL("../../packages/paperclip-runner/acpx-profiles.json", import.meta.url), "utf8"));
const claude = profiles.profiles.claude;
expect(server.dependencies[claude.agentServerPackage]).toBe(claude.agentServerVersion);
expect(server.bundleDependencies).toContain(claude.agentServerPackage);
});
@@ -25,6 +25,7 @@ vi.mock("@paperclipai/adapter-claude-local/server", () => ({
fetchClaudeQuota: mocks.claude,
}));
import { fetchCompanyQuotaWindows } from "../services/quota-windows.js";
import { idleWorkSnapshot } from "../services/task-admission.js";
let revision = 1;
const db = {
@@ -192,7 +193,8 @@ describe("connected account quotas", () => {
it("bounds a stalled credential lookup and preserves its account identity", async () => {
vi.useFakeTimers();
mocks.accounts.mockResolvedValue([account("stalled")]);
mocks.credential.mockReturnValue(new Promise(() => {}));
let finish!: (value: string) => void;
mocks.credential.mockReturnValue(new Promise<string>(resolve => { finish = resolve; }));
const pending = fetchCompanyQuotaWindows(db, "company-8", "user-8");
await vi.advanceTimersByTimeAsync(20_001);
const [result] = await pending;
@@ -203,6 +205,32 @@ describe("connected account quotas", () => {
});
expect(result.accountKey).toBeTruthy();
expect(result.error).not.toContain("timed out");
finish(JSON.stringify({ accessToken: "fixture" }));
await vi.advanceTimersByTimeAsync(0);
});
it("retains the actual credential refresh after its HTTP quota deadline expires", async () => {
vi.useFakeTimers();
const before = idleWorkSnapshot();
mocks.accounts.mockResolvedValue([account("tracked-refresh")]);
mocks.codex.mockRejectedValueOnce(new Error("401"));
let finish!: (value: string) => void;
mocks.refresh.mockReturnValue(new Promise<string>(resolve => { finish = resolve; }));
const pending = fetchCompanyQuotaWindows(db, "tracked-company", "tracked-user");
try {
await vi.advanceTimersByTimeAsync(0);
expect(mocks.refresh).toHaveBeenCalledOnce();
expect(idleWorkSnapshot().active).toBe(before.active + 1);
await vi.advanceTimersByTimeAsync(20_001);
expect((await pending)[0].ok).toBe(false);
expect(idleWorkSnapshot().active).toBe(before.active + 1);
} finally {
finish(JSON.stringify({ accessToken: "fixture" }));
await vi.advanceTimersByTimeAsync(0);
await pending;
}
expect(idleWorkSnapshot().active).toBe(before.active);
expect(idleWorkSnapshot().generation).toBe(before.generation + 2);
});
it("does not start a provider request when a credential resolves after its deadline", async () => {
vi.useFakeTimers();
@@ -0,0 +1,64 @@
import express from "express";
import request from "supertest";
import { beforeEach, describe, expect, it, vi } from "vitest";
import type { Db } from "@paperclipai/db";
import { costRoutes } from "../routes/costs.js";
import { errorHandler } from "../middleware/error-handler.js";
const mocks = vi.hoisted(() => ({ report: vi.fn(), retry: vi.fn(), decide: vi.fn(), canManage: vi.fn() }));
vi.mock("../services/cost-data-quality.js", () => ({ costDataQualityReport: mocks.report }));
vi.mock("../routes/ai-connections.js", () => ({ canManageAiConnections: mocks.canManage }));
vi.mock("../services/index.js", () => ({
heartbeatService: () => ({}), costService: () => ({}), financeService: () => ({}), budgetService: () => ({}),
companyService: () => ({}), agentService: () => ({}), issueService: () => ({}), accessService: () => ({ decide: mocks.decide }),
}));
vi.mock("../services/accounting-integrity.js", () => ({ accountingIntegrityService: () => ({ retry: mocks.retry }) }));
vi.mock("../services/billing-reconciliation.js", () => ({ billingReconciliationService: () => ({}) }));
const companyId = "11111111-1111-4111-8111-111111111111";
const runId = "22222222-2222-4222-8222-222222222222";
const path = `/api/companies/${companyId}/costs`;
function app(type: "board" | "agent" = "board", role = "member") {
const instance = express(); instance.use(express.json());
instance.use((req, _res, next) => {
req.actor = type === "board" ? { type: "board", source: "session", userId: "alice", companyIds: [companyId], memberships: [{ companyId, status: "active", membershipRole: role }] } as typeof req.actor : { type: "agent", agentId: "agent", companyId } as typeof req.actor;
next();
});
instance.use("/api", costRoutes({} as Db)); instance.use(errorHandler); return instance;
}
beforeEach(() => {
vi.clearAllMocks(); mocks.decide.mockResolvedValue({ allowed: true });
mocks.report.mockResolvedValue({ events: [], runs: [] }); mocks.retry.mockResolvedValue({ accounted: false });
});
describe("cost details and accounting retry authorization", () => {
it("allows cost readers, makes viewers read-only, and does not query connection-management permissions", async () => {
expect((await request(app()).get(`${path}/data-quality?period=all`)).status).toBe(200);
expect(mocks.report).toHaveBeenLastCalledWith(expect.anything(), companyId, true, { allTime: true });
expect((await request(app("board", "viewer")).get(`${path}/data-quality`)).status).toBe(200);
expect(mocks.report).toHaveBeenLastCalledWith(expect.anything(), companyId, false, undefined);
expect(mocks.canManage).not.toHaveBeenCalled();
});
it("rejects agents, foreign companies, invalid date bounds and denied cost visibility before reading details", async () => {
expect((await request(app("agent")).get(`${path}/data-quality`)).status).toBe(403);
expect((await request(app()).get("/api/companies/other/costs/data-quality")).status).toBe(403);
for (const query of ["from=invalid", "from=2026-10-09&to=2026-10-01", "period=all&from=2026-10-01"]) {
expect((await request(app()).get(`${path}/data-quality?${query}`)).status).toBe(400);
}
mocks.decide.mockResolvedValue({ allowed: false });
expect((await request(app()).get(`${path}/data-quality`)).status).toBe(403);
expect(mocks.report).not.toHaveBeenCalled();
});
it("passes authorized retry to the existing audited accounting service without executing agents", async () => {
const response = await request(app()).post(`${path}/retry-accounting`).send({ runId });
expect(response.status).toBe(200); expect(response.body).toEqual({ accounted: false });
expect(mocks.retry).toHaveBeenCalledExactlyOnceWith(companyId, runId, "alice");
});
it("rejects viewers, agents, cross-company requests, malformed IDs and denied cost visibility before retrying", async () => {
expect((await request(app("board", "viewer")).post(`${path}/retry-accounting`).send({ runId })).status).toBe(403);
expect((await request(app("agent")).post(`${path}/retry-accounting`).send({ runId })).status).toBe(403);
expect((await request(app()).post("/api/companies/other/costs/retry-accounting").send({ runId })).status).toBe(403);
expect((await request(app()).post(`${path}/retry-accounting`).send({ runId: "bad" })).status).toBe(400);
mocks.decide.mockResolvedValue({ allowed: false });
expect((await request(app()).post(`${path}/retry-accounting`).send({ runId })).status).toBe(403);
expect(mocks.retry).not.toHaveBeenCalled();
});
});
@@ -0,0 +1,216 @@
import { randomUUID } from "node:crypto";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import { eq, sql } from "drizzle-orm";
import { appearanceForPalette } from "@paperclipai/shared";
import { agents, companies, costEvents, createDb, heartbeatRuns, issues, nativeRunFinalizations } from "@paperclipai/db";
import { costDataQualityReport, RECEIPT_WAIT_MS } from "../services/cost-data-quality.js";
import { accountingIntegrityService } from "../services/accounting-integrity.js";
import { costService } from "../services/costs.js";
import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
const now = new Date("2026-10-09T13:00:00Z");
describe("actionable cost data quality", () => {
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let db: ReturnType<typeof createDb>;
beforeAll(async () => {
database = await startEmbeddedPostgresTestDatabase("paperclip-cost-quality-");
db = createDb(database.connectionString);
}, 90_000);
afterAll(async () => { await database?.cleanup(); });
async function company() {
const companyId = randomUUID(), agentId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Costs", issuePrefix: `Q${companyId.slice(0, 6)}` });
await db.insert(agents).values({ id: agentId, companyId, name: "Codie", role: "engineer", adapterType: "codex_local" });
return { companyId, agentId };
}
async function event(owner: Awaited<ReturnType<typeof company>>, overrides: Partial<typeof costEvents.$inferInsert> = {}) {
const [row] = await db.insert(costEvents).values({ ...owner, provider: "openai", model: "gpt-6-astra", billingType: "metered_api", costStatus: "unpriced", costCents: 0, occurredAt: now, ...overrides }).returning();
return row;
}
async function run(owner: Awaited<ReturnType<typeof company>>, overrides: Partial<typeof heartbeatRuns.$inferInsert> = {}) {
const [row] = await db.insert(heartbeatRuns).values({ ...owner, status: "failed", costAccountingPending: true, createdAt: now, finishedAt: now, usageJson: { accountingReceiptReady: false }, ...overrides }).returning();
return row;
}
it("distinguishes billing, model and price gaps, excluding normal subscription/credit/fixed usage", async () => {
const owner = await company();
const unknown = await event(owner, { billingType: "unknown", costStatus: "reported", costCents: 21 });
const missingModel = await event(owner, { model: "unknown" });
const missingPrice = await event(owner);
await event(owner, { billingType: "subscription_included" });
await event(owner, { billingType: "credits", costStatus: "reported" });
await event(owner, { billingType: "fixed", costStatus: "reported" });
await event(owner, { costStatus: "estimated", costCents: 3 });
const report = await costDataQualityReport(db, owner.companyId, true, undefined, now);
expect(report.events).toHaveLength(3);
expect(report.events.every(row => row.occurredAt === now.toISOString())).toBe(true);
expect(report.events).toEqual(expect.arrayContaining([
expect.objectContaining({ id: unknown.id, reason: "unknown_billing", agentName: "Codie" }),
expect.objectContaining({ id: missingModel.id, reason: "missing_model" }),
expect.objectContaining({ id: missingPrice.id, reason: "missing_price" }),
]));
const summary = await costService(db).summary(owner.companyId, { allTime: true });
expect(summary.unknownBillingEventCount).toBe(1);
expect(summary.spendCents).toBe(24); // Unknown billing retains its reported dollars.
expect(summary.unpricedEventCount).toBe(2); // Included subscription usage is not a price gap.
const providerRows = await costService(db).byProvider(owner.companyId, { allTime: true });
expect(providerRows.reduce((count, row) => count + row.eventCount!, 0)).toBe(7);
expect(providerRows.reduce((count, row) => count + row.estimatedEventCount!, 0)).toBe(1);
expect(providerRows.reduce((count, row) => count + row.unpricedEventCount!, 0)).toBe(2);
expect(await costService(db).byBiller(owner.companyId, { allTime: true })).toEqual([
expect.objectContaining({ eventCount: 7, estimatedEventCount: 1, unpricedEventCount: 2, costCents: 24 }),
]);
});
it("uses sparse ordered indexes for old gaps behind thousands of healthy records", async () => {
const owner = await company();
await db.execute(sql`insert into cost_events (company_id,agent_id,provider,model,billing_type,cost_status,cost_cents,occurred_at)
select ${owner.companyId}::uuid,${owner.agentId}::uuid,'openai','gpt-6-astra','metered_api','reported',1,
${now.toISOString()}::timestamptz - n * interval '1 second' from generate_series(1,5000) n`);
await db.execute(sql`insert into heartbeat_runs (company_id,agent_id,status,created_at,finished_at,cost_accounting_pending)
select ${owner.companyId}::uuid,${owner.agentId}::uuid,'succeeded',${now.toISOString()}::timestamptz,
${now.toISOString()}::timestamptz - n * interval '1 second',false from generate_series(1,5000) n`);
const old = new Date(now.getTime() - 10_000_000);
await event(owner, { occurredAt: old });
await event(owner, { occurredAt: old, billingType: "unknown", costStatus: "reported" });
await run(owner, { finishedAt: old });
await db.execute(sql`analyze cost_events`);
await db.execute(sql`analyze heartbeat_runs`);
const plans: string[] = [];
// Explain the actual production statements, with the normal planner enabled.
const explainingDb = new Proxy(db, {
get(target, key) {
if (key === "transaction") return (work: (tx: typeof db) => Promise<unknown>, options: Parameters<typeof db.transaction>[1]) =>
target.transaction(tx => work(new Proxy(tx, {
get(transaction, property) {
if (property === "execute") return async (query: Parameters<typeof tx.execute>[0]) => {
plans.push(JSON.stringify(await transaction.execute(sql`explain (analyze, buffers, format json) ${query}`)));
return transaction.execute(query);
};
const value = Reflect.get(transaction, property);
return typeof value === "function" ? value.bind(transaction) : value;
},
}) as unknown as typeof db), options);
const value = Reflect.get(target, key);
return typeof value === "function" ? value.bind(target) : value;
},
});
const report = await costDataQualityReport(explainingDb, owner.companyId, false, { allTime: true }, now);
expect(report.events).toHaveLength(2);
expect(report.runs).toHaveLength(1);
expect(plans).toHaveLength(2);
expect(plans[0]).toContain("cost_events_data_quality_idx");
expect(plans[1]).toContain("heartbeat_runs_cost_data_quality_idx");
expect(plans.join(" ")).not.toContain("Rows Removed by Filter\":5000");
});
it("enforces company and period boundaries, including month-to-date and inclusive endpoints", async () => {
const owner = await company(), other = await company();
const start = new Date("2026-10-01T00:00:00Z");
const first = await event(owner, { occurredAt: start });
const last = await event(owner);
const past = await event(owner, { occurredAt: new Date(start.getTime() - 1) });
await event(owner, { occurredAt: new Date(now.getTime() + 1) });
await event(other);
await run(other);
expect((await costDataQualityReport(db, owner.companyId, false, undefined, now)).events.map(row => row.id)).toEqual([last.id, first.id]);
expect((await costDataQualityReport(db, owner.companyId, true, { from: start, to: now }, now)).events).toHaveLength(2);
expect((await costDataQualityReport(db, owner.companyId, true, { from: new Date(past.occurredAt), to: new Date(past.occurredAt) }, now)).events.map(row => row.id)).toEqual([past.id]);
const all = await costDataQualityReport(db, owner.companyId, false, { allTime: true }, now);
expect(all.events).toHaveLength(4);
expect(all.runs).toEqual([]);
expect(all.canRetry).toBe(false);
await expect(costDataQualityReport(db, randomUUID(), true)).rejects.toThrow("Company not found");
});
it("includes the agent's saved appearance for both pricing and receipt gaps", async () => {
const owner = await company();
const appearance = appearanceForPalette("deep-tide");
await db.update(agents).set({ appearance }).where(eq(agents.id, owner.agentId));
await event(owner);
await run(owner);
const report = await costDataQualityReport(db, owner.companyId, true, undefined, now);
expect(report.events[0].agentAppearance).toEqual(appearance);
expect(report.runs[0].agentAppearance).toEqual(appearance);
});
it("changes waiting copy at fifteen minutes without settling or mutating the run", async () => {
const owner = await company();
const pending = await run(owner, { accountingLastError: "SECRET provider command", resultJson: { credential: "SECRET" } });
expect((await costDataQualityReport(db, owner.companyId, true, undefined, now)).runs[0].reason).toBe("waiting_for_receipt");
const report = await costDataQualityReport(db, owner.companyId, true, undefined, new Date(now.getTime() + RECEIPT_WAIT_MS));
expect(report.runs[0].reason).toBe("receipt_unavailable");
expect(Object.keys(report.runs[0]).sort()).toEqual(["agentId", "agentName", "agentAppearance", "id", "model", "occurredAt", "provider", "reason", "runId"].sort());
expect(JSON.stringify(report)).not.toContain("SECRET");
const [saved] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, pending.id));
expect(saved.costAccountingPending).toBe(true);
expect(saved.costAccountedAt).toBeNull();
expect(saved.accountingAttemptCount).toBe(0);
});
it("offers retry only at the writer's proven receipt boundary", async () => {
const owner = await company();
const ready = await run(owner, { usageJson: { accountingReceiptReady: true } });
const noWork = await run(owner, { usageJson: { accountingReceiptReady: false, accountingProviderWorkStarted: false } });
const resultProof = await run(owner, { resultJson: { executionRecovery: { providerWorkStarted: false } } });
const captureFailed = await run(owner, { usageJson: { accountingReceiptReady: true, accountingCaptureFailed: true, accountingProviderWorkStarted: false } });
const invalidProof = await run(owner, { usageJson: { accountingReceiptReady: false, accountingProviderWorkStarted: "false" } });
const acknowledged = await run(owner, { costAccountedAt: now });
await run(owner, { status: "running" });
await run(owner, { status: "queued" });
await run(owner, { costAccountingPending: false });
const report = await costDataQualityReport(db, owner.companyId, true, undefined, now);
expect(report.runs).toHaveLength(6);
for (const row of [ready, noWork, resultProof, acknowledged]) expect(report.runs.find(item => item.id === row.id)?.reason).toBe("retryable");
expect(report.runs.find(item => item.id === captureFailed.id)?.reason).toBe("waiting_for_receipt");
expect(report.runs.find(item => item.id === invalidProof.id)?.reason).toBe("waiting_for_receipt");
const integrity = accountingIntegrityService(db);
expect(await integrity.retry(owner.companyId, captureFailed.id, "board")).toEqual({ accounted: false });
expect(await integrity.retry(owner.companyId, noWork.id, "board")).toEqual({ accounted: true });
expect(await db.select().from(costEvents).where(eq(costEvents.companyId, owner.companyId))).toEqual([]);
});
it("does not offer retry while a native coordinator can still resume provider work", async () => {
const owner = await company();
const [issue] = await db.insert(issues).values({ companyId: owner.companyId, title: "Native task" }).returning();
const pending = await run(owner, { nativeIssueId: issue.id, runtimeMode: "native", usageJson: { accountingReceiptReady: true } });
await db.insert(nativeRunFinalizations).values({ companyId: owner.companyId, runId: pending.id, issueId: issue.id, phase: "recovering" });
expect((await costDataQualityReport(db, owner.companyId, true, undefined, now)).runs[0].reason).toBe("waiting_for_receipt");
expect(await accountingIntegrityService(db).retry(owner.companyId, pending.id, "board")).toEqual({ accounted: false });
await db.update(nativeRunFinalizations).set({ phase: "terminal_failure" }).where(eq(nativeRunFinalizations.runId, pending.id));
expect((await costDataQualityReport(db, owner.companyId, true, undefined, now)).runs[0].reason).toBe("retryable");
});
it("bounds each category independently and reports truncation", async () => {
const owner = await company();
for (let i = 0; i < 26; i++) {
await event(owner, { occurredAt: new Date(now.getTime() - i * 1000) });
await run(owner, { finishedAt: new Date(now.getTime() - i * 1000) });
}
const report = await costDataQualityReport(db, owner.companyId, true, undefined, now);
expect(report.events).toHaveLength(25);
expect(report.runs).toHaveLength(25);
expect(report.hasMoreEvents).toBe(true);
expect(report.hasMoreRuns).toBe(true);
expect(new Date(report.events[24].occurredAt).getTime()).toBe(now.getTime() - 24_000);
});
it("retry cannot invent a missing receipt; a late receipt is recorded exactly once", async () => {
const owner = await company();
const pending = await run(owner);
const integrity = accountingIntegrityService(db);
expect(await integrity.retry(owner.companyId, pending.id, "board")).toEqual({ accounted: false });
expect((await db.select().from(costEvents).where(eq(costEvents.companyId, owner.companyId)))).toEqual([]);
await db.update(heartbeatRuns).set({ usageJson: { accountingReceiptReady: true, provider: "openai", biller: "openai", billingType: "metered_api", model: "gpt-6-astra", costUsd: 0.21, inputTokens: 10, outputTokens: 2 } }).where(eq(heartbeatRuns.id, pending.id));
expect((await costDataQualityReport(db, owner.companyId, true, { allTime: true }, now)).runs[0].reason).toBe("retryable");
expect(await integrity.retry(owner.companyId, pending.id, "board")).toEqual({ accounted: true });
expect(await integrity.retry(owner.companyId, pending.id, "board")).toEqual({ accounted: false });
const charges = await db.select().from(costEvents).where(eq(costEvents.companyId, owner.companyId));
expect(charges).toHaveLength(1);
expect(charges[0].costCents).toBe(21);
expect((await costDataQualityReport(db, owner.companyId, true, { allTime: true }, now)).runs).toEqual([]);
const other = await company();
await expect(integrity.retry(other.companyId, pending.id, "board")).rejects.toThrow("Accounting run not found");
});
});
@@ -374,9 +374,10 @@ describe("effective snapshot gates the persistent-session execution decision", (
it("forces the persistent session when the snapshot grants persistent process sessions", async () => {
const { target, execute } = await buildSandboxTarget({ snapshot: FULL_GRANT, supportsSync: false });
await target.runner?.execute({ command: "node", args: ["agent.js"], useSession: true });
await target.runner?.execute({ command: "node", args: ["agent.js"], useSession: true, commandTimeoutMs: 12000 });
const call = execute.mock.calls[0]![0] as { forceSession?: boolean };
expect(call.forceSession).toBe(true);
expect(call).toHaveProperty("commandTimeoutMs", 12000);
});
it("never forces the persistent session when the snapshot removes persistent process sessions", async () => {
@@ -3822,6 +3822,8 @@ describeEmbeddedPostgres("environmentRuntimeService", () => {
cwd: "/workspace",
env: {},
timeoutMs: 1000,
commandTimeoutMs: 900,
forceSession: true,
});
await environmentService(db).update(environment.id, {
@@ -3835,7 +3837,11 @@ describeEmbeddedPostgres("environmentRuntimeService", () => {
expect(released[0]?.lease.status).toBe("released");
// The execute call carries the optional log sink as the fifth argument; it
// is undefined when the caller passes no sink.
expect(workerManager.call).toHaveBeenCalledWith(pluginId, "environmentExecute", expect.anything(), 31000, undefined);
expect(workerManager.call).toHaveBeenCalledWith(pluginId, "environmentExecute", expect.objectContaining({
timeoutMs: 1000,
commandTimeoutMs: 900,
bypassSession: false,
}), 31000, undefined);
expect(workerManager.call).toHaveBeenCalledWith(pluginId, "environmentReleaseLease", expect.anything(), 31234);
});
@@ -328,11 +328,6 @@ describe("GitHub external object provider", () => {
}),
{ ok: false, liveness: "unreachable", errorCode: "github_rate_limited" },
],
[
"not-found",
new Response("", { status: 404, headers: { etag: '"missing"' } }),
{ ok: true, snapshot: expect.objectContaining({ displayKey: "GitHub Pull Request", iconKey: "github", statusKey: "not_found", statusIconKey: "archive", statusCategory: "archived", statusTone: "muted" }) },
],
])("maps %s responses to provider-safe results", async (_name, githubResponse, expected) => {
const provider = createGitHubExternalObjectProvider({} as any, {
fetch: async () => githubResponse,
@@ -348,6 +343,191 @@ describe("GitHub external object provider", () => {
expect(result).toEqual(expect.objectContaining(expected));
expect(JSON.stringify(result)).not.toContain("http");
});
it("classifies a 404 as not-found when the repository is visible to the credentials", async () => {
const fetch = vi.fn(async (url: string) => {
if (url.endsWith("/repos/acme/app")) {
return new Response(JSON.stringify({ full_name: "acme/app" }), {
status: 200,
headers: { "content-type": "application/json" },
});
}
return new Response("", { status: 404, headers: { etag: '"missing"' } });
});
const provider = createGitHubExternalObjectProvider({} as any, { fetch, tokenProvider: null });
const resolver = provider.resolvers.find((entry) => entry.objectType === "pull_request")!;
const result = await resolver.resolve({
companyId: "company-1",
object: githubObject("pull/42", "pull_request"),
});
expect(fetch).toHaveBeenCalledWith(
"https://api.github.com/repos/acme/app",
expect.objectContaining({ headers: expect.not.objectContaining({ authorization: expect.any(String) }) }),
);
expect(result).toEqual({
ok: true,
snapshot: expect.objectContaining({
displayKey: "GitHub Pull Request",
iconKey: "github",
statusKey: "not_found",
statusIconKey: "archive",
statusCategory: "archived",
statusTone: "muted",
isTerminal: true,
data: expect.objectContaining({ provider: "github", notFound: true, probeVerified: true }),
}),
});
expect(JSON.stringify(result)).not.toContain("http");
});
it("caches repository-visibility probes per company and repo", async () => {
let probeCalls = 0;
const fetch = vi.fn(async (url: string) => {
if (url.endsWith("/repos/acme/app")) {
probeCalls += 1;
return new Response(JSON.stringify({ full_name: "acme/app" }), {
status: 200,
headers: { "content-type": "application/json" },
});
}
return new Response("", { status: 404, headers: { etag: '"missing"' } });
});
const provider = createGitHubExternalObjectProvider({} as any, { fetch, tokenProvider: null });
const resolver = provider.resolvers.find((entry) => entry.objectType === "pull_request")!;
await resolver.resolve({ companyId: "company-1", object: githubObject("pull/42", "pull_request") });
await resolver.resolve({ companyId: "company-1", object: githubObject("pull/99", "pull_request") });
expect(probeCalls).toBe(1);
expect(fetch.mock.calls.filter(([url]) => String(url).endsWith("/repos/acme/app"))).toHaveLength(1);
});
it("does not reuse a cached probe verdict across credential changes", async () => {
const fetch = vi.fn(async (url: string, init?: RequestInit) => {
if (url.endsWith("/repos/acme/app")) {
const auth = (init?.headers as Record<string, string> | undefined)?.authorization;
if (auth === "Bearer token-A" || auth === "Bearer token-B") {
return new Response(JSON.stringify({ full_name: "acme/app" }), {
status: 200,
headers: { "content-type": "application/json" },
});
}
return new Response("", { status: 404 });
}
return new Response("", { status: 404, headers: { etag: '"missing"' } });
});
let token: string | null = "token-A";
const provider = createGitHubExternalObjectProvider({} as any, {
fetch,
tokenProvider: async () => token,
});
const resolver = provider.resolvers.find((entry) => entry.objectType === "pull_request")!;
const visibleVerdict = await resolver.resolve({ companyId: "company-1", object: githubObject("pull/42", "pull_request") });
// Same repo, same credentials: probe result is reused.
await resolver.resolve({ companyId: "company-1", object: githubObject("pull/99", "pull_request") });
const probesUnderTokenA = fetch.mock.calls.filter(([url]) => String(url).endsWith("/repos/acme/app"));
expect(probesUnderTokenA).toHaveLength(1);
// Credential now only has access to a different repo — cache must not
// serve the stale "visible" verdict, or this object would be frozen as a
// probe-verified not-found forever.
token = "other-token";
const hiddenVerdict = await resolver.resolve({ companyId: "company-1", object: githubObject("pull/10", "pull_request") });
expect(visibleVerdict).toEqual(
expect.objectContaining({ ok: true, snapshot: expect.objectContaining({ isTerminal: true }) }),
);
expect(hiddenVerdict).toEqual(
expect.objectContaining({ ok: false, liveness: "auth_required", errorCode: "github_repo_not_accessible" }),
);
expect(fetch.mock.calls.filter(([url]) => String(url).endsWith("/repos/acme/app"))).toHaveLength(2);
});
it("classifies a 404 as auth-required when the repository itself is hidden", async () => {
const fetch = vi.fn(async (url: string) => {
if (url.endsWith("/repos/acme/app")) {
return new Response("", { status: 404 });
}
return new Response("", { status: 404, headers: { etag: '"missing"' } });
});
const provider = createGitHubExternalObjectProvider({} as any, { fetch, tokenProvider: null });
const resolver = provider.resolvers.find((entry) => entry.objectType === "pull_request")!;
const result = await resolver.resolve({
companyId: "company-1",
object: githubObject("pull/42", "pull_request"),
});
expect(result).toEqual(
expect.objectContaining({
ok: false,
liveness: "auth_required",
errorCode: "github_repo_not_accessible",
errorMessage: expect.stringContaining("acme/app"),
}),
);
expect(JSON.stringify(result)).not.toContain("http");
});
it("classifies a 404 as unreachable when the repository probe itself fails", async () => {
const fetch = vi.fn(async (url: string) => {
if (url.endsWith("/repos/acme/app")) {
throw new Error("network down");
}
return new Response("", { status: 404, headers: { etag: '"missing"' } });
});
const provider = createGitHubExternalObjectProvider({} as any, { fetch, tokenProvider: null });
const resolver = provider.resolvers.find((entry) => entry.objectType === "pull_request")!;
const result = await resolver.resolve({
companyId: "company-1",
object: githubObject("pull/42", "pull_request"),
});
expect(result).toEqual(
expect.objectContaining({
ok: false,
liveness: "unreachable",
errorCode: "github_repo_probe_failed",
}),
);
expect(JSON.stringify(result)).not.toContain("http");
});
it("classifies a 404 as unreachable when the repository probe is rate-limited", async () => {
const fetch = vi.fn(async (url: string) => {
if (url.endsWith("/repos/acme/app")) {
return new Response("", {
status: 403,
headers: { "x-ratelimit-remaining": "0", "x-ratelimit-reset": String(Math.floor(Date.now() / 1000) + 120) },
});
}
return new Response("", { status: 404, headers: { etag: '"missing"' } });
});
const provider = createGitHubExternalObjectProvider({} as any, { fetch, tokenProvider: null });
const resolver = provider.resolvers.find((entry) => entry.objectType === "pull_request")!;
const result = await resolver.resolve({
companyId: "company-1",
object: githubObject("pull/42", "pull_request"),
});
expect(result).toEqual(
expect.objectContaining({
ok: false,
liveness: "unreachable",
errorCode: "github_repo_probe_failed",
}),
);
expect(result).not.toEqual(
expect.objectContaining({ ok: false, liveness: "auth_required", errorCode: "github_repo_not_accessible" }),
);
expect(JSON.stringify(result)).not.toContain("http");
});
});
describeEmbeddedPostgres("externalObjectService", () => {
@@ -786,6 +966,183 @@ describeEmbeddedPostgres("externalObjectService", () => {
expect(resolve).toHaveBeenCalledTimes(1);
});
it("keeps terminal objects fresh-for-display after their refresh window passes", async () => {
const { companyId, issueId } = await createIssue();
const resolve = vi.fn(async () => ({
ok: true as const,
snapshot: {
statusCategory: "closed" as const,
statusTone: "muted" as const,
statusKey: "closed",
statusLabel: "Closed",
isTerminal: true,
ttlSeconds: 1,
},
}));
const svc = externalObjectService(db, {
resolvers: [{ providerKey: "url", objectType: "link", resolve }],
github: false,
});
await svc.syncIssue(issueId);
const object = await db.select().from(externalObjects).then((rows) => rows[0]!);
await svc.refreshObject(object.id, { companyId, force: true });
// Expire the refresh window so the display path computes liveness from
// scratch. Terminal objects must still show as fresh, never as stale.
await db
.update(externalObjects)
.set({ nextRefreshAt: new Date(0) })
.where(eq(externalObjects.id, object.id));
const summary = await svc.getIssueSummary(issueId);
const displayed = summary.objects[0];
expect(displayed?.liveness).toBe("fresh");
expect(displayed?.isTerminal).toBe(true);
const summaryLater = await svc.getIssueSummaries(companyId, [issueId]).then((map) => map.get(issueId));
expect(summaryLater?.objects[0]?.liveness).toBe("fresh");
expect(summaryLater?.staleCount).toBe(0);
expect(summaryLater?.byLiveness?.stale ?? 0).toBe(0);
});
it("re-checks previously frozen GitHub not-found objects so they can self-heal", async () => {
const { companyId } = await createIssue();
const canonical = canonicalizeExternalObjectUrl("https://github.com/acme/app/pull/42");
if (!canonical) throw new Error("expected canonical url");
const inserted = await db.insert(externalObjects).values({
companyId,
providerKey: "github",
objectType: "pull_request",
externalId: "acme/app#pull/42",
sanitizedCanonicalUrl: canonical.sanitizedCanonicalUrl,
canonicalIdentityHash: canonical.canonicalIdentityHash,
displayTitle: "acme/app#42",
statusKey: "not_found",
statusLabel: "Not found",
statusCategory: "archived",
statusTone: "muted",
liveness: "fresh",
isTerminal: true,
nextRefreshAt: new Date(0),
data: { provider: "github", notFound: true },
}).returning();
const object = inserted[0]!;
const snapshot = {
statusKey: "open" as const,
statusLabel: "Open" as const,
statusCategory: "open" as const,
statusTone: "info" as const,
statusIconKey: "git-pull-request" as const,
displayTitle: "acme/app#42",
isTerminal: false,
ttlSeconds: 300,
data: { provider: "github", owner: "acme", repo: "app", number: 42 },
};
const resolver: ExternalObjectResolver = {
providerKey: "github",
objectType: "pull_request",
resolve: vi.fn(async () => ({ ok: true as const, snapshot })),
};
const svc = externalObjectService(db, { resolvers: [resolver], github: false });
expect(object.isTerminal).toBe(true);
const refreshed = await svc.refreshDueObjects(companyId);
expect(refreshed).toHaveLength(1);
expect(refreshed[0]?.refreshed).toBe(true);
const updated = await db.select().from(externalObjects).then((rows) => rows[0]!);
expect(updated.statusLabel).toBe("Open");
expect(updated.isTerminal).toBe(false);
expect(updated.liveness).toBe("fresh");
});
it("does not re-check probe-verified GitHub not-found objects (deleted references are terminal)", async () => {
const { companyId } = await createIssue();
const canonical = canonicalizeExternalObjectUrl("https://github.com/acme/app/pull/42");
if (!canonical) throw new Error("expected canonical url");
await db.insert(externalObjects).values({
companyId,
providerKey: "github",
objectType: "pull_request",
externalId: "acme/app#pull/42",
sanitizedCanonicalUrl: canonical.sanitizedCanonicalUrl,
canonicalIdentityHash: canonical.canonicalIdentityHash,
displayTitle: "acme/app#42",
statusKey: "not_found",
statusLabel: "Not found",
statusCategory: "archived",
statusTone: "muted",
liveness: "fresh",
isTerminal: true,
nextRefreshAt: new Date(0),
data: { provider: "github", notFound: true, probeVerified: true },
});
const resolver: ExternalObjectResolver = {
providerKey: "github",
objectType: "pull_request",
resolve: vi.fn(async () => ({
ok: true as const,
snapshot: {
statusKey: "open",
statusLabel: "Open",
statusCategory: "open" as const,
statusTone: "info" as const,
isTerminal: false,
ttlSeconds: 300,
},
})),
};
const svc = externalObjectService(db, { resolvers: [resolver], github: false });
const refreshed = await svc.refreshDueObjects(companyId);
expect(refreshed).toEqual([]);
expect(resolver.resolve).not.toHaveBeenCalled();
});
it("does not re-check terminal objects that are not GitHub not-found verdicts", async () => {
const { companyId } = await createIssue();
const canonical = canonicalizeExternalObjectUrl("https://github.com/acme/app/pull/42");
if (!canonical) throw new Error("expected canonical url");
await db.insert(externalObjects).values({
companyId,
providerKey: "github",
objectType: "pull_request",
externalId: "acme/app#pull/42",
sanitizedCanonicalUrl: canonical.sanitizedCanonicalUrl,
canonicalIdentityHash: canonical.canonicalIdentityHash,
displayTitle: "acme/app#42",
statusKey: "merged",
statusLabel: "Merged",
statusCategory: "succeeded",
statusTone: "success",
liveness: "fresh",
isTerminal: true,
nextRefreshAt: new Date(0),
});
const resolver: ExternalObjectResolver = {
providerKey: "github",
objectType: "pull_request",
resolve: vi.fn(async () => ({
ok: true as const,
snapshot: {
statusKey: "open",
statusLabel: "Open",
statusCategory: "open" as const,
statusTone: "info" as const,
isTerminal: false,
ttlSeconds: 300,
},
})),
};
const svc = externalObjectService(db, { resolvers: [resolver], github: false });
const refreshed = await svc.refreshDueObjects(companyId);
expect(refreshed).toEqual([]);
expect(resolver.resolve).not.toHaveBeenCalled();
});
it("keeps external object identities company-scoped for duplicate urls", async () => {
const first = await createIssue();
const second = await createIssue();
+52 -1
View File
@@ -2,7 +2,7 @@ import { promises as fs } from "node:fs";
import os from "node:os";
import path from "node:path";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { inspectIdleSpool, readIdleLocalWork, markIdleIngressTracked, markIdleStartupComplete, registerIdleSpoolDirectory } from "../services/idle-local-work.js";
import { inspectIdleSpool, inspectIdleQuotaRecovery, idleQuotaRecoveryPath, readIdleLocalWork, markIdleIngressTracked, markIdleStartupComplete, registerIdleSpoolDirectory } from "../services/idle-local-work.js";
let root: string;
beforeEach(async () => { root = await fs.mkdtemp(path.join(os.tmpdir(), "idle-spool-")); });
@@ -51,3 +51,54 @@ describe("idle cleanup inspection", () => {
});
});
describe("idle credential recovery inspection", () => {
const company = "7238bc31-f3ca-42ba-b487-4122e77b8867";
it("allows missing roots and empty company directories after successful recovery", async () => {
expect(await inspectIdleQuotaRecovery(path.join(root, "missing"))).toBe("none");
expect(await inspectIdleQuotaRecovery(root)).toBe("none");
await fs.mkdir(path.join(root, company));
expect(await inspectIdleQuotaRecovery(root)).toBe("none");
});
it.each(["receipt.json", "malformed.json", "partial.tmp", "write.probe", "unexpected"])(
"retains every pending company entry without reading its contents: %s", async name => {
await fs.mkdir(path.join(root, company));
const file = path.join(root, company, name);
await fs.writeFile(file, "unreadable recovery fixture");
const read = vi.spyOn(fs, "readFile");
expect(await inspectIdleQuotaRecovery(root)).toBe("present");
expect(read).not.toHaveBeenCalled();
await fs.rm(file);
expect(await inspectIdleQuotaRecovery(root)).toBe("none");
},
);
it("retains unexpected root entries, nested directories and directory symlinks", async () => {
await fs.mkdir(path.join(root, "unexpected"));
expect(await inspectIdleQuotaRecovery(root)).toBe("present");
await fs.rm(path.join(root, "unexpected"), { recursive: true });
await fs.symlink(path.join(root, "missing"), path.join(root, company));
expect(await inspectIdleQuotaRecovery(root)).toBe("present");
await fs.rm(path.join(root, company));
await fs.mkdir(path.join(root, company, "nested"), { recursive: true });
expect(await inspectIdleQuotaRecovery(root)).toBe("present");
});
it("fails closed on invalid roots and directory read errors", async () => {
const file = path.join(root, "file"), link = path.join(root, "link");
await fs.writeFile(file, ""); await fs.symlink(root, link);
expect(await inspectIdleQuotaRecovery(file)).toBe("unknown");
expect(await inspectIdleQuotaRecovery(link)).toBe("unknown");
vi.spyOn(fs, "opendir").mockRejectedValue(Object.assign(new Error("private path"), { code: "EACCES" }));
expect(await inspectIdleQuotaRecovery(root)).toBe("unknown");
});
it("keeps persisted credential recovery awake after the request and process work have ended", async () => {
vi.stubEnv("PAPERCLIP_HOME", root);
markIdleIngressTracked(); markIdleStartupComplete({ scheduledBackups: false });
const directory = path.join(idleQuotaRecoveryPath(), company);
await fs.mkdir(directory, { recursive: true });
const file = path.join(directory, "pending.json");
await fs.writeFile(file, "encrypted journal fixture");
expect(await readIdleLocalWork()).toBe("present");
await fs.rm(file);
expect(await readIdleLocalWork()).toBe("none");
});
});
+242 -5
View File
@@ -6,7 +6,7 @@ import {
agentInstructionWorkingCopies, agents, agentApiKeys, agentWakeupRequests, companies, companySecretProposals, createDb,
adapterAuthSessions, environments, environmentLeases, executionWorkspaces,
heartbeatRuns, issues, issueWatchdogs, issueThreadInteractions, issueQuestionResponseDeliveries,
projects, routines, type Db,
projects, projectWorkspaces, routines, toolApplications, toolConnections, workspaceOperations, executionWorkspaceRuntimeLeases, type Db,
} from "@paperclipai/db";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { readIdleSleepSafety, type IdleSleepDrainStatus } from "../services/idle-sleep-safety.js";
@@ -185,6 +185,37 @@ if (!support.supported) console.warn(`Skipping idle sleep Postgres tests: ${supp
return { companyId, agentId, runId };
}
it.each(["succeeded", "failed", "cancelled", "timed_out", "interrupted"])(
"permits a finished %s run with a consumed retry date without changing history", async status => {
const { runId } = await seed();
const scheduledRetryAt = new Date(now - 1000);
await db.update(heartbeatRuns).set({ status, finishedAt: new Date(now), scheduledRetryAt }).where(eq(heartbeatRuns.id, runId));
expect(await read()).toEqual(none);
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)))
.toMatchObject([{ status, scheduledRetryAt }]);
},
);
it.each([
{ status: "scheduled_retry", finishedAt: new Date(now) },
{ status: "queued", finishedAt: new Date(now) },
{ status: "running", finishedAt: new Date(now) },
{ status: "failed", finishedAt: null },
{ status: "failed", finishedAt: new Date(now), costAccountingPending: true },
{ status: "failed", finishedAt: new Date(now - 2000) },
])("retains retry work without terminal consumption proof: %j", async change => {
const { runId } = await seed();
await db.update(heartbeatRuns).set({ scheduledRetryAt: new Date(now - 1000), ...change }).where(eq(heartbeatRuns.id, runId));
expect(await read()).toEqual(present);
});
it("keeps a live retry child awake after its parent's schedule is consumed", async () => {
const { companyId, agentId, runId } = await seed();
await db.update(heartbeatRuns).set({ finishedAt: new Date(now), scheduledRetryAt: new Date(now - 1000) }).where(eq(heartbeatRuns.id, runId));
await db.insert(heartbeatRuns).values({ companyId, agentId, invocationSource: "on_demand", status: "scheduled_retry", retryOfRunId: runId, scheduledRetryAt: new Date(now + 1000) });
expect(await read()).toEqual(present);
});
async function seedQuestionDelivery(change: Partial<typeof issueQuestionResponseDeliveries.$inferInsert> = {}) {
const seeded = await seed();
await db.update(heartbeatRuns).set({ finishedAt: new Date(now) }).where(eq(heartbeatRuns.id, seeded.runId));
@@ -443,6 +474,132 @@ if (!support.supported) console.warn(`Skipping idle sleep Postgres tests: ${supp
expect(await read()).toEqual(present);
});
async function seedAiConnection() {
const seeded = await seed();
const [application] = await db.insert(toolApplications).values({
companyId: seeded.companyId, name: "AI fixture", type: "rest_api",
}).returning();
const [connection] = await db.insert(toolConnections).values({
companyId: seeded.companyId, applicationId: application!.id,
name: "Saved AI login", uid: randomUUID(), connectionPurpose: "ai",
transport: "runtime_auth", status: "active", enabled: true,
}).returning();
return { ...seeded, connection: connection! };
}
it("allows saved AI credentials while preserving their connection", async () => {
const { connection } = await seedAiConnection();
expect(await read()).toEqual(none);
expect(await db.select().from(toolConnections).where(eq(toolConnections.id, connection.id)))
.toMatchObject([{ status: "active", enabled: true }]);
});
it.each(["draft", "active_login", "pending_run", "accepted_refresh"])(
"keeps AI connections awake for %s work", async kind => {
const { companyId, runId, connection } = await seedAiConnection();
if (kind === "draft") await db.update(toolConnections).set({ status: "draft" }).where(eq(toolConnections.id, connection.id));
if (kind === "pending_run") await db.update(heartbeatRuns).set({ status: "queued" }).where(eq(heartbeatRuns.id, runId));
if (kind === "active_login") {
const [environment] = await db.insert(environments).values({ name: "Active AI login" }).returning();
await db.insert(adapterAuthSessions).values({
companyId, environmentId: environment!.id, connectionId: connection.id,
adapterType: "codex_local", startedByUserId: "fixture-user",
publicSessionId: randomUUID(), status: "waiting_for_user",
});
}
const finish = kind === "accepted_refresh" ? beginIdleTrackedWork() : undefined;
try { expect(await read()).toEqual(kind === "accepted_refresh" ? unknown : present); }
finally { finish?.(); }
},
);
it("retains remote MCP connections alongside inert AI credentials", async () => {
const { companyId, connection } = await seedAiConnection();
await db.insert(toolConnections).values({
companyId, applicationId: connection.applicationId, name: "Remote tools", uid: randomUUID(),
connectionPurpose: "tool", transport: "mcp_remote", status: "active", enabled: true,
});
expect(await read()).toEqual(present);
});
async function seedFinishedFailedLease() {
const seeded = await seed();
await db.update(heartbeatRuns).set({ status: "failed", finishedAt: new Date(now) })
.where(eq(heartbeatRuns.id, seeded.runId));
const [environment] = await db.insert(environments).values({ name: "Failed turn environment", driver: "sandbox" }).returning();
const leaseId = randomUUID();
const receipt = {
schema: "paperclip.remote-termination.v1", companyId: seeded.companyId,
runId: seeded.runId, leaseId, provider: "plugin:demo:sandbox",
providerLeaseId: "fixture-resource", state: "destroyed", confirmedAt: new Date(now).toISOString(),
};
const [lease] = await db.insert(environmentLeases).values({
id: leaseId, companyId: seeded.companyId, environmentId: environment!.id,
heartbeatRunId: seeded.runId, status: "failed", leasePolicy: "ephemeral",
provider: receipt.provider, providerLeaseId: receipt.providerLeaseId,
acquiredAt: new Date(now - 1000), releasedAt: new Date(now), cleanupStatus: "success",
metadata: { remoteExecutionTermination: receipt },
}).returning();
return { ...seeded, environment: environment!, lease: lease!, receipt };
}
it.each(["destroyed", "stopped"])("allows failed lease history after exact provider %s proof", async state => {
const { lease, receipt } = await seedFinishedFailedLease();
await db.update(environmentLeases).set({ metadata: { remoteExecutionTermination: { ...receipt, state } } })
.where(eq(environmentLeases.id, lease.id));
expect(await read()).toEqual(none);
expect(await db.select().from(environmentLeases).where(eq(environmentLeases.id, lease.id)))
.toMatchObject([{ status: "failed", cleanupStatus: "success" }]);
});
it("permits consumed retry history only with exact failed-lease termination proof", async () => {
const { lease, runId } = await seedFinishedFailedLease();
await db.update(heartbeatRuns).set({ scheduledRetryAt: new Date(now - 1000) }).where(eq(heartbeatRuns.id, runId));
expect(await read()).toEqual(none);
await db.update(environmentLeases).set({ metadata: {} }).where(eq(environmentLeases.id, lease.id));
expect(await read()).toEqual(present);
});
it.each([
"cleanup_failed", "cleanup_unknown", "unreleased", "retained_policy", "reusable_policy",
"active", "pending_cleanup", "missing_receipt", "wrong_company", "wrong_run", "wrong_lease",
"wrong_provider", "wrong_resource", "not_stopped", "missing_run", "unfinished_run",
"retry", "accounting", "cleanup_in_flight", "pending_export", "pending_retain",
])("retains failed leases with %s evidence", async kind => {
const { lease, receipt, runId } = await seedFinishedFailedLease();
const change: Partial<typeof environmentLeases.$inferInsert> = {};
if (kind === "cleanup_failed") change.cleanupStatus = "failed";
if (kind === "cleanup_unknown") change.cleanupStatus = null;
if (kind === "unreleased") change.releasedAt = null;
if (kind === "retained_policy") change.leasePolicy = "retain_on_failure";
if (kind === "reusable_policy") change.leasePolicy = "reuse_by_environment";
if (kind === "active" || kind === "pending_cleanup") change.status = kind;
if (kind === "missing_receipt") change.metadata = {};
const fields: Record<string, string> = { wrong_company: "companyId", wrong_run: "runId", wrong_lease: "leaseId", wrong_provider: "provider", wrong_resource: "providerLeaseId", not_stopped: "state" };
if (fields[kind]) change.metadata = { remoteExecutionTermination: { ...receipt, [fields[kind]]: "wrong" } };
if (kind === "missing_run") change.heartbeatRunId = null;
if (kind === "unfinished_run") await db.update(heartbeatRuns).set({ finishedAt: null }).where(eq(heartbeatRuns.id, runId));
if (kind === "retry") await db.update(heartbeatRuns).set({ scheduledRetryAt: new Date(now + 1000) }).where(eq(heartbeatRuns.id, runId));
if (kind === "accounting") await db.update(heartbeatRuns).set({ costAccountingPending: true }).where(eq(heartbeatRuns.id, runId));
if (kind === "cleanup_in_flight") change.metadata = { remoteExecutionTermination: receipt, pendingCleanupInFlight: true };
if (kind === "pending_export") change.metadata = { remoteExecutionTermination: receipt, nativeWorkspaceExportResume: {} };
if (kind === "pending_retain") change.metadata = { remoteExecutionTermination: receipt, sandboxStopAndRetain: {} };
if (Object.keys(change).length) await db.update(environmentLeases).set(change).where(eq(environmentLeases.id, lease.id));
expect(await read()).toEqual(present);
});
it("uses the same failed-lease cleanup proof for finished login history", async () => {
const { companyId, environment, lease } = await seedFinishedFailedLease();
await db.insert(adapterAuthSessions).values({
companyId, environmentId: environment.id, adapterType: "codex_local", startedByUserId: "fixture-user",
publicSessionId: randomUUID(), providerLeaseId: lease.providerLeaseId,
status: "authenticated", finishedAt: new Date(now),
});
expect(await read()).toEqual(none);
await db.update(environmentLeases).set({ metadata: {} }).where(eq(environmentLeases.id, lease.id));
expect(await read()).toEqual(present);
});
it("reports a login sandbox awaiting cleanup", async () => {
const { companyId } = await seed();
const [environment] = await db.insert(environments).values({ name: "Login fixture" }).returning();
@@ -552,12 +709,92 @@ if (!support.supported) console.warn(`Skipping idle sleep Postgres tests: ${supp
expect(await read()).toEqual(present);
});
it("reports retained execution workspaces that may need delayed cleanup", async () => {
async function seedRetainedWorkspace(change: Partial<typeof executionWorkspaces.$inferInsert> = {}) {
const { companyId } = await seed();
const [project] = await db.insert(projects).values({ companyId, name: "Fixture project" }).returning();
await db.insert(executionWorkspaces).values({
companyId, projectId: project!.id, name: "Fixture workspace", mode: "isolated",
strategyType: "git_worktree", status: "closed", cleanupEligibleAt: new Date(now + 86_400_000),
const [workspace] = await db.insert(executionWorkspaces).values({
companyId, projectId: project!.id, name: "Fixture workspace", mode: "isolated_workspace",
strategyType: "git_worktree", status: "active", providerType: "git_worktree",
cwd: "/tmp/fixture-retained-worktree", providerRef: "/tmp/fixture-retained-worktree",
...change,
}).returning();
return workspace!;
}
it.each(["local_fs", "git_worktree"].flatMap(providerType =>
["active", "idle", "in_review", "archived"].map(status => ({ providerType, status }))))(
"permits a passive local directory without removing retained data: %j", async ({ providerType, status }) => {
const metadata = { config: {
environmentId: randomUUID(), provisionCommand: null, runtimeProvisionCommand: null,
teardownCommand: null, cleanupCommand: null, workspaceRuntime: null,
desiredState: null, serviceStates: null,
}, retainedFixture: true };
const strategyType = providerType === "local_fs" ? "project_primary" : "git_worktree";
const workspace = await seedRetainedWorkspace({ providerType, strategyType, status, metadata, cleanupEligibleAt: new Date(Date.UTC(2099, 0, 1)) });
expect(await read()).toEqual(none);
expect(await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspace.id)))
.toMatchObject([{ status, metadata, cwd: workspace.cwd, cleanupEligibleAt: workspace.cleanupEligibleAt }]);
},
);
it.each([null, {}, { config: null }, { config: {} }, { config: { desiredState: "stopped", serviceStates: {} } }])(
"permits passive local configuration: %j", async metadata => {
await seedRetainedWorkspace({ metadata });
expect(await read()).toEqual(none);
},
);
it.each([
{ providerType: "cloud_sandbox" }, { providerType: "future_provider" },
{ strategyType: "adapter_managed" }, { strategyType: "future_strategy" },
{ mode: "future_mode" },
{ providerRef: "another-resource" }, { status: "cleanup_failed" }, { status: "future_state" },
{ cleanupEligibleAt: new Date(now) }, { metadata: { reopenPendingConsumption: true } },
{ metadata: { reopenPendingConsumption: "unknown" } },
{ metadata: { config: [] } }, { metadata: { config: "malformed" } },
{ metadata: { config: { workspaceRuntime: {} } } },
{ metadata: { config: { workspaceRuntime: { services: [] } } } },
{ metadata: { config: { desiredState: "running" } } },
{ metadata: { config: { desiredState: "manual" } } },
{ metadata: { config: { serviceStates: { web: "stopped" } } } },
{ metadata: { config: { provisionCommand: "echo fixture" } } },
{ metadata: { config: { runtimeProvisionCommand: "echo fixture" } } },
{ metadata: { config: { teardownCommand: "echo fixture" } } },
{ metadata: { config: { cleanupCommand: "echo fixture" } } },
{ metadata: { config: { futureWorker: true } } },
])("retains a directory with work or unfamiliar state: %j", async change => {
await seedRetainedWorkspace(change);
expect(await read()).toEqual(present);
});
it("fails closed for malformed directory metadata", async () => {
const workspace = await seedRetainedWorkspace();
await db.execute(sql`UPDATE execution_workspaces SET metadata = '[]'::jsonb WHERE id = ${workspace.id}`);
expect(await read()).toEqual(present);
});
it("retains a shared directory that can inherit project runtime configuration", async () => {
const workspace = await seedRetainedWorkspace();
const [projectWorkspace] = await db.insert(projectWorkspaces).values({
companyId: workspace.companyId, projectId: workspace.projectId, name: "Shared fixture",
metadata: { runtimeConfig: { workspaceRuntime: { services: [] } } },
}).returning();
await db.update(executionWorkspaces).set({ mode: "shared_workspace", projectWorkspaceId: projectWorkspace!.id })
.where(eq(executionWorkspaces.id, workspace.id));
expect(await read()).toEqual(present);
});
it("retains a local directory with an unfinished operation", async () => {
const workspace = await seedRetainedWorkspace();
await db.insert(workspaceOperations).values({ companyId: workspace.companyId, executionWorkspaceId: workspace.id, phase: "cleanup", status: "running" });
expect(await read()).toEqual(present);
});
it("retains a local directory with a runtime owner", async () => {
const workspace = await seedRetainedWorkspace();
await db.insert(executionWorkspaceRuntimeLeases).values({
companyId: workspace.companyId, executionWorkspaceId: workspace.id,
ownerKey: "fixture-owner", lastAction: "use", expiresAt: new Date(Date.UTC(2099, 0, 1)),
});
expect(await read()).toEqual(present);
});
@@ -0,0 +1,388 @@
import { randomUUID } from "node:crypto";
import { eq, sql } from "drizzle-orm";
import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
import { activityLog, agentWakeupRequests, agents, approvals, authUsers, companies, companyMemberships, connectionGrants, connectionGrantDelegations,
connectionIntentDeliveries, createDb, environmentLeases, heartbeatRunEvents, heartbeatRuns, issueApprovals, issueRecoveryActions, issueThreadInteractions, issues, projects, toolApplications,
toolCatalogEntries, toolConnectionInstalls, toolConnections, toolProfileBindings, toolProfiles, toolPolicies,
} from "@paperclipai/db";
import { createPostgresRunDispatchAdapter } from "../modules/run-dispatch/adapters/postgres.js";
import { connectionIntentDeliveryService } from "../services/connection-intent-delivery.js";
import { findSatisfiedToolConnection, satisfiedConnectionIntentService } from "../services/satisfied-connection-intents.js";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { materializeNativeInteractionResponses } from "../services/native-runtime/native-interaction-bridge.js";
import { DELIVERY_QUEUES, subscribeDeliveryWork } from "../services/delivery-work-notifications.js";
import { issueThreadInteractionService } from "../services/issue-thread-interactions.js";
import { legacyExecutionNeedsReconciliationWithEvidence } from "../services/legacy-execution-recovery.js";
import { remoteTerminationReceipt } from "../services/remote-execution-termination.js";
const support = await getEmbeddedPostgresTestSupport();
const describePostgres = support.supported ? describe : describe.skip;
describePostgres("already available connection requests", () => {
let db!: ReturnType<typeof createDb>;
let temp!: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
beforeAll(async () => {
temp = await startEmbeddedPostgresTestDatabase("paperclip-satisfied-connections-");
db = createDb(temp.connectionString);
await db.insert(authUsers).values({ id: "connection-owner", name: "Owner", email: "owner@example.test", createdAt: new Date(), updatedAt: new Date() });
}, 20_000);
beforeEach(async () => { await db.execute(sql`truncate table companies cascade`); });
afterAll(async () => { await db.$client.end({ timeout: 1 }); await temp?.cleanup(); });
async function seed() {
const companyId = randomUUID();
const agentId = randomUUID();
const issueId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Connections", issuePrefix: "CON", defaultResponsibleUserId: "connection-owner" });
await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: "connection-owner", status: "active", membershipRole: "member" });
await db.insert(agents).values({ id: agentId, companyId, name: "Assignee", role: "engineer", adapterType: "claude_local", status: "idle", runtimeConfig: { heartbeat: { wakeOnDemand: true } } });
const [task] = await db.insert(issues).values({ id: issueId, companyId, title: "Use Sentry", status: "in_review", assigneeAgentId: agentId, responsibleUserId: "connection-owner" }).returning();
const [intent] = await db.insert(issueThreadInteractions).values({ companyId, issueId, kind: "connection_intent", status: "pending", addresseeUserId: "connection-owner",
requestedResolverPolicy: "human_only", effectiveResolverPolicy: "human_only", payload: { version: 1, serviceSlug: "sentry", serviceName: "Sentry", requestingAgentId: agentId, requestingAgentName: "Assignee", phase: "requested" } }).returning();
const [app] = await db.insert(toolApplications).values({ companyId, applicationKey: "sentry", name: "Sentry", type: "mcp_http", status: "active", metadata: { sourceTemplateKey: "sentry" } }).returning();
const [connection] = await db.insert(toolConnections).values({ companyId, applicationId: app!.id, name: "Existing Sentry", uid: "sentry", transport: "mcp_remote", authKind: "api_key", credentialPolicy: "per_user", status: "active", enabled: true, healthStatus: "ok", config: { sourceTemplateKey: "sentry" } }).returning();
const [grant] = await db.insert(connectionGrants).values({ companyId, connectionId: connection!.id, kind: "user", subjectUserId: "connection-owner", status: "active" }).returning();
await db.insert(toolConnectionInstalls).values({ companyId, connectionId: connection!.id, targetType: "agent", targetId: agentId });
const [tool] = await db.insert(toolCatalogEntries).values({ companyId, connectionId: connection!.id, toolName: "sentry-read", name: "sentry-read", versionHash: "v1", status: "active", entryKind: "tool" }).returning();
const [profile] = await db.insert(toolProfiles).values({ companyId, profileKey: "reads", name: "Reads", defaultAction: "allow", status: "active" }).returning();
await db.insert(toolProfileBindings).values({ companyId, profileId: profile!.id, targetType: "agent", targetId: agentId });
return { companyId, agentId, issueId, task: task!, intent: intent!, connection: connection!, grant: grant!, tool: tool!, profile: profile! };
}
async function retry(f: Awaited<ReturnType<typeof seed>>) {
const [run] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, issueId: f.issueId,
status: "scheduled_retry", scheduledRetryReason: "transient_failure", scheduledRetryAt: new Date(), runtimeMode: "legacy",
responsibleUserId: "connection-owner", contextSnapshot: { issueId: f.issueId }, resultJson: { conversationContinuation: "continue_conversation_v1" } }).returning();
return run!;
}
it("ignores an already available connection before the worker retires its card", async () => {
const f = await seed(); const run = await retry(f);
expect(await createPostgresRunDispatchAdapter(db).evaluateScheduledRetryGate({ companyId: f.companyId, runId: run.id, now: new Date() })).toMatchObject({ allowed: true });
expect((await db.select().from(issueThreadInteractions))[0].status).toBe("pending");
});
it("keeps a retry under another user's identity waiting for its response", async () => {
const f = await seed(); const run = await retry(f);
await db.update(heartbeatRuns).set({ responsibleUserId: "someone-else" }).where(eq(heartbeatRuns.id, run.id));
expect(await createPostgresRunDispatchAdapter(db).evaluateScheduledRetryGate({ companyId: f.companyId, runId: run.id, now: new Date() })).toMatchObject({ allowed: false, errorCode: "issue_waiting_for_response" });
});
it("notifies an idle worker when a new connection request commits", async () => {
const f = await seed(); const source = await retry(f);
await db.delete(issueThreadInteractions);
const wakeup = vi.fn().mockResolvedValue(null);
const worker = connectionIntentDeliveryService(db, { wakeup });
expect(await worker.hasPending()).toBe(false);
const notified = vi.fn();
const unsubscribe = subscribeDeliveryWork(db, DELIVERY_QUEUES.connection, notified);
try {
await issueThreadInteractionService(db).createConnectionIntent({ id: f.issueId, companyId: f.companyId }, {
payload: f.intent.payload, sourceRunId: source.id, addresseeUserId: "connection-owner", idempotencyKey: "new-card",
});
expect(notified).toHaveBeenCalledOnce();
expect(await worker.hasPending()).toBe(true);
await worker.sweepPending();
expect(wakeup).toHaveBeenCalledOnce();
} finally { unsubscribe(); }
});
it.each(["wrong_identity", "revoked", "disabled", "removed_member", "denied_policy"] as const)("rechecks %s after the card retires", async (reason) => {
const f = await seed(); const run = await retry(f);
await satisfiedConnectionIntentService(db).sweepPending();
if (reason === "wrong_identity") await db.update(heartbeatRuns).set({ responsibleUserId: "someone-else" });
if (reason === "revoked") await db.update(connectionGrants).set({ status: "revoked" });
if (reason === "disabled") await db.update(toolConnections).set({ enabled: false });
if (reason === "removed_member") await db.delete(companyMemberships);
if (reason === "denied_policy") await db.insert(toolPolicies).values({ companyId: f.companyId, name: "Block this task", policyType: "block", selectors: { issueId: f.issueId } });
expect(await createPostgresRunDispatchAdapter(db).evaluateScheduledRetryGate({ companyId: f.companyId, runId: run.id, now: new Date() })).toMatchObject({ allowed: false, errorCode: "issue_waiting_for_response" });
});
it.each(["company", "issue", "project"] as const)("uses %s policy when deciding whether tools are ready", async (scope) => {
const f = await seed();
const selectors = scope === "issue" ? { issueId: f.issueId } : scope === "project" ? { projectId: randomUUID() } : {};
if (scope === "project") {
const projectId = (selectors as { projectId: string }).projectId;
await db.insert(projects).values({ id: projectId, companyId: f.companyId, name: "Tools" });
await db.update(issues).set({ projectId });
}
await db.insert(toolPolicies).values({ companyId: f.companyId, name: "Block tools", policyType: "block", selectors });
expect(await satisfiedConnectionIntentService(db).sweepPending()).toMatchObject({ satisfied: 0 });
expect((await db.select().from(issueThreadInteractions))[0].status).toBe("pending");
});
it("retains invocation approval when the authorized connection's tools require it", async () => {
const f = await seed();
await db.insert(toolPolicies).values({ companyId: f.companyId, name: "Ask before use", policyType: "require_approval", selectors: { issueId: f.issueId } });
expect(await satisfiedConnectionIntentService(db).sweepPending()).toMatchObject({ satisfied: 1 });
expect((await db.select().from(toolPolicies))[0].policyType).toBe("require_approval");
expect(await db.select().from(approvals)).toHaveLength(0);
expect((await db.select().from(issueThreadInteractions))[0].resolvedByUserId).toBeNull();
});
async function addWait(f: Awaited<ReturnType<typeof seed>>, kind: "question" | "approval") {
if (kind === "question") {
const [question] = await db.insert(issueThreadInteractions).values({ companyId: f.companyId, issueId: f.issueId,
kind: "ask_user_questions", status: "pending", payload: { version: 1, questions: [] } }).returning();
return async () => { await db.update(issueThreadInteractions).set({ status: "answered" }).where(eq(issueThreadInteractions.id, question!.id)); };
}
const [approval] = await db.insert(approvals).values({ companyId: f.companyId, type: "tool_action", status: "pending", payload: {} }).returning();
await db.insert(issueApprovals).values({ companyId: f.companyId, issueId: f.issueId, approvalId: approval!.id });
return async () => { await db.update(approvals).set({ status: "approved" }).where(eq(approvals.id, approval!.id)); };
}
it.each(["question", "approval"] as const)("defers the saved system continuation for another %s", async (kind) => {
const f = await seed(); const clearWait = await addWait(f, kind);
const wakeup = vi.fn().mockResolvedValue(null);
const worker = connectionIntentDeliveryService(db, { wakeup });
await worker.sweepPending();
expect((await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, f.intent.id)))[0].status).toBe("expired");
expect(wakeup).not.toHaveBeenCalled();
expect((await db.select().from(connectionIntentDeliveries))[0].deliveredAt).toBeNull();
await clearWait();
await db.update(connectionIntentDeliveries).set({ nextAttemptAt: new Date() });
await worker.sweepPending();
expect(wakeup).toHaveBeenCalledOnce();
});
it.each(["question", "approval", "revoked"] as const)("rechecks a queued system continuation after %s changes", async (kind) => {
const f = await seed(); const run = await retry(f);
await satisfiedConnectionIntentService(db).sweepPending();
await db.update(heartbeatRuns).set({ status: "queued", contextSnapshot: { issueId: f.issueId,
connectionIntentResolution: "existing_connection", interactionKind: "connection_intent", interactionStatus: "expired",
mutation: "interaction", wakeReason: "issue_commented", source: "connection_intent.resolved" } }).where(eq(heartbeatRuns.id, run.id));
if (kind === "revoked") await db.update(connectionGrants).set({ status: "revoked" });
else await addWait(f, kind);
expect(await createPostgresRunDispatchAdapter(db).cancelStaleQueuedRun({ companyId: f.companyId, runId: run.id,
expectedStatus: "queued", now: new Date() })).toMatchObject({ outcome: "cancelled", errorCode: "issue_waiting_for_response" });
});
async function replaceAccount(f: Awaited<ReturnType<typeof seed>>) {
await db.update(toolConnections).set({ enabled: false }).where(eq(toolConnections.id, f.connection.id));
await db.update(connectionGrants).set({ status: "revoked" }).where(eq(connectionGrants.id, f.grant.id));
const [connection] = await db.insert(toolConnections).values({ ...f.connection, id: randomUUID(), uid: "sentry-replacement", name: "Replacement Sentry" }).returning();
await db.insert(connectionGrants).values({ ...f.grant, id: randomUUID(), connectionId: connection!.id, status: "active" });
await db.insert(toolConnectionInstalls).values({ companyId: f.companyId, connectionId: connection!.id, targetType: "agent", targetId: f.agentId });
await db.insert(toolCatalogEntries).values({ ...f.tool, id: randomUUID(), connectionId: connection!.id });
return connection!;
}
it("allows a later approved replacement instead of permanently binding retries to an old account", async () => {
const f = await seed(); const run = await retry(f);
await satisfiedConnectionIntentService(db).sweepPending();
const replacement = await replaceAccount(f);
await db.insert(issueThreadInteractions).values({ companyId: f.companyId, issueId: f.issueId, kind: "connection_intent",
status: "accepted", payload: f.intent.payload, addresseeUserId: "connection-owner", resolvedByUserId: "connection-owner",
result: { version: 1, outcome: "accepted", connectionId: replacement.id }, resolvedAt: new Date() });
expect(await createPostgresRunDispatchAdapter(db).evaluateScheduledRetryGate({ companyId: f.companyId, runId: run.id, now: new Date() })).toMatchObject({ allowed: true });
});
it("refreshes only the system observation for an already authorized replacement", async () => {
const f = await seed(); const run = await retry(f);
await satisfiedConnectionIntentService(db).sweepPending();
const replacement = await replaceAccount(f);
const input = { db, companyId: f.companyId, issueId: f.issueId, runId: run.id, agentId: f.agentId, interactionIds: [f.intent.id] };
await expect(materializeNativeInteractionResponses(input)).rejects.toMatchObject({ code: "native_interaction_unresolved" });
await connectionIntentDeliveryService(db, { wakeup: vi.fn().mockResolvedValue(null) }).sweepPending();
expect((await db.select().from(issueThreadInteractions))[0]).toMatchObject({ status: "expired", resolvedByUserId: null,
result: { outcome: "expired", connectionId: replacement.id } });
expect(await materializeNativeInteractionResponses(input)).toEqual([expect.objectContaining({
response: expect.objectContaining({ result: expect.objectContaining({ connectionId: replacement.id }) }) })]);
expect(await db.select().from(activityLog)).toEqual(expect.arrayContaining([expect.objectContaining({ actorType: "system",
details: expect.objectContaining({ previousConnectionId: f.connection.id, connectionId: replacement.id }) })]));
});
it.each(["queued", "running"] as const)("rearms an unstarted %s wait without manufacturing a workspace repair hold", async (status) => {
const f = await seed(); const run = await retry(f);
await satisfiedConnectionIntentService(db).sweepPending();
const key = `connection-intent:${f.intent.id}:expired`;
const [wake] = await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agentId,
source: "automation", status: "claimed", idempotencyKey: key, runId: run.id }).returning();
await db.update(connectionIntentDeliveries).set({ deliveredAt: new Date() });
await db.update(heartbeatRuns).set({ status, startedAt: status === "running" ? new Date() : null,
runtimeModeResolvedAt: status === "running" ? new Date() : null,
controllerBootId: status === "running" ? randomUUID() : null,
executionStage: status === "running" ? "preparing" : null, wakeupRequestId: wake!.id, resultJson: null,
contextSnapshot: { issueId: f.issueId, interactionId: f.intent.id, connectionIntentResolution: "existing_connection",
interactionKind: "connection_intent", interactionStatus: "expired", mutation: "interaction", wakeReason: "issue_commented", source: "connection_intent.resolved" } }).where(eq(heartbeatRuns.id, run.id));
await db.update(connectionGrants).set({ status: "revoked" });
const dispatch = createPostgresRunDispatchAdapter(db);
if (status === "queued") {
expect(await dispatch.cancelStaleQueuedRun({ companyId: f.companyId, runId: run.id,
expectedStatus: "queued", now: new Date() })).toMatchObject({ outcome: "cancelled", errorCode: "issue_waiting_for_response" });
} else {
const invoke = vi.fn(async () => null);
expect(await dispatch.dispatchResolvedInteractionIfCurrent({ companyId: f.companyId, runId: run.id,
expectedStatus: "running", now: new Date(), dispatch: invoke })).toMatchObject({ dispatched: false,
cancellation: { outcome: "cancelled", errorCode: "issue_waiting_for_response" } });
expect(invoke).not.toHaveBeenCalled();
}
const cancelled = (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0];
expect(await legacyExecutionNeedsReconciliationWithEvidence(db, cancelled)).toBe(false);
expect((await db.select().from(connectionIntentDeliveries))[0].deliveredAt).toBeNull();
expect((await db.select().from(agentWakeupRequests))[0].status).toBe("skipped");
await db.update(connectionGrants).set({ status: "active" });
const wakeup = vi.fn(async () => {
await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agentId, source: "automation", status: "queued", idempotencyKey: key });
return null;
});
await connectionIntentDeliveryService(db, { wakeup }).sweepPending();
expect(wakeup).toHaveBeenCalledOnce();
expect((await db.select().from(connectionIntentDeliveries))[0].deliveredAt).not.toBeNull();
await db.insert(heartbeatRunEvents).values({ companyId: f.companyId, runId: run.id, agentId: f.agentId,
seq: 2, eventType: "adapter.invoke" });
expect(await legacyExecutionNeedsReconciliationWithEvidence(db, cancelled)).toBe(true);
});
it("does not let an old delivery complete after cancellation rearms its continuation", async () => {
const f = await seed(); const run = await retry(f);
await satisfiedConnectionIntentService(db).sweepPending();
const key = `connection-intent:${f.intent.id}:expired`;
const [wake] = await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agentId,
source: "automation", status: "claimed", idempotencyKey: key, runId: run.id }).returning();
await db.update(heartbeatRuns).set({ status: "queued", wakeupRequestId: wake!.id,
contextSnapshot: { issueId: f.issueId, interactionId: f.intent.id, connectionIntentResolution: "existing_connection",
interactionKind: "connection_intent", interactionStatus: "expired", mutation: "interaction", source: "connection_intent.resolved" } }).where(eq(heartbeatRuns.id, run.id));
let cancelled = false;
// Pause precisely after durableWake() succeeds, before its completion write.
const racingDb = new Proxy(db, { get(target, property) {
if (property !== "update") return Reflect.get(target, property);
return (table: Parameters<typeof db.update>[0]) => {
const builder = db.update(table);
if (table !== connectionIntentDeliveries) return builder;
return { set(values: Parameters<typeof builder.set>[0]) {
const update = builder.set(values);
if (!values.deliveredAt) return update;
return { async where(condition: Parameters<typeof update.where>[0]) {
await db.update(connectionGrants).set({ status: "revoked" });
expect(await createPostgresRunDispatchAdapter(db).cancelStaleQueuedRun({ companyId: f.companyId,
runId: run.id, expectedStatus: "queued", now: new Date() })).toMatchObject({ outcome: "cancelled" });
cancelled = true;
return update.where(condition);
} };
} };
};
} });
await connectionIntentDeliveryService(racingDb, { wakeup: vi.fn() }).deliver(f.intent.id);
expect(cancelled).toBe(true);
expect((await db.select().from(connectionIntentDeliveries))[0].deliveredAt).toBeNull();
await db.update(connectionGrants).set({ status: "active" });
const wakeup = vi.fn(async () => { await db.insert(agentWakeupRequests).values({ companyId: f.companyId,
agentId: f.agentId, source: "automation", status: "queued", idempotencyKey: key }); return null; });
await connectionIntentDeliveryService(db, { wakeup }).sweepPending();
expect(wakeup).toHaveBeenCalledOnce();
expect((await db.select().from(connectionIntentDeliveries))[0].deliveredAt).not.toBeNull();
});
it("records a system completion and one durable continuation with a one-connection pool", async () => {
const f = await seed();
const single = createDb(temp.connectionString, { maxConnections: 1 });
try {
const outcomes = await Promise.all([satisfiedConnectionIntentService(single).sweepPending(), satisfiedConnectionIntentService(db).sweepPending()]);
expect(outcomes.reduce((sum, outcome) => sum + outcome.satisfied, 0)).toBe(1);
expect((await db.select().from(issueThreadInteractions))[0]).toMatchObject({ status: "expired", resolvedByUserId: null, result: { outcome: "expired", connectionId: f.connection.id } });
expect(await db.select().from(connectionIntentDeliveries)).toHaveLength(1);
expect(await db.select().from(activityLog)).toEqual([expect.objectContaining({ actorType: "system", action: "issue.thread_interaction_resolved" })]);
expect(await db.select().from(connectionGrantDelegations)).toHaveLength(0);
expect(await db.select().from(toolPolicies)).toHaveLength(0);
expect(await db.select().from(connectionGrants)).toHaveLength(1);
expect(await db.select().from(toolConnectionInstalls)).toHaveLength(1);
expect(await satisfiedConnectionIntentService(single).sweepPending()).toMatchObject({ satisfied: 0 });
} finally { await single.$client.end({ timeout: 1 }); }
});
it("discovers pending cards through the existing delivery worker", async () => {
const f = await seed(); const wakeup = vi.fn().mockResolvedValue(null);
const worker = connectionIntentDeliveryService(db, { wakeup });
expect(await worker.hasPending()).toBe(true);
await worker.sweepPending();
expect((await db.select().from(issueThreadInteractions))[0].status).toBe("expired");
expect(wakeup).toHaveBeenCalledOnce();
expect(wakeup).toHaveBeenCalledWith(f.agentId, expect.objectContaining({ requestedByActorType: "system", idempotencyKey: `connection-intent:${f.intent.id}:expired`, contextSnapshot: expect.objectContaining({ refreshTools: true, responsibleUserId: "connection-owner" }) }));
});
it("rechecks access after a restart before it delivers the saved continuation", async () => {
const f = await seed();
await satisfiedConnectionIntentService(db).sweepPending();
await db.update(connectionGrants).set({ status: "revoked" });
const wakeup = vi.fn().mockResolvedValue(null);
const worker = connectionIntentDeliveryService(db, { wakeup });
await worker.sweepPending();
expect(wakeup).not.toHaveBeenCalled();
expect((await db.select().from(connectionIntentDeliveries))[0].deliveredAt).toBeNull();
await db.update(connectionGrants).set({ status: "active" });
await db.update(connectionIntentDeliveries).set({ nextAttemptAt: new Date() });
await worker.sweepPending();
expect(wakeup).toHaveBeenCalledOnce();
});
it("delivers system retirement to the native runner only while the same connection remains usable", async () => {
const f = await seed(); const run = await retry(f);
await satisfiedConnectionIntentService(db).sweepPending();
const input = { db, companyId: f.companyId, issueId: f.issueId, runId: run.id, agentId: f.agentId, interactionIds: [f.intent.id] };
expect(await materializeNativeInteractionResponses(input)).toEqual([expect.objectContaining({
kind: "connection_intent", response: expect.objectContaining({ status: "expired", result: expect.objectContaining({ connectionId: f.connection.id }) }) })]);
await db.update(connectionGrants).set({ status: "revoked" });
await expect(materializeNativeInteractionResponses(input)).rejects.toMatchObject({ code: "native_interaction_unresolved" });
});
it.each(["missing_install", "denied_tools", "disabled", "unhealthy", "revoked", "wrong_user", "viewer", "removed_member", "additional_access", "ai", "channel", "upstream", "reassigned", "human_owner", "closed", "other_company"] as const)("preserves a request for %s", async (reason) => {
const f = await seed();
if (reason === "missing_install") await db.delete(toolConnectionInstalls);
if (reason === "denied_tools") await db.update(toolProfiles).set({ defaultAction: "deny" });
if (reason === "disabled") await db.update(toolConnections).set({ enabled: false });
if (reason === "unhealthy") await db.update(toolConnections).set({ healthStatus: "error" });
if (reason === "revoked") await db.update(connectionGrants).set({ status: "revoked" });
if (reason === "wrong_user") await db.update(connectionGrants).set({ subjectUserId: "someone-else" });
if (reason === "viewer") await db.update(companyMemberships).set({ membershipRole: "viewer" });
if (reason === "removed_member") await db.delete(companyMemberships);
if (["ai", "channel", "upstream", "additional_access"].includes(reason)) {
const extra = reason === "ai" || reason === "channel" ? { purpose: reason }
: reason === "upstream" ? { upstreamService: { slug: "sentry", name: "Sentry" } }
: { accessRequest: { connectionId: f.connection.id, connectionName: "Sentry", tools: [{ catalogEntryId: f.tool.id, toolName: f.tool.toolName, versionHash: "v1", permission: "allowed" }] } };
await db.update(issueThreadInteractions).set({ payload: { ...f.intent.payload, ...extra } as typeof f.intent.payload });
}
if (reason === "reassigned") await db.update(issues).set({ assigneeAgentId: null });
if (reason === "human_owner") await db.update(issues).set({ assigneeUserId: "connection-owner" });
if (reason === "closed") await db.update(issues).set({ status: "done" });
if (reason === "other_company") {
const [other] = await db.insert(companies).values({ name: "Other", issuePrefix: "OTH" }).returning();
await db.update(issueThreadInteractions).set({ companyId: other!.id });
}
const task = (await db.select().from(issues))[0]; const intent = (await db.select().from(issueThreadInteractions))[0];
expect(await findSatisfiedToolConnection(db, task, intent)).toBeNull();
expect(await satisfiedConnectionIntentService(db).sweepPending()).toMatchObject({ satisfied: 0 });
expect((await db.select().from(issueThreadInteractions))[0].status).toBe("pending");
expect(await db.select().from(connectionIntentDeliveries)).toHaveLength(0);
});
it("keeps a real question blocking even when a sibling connection is available", async () => {
const f = await seed(); const run = await retry(f);
await db.insert(issueThreadInteractions).values({ companyId: f.companyId, issueId: f.issueId, kind: "ask_user_questions", status: "pending", payload: { version: 1, questions: [] } });
expect(await createPostgresRunDispatchAdapter(db).evaluateScheduledRetryGate({ companyId: f.companyId, runId: run.id, now: new Date() })).toMatchObject({ allowed: false, errorCode: "issue_waiting_for_response" });
});
it("does not bypass a retained workspace hold after connecting", async () => {
const f = await seed();
const leaseId = randomUUID();
const workspaceRestoreRecovery = { schema: "paperclip.workspace-restore-recovery.v1", leaseIds: [leaseId] };
const [source] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, issueId: f.issueId, status: "interrupted", finishedAt: new Date(), runtimeMode: "legacy",
contextSnapshot: { issueId: f.issueId }, resultJson: { conversationContinuation: "continue_conversation_v1", workspaceRestoreFailure: "restore_failed", workspaceRestoreRecovery } }).returning();
const identity = { id: leaseId, companyId: f.companyId, heartbeatRunId: source!.id, provider: "daytona", providerLeaseId: `fixture-${leaseId}` };
await db.insert(environmentLeases).values({ ...identity, issueId: f.issueId, status: "released", leasePolicy: "retain_on_failure", releasedAt: new Date(), cleanupStatus: "success",
metadata: { remoteExecutionTermination: remoteTerminationReceipt(identity, { providerLeaseId: identity.providerLeaseId, state: "stopped" }) } });
await db.insert(issueRecoveryActions).values({ companyId: f.companyId, sourceIssueId: f.issueId,
kind: "active_run_watchdog", cause: "legacy_execution_requires_reconciliation", ownerType: "board",
fingerprint: `legacy-execution:${source!.id}`, status: "resolved", outcome: "blocked",
nextAction: "Repair the retained workspace before continuing.", evidence: { runId: source!.id,
workspaceRestoreFailure: "restore_failed", automaticRecovery: { replay: "blocked" },
workspaceRestoreRecovery } });
const run = await retry(f);
await db.update(heartbeatRuns).set({ retryOfRunId: source!.id, contextSnapshot: { issueId: f.issueId, retryOfRunId: source!.id } }).where(eq(heartbeatRuns.id, run.id));
await satisfiedConnectionIntentService(db).sweepPending();
const dispatch = createPostgresRunDispatchAdapter(db);
await dispatch.promoteOrCancelDueRetry({ companyId: f.companyId, runId: run.id, now: new Date() });
const outcome = await dispatch.cancelStaleQueuedRun({ companyId: f.companyId, runId: run.id, expectedStatus: "queued", now: new Date() });
expect(outcome).toMatchObject({ outcome: "cancelled", errorCode: "execution_reconciliation_required" });
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0].status).toBe("cancelled");
});
});
@@ -2,6 +2,7 @@ import { beforeEach, describe, expect, it, vi } from "vitest";
import type { Db } from "@paperclipai/db";
import { refreshCompanySubscriptions, refreshSubscriptionConnection } from "../services/subscription-refresh.js";
import type { SubscriptionConnection } from "../services/subscriptions.js";
import { idleWorkSnapshot } from "../services/task-admission.js";
const mocks = vi.hoisted(() => ({ accounts: vi.fn(), credential: vi.fn(), register: vi.fn(), probe: vi.fn() }));
vi.mock("../services/ai-connections.js", () => ({ aiConnectionService: () => ({ subscriptionAccounts: mocks.accounts, credential: mocks.credential }) }));
@@ -28,6 +29,40 @@ beforeEach(() => {
});
describe("subscription discovery capacity", () => {
it("tracks detached provider refresh through its final database write", async () => {
const db = database(), probe = deferred<{ credentialKey: string }>(), save = deferred<string>();
const before = idleWorkSnapshot();
mocks.probe.mockReturnValue(probe.promise);
mocks.register.mockReturnValue(save.promise);
const task = refreshSubscriptionConnection(db, input("tracked"), "tracked");
try {
await vi.waitFor(() => expect(mocks.probe).toHaveBeenCalledOnce());
expect(idleWorkSnapshot().active).toBe(before.active + 1);
expect(refreshSubscriptionConnection(db, input("tracked"), "tracked")).toBe(task);
expect(idleWorkSnapshot().active).toBe(before.active + 1);
probe.resolve({ credentialKey: "fixture-key" });
await vi.waitFor(() => expect(mocks.register).toHaveBeenCalledOnce());
expect(idleWorkSnapshot().active).toBe(before.active + 1);
} finally {
probe.resolve({ credentialKey: "fixture-key" }); save.resolve("tracked"); await task;
}
expect(idleWorkSnapshot().active).toBe(before.active);
expect(idleWorkSnapshot().generation).toBe(before.generation + 2);
});
it("tracks detached discovery while it waits for account credentials", async () => {
const db = database(), accounts = deferred<[]>();
const before = idleWorkSnapshot();
mocks.accounts.mockReturnValue(accounts.promise);
const task = refreshCompanySubscriptions(db, "tracked-company", "alice");
try {
expect(idleWorkSnapshot().active).toBe(before.active + 1);
expect(refreshCompanySubscriptions(db, "tracked-company", "alice")).toBe(task);
expect(idleWorkSnapshot().active).toBe(before.active + 1);
} finally { accounts.resolve([]); await task; }
expect(idleWorkSnapshot().active).toBe(before.active);
});
it.each([false, true])("waits for a provider slot without dropping accepted discovery (provider fails: %s)", async (fails) => {
const db = database();
const gates = Array.from({ length: 5 }, () => deferred<void>());
@@ -50,6 +50,7 @@ import {
resolveRuntimeProvisionCommand,
resolveWorkspaceRuntimeReadinessTimeoutSec,
resolveShell,
runWorkspaceJobForControl,
sanitizeRuntimeServiceBaseEnv,
setWorkspaceRuntimeExposureDepsForTests,
startRuntimeServicesForWorkspaceControl,
@@ -80,6 +81,7 @@ import {
import { resolvePaperclipConfigPath } from "../paths.ts";
import type { WorkspaceOperation } from "@paperclipai/shared";
import type { WorkspaceOperationRecorder } from "../services/workspace-operations.ts";
import { HttpError } from "../errors.ts";
import { deriveWorktreeInstanceId } from "../services/workspace-instance-cleanup.ts";
import {
getEmbeddedPostgresTestSupport,
@@ -10194,3 +10196,55 @@ describe("realizeExecutionWorkspace with an exact existing branch", () => {
});
});
});
describe("runWorkspaceJobForControl", () => {
const actor = { id: null, name: "Board", companyId: "company-1" };
it("reports a failing operator job as an unprocessable job result, not a server error", async () => {
const workspaceRoot = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-workspace-job-failure-"));
const { recorder, operations } = createWorkspaceOperationRecorderDouble();
const jobScript = "console.error('Restored patch differs from exact source'); process.exit(3);";
try {
const failure = await runWorkspaceJobForControl({
actor,
issue: null,
workspace: buildWorkspace(workspaceRoot),
command: {
name: "Recover retained files",
command: `${JSON.stringify(process.execPath)} -e ${JSON.stringify(jobScript)}`,
},
recorder,
}).then(() => null, (error: unknown) => error);
expect(failure).toBeInstanceOf(HttpError);
expect(failure).toMatchObject({
status: 422,
message: 'Workspace job "Recover retained files" failed: Restored patch differs from exact source',
details: { code: "workspace_job_failed", exitCode: 3 },
});
expect(operations).toHaveLength(1);
expect(operations[0]?.result).toMatchObject({ status: "failed", exitCode: 3 });
} finally {
await fs.rm(workspaceRoot, { recursive: true, force: true });
}
});
it("rejects a job without a command as unprocessable", async () => {
const workspaceRoot = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-workspace-job-missing-"));
const { recorder, operations } = createWorkspaceOperationRecorderDouble();
try {
await expect(runWorkspaceJobForControl({
actor,
issue: null,
workspace: buildWorkspace(workspaceRoot),
command: { name: "Empty job" },
recorder,
})).rejects.toMatchObject({ status: 422, message: 'Workspace job "Empty job" is missing command' });
expect(operations).toHaveLength(0);
} finally {
await fs.rm(workspaceRoot, { recursive: true, force: true });
}
});
});
+7
View File
@@ -118,6 +118,13 @@ describe("native ACPX environment checks", () => {
})]);
});
it("identifies an incomplete installation and gives the operator a repair", async () => {
probeInstallation.mockRejectedValue(Object.assign(new Error("The Claude ACP runtime package is missing. Reinstall or rebuild Paperclip."), { code: "ACPX_RUNTIME_PACKAGE_MISSING" }));
const result = await requireServerAdapter("paperclip_runner").testEnvironment!(context);
expect(result.status).toBe("fail");
expect(result.checks[0]).toMatchObject({ code: "acpx_runtime_package_missing", message: expect.stringContaining("Reinstall or rebuild") });
});
it("requires a successful installed runtime probe", async () => {
const result = await requireServerAdapter("paperclip_runner").testEnvironment!(context);
expect(result.status).toBe("pass");
+6 -1
View File
@@ -447,7 +447,12 @@ const paperclipRunnerAdapter: ServerAdapterModule = {
} catch (error) {
return {
adapterType: "paperclip_runner", status: "fail" as const, testedAt: new Date().toISOString(),
checks: [{ code: "acpx_runtime_unavailable", level: "error" as const, message: error instanceof Error ? error.message : "ACPX Claude runtime could not be verified." }],
checks: [{
code: error instanceof Error && "code" in error && error.code === "ACPX_RUNTIME_PACKAGE_MISSING"
? "acpx_runtime_package_missing" : "acpx_runtime_unavailable",
level: "error" as const,
message: error instanceof Error ? error.message : "The ACPX runtime could not be verified.",
}],
};
}
}
@@ -6,9 +6,7 @@ import type { Db } from "@paperclipai/db";
import {
agentWakeupRequests,
agents,
approvals,
issueApprovals,
issueThreadInteractions,
connectionIntentDeliveries,
heartbeatRuns,
issueRecoveryActions,
issueComments,
@@ -22,6 +20,8 @@ import { evaluateAgentInvokabilityFromDb } from "../../../services/agent-invokab
import { budgetService, budgetServiceInTransaction } from "../../../services/budgets.js";
import { isCompletedOnboardingHandoffWake } from "../../../services/chat-completion-delivery.js";
import { isHeartbeatWakeOnDemandEnabled } from "../../../services/heartbeat-policy.js";
import { DELIVERY_QUEUES, notifyDeliveryWork } from "../../../services/delivery-work-notifications.js";
import { connectionContinuationPendingResponse } from "../../../services/satisfied-connection-intents.js";
import { collectDispositionRepairSourceState } from "../../../services/recovery/disposition-repair.js";
import { legacyDispositionEpisode, legacyDispositionFingerprint } from "../../../services/recovery/legacy-continuation.js";
import { appendHeartbeatRunEvent } from "../../../services/heartbeat-run-events.js";
@@ -72,6 +72,7 @@ import { RunDispatchApplicationError } from "../application/types.js";
type HeartbeatRun = typeof heartbeatRuns.$inferSelect;
type LoadGateFactsInput = {
conversationContinuation: boolean;
responsibleUserId: string | null;
runId: string;
companyId: string;
agentId: string;
@@ -84,6 +85,8 @@ type LoadGateFactsResult =
| { agentFound: true; facts: ScheduledRetryFacts }
| { agentFound: false; issueId: string | null };
type LoadStalenessFactsInput = {
responsibleUserId: string | null;
conversationContinuation: boolean;
runId: string;
companyId: string;
agentId: string;
@@ -337,6 +340,7 @@ export function createPostgresRunDispatchAdapter(
.select({
id: issues.id,
companyId: issues.companyId,
projectId: issues.projectId,
status: issues.status,
assigneeAgentId: issues.assigneeAgentId,
assigneeUserId: issues.assigneeUserId,
@@ -364,19 +368,8 @@ export function createPostgresRunDispatchAdapter(
facts.issueExecutionRunId = issue.executionRunId;
facts.issueCheckoutRunId = issue.checkoutRunId;
if (input.conversationContinuation) {
const [interactions, linkedApprovals] = await Promise.all([
dbOrTx.select({ id: issueThreadInteractions.id }).from(issueThreadInteractions).where(and(
eq(issueThreadInteractions.companyId, input.companyId),
eq(issueThreadInteractions.issueId, issueId), eq(issueThreadInteractions.status, "pending"),
)).limit(1),
dbOrTx.select({ id: approvals.id }).from(issueApprovals).innerJoin(approvals, and(
eq(approvals.id, issueApprovals.approvalId), eq(approvals.companyId, issueApprovals.companyId),
)).where(and(
eq(issueApprovals.companyId, input.companyId), eq(issueApprovals.issueId, issueId),
inArray(approvals.status, ["pending", "revision_requested"]),
)).limit(1),
]);
facts.pendingResponse = interactions.length > 0 ? "interaction" : linkedApprovals.length > 0 ? "approval" : null;
facts.pendingResponse = await connectionContinuationPendingResponse(dbOrTx as unknown as Db,
issue, input.responsibleUserId, input.runId);
}
facts.reviewParticipant = await readNativeReviewParticipantFacts(dbOrTx, {
companyId: input.companyId, issueId, agentId: input.agentId,
@@ -464,6 +457,7 @@ export function createPostgresRunDispatchAdapter(
companyId: run.companyId,
agentId: run.agentId,
conversationContinuation: run.runtimeMode === "legacy" && hasConversationContinuationPolicy(run.resultJson),
responsibleUserId: run.responsibleUserId,
contextSnapshot: parseObject(run.contextSnapshot),
scheduledRetryReason: run.scheduledRetryReason,
retryReasonOverride: input.retryReasonOverride,
@@ -519,6 +513,9 @@ export function createPostgresRunDispatchAdapter(
const issueQuery = dbOrTx
.select({
id: issues.id,
companyId: issues.companyId,
projectId: issues.projectId,
assigneeUserId: issues.assigneeUserId,
status: issues.status,
assigneeAgentId: issues.assigneeAgentId,
executionRunId: issues.executionRunId,
@@ -600,6 +597,9 @@ export function createPostgresRunDispatchAdapter(
unresolvedBlockerIssueIds: readiness.unresolvedBlockerIssueIds,
unresolvedBlockerCount: readiness.unresolvedBlockerCount,
} : null,
pendingResponse: issue && (input.conversationContinuation || context.connectionIntentResolution === "existing_connection")
? await connectionContinuationPendingResponse(dbOrTx as unknown as Db, issue, input.responsibleUserId, input.runId)
: null,
issueFound: issue !== null,
issueStatus: issue?.status ?? null,
issueAssigneeAgentId: issue?.assigneeAgentId ?? null,
@@ -770,6 +770,7 @@ export function createPostgresRunDispatchAdapter(
companyId: run.companyId,
agentId: run.agentId,
conversationContinuation: run.runtimeMode === "legacy" && hasConversationContinuationPolicy(run.resultJson),
responsibleUserId: run.responsibleUserId,
contextSnapshot: parseObject(run.contextSnapshot),
scheduledRetryReason: run.scheduledRetryReason,
retryReasonOverride: run.scheduledRetryReason,
@@ -881,6 +882,7 @@ export function createPostgresRunDispatchAdapter(
decision: Extract<ReturnType<typeof decideQueuedRunStaleness>, { stale: true }>,
expectedStatus: "queued" | "running",
now: Date,
providerDispatchNotStarted = false,
): Promise<CancelStaleQueuedRunOutcome> {
const [row] = await tx
.update(heartbeatRuns)
@@ -892,6 +894,9 @@ export function createPostgresRunDispatchAdapter(
resultJson: {
...parseObject(run.resultJson),
stopReason: decision.errorCode,
...(providerDispatchNotStarted && decision.errorCode === "issue_waiting_for_response"
? { preDispatchResponseWait: { version: 1, providerWorkStarted: false } }
: {}),
...(decision.errorCode === "execution_reconciliation_required"
? { executionWait: decision.details }
: {}),
@@ -927,6 +932,20 @@ export function createPostgresRunDispatchAdapter(
);
}
const context = parseObject(row.contextSnapshot);
if (decision.errorCode === "issue_waiting_for_response" && context.connectionIntentResolution === "existing_connection"
&& context.source === "connection_intent.resolved" && context.interactionKind === "connection_intent"
&& typeof context.interactionId === "string") {
// The partial wake-key uniqueness fence permits another delivery after
// this skipped receipt. Rearm atomically so restored access or a later
// answer can resume the untouched request after a restart.
await tx.update(connectionIntentDeliveries).set({ deliveredAt: null, nextAttemptAt: now }).where(and(
eq(connectionIntentDeliveries.companyId, row.companyId),
eq(connectionIntentDeliveries.interactionId, context.interactionId),
));
await notifyDeliveryWork(tx, DELIVERY_QUEUES.connection);
}
await tx
.update(issues)
.set({
@@ -976,6 +995,8 @@ export function createPostgresRunDispatchAdapter(
runId: run.id,
companyId: run.companyId,
agentId: run.agentId,
responsibleUserId: run.responsibleUserId,
conversationContinuation: run.runtimeMode === "legacy" && hasConversationContinuationPolicy(run.resultJson),
issueId,
contextSnapshot,
scheduledRetryReason: run.scheduledRetryReason,
@@ -1129,6 +1150,7 @@ export function createPostgresRunDispatchAdapter(
decision,
input.expectedStatus,
input.now,
true, // This locked gate has not handed off to the provider callback.
);
return { dispatched: false as const, cancellation };
}
@@ -108,6 +108,7 @@ export type ScheduledRetryFacts = {
};
export type QueuedRunStalenessErrorCode =
| "issue_waiting_for_response"
| "execution_reconciliation_required"
| "issue_dependencies_blocked"
| "issue_not_found"
@@ -129,6 +130,7 @@ export type StalenessDecision =
};
export type QueuedRunFacts = {
pendingResponse?: "interaction" | "approval" | null;
/** Rechecked for automatic native replacements immediately before dispatch. */
dependenciesBlocked?: DependencyBlockFacts | null;
runId: string;
@@ -546,6 +548,12 @@ export function decideQueuedRunStaleness(
}
}
if (facts.pendingResponse) {
return { stale: true, errorCode: "issue_waiting_for_response",
reason: "Cancelled because the continuation is still waiting for an authorized response",
details: { issueId: facts.issueId, pendingResponse: facts.pendingResponse } };
}
if (facts.continuationParkApplies && facts.continuationParksExecutor) {
return {
stale: true,
@@ -9,6 +9,13 @@ import {
} from "./wake-context.js";
describe("wake context", () => {
it("recognizes a retired available connection without accepting ordinary expiry", () => {
const context = { interactionId: "connection", interactionStatus: "expired", interactionKind: "connection_intent",
connectionIntentResolution: "existing_connection", mutation: "interaction", wakeReason: "issue_commented" };
expect(isResolvedInteractionContinuationWakeContext(context)).toBe(true);
expect(isResolvedInteractionContinuationWakeContext({ ...context, interactionKind: "request_confirmation" })).toBe(false);
expect(isResolvedInteractionContinuationWakeContext({ ...context, connectionIntentResolution: undefined })).toBe(false);
});
it("requires the subscription-specific non-assignee receipt for a subscription wait", () => {
expect(isNonAssigneeWorkspaceBusyRetry("ai_connection_busy", {
aiConnectionBusyDeferredWhileAssignee: false,
@@ -95,7 +95,10 @@ export function isResolvedInteractionContinuationWakeContext(contextSnapshot: un
const interactionId = readNonEmptyString(context.interactionId);
const interactionStatus = readNonEmptyString(context.interactionStatus);
if (!interactionId || !interactionStatus) return false;
if (!RESOLVED_INTERACTION_CONTINUATION_STATUSES.has(interactionStatus)) return false;
const retiredConnection = interactionStatus === "expired"
&& context.interactionKind === "connection_intent"
&& context.connectionIntentResolution === "existing_connection";
if (!retiredConnection && !RESOLVED_INTERACTION_CONTINUATION_STATUSES.has(interactionStatus)) return false;
const mutation = readNonEmptyString(context.mutation);
const wakeReason = readNonEmptyString(context.wakeReason);
+21 -2
View File
@@ -6,6 +6,7 @@ import { refreshCompanySubscriptions } from "../services/subscription-refresh.js
import { canManageAiConnections } from "./ai-connections.js";
import { importProviderCostsSchema } from "@paperclipai/shared";
import { accountingIntegrityService } from "../services/accounting-integrity.js";
import { costDataQualityReport } from "../services/cost-data-quality.js";
import { billingReconciliationService } from "../services/billing-reconciliation.js";
import { adjustCostSchema, importBillingInvoiceSchema, repairAccountingSchema, retryAccountingSchema } from "@paperclipai/shared";
import { Router } from "express";
@@ -84,13 +85,31 @@ export function costRoutes(
const issues = issueService(db);
const access = accessService(db);
async function subscriptionActor(req: Parameters<typeof assertCompanyAccess>[0], companyId: string) {
function costReaderActor(req: Parameters<typeof assertCompanyAccess>[0], companyId: string) {
assertBoard(req);
assertCompanyAccess(req, companyId);
return { userId: getActorInfo(req).actorId, canManage: await canManageAiConnections(db, req, companyId),
return { userId: getActorInfo(req).actorId,
readOnly: req.actor.memberships?.some(member => member.companyId === companyId && member.membershipRole === "viewer") ?? false };
}
async function subscriptionActor(req: Parameters<typeof assertCompanyAccess>[0], companyId: string) {
return { ...costReaderActor(req, companyId), canManage: await canManageAiConnections(db, req, companyId) };
}
router.get("/companies/:companyId/costs/data-quality", async (req, res) => {
const companyId = req.params.companyId as string;
const actor = costReaderActor(req, companyId);
if (!(await assertCompanyCostReadAllowed(req, res, companyId))) return;
res.json(await costDataQualityReport(db, companyId, !actor.readOnly, parseCostDateRange(req.query)));
});
router.post("/companies/:companyId/costs/retry-accounting", validate(retryAccountingSchema), async (req, res) => {
const companyId = req.params.companyId as string;
const actor = costReaderActor(req, companyId);
if (!(await assertCompanyCostReadAllowed(req, res, companyId))) return;
if (actor.readOnly) throw forbidden("Viewers cannot retry cost recording");
res.json(await accountingIntegrityService(db, budgetHooks).retry(companyId, req.body.runId, actor.userId));
});
router.get("/companies/:companyId/costs/subscriptions", async (req, res) => {
const companyId = req.params.companyId as string;
const actor = await subscriptionActor(req, companyId);
+8
View File
@@ -5651,6 +5651,7 @@ const costSummaryPaths = [
"window-spend",
"quota-windows",
"subscriptions",
"data-quality",
] as const;
for (const segment of costSummaryPaths) {
@@ -5669,6 +5670,13 @@ for (const segment of costSummaryPaths) {
});
}
registry.registerPath({ method: "post", path: "/api/companies/{companyId}/costs/retry-accounting", tags: ["costs"],
summary: "Retry recording a run's saved cost data",
description: "Requires company cost-read and board write access. Viewers and agents cannot retry. Replays accounting evidence only; never starts provider work or invents missing usage.",
request: { params: z.object({ companyId: z.string() }), body: jsonBody(retryAccountingSchema) },
responses: { 200: r.ok(), 400: r.badRequest, 401: r.unauthorized, 403: r.forbidden, 404: r.notFound },
});
registry.registerPath({ method: "post", path: "/api/companies/{companyId}/costs/subscriptions/refresh", tags: ["costs"],
summary: "Refresh connected subscription estimates in the background",
description: "Board members only. Probes only credentials the caller can use. Successful observations are cached for six hours; provider failures retain prior estimates. No inference requests, invoices, or budget charges are created.",
@@ -24,6 +24,9 @@ describe("run failure diagnostics", () => {
expect(collectSetup(error, run({ ...fixture, errorCode: "adapter_failed" }))).not.toHaveProperty("environmentAcquisitionPhase");
expect(collectRunFailureDiagnostics(fixture, { error, phase: "execute" }).execution).not.toHaveProperty("environmentAcquisitionPhase");
expect(Object.getOwnPropertyDescriptors(error)).toEqual(properties);
expect(collectSetup(error)).not.toHaveProperty("environmentAcquisitionProviderState");
captureEnvironmentAcquisitionDiagnostic(error, scope, { phase: "create", elapsedMs: 2_000, budgetMs: 2_000, providerState: "pulling_snapshot" });
expect(collectSetup(error)).toMatchObject({ environmentAcquisitionPhase: "create", environmentAcquisitionProviderState: "pulling_snapshot" });
captureEnvironmentAcquisitionDiagnostic(error, scope, null);
expect(collectSetup(error)).not.toHaveProperty("environmentAcquisitionPhase");
});
@@ -1,6 +1,7 @@
import { connectionIntentService } from "./connection-intents.js";
import { connectionContinuationPendingResponse, findSatisfiedToolConnection, satisfiedConnectionIntentService } from "./satisfied-connection-intents.js";
import { isAiConnectionConfigurationFailure } from "./ai-auth-failure.js";
import { and, eq, isNull, lte, asc, notInArray, desc, sql } from "drizzle-orm";
import { and, eq, isNull, lte, asc, inArray, notInArray, desc, sql } from "drizzle-orm";
import { connectionIntentDeliveries, issueThreadInteractions, issues, agentWakeupRequests, companyMemberships, heartbeatRuns, chatConversations, chatEndpoints, type Db } from "@paperclipai/db";
import type { heartbeatService } from "./heartbeat.js";
import { issueService } from "./issues.js";
@@ -13,10 +14,11 @@ export async function wakeConnectionIntentAfterResolution(
input: {
loaded: {
issue: { id: string; assigneeAgentId: string | null; status: string };
interaction: { id: string; resolvedAt?: string | Date | null; payload?: unknown };
interaction: { id: string; resolvedAt?: string | Date | null; payload?: unknown; addresseeUserId?: string | null };
};
status: string;
actorId: string;
actorType?: "user" | "system";
},
) {
const agentId = input.loaded.issue.assigneeAgentId;
@@ -37,7 +39,7 @@ export async function wakeConnectionIntentAfterResolution(
mutation: "interaction",
},
idempotencyKey: `connection-intent:${input.loaded.interaction.id}:${input.status}`,
requestedByActorType: "user",
requestedByActorType: input.actorType ?? "user",
requestedByActorId: input.actorId,
contextSnapshot: {
issueId: input.loaded.issue.id,
@@ -48,6 +50,9 @@ export async function wakeConnectionIntentAfterResolution(
mutation: "interaction",
wakeReason: "issue_commented",
source: "connection_intent.resolved",
...(input.actorType === "system" && input.loaded.interaction.addresseeUserId
? { responsibleUserId: input.loaded.interaction.addresseeUserId, connectionIntentResolution: "existing_connection" }
: {}),
...(interactionResolvedAt
? { interactionResolvedAt }
: {}),
@@ -64,6 +69,7 @@ export async function wakeConnectionIntentAfterResolution(
export function connectionIntentDeliveryService(db: Db, heartbeat: Pick<Heartbeat, "wakeup">) {
const satisfiedIntents = satisfiedConnectionIntentService(db);
// Only a repaired AI-authentication failure may reopen a blocked task. An old
// card must never resume a newer failure, a reassignment, or a manual hold.
async function restoreAiBlockedTask(loaded: {
@@ -120,11 +126,16 @@ export function connectionIntentDeliveryService(db: Db, heartbeat: Pick<Heartbea
const [loaded] = await db.select({ interaction: issueThreadInteractions, issue: issues })
.from(issueThreadInteractions).innerJoin(issues, eq(issues.id, issueThreadInteractions.issueId))
.where(and(eq(issueThreadInteractions.id, interactionId), eq(issueThreadInteractions.companyId, claimed.companyId), eq(issues.companyId, claimed.companyId)));
const interaction = loaded?.interaction;
let interaction = loaded?.interaction;
const retiredForAvailableConnection = interaction?.status === "expired"
&& interaction.result?.outcome === "expired" && "connectionId" in interaction.result
&& typeof interaction.result.connectionId === "string";
const payload = interaction?.payload as { requestingAgentId?: string; serviceSlug?: string; purpose?: "ai" } | undefined;
if (!loaded || !interaction || !["accepted", "rejected"].includes(interaction.status)
if (!loaded || !interaction || (!retiredForAvailableConnection && !["accepted", "rejected"].includes(interaction.status))
|| ["done", "cancelled"].includes(loaded.issue.status) || loaded.issue.assigneeAgentId !== payload?.requestingAgentId) {
await db.update(connectionIntentDeliveries).set({ deliveredAt: new Date() }).where(eq(connectionIntentDeliveries.interactionId, interactionId));
await db.update(connectionIntentDeliveries).set({ deliveredAt: new Date() }).where(and(eq(connectionIntentDeliveries.interactionId, interactionId),
eq(connectionIntentDeliveries.companyId, claimed.companyId),
eq(connectionIntentDeliveries.nextAttemptAt, claimed.nextAttemptAt), isNull(connectionIntentDeliveries.deliveredAt)));
return;
}
const userId = interaction.addresseeUserId;
@@ -134,7 +145,9 @@ export function connectionIntentDeliveryService(db: Db, heartbeat: Pick<Heartbea
eq(companyMemberships.principalId, userId ?? ""), eq(companyMemberships.status, "active"),
)).limit(1);
if (!membership?.membershipRole || membership.membershipRole === "viewer") {
await db.update(connectionIntentDeliveries).set({ deliveredAt: new Date() }).where(eq(connectionIntentDeliveries.interactionId, interactionId));
await db.update(connectionIntentDeliveries).set({ deliveredAt: new Date() }).where(and(eq(connectionIntentDeliveries.interactionId, interactionId),
eq(connectionIntentDeliveries.companyId, claimed.companyId),
eq(connectionIntentDeliveries.nextAttemptAt, claimed.nextAttemptAt), isNull(connectionIntentDeliveries.deliveredAt)));
return;
}
}
@@ -143,6 +156,36 @@ export function connectionIntentDeliveryService(db: Db, heartbeat: Pick<Heartbea
agentId: payload!.requestingAgentId!, responsibleUserId: userId!, serviceSlug: payload!.serviceSlug!, purpose: payload!.purpose });
if (!ready) return;
}
if (retiredForAvailableConnection) {
const publications: ActivityPublication[] = [];
const current = await db.transaction(async tx => {
const [task] = await tx.select().from(issues).where(and(eq(issues.id, loaded.issue.id), eq(issues.companyId, claimed.companyId))).for("update");
const [card] = await tx.select().from(issueThreadInteractions).where(and(
eq(issueThreadInteractions.id, interactionId), eq(issueThreadInteractions.companyId, claimed.companyId),
eq(issueThreadInteractions.issueId, loaded.issue.id))).for("update");
if (!task || !card || card.status !== "expired" || card.result?.outcome !== "expired") return null;
const ready = await findSatisfiedToolConnection(tx as unknown as Db, task, { ...card, status: "pending" });
if (!ready) return null;
if (ready.id !== (card.result as { connectionId: string }).connectionId) {
const previousConnectionId = (card.result as { connectionId: string }).connectionId;
// This is a system observation of existing access. It grants nothing
// and never changes or impersonates a human decision.
const [updated] = await tx.update(issueThreadInteractions).set({ result: { ...card.result,
connectionId: ready.id }, updatedAt: new Date() }).where(eq(issueThreadInteractions.id, card.id)).returning();
Object.assign(card, updated);
await logActivity(tx as unknown as Db, { companyId: claimed.companyId, actorType: "system", actorId: "connection-reconciliation",
action: "issue.thread_interaction_resolved", entityType: "issue", entityId: task.id,
details: { interactionId, interactionKind: "connection_intent", status: "expired", connectionId: ready.id,
previousConnectionId, resolutionSource: "existing_connection_refreshed" } }, publications);
}
return { task, card, waiting: await connectionContinuationPendingResponse(tx as unknown as Db, task, card.addresseeUserId, card.sourceRunId) };
});
for (const publication of publications) publishActivity(publication);
if (!current || current.waiting) return;
loaded.issue = current.task;
loaded.interaction = current.card;
interaction = current.card;
}
if (interaction.status === "accepted" && payload?.purpose === "ai" && loaded.issue.status === "blocked") {
const restored = await restoreAiBlockedTask(loaded);
if (!restored) return;
@@ -157,21 +200,32 @@ export function connectionIntentDeliveryService(db: Db, heartbeat: Pick<Heartbea
// Check before dispatch: the previous worker may have crashed after enqueueing.
if (!(await durableWake()).length) {
try {
await wakeConnectionIntentAfterResolution(heartbeat, { loaded, status: interaction.status, actorId: interaction.resolvedByUserId ?? interaction.addresseeUserId! });
await wakeConnectionIntentAfterResolution(heartbeat, { loaded, status: interaction.status,
actorId: retiredForAvailableConnection ? "connection-reconciliation" : interaction.resolvedByUserId ?? interaction.addresseeUserId!,
...(retiredForAvailableConnection ? { actorType: "system" as const } : {}),
});
} catch (error) {
// The unique wake key also protects overlapping leases. Other failures retry.
if (!(await durableWake()).length) throw error;
}
}
if ((await durableWake()).length) {
await db.update(connectionIntentDeliveries).set({ deliveredAt: new Date() }).where(eq(connectionIntentDeliveries.interactionId, interactionId));
await db.update(connectionIntentDeliveries).set({ deliveredAt: new Date() }).where(and(eq(connectionIntentDeliveries.interactionId, interactionId),
eq(connectionIntentDeliveries.companyId, claimed.companyId),
eq(connectionIntentDeliveries.nextAttemptAt, claimed.nextAttemptAt), isNull(connectionIntentDeliveries.deliveredAt)));
}
}
async function hasPending() {
return (await db.select({ id: connectionIntentDeliveries.interactionId }).from(connectionIntentDeliveries)
.where(isNull(connectionIntentDeliveries.deliveredAt)).limit(1)).length > 0;
if ((await db.select({ id: connectionIntentDeliveries.interactionId }).from(connectionIntentDeliveries)
.where(isNull(connectionIntentDeliveries.deliveredAt)).limit(1)).length > 0) return true;
return (await db.select({ id: issueThreadInteractions.id }).from(issueThreadInteractions)
.innerJoin(issues, and(eq(issues.id, issueThreadInteractions.issueId), eq(issues.companyId, issueThreadInteractions.companyId)))
.where(and(eq(issueThreadInteractions.kind, "connection_intent"), eq(issueThreadInteractions.status, "pending"),
inArray(issues.status, ["in_progress", "in_review"]), isNull(issues.assigneeUserId)))
.limit(1)).length > 0;
}
return { deliver, hasPending, tryDeliver: async (id: string) => { try { await deliver(id); } catch { /* Persisted delivery remains due after its lease. */ } }, sweepPending: async () => {
await satisfiedIntents.sweepPending();
const rows = await db.select().from(connectionIntentDeliveries).where(and(isNull(connectionIntentDeliveries.deliveredAt), lte(connectionIntentDeliveries.nextAttemptAt, new Date())))
.orderBy(asc(connectionIntentDeliveries.nextAttemptAt)).limit(50);
let failed = 0;
+57
View File
@@ -0,0 +1,57 @@
import { sql } from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import type { CostDataQualityItem, CostDataQualityReport } from "@paperclipai/shared";
import { resolveCostDateRange, type CostDateRange } from "./cost-date-range.js";
import { withAccountingReadSnapshot } from "./accounting-transaction.js";
const PAGE_SIZE = 25;
// A display distinction, never authority to acknowledge missing usage as zero.
export const RECEIPT_WAIT_MS = 15 * 60 * 1000;
/** Safe explanations for cost readers: bounded metadata only, never raw errors,
* credential identities, prompts, environments, or untrusted provider payloads. */
export async function costDataQualityReport(db: Db, companyId: string, canRetry: boolean, range?: CostDateRange, now = new Date()): Promise<CostDataQualityReport> {
return withAccountingReadSnapshot(db, companyId, async tx => {
const period = resolveCostDateRange(range, now);
const eventFrom = period.from ? sql`and e.occurred_at >= ${period.from.toISOString()}::timestamptz` : sql``;
const eventTo = period.to ? sql`and e.occurred_at <= ${period.to.toISOString()}::timestamptz` : sql``;
const runFrom = period.from ? sql`and coalesce(r.finished_at,r.created_at) >= ${period.from.toISOString()}::timestamptz` : sql``;
const runTo = period.to ? sql`and coalesce(r.finished_at,r.created_at) <= ${period.to.toISOString()}::timestamptz` : sql``;
const [events, runs] = await Promise.all([
tx.execute<CostDataQualityItem & Record<string, unknown>>(sql`
select e.id, e.heartbeat_run_id as "runId", e.agent_id as "agentId", a.name as "agentName", a.appearance as "agentAppearance",
e.occurred_at::text as "occurredAt", e.provider, e.model,
case when e.billing_type = 'unknown' then 'unknown_billing'
when e.model in ('unknown','') then 'missing_model' else 'missing_price' end as reason
from cost_events e left join agents a on a.id = e.agent_id and a.company_id = e.company_id
where e.company_id = ${companyId} ${eventFrom} ${eventTo}
and (e.billing_type = 'unknown' or (e.cost_status = 'unpriced' and e.billing_type <> 'subscription_included'))
order by e.occurred_at desc,e.id limit ${PAGE_SIZE + 1}`),
tx.execute<CostDataQualityItem & Record<string, unknown>>(sql`
select r.id, r.id as "runId", r.agent_id as "agentId", a.name as "agentName", a.appearance as "agentAppearance",
coalesce(r.finished_at,r.created_at)::text as "occurredAt",
r.usage_json->>'provider' as provider,r.usage_json->>'model' as model,
case when r.cost_accounted_at is not null then 'retryable'
when r.usage_json->'accountingCaptureFailed' is distinct from 'true'::jsonb
and (r.usage_json->'accountingProviderWorkStarted' = 'false'::jsonb
or r.result_json->'executionRecovery'->'providerWorkStarted' = 'false'::jsonb) then 'retryable'
when r.usage_json->'accountingCaptureFailed' = 'true'::jsonb
or r.usage_json->'accountingReceiptReady' = 'false'::jsonb
or (r.runtime_mode = 'native' and exists (select 1 from native_run_finalizations f
where f.run_id = r.id and f.company_id = r.company_id and f.result_id is null and f.phase <> 'terminal_failure'))
then case when coalesce(r.finished_at,r.created_at) <= ${new Date(now.getTime() - RECEIPT_WAIT_MS).toISOString()}::timestamptz
then 'receipt_unavailable' else 'waiting_for_receipt' end
else 'retryable' end as reason
from heartbeat_runs r left join agents a on a.id = r.agent_id and a.company_id = r.company_id
where r.company_id = ${companyId} and r.cost_accounting_pending
and r.status in ('succeeded','failed','timed_out','cancelled','interrupted') ${runFrom} ${runTo}
order by coalesce(r.finished_at,r.created_at) desc,r.id limit ${PAGE_SIZE + 1}`),
]);
// PostgreSQL's text timestamp is not portable to every browser's Date parser.
const items = (rows: readonly CostDataQualityItem[]) => rows.slice(0, PAGE_SIZE).map(row => ({
...row, occurredAt: new Date(row.occurredAt).toISOString(),
}));
return { canRetry, events: items(Array.from(events)), runs: items(Array.from(runs)),
hasMoreEvents: events.length > PAGE_SIZE, hasMoreRuns: runs.length > PAGE_SIZE };
});
}
+9 -1
View File
@@ -186,10 +186,11 @@ export function costService(db: Db, budgetHooks: BudgetServiceHooks = {}) {
if (range?.from) conditions.push(gte(costEvents.occurredAt, range.from));
if (range?.to) conditions.push(lte(costEvents.occurredAt, range.to));
const [{ total, totalExact, eventCount, unpricedEventCount, estimatedEventCount }] = await db
const [{ total, totalExact, eventCount, unpricedEventCount, estimatedEventCount, unknownBillingEventCount }] = await db
.select({
eventCount: sql<number>`count(*)::int`,
estimatedEventCount: sql<number>`count(*) filter (where ${costEvents.costStatus} = 'estimated')::int`,
unknownBillingEventCount: sql<number>`count(*) filter (where ${costEvents.billingType} = 'unknown')::int`,
unpricedEventCount: sql<number>`count(*) filter (where ${costEvents.costStatus} = 'unpriced' and ${costEvents.billingType} <> 'subscription_included')::int`,
total: sumAsNumber(costEvents.costCents),
totalExact: sql<string>`coalesce(sum(${costEvents.costCents}), 0)::text`,
@@ -213,6 +214,7 @@ export function costService(db: Db, budgetHooks: BudgetServiceHooks = {}) {
companyId,
eventCount,
unpricedEventCount,
unknownBillingEventCount,
estimatedEventCount,
pendingRunCount,
pricingComplete: unpricedEventCount === 0 && pendingRunCount === 0,
@@ -492,6 +494,9 @@ export function costService(db: Db, budgetHooks: BudgetServiceHooks = {}) {
model: costEvents.model,
costCents: sumAsNumber(costEvents.costCents),
costCentsExact: sql<string>`coalesce(sum(${costEvents.costCents}), 0)::text`,
eventCount: sql<number>`count(*)::int`,
estimatedEventCount: sql<number>`count(*) filter (where ${costEvents.costStatus} = 'estimated')::int`,
unpricedEventCount: sql<number>`count(*) filter (where ${costEvents.costStatus} = 'unpriced' and ${costEvents.billingType} <> 'subscription_included')::int`,
inputTokens: sumAsNumber(ordinaryInputTokens),
cachedInputTokens: sumAsNumber(costEvents.cachedInputTokens),
outputTokens: sumAsNumber(costEvents.outputTokens),
@@ -523,6 +528,9 @@ export function costService(db: Db, budgetHooks: BudgetServiceHooks = {}) {
biller: costEvents.biller,
costCents: sumAsNumber(costEvents.costCents),
costCentsExact: sql<string>`coalesce(sum(${costEvents.costCents}), 0)::text`,
eventCount: sql<number>`count(*)::int`,
estimatedEventCount: sql<number>`count(*) filter (where ${costEvents.costStatus} = 'estimated')::int`,
unpricedEventCount: sql<number>`count(*) filter (where ${costEvents.costStatus} = 'unpriced' and ${costEvents.billingType} <> 'subscription_included')::int`,
inputTokens: sumAsNumber(ordinaryInputTokens),
cachedInputTokens: sumAsNumber(costEvents.cachedInputTokens),
outputTokens: sumAsNumber(costEvents.outputTokens),
@@ -484,6 +484,7 @@ export async function resolveEnvironmentExecutionTarget(input: {
env: commandInput.env,
stdin: commandInput.stdin,
timeoutMs: commandInput.timeoutMs,
commandTimeoutMs: commandInput.commandTimeoutMs,
onLog: commandInput.onLog ? onIncrementalLog : undefined,
// The ACP process session bridge sets `useSession` so its
// long-lived agent command opens the persistent session and
+38 -1
View File
@@ -1,3 +1,4 @@
import { allowLegacyShutdownWorkspaceCleanup, legacyShutdownWorkspaceResourceProtected } from "./legacy-shutdown-workspace-settlement.js";
import { beginIdleTrackedWork } from "./task-admission.js";
import { hasStopOnlyCleanup, prepareSandboxStopAndRetain, readStopOnlyCleanup, settleStopOnlyCleanup, stopOnlyCleanupKey } from "./sandbox-stop-and-retain.js";
import { JsonRpcCallError, readEnvironmentAcquisitionDiagnostic, readEnvironmentCreationCleanupError } from "@paperclipai/plugin-sdk";
@@ -538,6 +539,9 @@ export interface EnvironmentDriverExecuteInput extends EnvironmentDriverLeaseInp
env?: Record<string, string>;
stdin?: string;
timeoutMs?: number;
/** Total command execution budget for an explicitly requested session.
* Separate from the timeout of an individual status/log observation. */
commandTimeoutMs?: number;
/**
* Run this command outside the lease's persistent session. The run
* orchestrator sets this on the workspace provision command, which runs before
@@ -1943,6 +1947,9 @@ function createSandboxEnvironmentDriver(
lease.metadata?.agentId === input.agentId,
)
: [];
for (let index = reusableCandidateLeases.length - 1; index >= 0; index--) {
if (await legacyShutdownWorkspaceResourceProtected(db, reusableCandidateLeases[index]!)) reusableCandidateLeases.splice(index, 1);
}
const reusableExistingLeases = reusableCandidateLeases.filter((lease) =>
reusableSandboxLeaseScopeMatches({
lease,
@@ -2358,6 +2365,9 @@ function createSandboxEnvironmentDriver(
lease.metadata?.agentId === input.agentId,
)
: [];
for (let index = reusableCandidateLeases.length - 1; index >= 0; index--) {
if (await legacyShutdownWorkspaceResourceProtected(db, reusableCandidateLeases[index]!)) reusableCandidateLeases.splice(index, 1);
}
const reusableExistingLeases = reusableCandidateLeases.filter((lease) =>
reusableSandboxLeaseScopeMatches({
lease,
@@ -2955,6 +2965,7 @@ function createSandboxEnvironmentDriver(
env: input.env,
stdin: input.stdin,
timeoutMs: input.timeoutMs,
commandTimeoutMs: input.commandTimeoutMs,
// Forward the effective session-bypass flag so a provider that opens
// a persistent session skips it for a pre-run or context-less command
// (the workspace provision command, the CLI install command, the
@@ -3327,6 +3338,7 @@ function createSandboxEnvironmentDriver(
lease: EnvironmentLease;
failureReason: string;
}): Promise<EnvironmentLease | null> {
if (await legacyShutdownWorkspaceResourceProtected(db, input.lease)) return null;
let cleanupStatus: "success" | "failed" = "success";
let termination: ReturnType<typeof remoteTerminationReceipt>;
const metadata = input.lease.metadata ?? {};
@@ -3888,6 +3900,7 @@ export function environmentRuntimeService(
scopeCondition?: ReturnType<typeof and>;
failureReason: string;
}): Promise<EnvironmentLease | null> {
if (!(await allowLegacyShutdownWorkspaceCleanup(db, input.leaseRow))) return null;
const now = new Date();
const attemptId = randomUUID();
// Persist recovery ownership before any provider work. Re-check scope and
@@ -4065,8 +4078,16 @@ export function environmentRuntimeService(
// error through `onLeaseReleaseError` for its log path. Keep the order
// serial.
const released: EnvironmentRuntimeLeaseRecord[] = [];
for (const leaseRow of leaseRows) {
for (let leaseRow of leaseRows) {
try {
if (!(await allowLegacyShutdownWorkspaceCleanup(db, leaseRow, new Date(), {
ownerStopOnly: cancelActiveWork === true && providerResourceDisposition === "stop_and_retain",
}))) continue;
if (leaseRow.metadata?.legacyShutdownWorkspaceSettlement) {
leaseRow = (await db.select().from(environmentLeases).where(and(
eq(environmentLeases.id, leaseRow.id), eq(environmentLeases.companyId, leaseRow.companyId),
)))[0] ?? leaseRow;
}
const environment = leaseRow.environmentId
? await environmentsSvc.getById(leaseRow.environmentId)
: null;
@@ -4211,6 +4232,14 @@ export function environmentRuntimeService(
environment: Environment | null;
lease: EnvironmentLease;
}): Promise<unknown> {
if (!(await allowLegacyShutdownWorkspaceCleanup(db, input.lease))) {
throw new Error("Shutdown workspace settlement is still pending.");
}
if (input.lease.metadata?.legacyShutdownWorkspaceSettlement) {
const current = await environmentsSvc.getLeaseById(input.lease.id);
if (!current) throw new Error("Shutdown workspace source is unavailable.");
input = { ...input, lease: current };
}
const driver = requireDriverKey(getLeaseDriverKey(input.lease, input.environment));
if (!driver.retryPendingSandboxTeardown) {
throw new Error(
@@ -4407,6 +4436,7 @@ export function environmentRuntimeService(
},
async resumeRunLease(input: EnvironmentDriverLeaseInput): Promise<PluginEnvironmentLease | EnvironmentLease | null> {
if (await legacyShutdownWorkspaceResourceProtected(db, input.lease)) return null;
const driver = requireDriverKey(getLeaseDriverKey(input.lease, input.environment));
if (!driver.resumeRunLease) {
throw new Error(`Environment driver "${driver.driver}" does not support lease resume.`);
@@ -4415,6 +4445,13 @@ export function environmentRuntimeService(
},
async destroyRunLease(input: EnvironmentDriverLeaseInput): Promise<EnvironmentLease | null> {
if (await legacyShutdownWorkspaceResourceProtected(db, input.lease)) return null;
if (input.lease.metadata?.legacyShutdownWorkspaceSettlement) {
if (!(await allowLegacyShutdownWorkspaceCleanup(db, input.lease))) return null;
const current = await environmentsSvc.getLeaseById(input.lease.id);
if (!current || hasStopOnlyCleanup(current)) return null;
input = { ...input, lease: current };
}
const driver = requireDriverKey(getLeaseDriverKey(input.lease, input.environment));
if (!driver.destroyRunLease) {
throw new Error(`Environment driver "${driver.driver}" does not support lease destroy.`);
Loaded 100 of 148 files, more files were not shown because too many files have changed in this diff. Show more