Merge reviewed evidence repairs into native tracing controller

Apply the same bounded reader to newly introduced OTLP evidence. Preserve failed finalization accounting and relay closure.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip committed 2026-10-26 09:32:51 -05:00
commit edc7642773
315 files changed
+14146 -1145

No files matched your search

+1
View File
@@ -88,6 +88,7 @@ packages/grok-acp/bin
!packages/paperclip-runner/test-fixtures/pi-acp/profile-v27-identity.json
!packages/paperclip-runner/test-fixtures/pi-acp/profile-v28-identity.json
!packages/paperclip-runner/test-fixtures/pi-acp/profile-v29-identity.json
!packages/paperclip-runner/test/fixtures/copilot-profile-v23-identity.json
+10 -7
View File
@@ -63,13 +63,16 @@ reversals and Safari's edge bounce. It glides out and eases back in with shared
motion tokens; the task composer follows the same motion. Keep page padding
stable while the navigation moves, and honor reduced-motion preferences.
Task conversations reveal their initial comments, interaction cards, plan, and
relevant run history together after positioning the latest message. Keep the
mobile loading surface at a stable viewport height while that history loads;
concealed content must not stretch the document. Background refreshes keep an
already revealed conversation and composer mounted and visible.
Bound the initial wait to 15 seconds. If a request stalls, reveal the available
conversation and composer with a notice that some history is still loading.
Cold task opens reveal the instruction and recent conversation independently of
plans, related metadata, and historical run activity. Show the latest human
follow-up, recent saved replies, and text or pending requests from the latest run.
Put older messages and diagnostic activity behind **More messages**. Fetch those
only when expanded; never download all history just to calculate a count.
Retain messages and runs observed while the task stays open, preserve expansion
on return within the session, and load linked older comments when requested.
Background refreshes and history expansion keep the conversation and composer
mounted. A failed history request shows an inline retry without hiding readable
content. Full-history and classic views retain the bounded 15-second initial wait.
Only the viewer's own human comments sit on the right in blue bubbles. Agents
and other humans sit on the left on the page surface, with their own name and
+253 -9
View File
@@ -49,11 +49,42 @@ and leaves all existing cases passive. Source identities are not grants.
### Task query performance
Agent request authentication uses a compact run-identity read. It omits saved
execution payloads, results, and log excerpts from the query response. The read
still locks the task before the run and reconciles pending steering from durable
acknowledgements on every request. Default identity reads retain the full run.
Stored payloads and separate native tool authorization checks remain unchanged.
Native event acknowledgements read a count and maximum through the existing
source-sequence index. A stream with gaps uses indexed lookups for its first
missing successor. The acknowledgement keeps its contiguous-prefix rule without
returning every historical sequence number. All events stay stored. Counting the
index entries still grows with the stream length.
Conversation recovery reads only the saved run fields used to settle a turn.
Settlement locks and reads current conversation state without loading the task
description or comment bodies. It still checks the durable reply or pending
interaction, session generation, owning run, and newer user messages. Saved
execution payloads, task text, and comments remain stored in full.
The attention feed checks for newer runs with company-scoped existence probes.
The probes match the failed run's agent and issue or task context. They preserve
the existing millisecond comparison and empty-context rules. Each batch holds
at most 1,000 failed runs. Retained run history stays in the database; this check
returns only the identifiers of failures that have a newer matching run.
For failures without a task key, it reads the latest keyless timestamp once per
agent. Unique agents are batched separately, so repeated failures do not repeat
the same history scan. Both paths retain the current millisecond comparison.
The issue-lock adapter's release read omits `result_json.stdout` and
`result_json.stderr` before transferring a run to the controller. Every other result key, context field, and
stop, retry, lease, and workspace-recovery check stays intact. Full output remains
stored for execution history and detail reads. Terminal telemetry loads only its
run identity, timestamps, usage, and typed task identifier.
`result_json.stderr` before transferring a run to the controller. Native context
also omits saved launch text, execution-continuation bodies, workspace and
environment payloads, and the launch wake envelope. Task identity, retry and
continuation controls, interaction and comment references, execution policy, unknown context fields, and every other
result key stay intact. Legacy reads retain full context. Stop, retry, lease,
and workspace-recovery checks do not change. Full inputs and output remain stored
for execution history and detail reads. Terminal telemetry loads only its run
identity, timestamps, usage, and typed task identifier.
Recovery's latest-run queries return the context fields used by recovery
decisions. Prompts, execution continuation bodies, and workspace setup payloads
@@ -62,11 +93,48 @@ the PostgreSQL row version. Native blocked-wait repair compares that version
against a fresh full source read before applying changes. Any intervening row
update, including an edit outside the projection, defers repair to a fresh sweep.
The two preliminary reconciliation reads omit named launch renderings and
preparation payloads. Native profiles omit unused native execution input.
They retain all other context, full results, legacy profiles, task identity,
retry accounting, and unknown controls. Native runs still reach their native
recovery guard. Terminalization locks its current control state and returns
full rows from its writes. Stored input remains intact.
Execution-blocker reads select process, lifecycle, and task-binding fields rather
than full execution rows. Context retains typed task and pre-dispatch evidence;
Runner profiles retain the adapter type and complete native execution input and
workspace-sync receipts. Full result evidence stays available to the existing
retry and workspace checks. Stored context, profiles, and output are unchanged.
Runner profiles retain the adapter type, native input schema, binding, provider,
workspace-access evidence, and complete workspace-sync receipts. Native task and
runtime prompt payloads are not needed for these checks. Missing fields and
malformed input values retain their original shape. Full result evidence stays
available to the existing retry and workspace checks. Stored context, profiles,
and output are unchanged.
Committed native finalization reads run identity and ownership state before it
claims the coordinator. It loads complete saved input only when finalization
still needs to process an accepted result. Review presentation retains the full
result. Muse settlement reads its exact provider binding. Cleanup reads fresh
status and ownership metadata; cancelled native workspace retention and legacy
Stop settlement still load complete run evidence. Workspace settlement reads the
saved sync descriptor, export retry, and first-heartbeat flag. These projections
leave stored prompts, logs, results, and recovery state unchanged.
Settled native process release locks run identity, terminal state, process
identity, and checkpoint presence without loading the checkpoint or execution
payloads. It retains current coordinator checks, provider evidence, local-host
identity checks, and every lease's termination receipt. The stop receipt and
process-field update remain in the same transaction.
Legacy terminalization locks runtime mode, status, full result evidence, and the
status-delivery token without loading unused input, profile, or log bodies.
It keeps the existing lock, full returned run, workspace-retention
receipts, and source-bound recovery holds. Stored payloads remain unchanged.
Conversation ownership checks select process identity and full recovery result
evidence without execution prompts or profiles. Committed native reply repair
selects its persisted result and identity bindings plus typed presentation flags.
It keeps its existing row locks, accepted-result digest checks, publication
authorization, and selected-answer protections. Full execution history stays in
storage.
Run-context lookups use expression indexes for both `context_snapshot.issueId`
and `context_snapshot.paperclipIssue.id`. Task creation provenance has its own
@@ -107,11 +175,77 @@ launch context does not travel with each policy or parent-report check. JSON
types, task-key precedence, and company and agent identity checks are preserved.
Each decision still reads current data; this is not an authorization cache.
Restoring a saved monitor's receipt or completion proof creates a new native
assessment identity even when its scheduled deadline is unchanged. The saved
proof is part of the assessment input. Prior decisions and accepted results stay
intact, and an unchanged replay reuses the same assessment and decision.
Native tool authority lookups select current run identity, status, invocation,
responsible user, context, and receipt data. Saved profiles and output excerpts
stay in storage and are omitted from that read. Task and agent binding, current
status, budget, and task privacy checks retain their existing predicates.
Mutation transactions still lock and read complete run records. The unlocked
authority read also omits named launch renderings and preparation data that its
tools do not consume. Ordinary tool checks also omit the wake payload. Mail
checks load the current full wake to enforce external-chat restrictions. Other
context and result receipts remain available. The complete saved context stays
in storage for execution.
WebSocket delivery uses one fresh authorization batch for an event's task and
run bindings. When both bindings name the same task, the batch reads its current
authority once. Different task bindings must both pass. The batch is discarded
after that delivery; later events still recheck grants, membership, task
existence, and company boundaries.
Browser progress events update cached live run status directly. A successful
patch does not refetch task details, activity, or run history. Unknown runs
retain the existing refresh fallback. Run transitions, activity events, and
reconnect recovery still refresh their affected data.
Task run queries stop interval polling only after a successful empty response
and an open shared live connection for the current company and session. Initial
reads, active work, errors, connection setup, and disconnections retain their
existing polling cadence. Company or session changes require a new connection.
Views without an assigned agent retain polling. Matching run lifecycle events
refresh the displayed task's run queries independently of route and window
focus. Events that omit task identity conservatively refresh these queries,
including reviewer runs from another agent. One parent view owns both query
observers and reuses refreshes already covered by the focused-route handler. A
matching event that overlaps an older request queues a fresh read after that
request settles. Connection readiness also reconciles the displayed run
queries after any older request. Follow-up reads belong to the current mounted
scope and stop when it changes or unmounts. The first open and reconnect still
reconcile missed events.
Bundled skill release checks read the current stored version ID and a SHA-256
fingerprint of its file inventory. A per-database memo keeps at most 64 verified
snapshots. It reuses a verification only when the current database fingerprint
and current on-disk inventory hash both match. A changed or missing snapshot
still follows full validation or seeding. Company access and local source
reconciliation still run for each request; authorization is not cached.
Connection-assignment checks select profile bindings for the current company or
agent and read rules that can match the selected connection's catalog rows.
All five selector kinds, null application matching, exclusions, and profile
precedence retain their existing evaluator. Rule filtering uses the catalog
values already read by that check, so a later catalog change cannot drop an
exclusion from the decision being evaluated. There is no cross-call cache.
Filesystem preparation uses current task/run and selected-configuration checks.
Historical root-reader authorization has been retired; no retained-reader batch is
loaded at admission or publication. Workspace-record disclosure still uses task
privacy checks, independently of filesystem access.
Saved-message execution-wait scans read current run state, process ownership,
startup and cancellation receipts, task binding, and the recorded adapter kind.
Saved profiles, output excerpts, and unrelated launch context stay in storage.
Continuation admission and its locked mutation reads retain their existing gates.
Task creation provenance reads current run identity, scope, and task binding.
Legacy task references retain their typed values and existing precedence. Saved
profiles, output, and unrelated context stay in storage. Viewer access to the
source task and run is still checked before provenance is returned.
The initial agent JWT run lookup reads only the cancellation state and
conversation flag from saved run JSON. It does not transfer the full context or
result. Run status, responsible-user identity, company and agent boundaries,
@@ -176,6 +310,32 @@ The benchmark loads the baseline authorization module with current dependencies,
compares complete workspace rows, and prints five warm samples and query plans.
It measures SQL execution, not the full repository admission or agent startup.
Native finalization retries preserve a run's terminal completion time when its
status is unchanged. A retry mints another status delivery only when the current
run's status, finalization metadata, phase, error fields, or missing completion
time changes. An unchanged projection retains its existing pending delivery ID,
including after a publish failure. Status delivery stays at least once; retry
deadlines, recovery, accepted results, and workspace cleanup still run normally.
Legacy recovery also retains the pending delivery when status is unchanged and
no patch was supplied. Explicit patches, new terminal statuses, and newly
retained workspace evidence still create a delivery. File retention, ownership
checks, failed write conditions, lock cleanup, and recovery holds still apply.
The retention-evidence write returns another full run only when that evidence
changes. An unchanged retry keeps the full run returned by the first write and
still processes retained leases and recovery holds.
Agent lists read the complete company roster once and use that same snapshot for
termination filtering and organization-chain normalization. Detail and plugin runtime reads
load only identity, company, name, status, and manager from other company agents
for chain checks. The selected agent's full configuration, current monthly spend,
and existing response redaction remain available. Every call reads current data;
these lookups do not cache authorization or agent configuration.
Queued-comment display filters wake requests by the current company, assigned
agent, status, and exact JSON string task identity in SQL. Other tasks' comment
identifiers stay in the database. The display projection, ordering, current run
ownership checks, and persisted wake payloads retain their existing contract.
## 1. Embedded PostgreSQL — zero config
If you don't set `DATABASE_URL`, the server automatically starts an embedded PostgreSQL instance and manages a local data directory.
@@ -298,14 +458,14 @@ DATABASE_URL=postgres://postgres.[PROJECT-REF]:[PASSWORD]@aws-0-[REGION].pooler.
DATABASE_MIGRATION_URL=postgres://postgres.[PROJECT-REF]:[PASSWORD]@aws-0-[REGION].pooler.supabase.com:5432/postgres
```
If your hosted database requires transaction-pooling-only connections (pgbouncer transaction mode, Supavisor port 6543, Neon `-pooler` endpoints), set `DATABASE_PREPARED_STATEMENTS=false` so the client does not rely on session-scoped prepared statements, and keep `DATABASE_MIGRATION_URL` on a direct connection. Do not edit database client source files as part of deployment setup.
If your transaction-mode pooler does not support protocol-level named prepared statements, set `DATABASE_PREPARED_STATEMENTS=false`. If your pooler supports them, you can opt in with `DATABASE_PREPARED_STATEMENTS=true`. Check your provider's pooler configuration before you enable this setting. Keep `DATABASE_MIGRATION_URL` on a direct connection. Do not edit database client source files as part of deployment setup.
### Client tuning (optional)
All of these are optional; when unset, the driver defaults apply and behavior is unchanged — typical self-hosted setups need none of them:
```sh
DATABASE_PREPARED_STATEMENTS=false # required for transaction-mode poolers; default: enabled
DATABASE_PREPARED_STATEMENTS=false # use false when the pooler does not support named prepared statements
DATABASE_POOL_MAX=25 # connection pool size; default: 10
DATABASE_IDLE_TIMEOUT_SECONDS=60 # close idle pooled connections; default: 60 (0 = keep open)
DATABASE_CONNECT_TIMEOUT_SECONDS=10 # default: 30
@@ -313,6 +473,70 @@ DATABASE_MAX_LIFETIME_SECONDS=1800 # recycle a pooled connection after this lo
DATABASE_APPLICATION_NAME=paperclip # application_name in pg_stat_activity; default: paperclip
```
An explicit `DATABASE_PREPARED_STATEMENTS=true` enables named prepared statements
for parameterized ORM queries, transactions, reserved connections, and dedicated
clients. Postgres.js manages the statement cache. This does not cache query
results. Bound values and returned data stay fresh. Explicit per-query options
still apply, including `prepare: false` and simple-protocol execution. With the
setting unset, ORM queries keep their existing unprepared behavior and tagged
SQL uses the driver default. With `false`, preparation stays disabled.
### Bounded response diagnostics
`PAPERCLIP_DB_RESPONSE_CAPTURE_WINDOW=<start>/<end>` enables one explicit UTC
window. Both timestamps use ISO format ending in `Z`. The window lasts at most
60 seconds and starts within 30 minutes. Unset, invalid, and expired windows
leave database construction unchanged. An expired window stays inactive after a
restart; remove the setting after the investigation.
The diagnostic measures awaited Postgres.js `unsafe` responses before Drizzle
column decoding. It retains the Drizzle builder caller for normal, prepared,
transaction, and dedicated-pool execution. It issues no queries.
It logs the public server caller locations, static schema table, operation,
query/row counts, and response value sizes. It emits at most 2,048 aggregate
groups and a summary. Measurement stops at the deadline. Environment-enabled
captures emit at most 64 records per second and can take another 33 seconds to
finish writing the bounded output. Wait for the summary before collecting it.
It never logs SQL, parameters, result
values, connection strings, or query errors. In-flight queries continue normally.
For a bundled server, keep the linked source maps and enable Node's
`--enable-source-maps` option so the diagnostic can resolve public caller
locations. Restore the prior Node options after capture. Sizes still count when
caller locations are unavailable.
Look for `paperclip.db_response_capture.entry.v1` and
`paperclip.db_response_capture.summary.v1` in the process logs. A log provider
can store `event` as a structured field instead of message text. Filter by that
field or read the bounded window from the exact deployment. Check the summary
and group totals, including the unattributed overflow group.
Fetch a complete capture as follows:
1. Read the summary events from the exact deployment and window. Keep each
distinct `captureId` separately.
2. Each entry has a unique `groupIndex` within that capture and a `groupPage`.
The summary gives `groupPageSize` (128) and `groupPages`. Fetch every page
from zero through `groupPages - 1`, filtering on the entry event, `captureId`,
and `groupPage`. Request at least `groupPageSize` matching records per page.
A structured query can use
`@event:paperclip.db_response_capture.entry.v1 AND @captureId:<id> AND @groupPage:0`.
3. If the provider's record limit is smaller than a page, fetch each
`groupIndex` separately with the same capture ID. Each index matches one
entry. Do not split by timestamp: entries can share a timestamp.
4. Deduplicate by `(captureId, groupIndex)`. Require every index from zero
through `groups - 1`, with no gaps. Entry count must equal `groups`, summed
`calls` must equal `completedQueries`, and summed `responseValueBytes` must
equal the summary total. A missing page is an incomplete measurement.
Tagged raw-driver calls, streams, unawaited raw-driver
dispatch, and protocol overhead are not measured. Relational queries use a driver-level caller fallback.
Queries still in flight at the deadline are reported separately. Sizes describe
driver values before column decoding, not billed network bytes. Complex parsed
values use JSON size estimates, counted in `estimatedValues`. Reconcile them with
provider metering and socket measurements. Serialization adds temporary CPU and memory work;
measure the final transfer rate after the window expires.
### Push the schema
```sh
@@ -402,6 +626,13 @@ bindings, selectors, exclusions, and installed connections on every call. The
public effective-profile response continues to include complete tool schemas
and annotations.
Connection readiness checks use a scoped assignment reader. It applies the same
current profile precedence, selectors, and exclusions to the candidate connection
IDs. The reader selects rule match fields and catalog identities. It does not
load rule conditions, catalog schemas, profile summaries, or the company-agent
list. Installed connections and later identity and gateway checks remain fresh.
Full effective-profile and effective-tool responses keep their complete shape.
## Execution identity row locks
Identity initialization, credential acquisition, and steering reconciliation lock
@@ -741,6 +972,19 @@ pnpm secrets:migrate-inline-env --apply
Hosted AWS provider notes live in [SECRETS-AWS-PROVIDER.md](./SECRETS-AWS-PROVIDER.md).
Sandbox runtime configuration resolves a provider's secret-reference fields
through its schema-only lookup. Each lookup reads the selected plugin ID
and a SHA-256 hash of its current schema in one database statement. A matching
retained schema is omitted from the query result. Selection order, schema
changes, and deletion remain visible on the next lookup, without a time-based
cache. A database error still fails the lookup.
The retained schemas are private to one `Db` object, with at most 32 entries of
64 KiB each in serialized form. Larger schemas still work and are read in full.
Callers receive independent schema objects. This reuse stores neither resolved
secrets nor authorization decisions. Default driver lookups still read complete
current plugin records and enforce their existing readiness and worker checks.
### Persistent agent conversations
Migration `0274_agent_chat.sql` adds conversation identity/state and session generation/boundary columns to `issues`, plus idempotent client request IDs and processed session-boundary generations to `issue_comments`. The company/agent/user unique index resolves concurrent first writes to one issue. A check constraint preserves the assigned-agent identity and prevents terminal conversation status. Comment request IDs are unique per issue and user. There is no separate chat/message store. Provider sessions continue to use `agent_task_sessions`; `/new` removes only the matching conversation session, and session writers fence stale generations against the issue row.
+24
View File
@@ -907,6 +907,15 @@ can update an entry only after the current account's summary confirms access.
The first healthy live subscription reconciles domain data without re-fetching
session and health; reconnects and visibility recovery retain full reconciliation.
### Run history authorization reads
Run history reuses the authorization service's batch read API for distinct task
bindings. Existence checks remain company-scoped and use batches of at most 500
IDs. Missing or foreign tasks stay unreadable in every privacy mode. The batch
snapshot lasts for one request only; the next request checks current grants,
membership, and run restrictions again. Adapters that expose only `decide`
retain the individual-read path with duplicate bindings shared within the request.
### Local Linux runner qualification
Before serving requests, the server reads `getconf GNU_LIBC_VERSION` once with
@@ -1807,11 +1816,21 @@ dispatch with cancellation and execution, and re-exports the existing
public helpers and configuration-incomplete error class. Keep preparation policy
changes in this module and its tests.
Queued-comment display reads select only wake identity, queue state, typed comment
and interaction bindings, the displayed wait reason, and fresh-session flags.
They read only the queued run's runtime mode. Mutation locks retain complete
wake and run rows, so edits, ordering, interruption, and saved execution inputs
keep their existing write contract.
Run retrieval and session state are in `server/src/services/heartbeat/run-state.ts`.
It owns bounded run projections, database encoding checks, task session reads and
writes, explicit resumes, session compaction, and usage/billing helpers.
`createHeartbeatRunState(db)` binds these operations without doing database work
during construction. The encoding-check cache belongs to each factory instance.
Queued-comment polling reads current issue control fields, the typed issue binding,
and the selected adapter type. Mutation and execution paths retain their full run reads.
Workspace-operation polling reads the current run access scope and typed workspace
binding, then applies the existing run and operation permission checks.
`heartbeat.ts` keeps run execution and re-exports the
existing public helpers. Keep session policy changes separate from run
orchestration changes.
@@ -2608,6 +2627,11 @@ Environment overrides:
negative, empty, or non-numeric value falls back to the default.
`PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_ENABLED=false` turns expiry off, and
`PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_DRY_RUN=true` only reports candidates.
- `PAPERCLIP_NATIVE_RUNNER_EVIDENCE_RETENTION_DAYS=<days>` sets how long native
session continuity-break archives and quarantined session state are kept (see
`doc/recoverable-disk-cleanup.md`). The default is `7` and the minimum is `1`. A
negative, empty, or non-numeric value falls back to the default. The backup
expiration enabled and dry-run switches apply to it as well.
- `PAPERCLIP_NATIVE_WORKSPACE_SYNC_RETENTION_DAYS=<days>` sets how long a settled
run's native workspace sync folder (descriptor, manifest and seed archives under
`<instance>/native-workspace-sync/<run>`) is kept (see
+22
View File
@@ -891,6 +891,21 @@ grant privileged tools. Backlog assignments retain the existing human requester
without starting a run. Reassignment transactionally cancels prior human requests,
including service/plugin writes; cancelled runs cannot authorize later retries. See `doc/LOW-TRUST-PRESETS.md` for containment details.
Project or repository access is a separate boundary from exact-task direction.
A workspace project denial appears on the task with a link to the agent's
Permissions / Trust settings. Board operators with `agents:create` can add
same-company projects through `POST /api/agents/:id/project-access`; the operation
locks the agent, preserves existing permissions and audits added projects.
Saving does not retry work. The task offers explicit Retry after the required
projects are allowed, and dispatch rechecks authorization. Readiness checks the
task's current organizational project and selected workspace source, including
a valid pending selection. Moving the task to an allowed project does not clear
a retained source's access requirement. Existing setup failures with the exact
project-boundary denial also receive the explanation on reload, without assuming
that the task's current project caused the historical failure. A stale or
unavailable workspace selection stays in the non-retryable failure state; it
does not ask the user to grant project access that cannot repair the selection.
Cross-issue writes are contained per heartbeat run. An agent-authored comment
may wake the target assignee, including an explicit `resume: true` comment on a
`done` or `cancelled` issue, but the wake remains agent-class and is subject to
@@ -1887,6 +1902,13 @@ Required UX behaviors:
- conflict toasts on atomic checkout failure
- no silent background failures; every failed run visible in UI
Cold opens of ordinary tasks show the instruction and recent conversation before
related metadata and historical run activity. Earlier messages and diagnostics
load through **More messages**. Messages observed while the task stays open remain
visible; deep links still load their target. Compact reads retain the full reads'
authorization and redaction rules. Agent Chat and the classic task interface keep
their existing history behavior.
Regular task chats and Agent Chat keep requests as compact, reopenable entries
at their original position in the feed. Dismissing any composer request clears
its form and pending reminder, including completion reviews, confirmations,
@@ -374,3 +374,27 @@ and records a recovery action for the reviewer without an automatic retry loop.
Both Codex and ACPX providers wait for server completion feedback before their
terminal tool call succeeds. A rejected completion report stays correctable in
the same provider turn. The runner does not latch it as an accepted result.
## Completed runs with saved monitors
A successful accepted result can report a monitor wait after the run has
already released execution and checkout. Finalization can preserve that wait
when the same agent still owns a valid ordinary task monitor and the completed
run has a server-recorded `set_task_monitor` receipt for the task. The current
monitor can have a later deadline; finalization does not restore an older timer.
The server rechecks the accepted source, current monitor, empty execution and
checkout slots, and governance under the coordinator and task locks. The
`scheduled_monitor_preserved` decision keeps task status and has no task effects.
It records the completed result without dispatching another provider turn.
The existing monitor scheduler receives a transaction-aware notification when
this finalization blocker clears. Replaying the committed decision sends no
second notification.
Live response selection and committed response repair retain the accepted
monitor explanation under the existing reply and publication authorization rules.
Missing receipts, changed ownership, cancellation intents, cleared or quota
monitors, and unfinished workspace finalization retain their recovery paths.
Due and exhausted monitors still belong to the scheduler. This does not change
their deadlines or attempts, clear unrelated recovery holds, or relax the
Cloud sleep and wake contract.
+7
View File
@@ -40,6 +40,13 @@ Installation may return before GitHub's separately delivered signed webhook
ping. Paperclip keeps the same draft and credentials, shows a waiting state,
and continues automatically after authenticating the ping. Refreshing or
resuming that draft must not create another App or bypass signature verification.
While this Cloud setup screen is waiting, its authenticated advance request also
drains the existing sealed event queue. It can bypass an older idle-connection
backoff, at most once every five seconds. Setup and background requests share
one in-flight lease. The normal stack webhook handler still authenticates the
original GitHub bytes before setup can continue. This adds no background timer;
closing the screen stops these readiness requests. Existing background recovery
and its sleep restrictions remain unchanged.
New connections default to authorized mentions, advisory reviews, linked-member
access, guests off, and enabled bot GitHub tools. Existing instructions, explicit
+6
View File
@@ -1,5 +1,11 @@
# Paperclip evaluation guide
The explicit-only [Daytona restart continuity journey](../tests/runner-e2e/README.md)
extends the existing native restart oracle to a 125-second controller outage,
using the recorded Claude and Codex models in disposable development fixtures.
It requires same-process adoption, saved files and finalization without replay;
local regression checks and catalog discovery are separate from live acceptance.
The explicit-only [GitHub review bot journey](../tests/runner-e2e/GITHUB-REVIEW-JOURNEY.md)
converts the Ponytail walkthrough into attended App setup, scoped installation,
automatic PR review, and independent two-service checks. It supports attached
+6
View File
@@ -14,6 +14,12 @@ Delegated work and interactions persist their originating context. Retries retai
Executions with managed GitHub configured receive token-free `git` and `gh` launchers. Each launcher invocation resolves one eligible credential from the current run's accepted identity at operation start. A `gh` command's child Git processes inherit that command's captured identity; later steering does not change already-started operations.
GitHub credential capture reads the current task, project, and retained trust
policy bindings without transferring saved prompts, runner profiles, results,
or log excerpts. It keeps the existing task/run lock order and accepted-identity
reconciliation. Other identity callers retain their complete execution input,
including the native input required by project creation authorization.
Native runners retain a session-owned broker and launcher path across warm turns. The provider keeps an opaque transport token, not a GitHub credential or the previous run's signed capability. After acquiring exclusive session ownership, the controller binds the broker to the current company, agent, task, and run. Requests cannot choose another run or responsible person. The broker rejects requests while idle and discards credential responses if their run binding changed during acquisition. Each operation still rechecks the live run, accepted identity, grants, and trust policy through the shared credential resolver. Changing run IDs alone no longer replaces the provider process; changes to authentication mode, provider credentials, permissions, or other session configuration retain their existing retirement rules.
The session broker listens only on controller loopback and accepts only its authenticated GitHub credential operation. Remote executions reach it through the existing authenticated callback bridge. Its transport and launchers are retired with the provider session. If remote bridge startup fails, anonymous launchers keep ordinary work available; the next run retries setup with a fresh session. Controller restart/cold recovery still uses the existing checkpoint and process-recovery rules; the broker's in-memory authority is not persisted for adoption.
+41 -1
View File
@@ -1676,6 +1676,30 @@ waiting for provider startup. An abrupt controller exit during session opening
can then recover through the same exact process-identity checks as an active turn.
Both graceful and hot restarts detach the old controller from native sessions.
Remote Runner processes use their authenticated connection lease as the bound
on disconnected lifetime. The controller's shorter connection-recovery budget
limits its own attempt; it must not terminate the remote owner while another
controller is starting or classifying runs. Lease expiry still shuts down the
provider, and exact identity, authentication, Stop and revocation checks remain
required before any recovery or new execution.
Native ACPX semantic tool calls wait on their durable PRP invocation across a
controller outage. Their provider-side MCP bridge does not impose a separate
request deadline that could make the provider retry an already queued write.
Runner lease expiry, turn cancellation and bridge closure still end those waits.
Ordinary bridge callers retain their request deadline. Repeating the same call
identity returns its one retained result; it does not execute the operation again.
The native Claude sidecar also sets its HTTP MCP client budget to one hour,
matching the Runner connection lease budget. This prevents Claude's default
60-second HTTP deadline from aborting an otherwise pending durable request.
The setting applies to that native Claude session's MCP client; other provider
sessions and ordinary bridge deadlines remain unchanged. Explicit cancellation
and Runner shutdown still take precedence over that client budget.
An adopted transport can receive authenticated queued semantic calls before
the driver publishes its recovered session and tool handler. It holds those
calls until the handler is installed, then rechecks the original authority and
turn epoch before dispatch. Transport failure, close or controller detachment
ends that wait without invoking a late handler. Missing startup wiring is not
reported to the provider as the result of an external operation.
If shutdown begins while a provider session is opening, its eventual publication
honors the pending detachment before dispatching a turn. Once detached, an old
execution finalizer cannot suspend or signal the durable runner: the next
@@ -1683,6 +1707,18 @@ controller must recover it through the authenticated ownership checks. This
preserves active work and queued messages without treating a server restart as
user cancellation.
During controller shutdown, already registered provider startups share one
30-second sweep deadline to finish read-only run admission and publish their exact
sessions for detachment. Its company, run, session, runner and completion-contract binding
must still match persisted authority. The draining controller cannot checkpoint,
publish events, submit a turn or settle the run. A new startup after the shutdown
fence, or admission after the window expires, remains rejected. A Stop pending
before or after session publication keeps cancellation and cleanup with the
current owner. Shutdown waits for that owner only until the same sweep deadline;
it does not detach the session or submit a prompt while Stop is pending. An
unsettled owner remains explicitly pending for reconciliation. Its timeout does
not abort detachment of healthy sessions or trigger a global graceful drain.
Recovery restores a structured result proposal from the durable run events when
its append survived but the active turn's session checkpoint did not. Advancing
the replay cursor must preserve that proposal for the exact run, session, source,
@@ -1807,7 +1843,11 @@ experimental isolated-workspaces UI. Ordinary workspace updates remain gated;
the runtime can bind only a validated shared workspace in the issue's company
and project. Follow-ups can therefore reuse the same sandbox and provider
session. A staged provider package is reused only after the complete expected
manifest and artifact hashes verify. A missing, changed, or incompatible package
manifest and artifact hashes verify. Exact live-process reattachment uses the
controller-owned pack and runner identities recorded after the original session
passed authentication. It preserves protocol and capability checks. Fresh or
replacement launches use current release authority. A missing historical receipt
retains the current release gate; guest files cannot establish old authority. A missing, changed, or incompatible package
must be replaced and verified before launch.
Warm attachment requires two consecutive authenticated readiness snapshots.
+74 -1
View File
@@ -1,5 +1,15 @@
# Native workspace finalization ownership and recovery
Periodic heartbeat recovery runs one pass at a time in each server process.
A tick that arrives during a slow pass reports its age and starts no duplicate.
A later tick can start a fresh pass after the current work settles. Shutdown
and idle sleep still track the actual recovery promise through cleanup. Direct
run operations and scheduled work delivery keep their existing paths. A pass
that remains active for three scheduler intervals produces an error log with its
age. A waiting timeout does not admit another pass: the actual work must settle
before ownership is released. A stuck operation requires investigation and
recovery at the operation or provider boundary.
A native provider can fail before it submits a result while its workspace still
contains unpublished edits. A prepared workspace reference is not an export
receipt. Terminal provider failure pins the exact bound sandbox and records a
@@ -9,6 +19,17 @@ cleanup run. This includes descriptor reads that fail inside runtime selection.
preparation failure; another owner or a stale claim prevents cleanup. Settling
that claim and recording file retention share a transaction.
The reconciliation candidate scan selects the complete workspace-sync record and
the native status wake identifier from each saved run. It keeps their JSON types
and reads full evidence later where finalization or recovery needs it. Saved
execution input, runner profiles, and logs stay in the stored run.
Wake-queue release reads keep complete context and result receipts. Native runs
omit runner profiles and log excerpts from this read; cleanup and execution
read their own current evidence. Legacy runs retain the full release row so
workspace-restore and pre-dispatch proofs keep their existing inputs. The
selection is one scoped database query for either runtime.
The recovery card names the original run and requires evidence that its exact
allocation stopped and its files were recovered. Recording that evidence does
not retry the old turn, certify unknown external actions, change the current task
@@ -371,6 +392,26 @@ and operator guidance remain under Recovery details. Source-only repair remains
available through the board-authorized recovery resolution API and its original
stop-receipt and evidence checks.
After a remote provider pack passes full integrity verification and the session
passes authenticated admission, the controller saves its manifest and runner
binary identity in `provider-pack-authority.json`. The receipt binds the company,
agent, workspace, session, runner instance, environment lease, remote working
directory, and provider descriptor. It stays in private controller state, outside
guest workspace and harness backup directories. Projectless runs use their
original environment lease as the workspace identity because each run otherwise
has a new workspace placeholder. Real workspace identities still require an exact
match. A receipt write failure warns and returns the admitted session to its
normal owner; it does not discard the live session handle. Future recovery still
fails closed without matching authority.
Exact process reattachment uses that saved release authority. It still verifies
the full remote pack, runner protocol and capabilities, exact process identity,
and authenticated PRP handshake. A new controller release alone does not require
a live provider to change its executable files. Fresh launches and explicit
session replacement select current release artifacts. Invalid or mismatched
receipts fail closed. Older sessions without a receipt retain the strict current
release gate; a sandbox's own manifest cannot supply missing launch authority.
Remote provider-pack verification retries an interrupted read at most once.
Each command retains its 30-second transport timeout. A timeout, signal, absent
exit status, or transport exception does not prove artifact incompatibility.
@@ -380,7 +421,9 @@ a new provider attempt, so it cannot renew this exhausted read budget. A
completed nonzero verification, digest mismatch, or
wrong OpenCode version remains incompatible and cannot authorize a new provider
turn. Recovery retains the exact source workspace and existing ownership gates;
these read-only retries do not replay tools or alter a retained pack.
these read-only retries do not replay tools or alter a retained pack. Verification
errors include bounded, redacted output from both transport streams, including
providers that combine stderr into stdout.
### Continue after confirmed source destruction
@@ -443,3 +486,33 @@ For a cancelled native run, the file-recovery form also offers an optional
confirmation that no external actions were performed. Leave it unchecked unless
the run history verifies that fact. The default records files without certifying
action outcomes. Changing the selected source clears the evidence and confirmations.
Committed chat response authorization reads current run identity, completion
binding, and chat source fields under the existing nonblocking share lock.
It excludes saved execution input, runner profiles, and log excerpts. Actual
retry admission still reads the complete run and its recovery evidence. These
reads preserve stored run data, source revocation checks, and JSON value types.
External review-wait discovery selects only chat authorization context fields.
It keeps the current endpoint, membership, resource, and question-receipt checks;
saved prompts remain stored and do not cross this discovery query.
Legacy continuation recovery rejects native runs before it loads their saved
execution input. The persisted disposition check reads only the task binding
for a native run. It keeps the same task-binding and native-finalization skip
reasons, including the original nonblank binding values. Native input and history
stay stored. Real legacy recovery still reads
its complete context, profile, excerpts, and required failure evidence.
The native Stop continuation recheck reads current run identity, the full Stop
acknowledgement receipt, and process identity. It does not load saved execution
input, runner profiles, excerpts, or unrelated result fields. Lease cleanup,
remote termination receipts, local process checks, task ownership, and admission
checks keep their existing behavior. Stored runs retain all of their data.
One periodic stranded-message pass coalesces successful native release checks
for the same company, task, source run, and current source row version. The release
path already drains that
task's queue through current admission checks. Each saved request still advances
its recovery cursor. Rejected attempts, changed source runs, and updated source evidence remain eligible,
and the next pass checks current state again. Legacy per-request handling stays
unchanged. Pending work is awaited before a source is marked checked.
+46
View File
@@ -0,0 +1,46 @@
# Task cold-open performance
Branch: `codex/task-cold-open`. Owner: task detail UI and existing issue/run read services.
Endpoint: tested, reviewable PR; deployment and staging acceptance are separate.
Cold opens show the instruction and recent exchange without hydrating historical
run diagnostics. Full issue enrichment stays on its existing authorized route.
The core view skips enrichment. Recent comments retain the latest human input
and four recent replies, plus the first human instruction when no description
exists. Recent activity preserves queue/steering metadata. Recent run summaries
select the latest run; conversation event reads exclude tools and traces.
Original comment/steering facts are read separately for visible comments, with
at most two facts per comment and 100 comment IDs per request. A reconnect that
leaves a watched unfinished run outside the recent window reconciles run summaries.
Core snapshots share task-detail mutation updates, rollback and invalidation.
The browser retains messages and runs observed while open. New native runs use
the normal event path. Expansion fetches full history, preserves existing rows,
and remains expanded on return in the session. Deep links and classic/Agent Chat
retain their full-history behavior. No database migration or permissions change.
| Case | Evidence |
| --- | --- |
| Cold task | Page tests hold enrichment/history pending while recent messages render; desktop/mobile thread tests keep content and composer visible. |
| Long history | Disposable app with 80 comments and six large run histories; cold network uses recent comments, recent runs, and one conversation-events request. Expansion reveals older comments. |
| Active page | Burst catch-up tests retain 60 messages across cursor pages; new-run tests retain previously visible runs. |
| Native history | Real PostgreSQL test filters tool payloads, preserves request lifecycles and message deltas, and pages by sequence. Expansion test resets cursor without duplicates. |
| Access | Core and preview route tests reject a foreign authenticated tenant. |
| Compatibility | Existing task UI, full event reads, and full activity/run reads remain covered. |
Measurements on the disposable local fixture: core read 50 ms; recent comments
24 ms; recent run summaries 17 ms. Latest native conversation events: 839 JSON
characters / 8 ms versus full latest-run events: 1,099,005 / 86 ms. These are local
HTTP samples, not staging latency promises. Browser navigation showed the initial
content paint at 1.34 seconds, with development assets already cached.
Verification: affected UI/API tests, real PostgreSQL tests, workspace typecheck,
production build, token gates, and map checks. The full suite was attempted but
this host exhausts macOS PostgreSQL semaphores. Targeted database tests were run
against a separate disposable database on an existing local server; the temporary
test setup override was restored before typecheck/build.
Related open PRs: #14667 changes the reveal gate and scroll anchors; #16229 adds
general history pagination. This change adds a core read and a folded cold-open
projection, while preserving the default full API contracts. Coordinate those
shared files if either PR lands first.
+33
View File
@@ -89,10 +89,43 @@ The rest of the session directory, including its control-plane state, is kept.
| `PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_ENABLED` | `true` | Set `false` to stop expiring copies |
| `PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_DRY_RUN` | `false` | Set `true` to report candidates without deleting copies or task sessions |
| `PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_DAYS` | `7` | Days an issue must be closed before its sessions' copies expire (minimum `1`) |
| `PAPERCLIP_NATIVE_RUNNER_EVIDENCE_RETENTION_DAYS` | `7` | Days continuity-break archives and quarantined session state are kept (minimum `1`) |
Each batch that finds candidates logs them with candidate and reclaimed allocated
bytes, cleared task sessions, and skip reasons.
### Continuity-break archives and quarantined session state
The same sweep also removes two kinds of evidence that native sessions leave behind
under the same root:
- `<session>/continuity-breaks/<epoch ms>-<UUID>/`, written when a session starts a
fresh provider session and moves the retired session's state aside. Nothing reads
these archives again.
- `quarantine/<session>.<reason>.<epoch ms>.<pid>.<UUID>/`, written when a session's
state loses authority. Recovery and failed-run retry still consult these entries.
An entry's age comes from the epoch in its name, or from its modification time when
that epoch is in the future. An entry is removed once it is older than
`PAPERCLIP_NATIVE_RUNNER_EVIDENCE_RETENTION_DAYS` and its session is not executing,
initializing, warm, or closing in this process. Quarantine entries created by
native cleanup (`<session>.identity_indeterminate.cleanup.<UUID>`) are referenced
from recovery history and kept. Symlinks are never followed or removed. Once a
quarantined entry is removed, recovery and failed-run retry no longer see it, so a
later run of that session may start a fresh provider session instead of refusing.
The evidence check runs on the sweep only when something is due: once at startup,
then when the oldest retained entry, or an entry the executor has written since,
reaches the window, and on the next sweep after a busy session, a failed removal,
or a full batch. It only reads the filesystem and never queries the
database. A dry run reports each candidate once until newly written evidence comes due.
An entry is renamed to `.expired-<UUID>` in its folder before it is deleted, and a
later sweep finishes an interrupted deletion. An empty `continuity-breaks` folder
is removed. Archives, quarantine entries, and leftovers count toward the batch
limits of 200 entries and 50 removals shared with backup copies, and
`PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_ENABLED` and `_DRY_RUN` apply to both.
## Native workspace sync folders
Each native run that syncs a workspace with a remote sandbox writes a folder at
+9 -2
View File
@@ -56,12 +56,19 @@ For production, use a hosted provider like [Supabase](https://supabase.com/).
Use the **direct connection** (port 5432) for migrations and the **pooled connection** (port 6543) for the application.
If using connection pooling (transaction mode), disable prepared statements via the environment — no source edits needed:
If your transaction-mode pooler does not support protocol-level named prepared statements, disable them via the environment:
```sh
DATABASE_PREPARED_STATEMENTS=false
```
For a direct connection or a pooler that supports named prepared statements,
`DATABASE_PREPARED_STATEMENTS=true` also prepares parameterized ORM queries and
queries inside transactions and reserved or dedicated connections. Check the
provider's configuration before you enable it. Preparation caches statement
metadata, not query results. Unset preserves the existing ORM behavior and the
driver default for tagged SQL. Explicit per-query options still apply.
Related optional client tuning: `DATABASE_POOL_MAX`, `DATABASE_IDLE_TIMEOUT_SECONDS`, `DATABASE_CONNECT_TIMEOUT_SECONDS`, `DATABASE_MAX_LIFETIME_SECONDS`, `DATABASE_APPLICATION_NAME`. Driver defaults apply when unset, except that idle pooled connections close after 60 seconds (`DATABASE_IDLE_TIMEOUT_SECONDS=0` keeps them open) and the pool reports `application_name=paperclip`. See [Connection pool settings](#connection-pool-settings).
## Connection Pool Settings
@@ -75,7 +82,7 @@ The server opens one postgres.js pool for its own queries (and a second one when
| `DATABASE_CONNECT_TIMEOUT_SECONDS` | `30` (driver) | Give up on a connection attempt after this long. |
| `DATABASE_MAX_LIFETIME_SECONDS` | 30–60 min, randomized (driver) | Recycle a pooled connection once it is this old. |
| `DATABASE_APPLICATION_NAME` | `paperclip` | Value of `application_name` in `pg_stat_activity`, so you can find Paperclip's backends: `SELECT * FROM pg_stat_activity WHERE application_name = 'paperclip';` |
| `DATABASE_PREPARED_STATEMENTS` | `true` (driver) | Set `false` behind a transaction-mode pooler (see above). |
| `DATABASE_PREPARED_STATEMENTS` | Unset | `true` opts parameterized ORM queries into preparation. `false` disables preparation. Use `false` when the pooler does not support named statements. Unset preserves existing behavior. |
The server ends its pools during shutdown (SIGINT/SIGTERM) and when startup fails after the pool was opened, so a restarting server does not leave idle backends behind. Size `max_connections` on the PostgreSQL side for at least `DATABASE_POOL_MAX` per server process plus your other clients. Each server process also opens up to 2 connections (`application_name=paperclip-controller-lease`) for agent-run lease renewals, so they never queue behind slow application queries; count those too.
+6
View File
@@ -104,6 +104,12 @@ Review: `baseline`.
API: `present` — representative operation IDs:
- `GET /api/agents/{id}/instructions-bundle/file`.
- `GET /api/agents/{id}/project-access` and `POST /api/agents/{id}/project-access`
are board-only. The POST requires `agents:create`, checks selected projects in
the same company, and adds IDs under an agent row lock. It preserves existing
containment, audits new grants and rejects pending-approval agents. Agent keys
cannot use this route to grant themselves access; a read-only board viewer
receives `canManage: false`. Saving never creates a run or wakeup.
- `PUT /api/agents/{id}/instructions-bundle/file`.
- `PATCH /api/agents/{id}` with an explicit `runner: "paperclip"` upgrade.
- `POST /api/agents/{id}/qualify-runner` to qualify the saved effective sandbox before upgrading.
+2 -2
View File
@@ -209,8 +209,8 @@ Live latency verification remains separate from the existing access baseline.
API: `present` — representative operation IDs:
- `GET /api/heartbeat-runs/{runId}`.
- `GET /api/heartbeat-runs/{runId}/log`.
- `GET /api/heartbeat-runs/{runId}/events`.
- `GET /api/heartbeat-runs/{runId}/log` (optional `alignStartToRecord=true` extends a tail backward to its first complete record; this can exceed `limitBytes` by that record's prefix and retains the same access/redaction checks).
- `GET /api/heartbeat-runs/{runId}/events` (optional `view=conversation` returns assistant messages and request lifecycles through the same access/redaction checks; `afterSeq` remains a sequence cursor).
- `GET /api/heartbeat-runs/{runId}/workspace-operations`.
- `GET /api/heartbeat-runs/{runId}/issues`.
- `GET /api/companies/{companyId}/heartbeat-runs`.
+5 -1
View File
@@ -27,7 +27,11 @@ API: `present` — representative operation IDs:
Agent access: `available` — Native create_task/reassign_task; REST / CLI / MCP.
Policy: Task visibility/mutation, task-assignment and ownership policy; review audience
independent; tree controls board-only.
independent; tree controls board-only. Resume preview counts only current agent
assignees on todo, in-progress, and in-review tasks as wake targets. Release keeps
blocked and backlog statuses intact; existing execution gates still apply.
Reopened tasks use current eligibility rather than the pause snapshot's skipped
flag. Other active pause holds and execution reconciliation gates remain effective.
Evidence: `source-only` —
[implementation](../../server/src/routes/issue-tree-control.ts#L95) at the reviewed
+5 -3
View File
@@ -84,7 +84,7 @@ Gaps: Bounded native search and one-page CLI local match are not complete compan
### `read-task-and-summaries`
API: `present` — GET /api/issues/{id}; GET /api/companies/{companyId}/issues/summaries?ids=... .
API: `present` — GET /api/issues/{id} (optional `view=core` omits related enrichment); GET /api/companies/{companyId}/issues/summaries?ids=... .
Agent access: `partial` — REST/CLI issue get; native get_task_context reads CURRENT task only; call_api for eligible arbitrary task/summaries. No dedicated summaries CLI/tool.
@@ -254,7 +254,7 @@ Read, append/retry, author-delete or queued-cancel messages, attach allowed file
### `read-history-comment`
API: `present` — GET /api/issues/{id}/comments and /comments/{commentId}.
API: `present` — GET /api/issues/{id}/comments and /comments/{commentId}; GET /api/issues/{id}/recent-comments returns four latest comments plus the latest human comment, a cursor, and `hasMore`. The compact read also includes the first human instruction when there is no description. Cold task views use `GET /api/issues/{id}/activity?view=recent` (50 rows) and `/runs?view=recent` (two summaries) without changing the default full-read contracts. It uses the same issue authorization and run/trust redaction.
Agent access: `available` — REST/call_api, CLI issue comments/comment:get; native get_task_history CURRENT task only, declared limit 1..200 but production clamps to 1..100 (default 50).
@@ -264,7 +264,9 @@ Evidence: `source-only` — reviewed at `3bbc2104b65caf9f7763938a01260a5673e12f7
Verification: `not-run` — Read multipage history and exact comment, reload same content; reject inaccessible task, foreign/mismatched comment and inspect trust redaction. Request native limit 200 and confirm at most 100 comments.
Gaps: Native bounded history is not an arbitrary-task/paginated API substitute.
The same authorized activity route supports `view=comment-inputs&commentIds=<comma-separated UUIDs>` for original comment/steering facts. Each request accepts at most 100 comment IDs and returns at most two facts per comment, scoped to the accessible issue and company. This preserves delivered-input state when its receipt is older than the recent activity window.
Gaps: Native bounded history is not an arbitrary-task/paginated API substitute. The new UI preview route has no dedicated native tool or CLI command. Existing full comment reads remain available.
### `append-retry-comment`
+6
View File
@@ -41,6 +41,12 @@ existing preparation and verification lifecycle.
Open `/agents/:agentId` and the configuration/instruction surfaces it exposes.
For a low-trust agent, open **Permissions / Trust → Allowed projects**. An operator
with agent-management permission can select additional projects and explicitly
save them. Existing projects and other containment restrictions remain intact.
Cancel discards the draft; a failed save retains it. Reload to verify persistence.
This does not start or retry a task.
In `/agents/all`, select roster checkboxes and choose **Edit connection & model**.
A mixed selection offers one action per harness. Each field is opt-in. The editor
previews each changed value and excludes agents already at the target value.
+9
View File
@@ -205,6 +205,15 @@ configuration changes, saved retry deadlines, rollback, and parking pending
repair while an endpoint or connection is inactive. The provider responses are
fixtures. Settled maintenance leaves no discovery timer or database scans.
GitHub Cloud setup also covers an older empty-queue backoff, concurrent setup and
background leases, and retry after a Cloud error in the
[event poller regression](../../server/src/services/github-connection-events.test.ts).
Channel integration checks that active setup drains its signed ping through the
normal webhook handler; a forged ping still cannot verify the endpoint. These
fixtures do not establish live Cloud delivery. For live proof, start a disposable
App after an idle personal GitHub connection, complete installation, and confirm
that the same draft advances after authenticating the queued ping.
Manual: complete setup for one named provider, reload endpoint settings, link
the intended sender, and send a disposable message. Verify one task and a useful
reply in the originating conversation. Inspect an unlinked sender's denial and
+20
View File
@@ -48,6 +48,18 @@ Open the affected task from Tasks, Inbox/Blocked, search, or a direct task link.
Observe one working, backoff, failed or waiting-for-user state. Use Retry after
an infrastructure failure. No process or file certification is required.
A workspace project denial names the blocked access and links to the agent's
**Permissions / Trust** settings. After an authorized person adds the required
project, return to the saved task for an explicit **Retry task** action. Saving
permissions alone does not wake the agent. Historical setup denials receive the
same notice on reload; unrelated errors and prior assignees do not.
Move a denied task to a project the agent can access while retaining a source
from another project: the notice must stay blocked until the selected source is
also allowed or an authorized workspace selection replaces it. A valid pending
selection is checked for the next admission; a stale or unavailable selection
must show a non-retryable failure without a project-permission repair action.
Project changes alone preserve selected files.
Thinking appears immediately after Retry, before the request completes.
Technical source/run details remain in deliberate diagnostics. The
[task recovery contract](../../doc/task-recovery.md) describes receipt identity,
@@ -170,6 +182,14 @@ and rejected identity evidence. The Runner restart tests require positive exit
evidence before replacement. This coverage does not prove recovery of a
historical retained sandbox or behavior during a live provider outage.
For a release change during a remote provider session, verify that reattachment
uses the controller's admitted pack manifest and runner digest. Check useful
output from the same process without replaying a turn. Repeat with a modified
sidecar, mismatched receipt scope, replacement lease, and incompatible runner
protocol. The [authority tests](../../server/src/services/native-runtime/remote-provider-pack-authority.test.ts)
and executor wiring regressions cover selection and fail-closed scope checks.
They do not prove a historical retained sandbox was recovered.
For completion recovery, restart after a native completion proposal is appended
but before its session checkpoint. Verify that the original result is finalized
after provider termination, with no new provider turn or repeated external action.
+7
View File
@@ -49,6 +49,13 @@ Automated: [execution policy](../../server/src/__tests__/issue-execution-policy.
Manual: Complete implementation under a review-required policy and confirm it waits for the required review outcome. Exercise rejection and later approval. Pause the parent while child work is pending, then verify resumption respects budget and ownership gates.
Resume removes the pause without changing task status. Its wake count includes only
current agent assignees on todo, in-progress, or in-review tasks. The dialog explains
that blocked, backlog, and unassigned tasks remain idle. Verify this in both task
interfaces with [the paused composer browser suite](../../tests/e2e/paused-composer.spec.ts).
Release rechecks tasks reopened after the pause; another active pause still prevents
their execution, and uncertain prior executions must be reconciled before waking.
## Gotchas
- Review completion and task completion are different events under some policies.
+4 -2
View File
@@ -39,7 +39,7 @@ Open `/issues` or a project’s Issues view; legacy `/tasks` links redirect.
### `task-detail`
Open `/issues/:issueId` from a row, mention, or direct link.
Open `/issues/:issueId` from a row, mention, or direct link. Cold opens show the instruction and recent exchange first. **More messages** loads earlier messages and run activity; incoming messages remain visible while the task stays open.
### `task-sharing`
@@ -87,7 +87,9 @@ Manual: Filter by status and assignee, change grouping/sort, open a task, and re
### `task-detail`
Automated: [task detail](../../ui/src/pages/IssueDetail.test.tsx) covers component behavior, while [issue service](../../server/src/__tests__/issues-service.test.ts) covers server contracts.
Manual: Open a long task with slow historical requests. Read the instruction and recent reply before those requests finish. Edit the title/status while enrichment is pending and verify saves and rollbacks. Expand **More messages**, verify scroll position and no duplicate replies, return to the task, and follow an older comment link. Leave a task open across new messages and runs and confirm none disappear. Reconnect after several turns and verify watched runs settle. Reopen after steering with more than 50 newer activity rows; the delivered message must not reappear in the queue.
Automated: [cold-open reads](../../server/src/__tests__/task-cold-open.test.ts) cover bounded reads, event filtering and company denial. [Task detail](../../ui/src/pages/IssueDetail.test.tsx) covers component behavior, while [issue service](../../server/src/__tests__/issues-service.test.ts) covers server contracts.
Failure markers have no dashed divider rules. An adjacent run error and
no-live-execution-path notice show only the last error, with its details and
+4 -7
View File
@@ -4,6 +4,8 @@ import { readFile, readdir } from "node:fs/promises";
import { fileURLToPath } from "node:url";
import postgres from "postgres";
import { installDatabaseWorkSignals } from "./work-signals.js";
import { installDatabaseResponseCapture } from "./response-capture.js";
import { installPreparedQueries } from "./prepared-queries.js";
import * as schema from "./schema/index.js";
const MIGRATIONS_FOLDER = fileURLToPath(new URL("./migrations", import.meta.url));
@@ -128,13 +130,6 @@ export type MigrationState =
};
export interface DatabaseClientOptions {
/**
* postgres.js `prepare`. Set false when connecting through a
* transaction-mode pooler (pgbouncer / Neon `-pooler` endpoints /
* Supabase Supavisor transaction ports) so the client does not rely on
* session-scoped prepared statements. Defaults to the driver default
* (enabled), preserving existing behavior on direct connections.
*/
prepare?: boolean;
/** postgres.js `max` — connection pool size (driver default: 10). */
maxConnections?: number;
@@ -300,6 +295,7 @@ export function createDb(url: string, options?: DatabaseClientOptions) {
function createDbClient(url: string, resolved: DatabaseClientOptions, owner?: object) {
const sql = postgres(url, postgresJsOptions(resolved));
if (resolved.prepare === true) installPreparedQueries(sql);
const key = hostPortKeyOrNull(url);
if (key) registerClient(key, sql);
// A disconnect can lose the response after a statement has committed.
@@ -307,6 +303,7 @@ function createDbClient(url: string, resolved: DatabaseClientOptions, owner?: ob
// message cannot establish that replay is safe. Leave retries to callers
// that know the complete operation is idempotent.
const db = installDatabaseWorkSignals(drizzlePg(sql, { schema }), owner);
installDatabaseResponseCapture(db, schema);
dedicatedDbFactories.set(db, () => createDbClient(url, {
...resolved, maxConnections: 1, applicationName: "paperclip-workspace-finalization-lock",
}, db));
@@ -0,0 +1,175 @@
import { eq, sql } from "drizzle-orm";
import { integer, jsonb, pgTable, text } from "drizzle-orm/pg-core";
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it } from "vitest";
import { closeLivenessDb, createDb, databaseClientOptionsFromEnv, livenessDb, withDedicatedDbConnection, type DatabaseClientOptions, type Db } from "./client.js";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./test-embedded-postgres.js";
import { runEventPayload } from "./run-event-payload.js";
const records = pgTable("prepared_query_fixture", { id: text("id").primaryKey(), companyId: text("company_id").notNull(), value: integer("value").notNull(), payload: jsonb("payload") });
const encoded = pgTable("prepared_encoded_fixture", { id: text("id").primaryKey(), payload: runEventPayload("payload") });
const support = await getEmbeddedPostgresTestSupport();
describe.skipIf(!support.supported)("explicit ORM prepared statements on PostgreSQL", () => {
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let setup: Db;
const clients: Db[] = [];
beforeAll(async () => {
temporary = await startEmbeddedPostgresTestDatabase("prepared-orm-");
setup = createDb(temporary.connectionString, { prepare: false, maxConnections: 1 });
await setup.execute(sql`create table prepared_query_fixture(id text primary key, company_id text not null, value integer not null, payload jsonb)`);
await setup.execute(sql`create table prepared_encoded_fixture(id text primary key, payload jsonb)`);
await setup.execute(sql`create table prepared_schema_change_fixture(id text primary key)`);
await setup.execute(sql`insert into prepared_schema_change_fixture(id) values ('retained')`);
}, 90_000);
beforeEach(async () => {
await setup.execute(sql`truncate prepared_query_fixture, prepared_encoded_fixture`);
await setup.insert(records).values([
{ id: "a", companyId: "company-a", value: 1, payload: { owner: "a" } },
{ id: "b", companyId: "company-b", value: 2, payload: { owner: "b" } },
]);
});
afterEach(async () => {
for (const db of clients.splice(0)) {
await closeLivenessDb(db, 2);
await db.$client.end();
}
});
afterAll(async () => { await setup?.$client.end(); await temporary?.cleanup(); });
function fixture(options: DatabaseClientOptions = {}) {
const db = createDb(temporary.connectionString, { ...options, maxConnections: 1 });
clients.push(db); return db;
}
async function statements(db: Db, pattern = '%"prepared_query_fixture"%') {
return db.$client.unsafe("select name,statement from pg_prepared_statements where statement like $1", [pattern]);
}
it.each([undefined, false])("keeps the existing ORM mode when prepare is %s", async prepare => {
const db = fixture({ prepare });
expect(await db.select().from(records).where(eq(records.companyId, "company-a"))).toEqual([
{ id: "a", companyId: "company-a", value: 1, payload: { owner: "a" } },
]);
expect(await statements(db)).toHaveLength(0);
});
it("honors the explicit environment opt-in for ORM queries with fresh company bindings", async () => {
const db = fixture(databaseClientOptionsFromEnv({ DATABASE_PREPARED_STATEMENTS: "true" }));
for (let i = 0; i < 12; i++) {
const companyId = i % 2 ? "company-b" : "company-a";
const rows = await db.select().from(records).where(eq(records.companyId, companyId));
expect(rows).toEqual([{ id: i % 2 ? "b" : "a", companyId, value: i % 2 ? 2 : 1, payload: { owner: i % 2 ? "b" : "a" } }]);
}
expect(await statements(db)).toHaveLength(1);
});
it("preserves explicit per-query opt-out", async () => {
const db = fixture({ prepare: true });
expect((await db.$client.unsafe("select $1::integer as explicit_opt_out_marker", [7], { prepare: false }))[0].explicit_opt_out_marker).toBe(7);
expect(await statements(db, "%explicit_opt_out_marker%")).toHaveLength(0);
});
it("preserves simple queries and the two-argument options overload", async () => {
const db = fixture({ prepare: true });
const simpleOptions = { prepare: true, simple: true };
const simple = await db.$client.unsafe("select 1 as first; select 2 as second", [], simpleOptions);
expect(simple).toEqual([[{ first: 1 }], [{ second: 2 }]]);
const unsafe = db.$client.unsafe as unknown as (query: string, options: { simple: boolean }) => Promise<unknown>;
expect(await unsafe("select 3 as third; select 4 as fourth", { simple: true }))
.toEqual([[{ third: 3 }], [{ fourth: 4 }]]);
});
it("does not replay a rejected prepared write", async () => {
const db = fixture({ prepare: true });
await expect(db.insert(records).values({ id: "a", companyId: "company-b", value: 9 }))
.rejects.toMatchObject({ cause: { code: "23505" } });
expect(await db.select({ companyId: records.companyId, value: records.value }).from(records).where(eq(records.id, "a")))
.toEqual([{ companyId: "company-a", value: 1 }]);
});
it("keeps write queries lazy and repeated awaits from repeating writes", async () => {
const db = fixture({ prepare: true });
const pending = db.$client.unsafe("insert into prepared_query_fixture(id,company_id,value) values ($1,$2,$3) returning id", ["lazy", "company-a", 3]);
expect(typeof pending.values).toBe("function");
expect(await setup.select().from(records).where(eq(records.id, "lazy"))).toEqual([]);
expect(await pending).toEqual([{ id: "lazy" }]);
expect(await pending).toEqual([{ id: "lazy" }]);
expect(await setup.select({ id: records.id }).from(records).where(eq(records.id, "lazy"))).toEqual([{ id: "lazy" }]);
expect(await statements(db, "%insert into prepared_query_fixture%")).toHaveLength(1);
});
it("preserves transaction commits, nested savepoint rollback, and scoped reads", async () => {
const db = fixture({ prepare: true });
await db.transaction(async tx => {
expect(await tx.select({ value: records.value }).from(records).where(eq(records.companyId, "company-a"))).toEqual([{ value: 1 }]);
await expect(tx.transaction(async nested => {
await nested.update(records).set({ value: 99 }).where(eq(records.companyId, "company-a"));
throw new Error("rollback savepoint");
})).rejects.toThrow("rollback savepoint");
expect(await tx.select({ value: records.value }).from(records).where(eq(records.companyId, "company-a"))).toEqual([{ value: 1 }]);
await tx.insert(records).values({ id: "committed", companyId: "company-b", value: 3 });
const names = await tx.execute(sql`select statement from pg_prepared_statements`);
expect(names.some(row => String(row.statement).includes('"prepared_query_fixture"'))).toBe(true);
}, { isolationLevel: "serializable" });
expect(await db.select({ value: records.value }).from(records).where(eq(records.id, "committed"))).toEqual([{ value: 3 }]);
});
it("rolls back failed outer transactions without replay", async () => {
const db = fixture({ prepare: true });
await expect(db.transaction(async tx => {
await tx.update(records).set({ value: 42 }).where(eq(records.id, "a"));
throw new Error("rollback outer transaction");
})).rejects.toThrow("rollback outer transaction");
expect(await db.select({ value: records.value }).from(records).where(eq(records.id, "a"))).toEqual([{ value: 1 }]);
});
it("prepares reserved-client queries and keeps their bound values fresh", async () => {
const db = fixture({ prepare: true });
const reserved = await db.$client.reserve();
try {
for (const marker of [1, 2, 3]) {
expect((await reserved.unsafe("select $1::integer as reserved_prepared_marker", [marker]))[0].reserved_prepared_marker).toBe(marker);
}
const names = await reserved.unsafe("select statement from pg_prepared_statements where statement like $1", ["%reserved_prepared_marker%"]);
expect(names).toHaveLength(1);
} finally { reserved.release(); }
});
it("preserves object and array results for the same prepared shape", async () => {
const db = fixture({ prepare: true });
expect(await db.$client.unsafe("select $1::integer as result_mode_marker", [41]).values()).toEqual([[41]]);
expect(await db.$client.unsafe("select $1::integer as result_mode_marker", [42])).toEqual([{ result_mode_marker: 42 }]);
});
it("keeps custom stored-payload decoding fresh", async () => {
const db = fixture({ prepare: true });
const first = { text: "encoded first payload🙂".repeat(2048) };
const second = { text: "encoded second payload🙂".repeat(2048) };
await setup.insert(encoded).values({ id: "encoded", payload: first });
expect((await db.select().from(encoded).where(eq(encoded.id, "encoded")))[0].payload).toEqual(first);
await setup.update(encoded).set({ payload: second }).where(eq(encoded.id, "encoded"));
expect((await db.select().from(encoded).where(eq(encoded.id, "encoded")))[0].payload).toEqual(second);
expect(await statements(db, '%"prepared_encoded_fixture"%')).toHaveLength(1);
});
it("inherits the opt-in in dedicated and liveness clients", async () => {
const db = fixture({ prepare: true });
await withDedicatedDbConnection(db, async dedicated => {
expect(await dedicated.select({ value: records.value }).from(records).where(eq(records.companyId, "company-b"))).toEqual([{ value: 2 }]);
expect(await statements(dedicated)).toHaveLength(1);
});
const live = livenessDb(db);
expect(await live.select({ value: records.value }).from(records).where(eq(records.companyId, "company-a"))).toEqual([{ value: 1 }]);
expect(await statements(live)).toHaveLength(1);
});
it("refreshes prepared result metadata after a schema change", async () => {
const db = fixture({ prepare: true });
const query = "select * from prepared_schema_change_fixture where id=$1";
expect(await db.$client.unsafe(query, ["retained"])).toEqual([{ id: "retained" }]);
try {
await setup.execute(sql`alter table prepared_schema_change_fixture add column extra text`);
expect(await db.$client.unsafe(query, ["retained"])).toEqual([{ id: "retained", extra: null }]);
} finally {
await setup.execute(sql`alter table prepared_schema_change_fixture drop column if exists extra`);
}
});
});
+47
View File
@@ -0,0 +1,47 @@
type Method = (this: unknown, ...args: unknown[]) => unknown;
export function installPreparedQueries(client: unknown): void {
const visited = new WeakSet<object>();
function visit(value: unknown): void {
if ((typeof value !== "object" || value === null) && typeof value !== "function") return;
if (visited.has(value)) return;
visited.add(value);
const target = value as Record<string, unknown>;
const unsafe = target.unsafe;
if (typeof unsafe === "function") {
target.unsafe = function (this: unknown, ...args: unknown[]) {
const index = args.length === 2 && !Array.isArray(args[1]) ? 1 : 2;
const options = args[index];
if (options !== undefined && (options === null || typeof options !== "object")) {
return Reflect.apply(unsafe, this, args);
}
if (index === 2 && args.length < 2) args[1] = [];
args[index] = { prepare: true, ...options };
return Reflect.apply(unsafe, this, args);
};
}
for (const method of ["begin", "savepoint"]) {
const original = target[method];
if (typeof original !== "function") continue;
target[method] = function (this: unknown, ...args: unknown[]) {
const wrapped = args.map(arg => typeof arg !== "function" ? arg : function (this: unknown, ...inner: unknown[]) {
visit(inner[0]);
return Reflect.apply(arg, this, inner);
});
return Reflect.apply(original, this, wrapped);
} satisfies Method;
}
const reserve = target.reserve;
if (typeof reserve === "function") {
target.reserve = function (this: unknown, ...args: unknown[]) {
return Promise.resolve(Reflect.apply(reserve, this, args)).then(reserved => {
visit(reserved);
return reserved;
});
} satisfies Method;
}
}
visit(client);
}
@@ -0,0 +1,204 @@
import { randomUUID } from "node:crypto";
import { compileFunction } from "node:vm";
import { eq, sql } from "drizzle-orm";
import { pgTable, text } from "drizzle-orm/pg-core";
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
import { createDb, withDedicatedDbConnection } from "./client.js";
import { createDatabaseResponseCapture, databaseResponseCaptureFromEnv, installDatabaseResponseCapture } from "./response-capture.js";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./test-embedded-postgres.js";
import { encodeRunEventPayload, runEventPayload } from "./run-event-payload.js";
import { databaseWorkPending, reconcileDatabaseWork, signalDatabaseWork, subscribeDatabaseWork } from "./work-signals.js";
const table = pgTable("response_capture_fixture", { id: text("id").primaryKey(), payload: text("payload").notNull() });
const encodedTable = pgTable("response_capture_encoded_fixture", { id: text("id").primaryKey(), payload: runEventPayload("payload") });
const support = await getEmbeddedPostgresTestSupport();
describe.skipIf(!support.supported)("database response capture PostgreSQL behavior", () => {
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
const clients: Array<ReturnType<typeof createDb>> = [];
const captures: Array<ReturnType<typeof createDatabaseResponseCapture>> = [];
beforeAll(async () => {
temporary = await startEmbeddedPostgresTestDatabase("response-capture-");
const db = createDb(temporary.connectionString);
try {
await db.execute(sql`create table response_capture_fixture (id text primary key, payload text not null)`);
await db.execute(sql`create table response_capture_encoded_fixture (id text primary key, payload jsonb not null)`);
}
finally { await db.$client.end(); }
}, 90_000);
afterEach(async () => {
for (const capture of captures.splice(0)) capture.flush();
for (const client of clients.splice(0)) await client.$client.end();
vi.restoreAllMocks(); vi.unstubAllEnvs();
});
afterAll(async () => { await temporary?.cleanup(); });
function fixture(prepare?: boolean) {
const events: Array<Record<string, unknown>> = [];
const now = Date.now();
const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event));
const db = createDb(temporary.connectionString, { prepare });
clients.push(db); captures.push(capture);
expect(installDatabaseResponseCapture(db, { table, encodedTable }, capture)).toBe(db);
expect(installDatabaseResponseCapture(db, { table, encodedTable }, capture)).toBe(db);
return { db, capture, events };
}
function summary(events: Array<Record<string, unknown>>) { return events.find(event => event.event === "paperclip.db_response_capture.summary.v1")!; }
it("retains capture counts when ORM preparation is enabled", async () => {
const f = fixture(true), id = randomUUID(), payload = "prepared-capture-value";
await f.db.insert(table).values({ id, payload });
await f.db.transaction(async tx => {
expect(await tx.select().from(table).where(eq(table.id, id))).toEqual([{ id, payload }]);
});
f.capture.flush();
expect(summary(f.events)).toMatchObject({ startedQueries: 2, completedQueries: 2 });
expect(JSON.stringify(f.events)).not.toContain(payload);
expect(JSON.stringify(f.events)).not.toContain(id);
});
it("keeps rows and builder identity through lazy select and write returning", async () => {
const f = fixture(), id = randomUUID(), payload = "sentinel-result-and-parameter-value🙂\n\"";
const insert = f.db.insert(table).values({ id, payload });
expect(insert.returning()).toBe(insert);
expect(await insert).toEqual([{ id, payload }]);
const query = f.db.select().from(table);
expect(query.where(eq(table.id, id))).toBe(query);
expect(await query).toEqual([{ id, payload }]);
const updated = await f.db.update(table).set({ payload: "changed" }).where(eq(table.id, id)).returning();
expect(updated).toEqual([{ id, payload: "changed" }]);
expect(await f.db.delete(table).where(eq(table.id, id)).returning()).toEqual([{ id, payload: "changed" }]);
f.capture.flush();
expect(summary(f.events)).toMatchObject({ startedQueries: 4, completedQueries: 4 });
expect(f.events.filter(event => event.event === "paperclip.db_response_capture.entry.v1").map(event => event.operation).sort())
.toEqual(["delete", "insert", "select", "update"]);
expect(JSON.stringify(f.events)).not.toContain(payload);
expect(JSON.stringify(f.events)).not.toContain(id);
});
it("counts the stored encoding before a custom column decoder changes the value", async () => {
const setup = createDb(temporary.connectionString); clients.push(setup);
const id = randomUUID(), payload = { text: "codec-payload-sentinel\u0000value".repeat(512) };
await setup.insert(encodedTable).values({ id, payload });
const f = fixture();
expect(await f.db.select().from(encodedTable).where(eq(encodedTable.id, id))).toEqual([{ id, payload }]);
f.capture.flush();
const entry = f.events.find(event => event.table === "response_capture_encoded_fixture")!;
expect(entry.responseValueBytes).toBe(Buffer.byteLength(id) + Buffer.byteLength(encodeRunEventPayload(payload)));
expect(entry.responseValueBytes).toBeGreaterThan(Buffer.byteLength(JSON.stringify({ id, payload })));
expect(JSON.stringify(f.events)).not.toContain("codec-payload-sentinel");
});
it("keeps a raw driver query lazy and counts repeated awaits without replay", async () => {
const f = fixture(), id = randomUUID();
const observer = createDb(temporary.connectionString); clients.push(observer);
const pending = f.db.$client.unsafe("insert into response_capture_fixture (id, payload) values ($1, $2) returning id", [id, "raw-lazy-sentinel"]);
expect(await observer.select().from(table).where(eq(table.id, id))).toEqual([]);
const first = await pending;
expect(await pending).toBe(first);
expect(first).toEqual([{ id }]);
expect(await observer.select().from(table).where(eq(table.id, id))).toEqual([{ id, payload: "raw-lazy-sentinel" }]);
f.capture.flush();
expect(summary(f.events)).toMatchObject({ startedQueries: 1, completedQueries: 1 });
expect(JSON.stringify(f.events)).not.toContain("raw-lazy-sentinel");
expect(JSON.stringify(f.events)).not.toContain(id);
});
it("counts prepared executions once and retains their parameter bindings", async () => {
const f = fixture(), id = randomUUID();
await f.db.insert(table).values({ id, payload: "prepared-value" });
const prepared = f.db.select().from(table).where(eq(table.id, sql.placeholder("id"))).prepare("response_capture_prepared");
expect(await prepared.execute({ id })).toEqual([{ id, payload: "prepared-value" }]);
expect(await prepared.execute({ id: randomUUID() })).toEqual([]);
f.capture.flush();
expect(summary(f.events)).toMatchObject({ startedQueries: 3, completedQueries: 3 });
expect(JSON.stringify(f.events)).not.toContain("prepared-value");
});
it("retains distinct builder callers through normal, prepared, and transaction execution", async () => {
const f = fixture(), id = randomUUID();
const setup = createDb(temporary.connectionString); clients.push(setup);
await setup.insert(table).values({ id, payload: "caller-sentinel" });
const shape = f.db.select().from(table);
const caller = (name: string) => compileFunction(
"return db.select().from(table).where(eq(table.id, id))", ["db", "id"],
{ filename: `/fixture/server/src/services/${name}.ts`, contextExtensions: [{ table, eq }] },
) as (db: Pick<typeof f.db, "select">, id: string) => typeof shape;
const fromA = caller("capture-caller-a"), fromB = caller("capture-caller-b");
const normalA = fromA(f.db, id), normalB = fromB(f.db, id);
expect(await normalA).toEqual([{ id, payload: "caller-sentinel" }]);
expect(await normalB).toEqual([{ id, payload: "caller-sentinel" }]);
const prepared = fromA(f.db, id).prepare("capture_caller_prepared");
expect(await prepared.execute()).toEqual([{ id, payload: "caller-sentinel" }]);
await f.db.transaction(async tx => {
expect(await fromB(tx, id)).toEqual([{ id, payload: "caller-sentinel" }]);
});
f.capture.flush();
const entries = f.events.filter(event => event.event === "paperclip.db_response_capture.entry.v1");
expect(entries).toHaveLength(2);
for (const name of ["a", "b"]) {
expect(entries.find(event => (event.callers as string[]).some(location => location.startsWith(`services/capture-caller-${name}.ts:`))))
.toMatchObject({ operation: "select", table: "response_capture_fixture", calls: 2, rows: 2 });
}
expect(summary(f.events)).toMatchObject({ startedQueries: 4, completedQueries: 4 });
expect(JSON.stringify(f.events)).not.toContain("/fixture/");
expect(JSON.stringify(f.events)).not.toContain("caller-sentinel");
expect(JSON.stringify(f.events)).not.toContain(id);
});
it("records the executed CTE rather than its unexecuted inner select", async () => {
const f = fixture(), id = randomUUID();
await f.db.insert(table).values({ id, payload: "cte-value" });
const cte = f.db.$with("capture_cte").as(f.db.select().from(table).where(eq(table.id, id)));
expect(await f.db.with(cte).select().from(cte)).toEqual([{ id, payload: "cte-value" }]);
f.capture.flush();
expect(summary(f.events)).toMatchObject({ startedQueries: 2, completedQueries: 2 });
});
it("keeps raw execute results and rejection behavior without logging SQL or errors", async () => {
const f = fixture();
expect(await f.db.execute(sql`select ${"sentinel-sql-parameter"}::text as value`)).toEqual([{ value: "sentinel-sql-parameter" }]);
await expect(f.db.execute(sql`select 1/0`)).rejects.toThrow();
f.capture.flush();
expect(summary(f.events)).toMatchObject({ startedQueries: 2, completedQueries: 2 });
expect(f.events.find(event => event.operation === "execute")).toMatchObject({ queryErrors: 1 });
expect(JSON.stringify(f.events)).not.toContain("sentinel-sql-parameter");
expect(JSON.stringify(f.events)).not.toContain("select 1/0");
});
it("preserves nested commit, rollback, and durable work-signal boundaries", async () => {
const f = fixture(), id = randomUUID(), events: string[] = [];
const unsubscribe = subscribeDatabaseWork(f.db, "capture-topic", event => events.push(event));
try {
expect(await f.db.transaction(async tx => {
await tx.transaction(async nested => {
await signalDatabaseWork(nested, "capture-topic");
await nested.insert(table).values({ id, payload: "committed" });
});
expect(events).toEqual(["intent"]); expect(databaseWorkPending(f.db, "capture-topic")).toBe(true);
return 42;
})).toBe(42);
expect(events).toEqual(["intent", "settled"]);
expect(databaseWorkPending(f.db, "capture-topic")).toBe(false);
const rolledBack = randomUUID();
await expect(f.db.transaction(async tx => {
await signalDatabaseWork(tx, "capture-topic");
await tx.insert(table).values({ id: rolledBack, payload: "rolled-back" });
throw new Error("rollback fixture");
})).rejects.toThrow("rollback fixture");
await reconcileDatabaseWork(f.db, "capture-topic");
expect(databaseWorkPending(f.db, "capture-topic")).toBe(false);
expect(await f.db.select().from(table).where(eq(table.id, rolledBack))).toEqual([]);
expect(await f.db.select().from(table).where(eq(table.id, id))).toEqual([{ id, payload: "committed" }]);
} finally { unsubscribe(); }
f.capture.flush();
expect(summary(f.events).completedQueries).toBeGreaterThan(4);
});
it.each([undefined, true])("covers automatically created dedicated pools with prepare %s and retains owner identity", async prepare => {
const logs: string[] = [], now = Date.now();
vi.spyOn(console, "info").mockImplementation(value => logs.push(String(value)));
vi.stubEnv("PAPERCLIP_DB_RESPONSE_CAPTURE_WINDOW", `${new Date(now - 100).toISOString()}/${new Date(now + 59_000).toISOString()}`);
const db = createDb(temporary.connectionString, { prepare }); clients.push(db);
const capture = databaseResponseCaptureFromEnv()!; captures.push(capture);
expect(await withDedicatedDbConnection(db, async dedicated => dedicated.execute(sql`select ${"dedicated-value"}::text as value`)))
.toEqual([{ value: "dedicated-value" }]);
capture.flush();
await vi.waitFor(() => expect(logs.some(line => JSON.parse(line).event === "paperclip.db_response_capture.summary.v1")).toBe(true),
{ timeout: 1_000, interval: 5 });
const events = logs.map(line => JSON.parse(line) as Record<string, unknown>);
expect(summary(events)).toMatchObject({ startedQueries: 1, completedQueries: 1 });
expect(logs.join("\n")).not.toContain("dedicated-value");
expect(logs.join("\n")).not.toContain(temporary.connectionString);
});
});
+203
View File
@@ -0,0 +1,203 @@
import { integer, pgTable } from "drizzle-orm/pg-core";
import { afterEach, describe, expect, it, vi } from "vitest";
import { createDatabaseResponseCapture, databaseResponseCaptureFromEnv, installDatabaseResponseCapture, publicDatabaseCallers, responseCaptureWindow } from "./response-capture.js";
const epoch = Date.parse("2020-01-01T00:00:00Z");
const metadata = { operation: "select", table: "capture_fixture", callers: ["services/fixture.ts:4:2"] };
const table = pgTable("capture_fixture", { id: integer("id") });
const captures: Array<ReturnType<typeof createDatabaseResponseCapture>> = [];
afterEach(() => { for (const capture of captures.splice(0)) capture.flush(); vi.unstubAllEnvs(); });
function fixture(emitOverride?: (event: Record<string, unknown>) => void) {
let clock = epoch;
const events: Array<Record<string, unknown>> = [];
const capture = createDatabaseResponseCapture({ start: epoch, end: epoch + 60_000 }, emitOverride ?? (event => events.push(event)), () => clock);
captures.push(capture);
return { capture, events, now: (value: number) => { clock = value; } };
}
describe("bounded database response capture", () => {
it("paces a complete environment capture without extending the measurement window or leaving timers", () => {
vi.useFakeTimers();
vi.setSystemTime(epoch);
const emitted: Array<{ at: number; event: Record<string, unknown> }> = [];
const log = vi.spyOn(console, "info").mockImplementation(value => emitted.push({ at: Date.now(), event: JSON.parse(value) }));
try {
const capture = databaseResponseCaptureFromEnv({ PAPERCLIP_DB_RESPONSE_CAPTURE_WINDOW: "2020-01-01T00:00:00Z/2020-01-01T00:01:00Z" })!;
captures.push(capture);
const rows = [{ secret: "capture-response-sentinel" }];
for (let index = 0; index < 2200; index++) {
expect(capture.execute(() => rows, { ...metadata, callers: [`services/capture.ts:${index + 1}:1`] })).toBe(rows);
}
vi.advanceTimersByTime(60_001);
expect(emitted.length).toBeGreaterThan(0);
expect(emitted.length).toBeLessThanOrEqual(64);
expect(capture.active()).toBe(false);
expect(capture.execute(() => rows, metadata)).toBe(rows);
capture.flush();
vi.advanceTimersByTime(32_000);
expect(emitted).toHaveLength(2049);
const groups = emitted.filter(item => item.event.event === "paperclip.db_response_capture.entry.v1");
const summary = emitted.at(-1)!.event;
expect(groups).toHaveLength(2048);
expect(summary).toMatchObject({ event: "paperclip.db_response_capture.summary.v1", completedQueries: 2200,
groups: 2048, closedAt: "2020-01-01T00:01:00.000Z" });
expect(groups.reduce((calls, item) => calls + Number(item.event.calls), 0)).toBe(summary.completedQueries);
expect(groups.reduce((bytes, item) => bytes + Number(item.event.responseValueBytes), 0)).toBe(summary.responseValueBytes);
expect(groups.map(item => item.event.groupIndex)).toEqual(Array.from({ length: 2048 }, (_, index) => index));
for (const { at } of emitted) expect(emitted.filter(item => item.at >= at && item.at < at + 1000).length).toBeLessThanOrEqual(64);
expect(JSON.stringify(emitted)).not.toContain("capture-response-sentinel");
expect(vi.getTimerCount()).toBe(0);
vi.advanceTimersByTime(60_000);
expect(emitted).toHaveLength(2049);
} finally { log.mockRestore(); vi.clearAllTimers(); vi.useRealTimers(); }
});
it("contains deferred logger failures and finishes its bounded output", () => {
vi.useFakeTimers();
vi.setSystemTime(epoch + 120_000);
const log = vi.spyOn(console, "info").mockImplementation(() => { throw new Error("fixture logger failure"); });
try {
const capture = databaseResponseCaptureFromEnv({ PAPERCLIP_DB_RESPONSE_CAPTURE_WINDOW: "2020-01-01T00:02:00Z/2020-01-01T00:03:00Z" })!;
captures.push(capture);
expect(capture.execute(() => 7, metadata)).toBe(7);
expect(() => vi.advanceTimersByTime(60_001)).not.toThrow();
expect(log).toHaveBeenCalledTimes(2);
expect(vi.getTimerCount()).toBe(0);
} finally { log.mockRestore(); vi.clearAllTimers(); vi.useRealTimers(); }
});
it.each([undefined, "", "sentinel-invalid-input", "2020-01-01T00:00:00Z", "2020-01-01T00:00:00Z/2020-01-01T00:00:00Z",
"2020-01-01T00:00:00Z/2020-01-01T00:01:01Z", "2020-01-01T01:00:00Z/2020-01-01T01:01:00Z",
"2020-01-01T00:00:00Z/2020-01-01T00:01:00Z/extra",
])("disables absent, malformed, expired, or unbounded windows: %s", value => {
expect(responseCaptureWindow(value, epoch + 60_000)).toBeUndefined();
});
it("accepts a fixed future window of at most one minute", () => {
expect(responseCaptureWindow("2020-01-01T00:01:00Z/2020-01-01T00:02:00Z", epoch))
.toEqual({ start: epoch + 60_000, end: epoch + 120_000 });
expect(() => createDatabaseResponseCapture({ start: epoch, end: epoch + 60_001 }, () => {})).toThrow("Invalid");
});
it("keeps only public server file locations from stacks", () => {
const stack = ["Error: sentinel-message", "at f (/private/fixture/server/src/services/example.ts:4:2)",
"at f (/private/fixture/server/src/services/example.ts:4:2)", "at f (/private/fixture/server/dist/routes/example.js:5:3)",
"at f (/private/fixture/customer-file.js:7:9)", "at f (/app/server/src/services/../../customer-file.ts:8:1)"].join("\n");
expect(publicDatabaseCallers(stack)).toEqual(["services/example.ts:4:2", "routes/example.js:5:3"]);
});
it("leaves methods and identity unchanged when disabled", () => {
vi.stubEnv("PAPERCLIP_DB_RESPONSE_CAPTURE_WINDOW", undefined);
const db = { select: vi.fn(), transaction: vi.fn() }, select = db.select, transaction = db.transaction;
expect(installDatabaseResponseCapture(db, { table })).toBe(db);
expect(db.select).toBe(select); expect(db.transaction).toBe(transaction);
});
it("preserves exact values and only emits counts, sizes, and caller metadata", async () => {
const f = fixture(), rows = [{ text: "sentinel-result-value🙂\n\"", n: 4n }];
expect(await f.capture.execute(() => Promise.resolve(rows), metadata)).toBe(rows);
f.capture.flush();
const entry = f.events.find(event => event.event === "paperclip.db_response_capture.entry.v1")!;
expect(entry).toMatchObject({ calls: 1, rows: 1, queryErrors: 0, serializationErrors: 0,
responseValueBytes: Buffer.byteLength(JSON.stringify(rows, (_, value) => typeof value === "bigint" ? value.toString() : value)) });
expect(JSON.stringify(f.events)).not.toContain("sentinel-result-value");
expect(f.events.at(-1)).toMatchObject({ startedQueries: 1, completedQueries: 1, inFlightAtEnd: 0,
rowValuesLogged: false, sqlOrParametersLogged: false, wireBytesMeasured: false });
});
it("counts driver text values without adding JSON escaping or column names", async () => {
const f = fixture(), text = "sentinel-driver-text\n\"";
const rows = Object.assign([[text, true, 42n, null]], { columns: [
{ name: "sentinel-column-name" }, { name: "b" }, { name: "n" }, { name: "empty" } ] });
expect(await f.capture.execute(() => Promise.resolve(rows), metadata)).toBe(rows);
f.capture.flush();
expect(f.events[0]).toMatchObject({ responseValueBytes: Buffer.byteLength(text) + 1 + 2, estimatedValues: 0 });
expect(JSON.stringify(f.events)).not.toContain("sentinel-driver-text");
expect(JSON.stringify(f.events)).not.toContain("sentinel-column-name");
});
it("does not alter synchronous or asynchronous errors or retry them", async () => {
const f = fixture(), error = new Error("sentinel-private-error");
const sync = vi.fn(() => { throw error; }), asyncFailure = vi.fn(() => Promise.reject(error));
expect(() => f.capture.execute(sync, metadata)).toThrow(error);
await expect(f.capture.execute(asyncFailure, metadata)).rejects.toBe(error);
expect(sync).toHaveBeenCalledTimes(1); expect(asyncFailure).toHaveBeenCalledTimes(1);
f.capture.flush();
expect(f.events[0]).toMatchObject({ calls: 2, queryErrors: 2, responseValueBytes: 0 });
expect(JSON.stringify(f.events)).not.toContain("sentinel-private-error");
});
it("keeps unserializable results and logging failures out of the query outcome", () => {
const f = fixture(), circular: Record<string, unknown> = {}; circular.self = circular;
expect(f.capture.execute(() => circular, metadata)).toBe(circular);
f.capture.flush();
expect(f.events[0]).toMatchObject({ calls: 1, serializationErrors: 1 });
const brokenLogger = fixture(() => { throw new Error("logger fixture"); });
expect(brokenLogger.capture.execute(() => circular, metadata)).toBe(circular);
expect(() => brokenLogger.capture.flush()).not.toThrow();
});
it("stops accounting at the deadline and leaves in-flight work untouched", async () => {
const f = fixture();
let resolve!: (rows: unknown[]) => void;
const pending = new Promise<unknown[]>(done => { resolve = done; });
const result = f.capture.execute(() => pending, metadata);
f.now(epoch + 60_000); f.capture.flush();
expect(f.events.at(-1)).toMatchObject({ startedQueries: 1, completedQueries: 0, inFlightAtEnd: 1 });
const rows = [{ marker: "late-result-value" }]; resolve(rows);
expect(await result).toBe(rows);
const direct = Promise.resolve(rows);
expect(f.capture.execute(() => direct, metadata)).toBe(direct);
expect(f.events).toHaveLength(1);
});
it("retains the deadline count when a query resolves before a delayed flush", async () => {
const f = fixture();
let resolve!: (rows: unknown[]) => void;
const pending = new Promise<unknown[]>(done => { resolve = done; });
const result = f.capture.execute(() => pending, metadata);
f.now(epoch + 60_001);
const rows = [{ marker: "late-before-flush-sentinel" }]; resolve(rows);
expect(await result).toBe(rows);
f.capture.flush();
expect(f.events).toHaveLength(1);
expect(f.events[0]).toMatchObject({ startedQueries: 1, completedQueries: 0, inFlightAtEnd: 1, responseValueBytes: 0 });
expect(JSON.stringify(f.events)).not.toContain("late-before-flush-sentinel");
});
it("bounds cardinality without dropping response-byte totals", () => {
const f = fixture();
for (let i = 0; i < 2200; i += 1) f.capture.execute(() => [i], { ...metadata, callers: [`services/f${i}.ts:1:1`] });
f.capture.flush();
const summary = f.events.at(-1)!;
expect(summary).toMatchObject({ startedQueries: 2200, completedQueries: 2200, groups: 2048, overflowCalls: 153,
groupPageSize: 128, groupPages: 16 });
expect(f.events).toHaveLength(2049);
const entries = f.events.slice(0, -1);
expect(entries.map(entry => entry.groupIndex)).toEqual(Array.from({ length: 2048 }, (_, index) => index));
for (let page = 0; page < 16; page += 1) {
expect(entries.filter(entry => entry.groupPage === page)).toHaveLength(128);
}
expect(entries.reduce((sum, entry) => sum + Number(entry.calls), 0)).toBe(2200);
expect(summary.responseValueBytes).toBe(Array.from({ length: 2200 }, (_, i) => Buffer.byteLength(JSON.stringify([i]))).reduce((a, b) => a + b, 0));
});
it.each([0, 129])("numbers every page including an empty or partial final page (%s)", count => {
const f = fixture();
for (let index = 0; index < count; index += 1) {
f.capture.execute(() => [index], { ...metadata, callers: [`services/page${index}.ts:1:1`] });
}
f.capture.flush();
const entries = f.events.slice(0, -1), summary = f.events.at(-1)!;
expect(summary).toMatchObject({ groups: count, groupPageSize: 128, groupPages: Math.ceil(count / 128) });
expect(entries.map(entry => entry.groupIndex)).toEqual(Array.from({ length: count }, (_, index) => index));
expect(entries.filter(entry => entry.groupPage === 0)).toHaveLength(Math.min(count, 128));
expect(entries.filter(entry => entry.groupPage === 1)).toHaveLength(Math.max(0, count - 128));
});
it("does not execute builders or prepared statements until requested", async () => {
const f = fixture(), rows = [{ id: 1 }], execute = vi.fn(async () => rows);
const query = { execute, prepare: () => ({ execute }) };
const builder = { from: () => query };
const db = { select: () => builder };
expect(installDatabaseResponseCapture(db, { table }, f.capture)).toBe(db);
expect(db.select()).toBe(builder);
expect(db.select().from()).toBe(query);
expect(execute).not.toHaveBeenCalled();
const prepared = query.prepare();
expect(execute).not.toHaveBeenCalled();
expect(await prepared.execute()).toBe(rows);
expect(await query.execute()).toBe(rows);
expect(execute).toHaveBeenCalledTimes(2);
f.capture.flush();
expect(f.events.at(-1)).toMatchObject({ completedQueries: 0 });
});
});
+299
View File
@@ -0,0 +1,299 @@
import { AsyncLocalStorage } from "node:async_hooks";
import { randomUUID } from "node:crypto";
import { getTableName, is, Table } from "drizzle-orm";
type Metadata = { operation: string; table: string; callers: string[] };
type Entry = Metadata & { calls: number; rows: number; bytes: number; maxBytes: number; queryErrors: number; serializationErrors: number; estimatedValues: number };
type Event = Record<string, unknown>;
type Method = (this: unknown, ...args: unknown[]) => unknown;
export type ResponseCaptureWindow = { start: number; end: number };
const GROUP_PAGE_SIZE = 128;
export function responseCaptureWindow(value: string | undefined, now = Date.now()): ResponseCaptureWindow | undefined {
if (!value) return;
const parts = value.split("/");
if (parts.length !== 2 || parts.some(part => !/^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d{3})?Z$/.test(part))) return;
const start = Date.parse(parts[0]!), end = Date.parse(parts[1]!);
if (!Number.isFinite(start) || !Number.isFinite(end) || end <= start || end - start > 60_000 || end <= now || start > now + 30 * 60_000) return;
return { start, end };
}
export function publicDatabaseCallers(stack: string | undefined): string[] {
const callers: string[] = [];
for (const line of (stack ?? "").split("\n")) {
const match = line.match(/[/\\]server[/\\](?:src|dist)[/\\]((?:services|routes|modules|middleware)[/\\][A-Za-z0-9_./\\-]+\.[cm]?[jt]s):(\d+):(\d+)/);
if (!match) continue;
const relative = match[1]!.replaceAll("\\", "/");
if (relative.split("/").includes("..")) continue;
const caller = `${relative}:${match[2]}:${match[3]}`;
if (!callers.includes(caller)) callers.push(caller);
if (callers.length === 4) break;
}
return callers;
}
function responseValueSize(value: unknown): { bytes: number; estimatedValues: number } {
const columns = value && typeof value === "object" ? (value as { columns?: unknown }).columns : undefined;
if (!Array.isArray(value) || !Array.isArray(columns)) {
const serialized = JSON.stringify(value, (_, item: unknown) => typeof item === "bigint" ? item.toString() : item);
return { bytes: serialized === undefined ? 0 : Buffer.byteLength(serialized), estimatedValues: 1 };
}
let bytes = 0, estimatedValues = 0;
for (const row of value) {
const cells = Array.isArray(row) ? row : columns.map(column => row[column.name]);
for (const cell of cells) {
if (cell === null || cell === undefined) continue;
if (typeof cell === "string") bytes += Buffer.byteLength(cell);
else if (typeof cell === "boolean") bytes += 1;
else if (typeof cell === "number" || typeof cell === "bigint") bytes += Buffer.byteLength(String(cell));
else {
estimatedValues += 1;
const serialized = JSON.stringify(cell, (_, item: unknown) => typeof item === "bigint" ? item.toString() : item);
if (serialized !== undefined) bytes += Buffer.byteLength(serialized);
}
}
}
return { bytes, estimatedValues };
}
export function createDatabaseResponseCapture(
window: ResponseCaptureWindow,
emit: (event: Event) => void,
now: () => number = Date.now,
) {
if (!Number.isFinite(new Date(window.start).getTime()) || !Number.isFinite(new Date(window.end).getTime()) ||
window.end <= window.start || window.end - window.start > 60_000) throw new Error("Invalid database response capture window");
const id = randomUUID(), entries = new Map<string, Entry>();
const roots = new WeakSet<object>(), queries = new WeakSet<object>(), factories = new WeakSet<object>();
const clients = new WeakSet<object>(), driverQueries = new WeakSet<object>();
const scope = new AsyncLocalStorage<Metadata>();
let closed = false, inFlight = 0, started = 0, completed = 0, overflowCalls = 0;
const send = (event: Event) => { try { emit(event); } catch {} };
const expired = () => closed || now() >= window.end;
const active = () => !expired() && now() >= window.start;
const flush = () => {
if (closed) return;
closed = true;
clearTimeout(timer);
scope.disable();
const ordered = [...entries.values()].sort((a, b) => b.bytes - a.bytes);
for (const [groupIndex, entry] of ordered.entries()) send({ event: "paperclip.db_response_capture.entry.v1", captureId: id,
groupIndex, groupPage: Math.floor(groupIndex / GROUP_PAGE_SIZE),
operation: entry.operation, table: entry.table, callers: entry.callers, calls: entry.calls, rows: entry.rows,
responseValueBytes: entry.bytes, maxResponseValueBytes: entry.maxBytes, estimatedValues: entry.estimatedValues,
queryErrors: entry.queryErrors, serializationErrors: entry.serializationErrors });
send({ event: "paperclip.db_response_capture.summary.v1", captureId: id,
windowStart: new Date(window.start).toISOString(), windowEnd: new Date(window.end).toISOString(),
closedAt: new Date(now()).toISOString(),
startedQueries: started, completedQueries: completed, inFlightAtEnd: inFlight, groups: entries.size,
groupPageSize: GROUP_PAGE_SIZE, groupPages: Math.ceil(ordered.length / GROUP_PAGE_SIZE),
overflowCalls, responseValueBytes: ordered.reduce((sum, entry) => sum + entry.bytes, 0),
rowValuesLogged: false, sqlOrParametersLogged: false, wireBytesMeasured: false,
measurementBoundary: "postgres-js results before Drizzle column decoding",
relationalQueriesMeasured: true, unsafeDriverQueriesMeasured: true,
taggedDriverQueriesMeasured: false, streamingQueriesMeasured: false });
entries.clear();
};
const timer = setTimeout(flush, Math.max(0, window.end - now()));
timer.unref();
function finish(metadata: Metadata, value: unknown, failed: boolean) {
if (!active()) return;
inFlight -= 1;
completed += 1;
try {
let key = JSON.stringify(metadata);
if (!entries.has(key) && entries.size >= 2047) {
overflowCalls += 1;
metadata = { operation: "other", table: "other", callers: [] };
key = "overflow";
}
let entry = entries.get(key);
if (!entry) {
entry = { ...metadata, calls: 0, rows: 0, bytes: 0, maxBytes: 0, queryErrors: 0, serializationErrors: 0, estimatedValues: 0 };
entries.set(key, entry);
}
entry.calls += 1;
if (failed) { entry.queryErrors += 1; return; }
entry.rows += Array.isArray(value) ? value.length : 0;
try {
const size = responseValueSize(value);
entry.bytes += size.bytes;
entry.maxBytes = Math.max(entry.maxBytes, size.bytes);
entry.estimatedValues += size.estimatedValues;
} catch { entry.serializationErrors += 1; }
} catch {}
}
function execute(action: () => unknown, metadata: Metadata) {
if (!active()) return action();
inFlight += 1; started += 1;
let result: unknown;
try { result = action(); }
catch (error) { finish(metadata, undefined, true); throw error; }
let asynchronous = false;
try { asynchronous = Boolean(result && typeof (result as { then?: unknown }).then === "function"); } catch {}
if (asynchronous) {
return Promise.resolve(result).then(value => { finish(metadata, value, false); return value; },
error => { finish(metadata, undefined, true); throw error; });
}
finish(metadata, result, false);
return result;
}
return { window, roots, queries, factories, clients, driverQueries, scope, active, expired, execute, flush };
}
type Capture = ReturnType<typeof createDatabaseResponseCapture>;
let environmentCapture: { key: string; capture: Capture } | undefined;
function pacedCaptureLogger() {
const pending: Event[] = [];
let timer: ReturnType<typeof setTimeout> | undefined;
const drain = () => {
timer = undefined;
for (const event of pending.splice(0, 64)) {
try { console.info(JSON.stringify(event)); } catch {}
}
if (pending.length) {
timer = setTimeout(drain, 1_000);
timer.unref();
}
};
return (event: Event) => {
pending.push(event);
if (!timer) {
timer = setTimeout(drain, 0);
timer.unref();
}
};
}
export function databaseResponseCaptureFromEnv(env: NodeJS.ProcessEnv = process.env): Capture | undefined {
const key = env.PAPERCLIP_DB_RESPONSE_CAPTURE_WINDOW;
const window = responseCaptureWindow(key);
if (!key || !window) return;
if (environmentCapture?.key === key) return environmentCapture.capture;
const capture = createDatabaseResponseCapture(window, pacedCaptureLogger());
environmentCapture = { key, capture };
return capture;
}
export function installDatabaseResponseCapture<T extends object>(db: T, schema: Record<string, unknown>, capture = databaseResponseCaptureFromEnv()): T {
if (!capture || capture.expired()) return db;
const tables = new Set(Object.values(schema).filter(value => is(value, Table)).map(table => getTableName(table)));
const object = (value: unknown): value is Record<string, unknown> => value !== null && typeof value === "object";
function replace(target: Record<string, unknown>, name: string, replacement: (original: Method) => Method) {
const original = target[name];
if (typeof original !== "function") return;
try { target[name] = replacement(original as Method); } catch {}
}
function tableName(value: unknown) {
try { if (is(value, Table)) { const name = getTableName(value); if (tables.has(name)) return name; } } catch {}
return "other";
}
function query(value: unknown, metadata: Metadata) {
if (!object(value) || capture!.queries.has(value)) return;
capture!.queries.add(value);
replace(value, "execute", original => function (this: unknown, ...args: unknown[]) {
if (capture!.expired()) return Reflect.apply(original, this, args);
return capture!.scope.run(metadata, () => Reflect.apply(original, this, args));
});
replace(value, "prepare", original => function (this: unknown, ...args: unknown[]) {
const prepared = Reflect.apply(original, this, args);
try { query(prepared, metadata); } catch {}
return prepared;
});
}
function factory(value: unknown, step: string, metadata: Metadata, from = false) {
if (!object(value) || capture!.factories.has(value)) return;
capture!.factories.add(value);
replace(value, step, original => function (this: unknown, ...args: unknown[]) {
const result = Reflect.apply(original, this, args);
try { query(result, { ...metadata, table: from ? tableName(args[0]) : metadata.table }); } catch {}
return result;
});
}
function driverQuery(value: unknown, metadata: Metadata) {
if (!object(value) || capture!.driverQueries.has(value)) return;
capture!.driverQueries.add(value);
let consumed = false;
replace(value, "then", original => function (this: unknown, ...handlers: unknown[]) {
if (consumed || !capture!.active()) {
consumed = true;
return Reflect.apply(original, this, handlers);
}
consumed = true;
const result = capture!.execute(() => Reflect.apply(original, this, []), metadata) as Promise<unknown>;
return Reflect.apply(result.then, result, handlers);
});
for (const method of ["execute", "cursor", "forEach", "readable", "writable"]) {
replace(value, method, original => function (this: unknown, ...args: unknown[]) {
consumed = true;
return Reflect.apply(original, this, args);
});
}
}
function sqlMetadata(value: unknown): Metadata {
const scoped = capture!.scope.getStore();
const text = typeof value === "string" ? value : "";
let table = "other";
for (const match of text.matchAll(/\b(?:from|update|into)\s+"([A-Za-z0-9_]+)"/gi)) {
if (tables.has(match[1]!)) { table = match[1]!; break; }
}
return scoped ? { ...scoped, table: scoped.table === "other" ? table : scoped.table }
: { operation: text.match(/^\s*(select|insert|update|delete)\b/i)?.[1]?.toLowerCase() ?? "execute",
table, callers: publicDatabaseCallers(new Error().stack) };
}
function client(value: unknown) {
if ((!object(value) && typeof value !== "function") || capture!.clients.has(value)) return;
capture!.clients.add(value);
const target = value as Record<string, unknown>;
replace(target, "unsafe", original => function (this: unknown, ...args: unknown[]) {
const result = Reflect.apply(original, this, args);
if (!capture!.expired()) { try { driverQuery(result, sqlMetadata(args[0])); } catch {} }
return result;
});
for (const method of ["begin", "savepoint"]) {
replace(target, method, original => function (this: unknown, ...args: unknown[]) {
const wrapped = args.map(arg => typeof arg !== "function" ? arg : function (this: unknown, ...inner: unknown[]) {
try { client(inner[0]); } catch {}
return Reflect.apply(arg, this, inner);
});
return Reflect.apply(original, this, wrapped);
});
}
replace(target, "reserve", original => function (this: unknown, ...args: unknown[]) {
return Promise.resolve(Reflect.apply(original, this, args)).then(reserved => {
try { client(reserved); } catch {}
return reserved;
});
});
}
function root(value: unknown) {
if (!object(value) || capture!.roots.has(value)) return;
capture!.roots.add(value);
for (const method of ["select", "selectDistinct", "selectDistinctOn", "insert", "update", "delete", "execute", "with"] as const) {
replace(value, method, original => function (this: unknown, ...args: unknown[]) {
const result = Reflect.apply(original, this, args);
if (capture!.expired()) return result;
try {
if (method === "with") { root(result); return result; }
const operation = method.startsWith("select") ? "select" : method;
const metadata = { operation, table: operation === "select" || method === "execute" ? "other" : tableName(args[0]), callers: publicDatabaseCallers(new Error().stack) };
if (operation === "select") factory(result, "from", metadata, true);
else if (method === "insert") factory(result, "values", metadata);
else if (method === "update") factory(result, "set", metadata);
else query(result, metadata);
} catch {}
return result;
});
}
replace(value, "transaction", original => function (this: unknown, callback: unknown, ...options: unknown[]) {
if (typeof callback !== "function" || capture!.expired()) return Reflect.apply(original, this, [callback, ...options]);
return Reflect.apply(original, this, [function (this: unknown, ...args: unknown[]) {
try { root(args[0]); } catch {}
return Reflect.apply(callback, this, args);
}, ...options]);
});
}
try { client((db as Record<string, unknown>).$client); root(db); } catch {}
return db;
}
+4 -4
View File
@@ -14,12 +14,12 @@
"requiresProviderPolicy": false
},
"pi": {
"agentProfileVersion": 28,
"agentProfileVersion": 29,
"agentServerPackage": "pi-acp",
"agentServerVersion": "0.0.33",
"agentRuntimePackage": "@earendil-works/pi-coding-agent",
"agentRuntimeVersion": "1.0.0",
"commandDigest": "sha256:4918a4d5472d66b22b1ec642b68b5c60a8092a39c84ec1ab1c9ad0289cb8c68c",
"commandDigest": "sha256:e80c6dc7bde12790371345ee5a12f0201d30941d26f1a780aec3d090545b2a06",
"requiresProviderPolicy": true
},
"cursor": {
@@ -32,12 +32,12 @@
"requiresProviderPolicy": true
},
"copilot": {
"agentProfileVersion": 58,
"agentProfileVersion": 59,
"agentServerPackage": "@github/copilot",
"agentServerVersion": "1.0.88",
"agentRuntimePackage": null,
"agentRuntimeVersion": null,
"commandDigest": "sha256:d490dacdab02051d24c84aae49ac6fb12b8226cf8a47b014ec53f3d3a8410a94",
"commandDigest": "sha256:2f39ffe54a6af832d044b5098c31df9e08d4d27aee0ac675ba4b0c9da7445c40",
"requiresProviderPolicy": true
},
"claude": {
@@ -159,6 +159,84 @@ change the hosted matrix. The controller must enforce request admission,
deadlines, and limits before forwarding traffic. A local fake-API probe only
proves routing; it does not qualify live provider or Braintrust behavior.
## Controlled request budgets
The source-checkout controller is
`packages/paperclip-runner/scripts/controlled-evals/run.mjs`. Runner/evals owns
this orchestration. It reuses the existing eval request proxy and ledger from
`tests/runner-e2e` through one import graph; it does not copy their reservation
or provider-usage rules. It requires the workspace's installed `tsx` runtime.
It is not a published Runner package entry point.
The controller accepts the existing `--request` and `--output` contract. Set
`PAPERCLIP_RUNNER_EVAL_CONTROL_MANIFEST` to an explicit reviewed manifest and
supply the selected upstream credential only to this trusted controller. For
Runner Evals, pass this controller path as `eval_program.py run --runner-cli`;
the manifest identifies the actual built `eval-session.js`. The provider child
receives only its temporary proxy credential and private home.
The manifest schema is `paperclip.runner-eval.controller/v1`. It contains:
- `campaignId`, `maxUsd`, `deadlineAt`, `pricingVerifiedAt`, and
`pricingExpiresAt`; admission must fit a current four-hour window.
- `sourceRevision` and `definitionsRevision`, each an exact commit ID.
These are the trusted preparer's provenance declarations.
- `runnerCli`, `runnerPackage`, and `runnerd`, each an absolute `path` and
content `sha256`. The controller checks the three files before launch.
- An absolute durable `outputDirectory`, outside the disposable case workspace.
- `route` (`direct` or `braintrust`), plus `projectId` for Braintrust.
- `models`, using the existing exact-model price, request-limit, and reviewed
full-lifecycle `provisionalBound` contract.
- `overhead`, using the existing four-component operating allowance with a
current full-lifecycle bound for every component. This is reserved once.
- `attempts`, each with a unique `executionId` and `attemptId`,
`requestSha256`, `provider`, optional `acpxAgent`, exact `model`, `maxUsd`,
`maxRequests`, `maxConcurrent`, and `timeoutMs`. Native Codex and ACPX
Claude are supported. Each execution has one explicit attempt and no retry.
Use `requestBindingSha256` from `controlled-evals/manifest.mjs` to bind the
complete compiled request. It excludes only the temporary working directory
and native executable path; the executable digest remains bound and its path
is checked separately. The controller gives the child a retained copy of the
validated request. Changing prompts, fixtures, models, limits, or attempts
requires a new reviewed binding.
The sum of every cell's allocation and the once-only operating allowance must
fit `maxUsd` before any cell starts. A request reserves against its cell's
ledger before forwarding. Cells cannot borrow unused allocations. Unknown
requests keep their reservations, and failed or missing cell evidence keeps
that cell's full allocation in the campaign report. An existing cell directory
or output is never reset for a retry. Cancellation closes admission, drains
upstream receipts within the proxy limit, and stops the owned CLI process
group. Unverified provider cleanup remains explicit and retains the allocation.
Each result retains `controllerAccounting` alongside the original provider
outcome and usage. Retain the durable controller directory as private evidence:
its admission, request, gateway descriptor, request receipts, ledger, original
provider artifact, and final result are required for reconciliation. The
private program's existing immutable attempt copy includes the result metadata;
its current default history publisher does not archive this separate directory.
Do not infer receipt retention from that metadata alone.
```sh
node packages/paperclip-runner/scripts/controlled-evals/report.mjs \
/private/path/controller-manifest.json > /private/path/controller-report.json
```
The report recomputes ledger costs, checks their attempt/manifest bindings, and
preserves known partial charges. It reports operating costs as held and total
costs as provisional when evidence is intact, or incomplete otherwise. It does
not release historical holds or claim invoice
settlement. Existing provider usage is retained separately; do not add the
provider and controller cost observations as if they were separate charges.
This local controller is not yet wired into the hosted matrix. Its durable
attempt-directory guard applies to one shared filesystem, not independent
GitHub workers. Hosted integration must bind unique cells to the prepared
allocation, reject workflow/job reruns, and preserve the controller evidence
before enabling paid work. Native OTLP routing, report integration, vendor
metering bounds, and actual paid qualification remain separate gates.
## Controller-assigned native observability
The same CLI accepts `--observability-config /absolute/path/observability.json`
@@ -292,6 +370,12 @@ hashes. These detect corruption; they are not signatures or proof against a
host that can rewrite all evidence. Known configured credential values are
checked before encoding and after decoding. Keep bundles private.
Controller post-child reads, archive creation, and durable reports use the same
nonblocking, no-follow, byte-capped regular-file reader. Configured private/provider
keys and credentials inside JSON configuration contents are checked in both raw
and JSON-escaped form. Attempt IDs contain only letters, digits, `_`, `.`, and `-`
so each retained attempt has one directory name.
The reader accepts only bounded regular files and fixed evidence paths. A
bundle is limited to 64 MiB of decoded content plus encoding overhead; this
storage-format limit does not qualify a vendor charge or operating-cost bound.
@@ -3212,7 +3212,7 @@ mod tests {
#[test]
fn pi_profile_matches_published_identity_and_rejects_prior_profiles() {
let published: Value = serde_json::from_str(include_str!(
"../../../../test-fixtures/pi-acp/profile-v28-identity.json"
"../../../../test-fixtures/pi-acp/profile-v29-identity.json"
))
.unwrap();
let mut value = descriptor("codex");
@@ -3229,6 +3229,7 @@ mod tests {
current.validate(&context()).unwrap();
for prior in [
include_str!("../../../../test-fixtures/pi-acp/profile-v28-identity.json"),
include_str!("../../../../test-fixtures/pi-acp/profile-v24-identity.json"),
include_str!("../../../../test-fixtures/pi-acp/profile-v23-identity.json"),
include_str!("../../../../test-fixtures/pi-acp/profile-v15-identity.json"),
@@ -29,7 +29,7 @@ pub(crate) fn acpx_release_profile(agent: &str) -> Option<AcpxReleaseProfile> {
agent_runtime_package: Some("@earendil-works/pi-coding-agent"),
agent_runtime_version: Some("1.0.0"),
command_digest:
"sha256:4918a4d5472d66b22b1ec642b68b5c60a8092a39c84ec1ab1c9ad0289cb8c68c",
"sha256:e80c6dc7bde12790371345ee5a12f0201d30941d26f1a780aec3d090545b2a06",
requires_provider_policy: true,
},
"cursor" => AcpxReleaseProfile {
@@ -47,7 +47,7 @@ pub(crate) fn acpx_release_profile(agent: &str) -> Option<AcpxReleaseProfile> {
agent_runtime_package: None,
agent_runtime_version: None,
command_digest:
"sha256:d490dacdab02051d24c84aae49ac6fb12b8226cf8a47b014ec53f3d3a8410a94",
"sha256:2f39ffe54a6af832d044b5098c31df9e08d4d27aee0ac675ba4b0c9da7445c40",
requires_provider_policy: true,
},
"claude" => AcpxReleaseProfile {
@@ -17,7 +17,7 @@ use sha2::{Digest, Sha256};
const CODEX_ACPX_DIGEST: &str =
"sha256:443491f4e814501a5dc47347e4510e84faeec9c9382e2d4e53bfe69386d5aabc";
const PI_ACPX_DIGEST: &str =
"sha256:4918a4d5472d66b22b1ec642b68b5c60a8092a39c84ec1ab1c9ad0289cb8c68c";
"sha256:e80c6dc7bde12790371345ee5a12f0201d30941d26f1a780aec3d090545b2a06";
fn temporary_directory(label: &str) -> PathBuf {
let nonce = SystemTime::now()
@@ -1,47 +1,39 @@
import { constants } from 'node:fs';
import { lstat, open, readdir, writeFile } from 'node:fs/promises';
import { lstat, readdir, writeFile } from 'node:fs/promises';
import { basename, join, resolve } from 'node:path';
import { pathToFileURL } from 'node:url';
import { bindRequest, canonical, fields, readJson, requestBindingSha256, sha256, units, usd, validateManifest } from './manifest.mjs';
import { attemptDirectory, controlledResultArtifact } from './run.mjs';
import { summarizeControlledCampaign } from './report.mjs';
import { FILE_LIMITS, RECEIPT, limitFor, readRegularFile } from './evidence-files.mjs';
const MiB = 1_048_576;
const MAX_RAW_BYTES = 64 * MiB;
const MAX_ARCHIVE_BYTES = 96 * MiB;
const FILE_LIMITS = new Map([
['admission.json', MiB], ['request.json', 8 * MiB], ['gateway.json', 8192],
['ledger.jsonl', 16 * MiB], ['result.json', MiB], ['provider-artifact.json', 16 * MiB],
['observability.json', 131_072], ['otlp-native.jsonl', 2 * MiB], ['otlp-controller.jsonl', 2 * MiB], ['otlp-root.json', 16_384],
['stdout.log', MiB], ['stderr.log', MiB],
]);
const RECEIPT = /^receipts\/[a-f0-9]{8}(?:-[a-f0-9]{4}){3}-[a-f0-9]{12}\.json$/;
const limitFor = name => FILE_LIMITS.get(name) ?? (RECEIPT.test(name) ? MiB : 0);
function secretValues({ environment = process.env, secrets = [] } = {}) {
return [...new Set([...secrets, ...Object.entries(environment)
.filter(([name]) => /(?:API_KEY|TOKEN|SECRET|PASSWORD|ACCESS_KEY_ID)$/.test(name))
.map(([, value]) => value)].filter(value => typeof value === 'string' && value.length > 0))];
const configured = [...secrets, ...Object.entries(environment)
.filter(([name]) => /(?:KEY|TOKEN|SECRET|PASSWORD|ACCESS_KEY_ID|CONFIG_CONTENT)$/i.test(name))
.map(([, value]) => value)].filter(value => typeof value === 'string' && value.length > 0);
for (const [name, value] of Object.entries(environment)) {
if (!/CONFIG_CONTENT$/i.test(name) || typeof value !== 'string') continue;
try {
const visit = item => {
if (!item || typeof item !== 'object') return;
for (const [key, entry] of Object.entries(item)) {
if (/(?:key|token|secret|password)$/i.test(key) && typeof entry === 'string' && entry) configured.push(entry);
else visit(entry);
}
};
visit(JSON.parse(value));
} catch {}
}
return [...new Set(configured.flatMap(value => [value, JSON.stringify(value).slice(1, -1)]))];
}
function excludeSecrets(bytes, secrets) {
if (secrets.some(secret => bytes.includes(Buffer.from(secret)))) throw new Error('Controller evidence contains a configured credential');
}
async function readRegularFile(path, maximum) {
const handle = await open(path, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK);
try {
const stat = await handle.stat();
if (!stat.isFile() || stat.size > maximum) throw new Error('Controller evidence is not a bounded regular file');
const bytes = Buffer.alloc(stat.size + 1);
let length = 0;
while (length < bytes.length) {
const { bytesRead } = await handle.read(bytes, length, bytes.length - length, null);
if (!bytesRead) break;
length += bytesRead;
}
if (length !== stat.size) throw new Error('Controller evidence changed while being read');
return bytes.subarray(0, length);
} finally { await handle.close(); }
}
async function realDirectory(path) {
try {
const stat = await lstat(path);
@@ -1,6 +1,7 @@
import assert from 'node:assert/strict';
import { test } from 'node:test';
import { mkdtemp, mkdir, readFile, readdir, rm, symlink, writeFile } from 'node:fs/promises';
import { execFileSync } from 'node:child_process';
import { mkdtemp, mkdir, readFile, readdir, rm, symlink, truncate, writeFile } from 'node:fs/promises';
import { writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
@@ -15,7 +16,9 @@ import assert from 'node:assert/strict';
import {createHash} from 'node:crypto';
import {spawn} from 'node:child_process';
import {once} from 'node:events';
import {readFile,writeFile} from 'node:fs/promises';
import {readFile,writeFile,rm,readdir,symlink,truncate} from 'node:fs/promises';
import {dirname,join} from 'node:path';
import {execFileSync} from 'node:child_process';
const args=new Map();for(let i=2;i<process.argv.length;i+=2)args.set(process.argv[i],process.argv[i+1]);
const request=JSON.parse(await readFile(args.get('--request'),'utf8'));
if(${Boolean(options.resistantDescendant)}) {
@@ -48,6 +51,17 @@ if(!${Boolean(options.missingArtifact)})await writeFile(args.get('--output'),JSO
requestRouting:${options.missingRoute ? 'null' : "{mode:'controller_gateway',provider:descriptor.provider,declaredUpstreamRoute:descriptor.upstreamRoute,campaignId:descriptor.campaignId,executionId:descriptor.executionId,attemptId:descriptor.attemptId,descriptorSha256:createHash('sha256').update(descriptorRaw).digest('hex')}"},
nativeObservability, snapshot:{status:'closed',process:{runnerExited:true,runnerPid:process.pid,providerPid:process.pid}},statuses,
leaked:${options.leakToken ? 'process.env.PAPERCLIP_RUNNER_EVAL_GATEWAY_TOKEN' : options.leakOtlpToken ? 'process.env.PAPERCLIP_RUNNER_EVAL_OTLP_TOKEN' : 'null'}}));
if(${Boolean(options.invalidEvidence)}) {
const root=dirname(args.get('--output'));
const name=${JSON.stringify(options.invalidEvidence ?? '')};
const file=name==='receipt'?join(root,'receipts',(await readdir(join(root,'receipts')))[0]):join(root,name);
await rm(file);
const kind=${JSON.stringify(options.invalidFileType ?? '')};
if(kind==='fifo')execFileSync('mkfifo',[file]);
else if(kind==='symlink')await symlink('/dev/zero',file);
else {await writeFile(file,'');await truncate(file,16*1024*1024+1);}
}
if(${Boolean(options.orphanExit)})process.exit(0);
`;
@@ -138,6 +152,7 @@ test('disjoint concurrent cells cannot borrow an allowance and unknown requests
test('rejects oversized or unqualified allocations, changed requests and changed executables before dispatch', async t => {
const f = await fixture(t, { count: 2 });
for (const mutate of [
value => { value.attempts[0].attemptId = 'nested/attempt'; },
value => { value.maxUsd = '0.01'; }, value => { value.attempts[1].executionId = value.attempts[0].executionId; },
value => { value.attempts[0].provider = 'acpx'; value.attempts[0].acpxAgent = 'codex'; },
value => { delete value.overhead.components[0].provisionalBound; }, value => { value.models[0].secret = 'unexpected'; },
@@ -395,6 +410,51 @@ test('configured credentials are excluded before encoding and after archive deco
await assert.rejects(readControlledEvidence(file, { environment }), /configured credential/);
});
for (const name of ['provider-artifact.json', 'receipt', 'ledger.jsonl']) for (const kind of ['fifo', 'symlink', 'oversized']) {
test(`post-child ${name} rejects ${kind} without blocking or releasing its allocation`, { timeout: 20_000 }, async t => {
const f = await fixture(t, { requests: 1, invalidEvidence: name, invalidFileType: kind });
const result = await f.run();
assert.equal(result.report.status, 'incomplete');
assert.equal(result.report.accounting.knownUsd, '0.000068000');
const summary = await summarizeControlledCampaign(f.manifest);
assert.equal(summary.incompleteAttempts, 1);
assert.equal(summary.committedUsd, '0.014000000');
});
}
test('durable reports bound every evidence file and retain full holds for invalid proof', { timeout: 20_000 }, async t => {
const f = await fixture(t, { requests: 1, observability: true }); await f.run();
const root = attemptDirectory(f.manifest, 'attempt-0');
const receipt = `receipts/${(await readdir(join(root, 'receipts')))[0]}`;
for (const name of ['admission.json', 'request.json', 'result.json', 'provider-artifact.json', 'gateway.json', 'ledger.jsonl', receipt, 'observability.json', 'otlp-native.jsonl', 'otlp-controller.jsonl', 'otlp-root.json']) {
const path = join(root, name), original = await readFile(path);
for (const kind of ['fifo', 'symlink', 'oversized']) {
await rm(path);
if (kind === 'fifo') execFileSync('mkfifo', [path]);
else if (kind === 'symlink') await symlink('/dev/zero', path);
else { await writeFile(path, ''); await truncate(path, 16 * 1024 * 1024 + 1); }
const summary = await summarizeControlledCampaign(f.manifest);
assert.equal(summary.incompleteAttempts, 1, `${name}:${kind}`);
assert.equal(summary.committedUsd, '0.014000000', `${name}:${kind}`);
}
await rm(path); await writeFile(path, original);
}
assert.equal((await summarizeControlledCampaign(f.manifest)).incompleteAttempts, 0);
});
for (const name of ['PAPERCLIP_AGENT_PRIVATE_KEY', 'PAPERCLIP_AI_PROVIDER_KEY', 'AWS_SECRET_ACCESS_KEY', 'OPENCODE_CONFIG_CONTENT']) {
test(`archive excludes configured ${name} before and after encoding`, async t => {
const f = await fixture(t); await f.run();
const secret = name === 'PAPERCLIP_AGENT_PRIVATE_KEY' ? '-----BEGIN PRIVATE KEY-----\nsynthetic-private-material\n-----END PRIVATE KEY-----' : 'synthetic-extra-credential';
const environment = { [name]: name === 'OPENCODE_CONFIG_CONTENT' ? JSON.stringify({ provider: { apiKey: secret } }) : secret };
const root = attemptDirectory(f.manifest, 'attempt-0');
await writeFile(join(root, 'stderr.log'), JSON.stringify({ retained: secret }));
await assert.rejects(packControlledAttempt({ manifest: f.manifest, attemptId: 'attempt-0', environment }), /configured credential/);
const unchecked = await packControlledAttempt({ manifest: f.manifest, attemptId: 'attempt-0', environment: {} });
assert.throws(() => decodeControlledEvidence(unchecked, { environment }), /configured credential/);
});
}
test('retention annotations preserve an already-failed controller result and its known costs', async t => {
const f = await fixture(t, { requests: 1, missingRoute: true }); await f.run();
const bundle = await packControlledAttempt({ manifest: f.manifest, attemptId: 'attempt-0' });
@@ -579,7 +639,9 @@ test('cancellation aborts a pending native export and retains its uncertain rece
test('lost ledger finalization still closes both OTLP relays', async t => {
const f = await fixture(t, { observability: true, requests: 1 });
const root = attemptDirectory(f.manifest, 'attempt-0');
await assert.rejects(f.run(0, async () => { await rm(join(root, 'ledger.jsonl')); return success(); }), { code: 'ENOENT' });
const result = await f.run(0, async () => { await rm(join(root, 'ledger.jsonl')); return success(); });
assert.equal(result.code, 2); assert.equal(result.report.failure, 'ledger_evidence_invalid');
assert.equal((await summarizeControlledCampaign(f.manifest)).committedUsd, '0.014000000');
for (const file of ['otlp-native.jsonl', 'otlp-controller.jsonl']) {
const rows = (await readFile(join(root, file), 'utf8')).trim().split('\n').map(line => JSON.parse(line));
assert.equal(rows.at(-1).type, 'closed');
@@ -619,3 +681,15 @@ test('cancellation during root export stops forwarding and cannot close the atte
const summary = await summarizeControlledCampaign(f.manifest);
assert.equal(summary.committedUsd, '0.014000000'); assert.equal(summary.knownUsd, '0.000068000');
});
for (const name of ['observability.json', 'otlp-native.jsonl', 'otlp-controller.jsonl']) for (const kind of ['fifo', 'symlink', 'oversized']) {
test(`native post-child ${name} rejects ${kind} and retains model cost plus the full hold`, { timeout: 20_000 }, async t => {
const f = await fixture(t, { requests: 1, observability: true, invalidEvidence: name, invalidFileType: kind });
const result = await f.run();
assert.equal(result.code, 2);
const summary = await summarizeControlledCampaign(f.manifest);
assert.equal(summary.incompleteAttempts, 1);
assert.equal(summary.knownUsd, '0.000068000');
assert.equal(summary.committedUsd, '0.014000000');
});
}
@@ -0,0 +1,29 @@
import { constants } from 'node:fs';
import { open } from 'node:fs/promises';
const MiB = 1_048_576;
export const FILE_LIMITS = new Map([
['admission.json', MiB], ['request.json', 8 * MiB], ['gateway.json', 8192],
['ledger.jsonl', 16 * MiB], ['result.json', MiB], ['provider-artifact.json', 16 * MiB],
['observability.json', 131_072], ['otlp-native.jsonl', 2 * MiB], ['otlp-controller.jsonl', 2 * MiB], ['otlp-root.json', 16_384],
['stdout.log', MiB], ['stderr.log', MiB],
]);
export const RECEIPT = /^receipts\/[a-f0-9]{8}(?:-[a-f0-9]{4}){3}-[a-f0-9]{12}\.json$/;
export const limitFor = name => FILE_LIMITS.get(name) ?? (RECEIPT.test(name) ? MiB : 0);
export async function readRegularFile(path, maximum) {
const handle = await open(path, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK);
try {
const stat = await handle.stat();
if (!stat.isFile() || stat.size > maximum) throw new Error('Controller evidence is not a bounded regular file');
const bytes = Buffer.alloc(stat.size + 1);
let length = 0;
while (length < bytes.length) {
const { bytesRead } = await handle.read(bytes, length, bytes.length - length, null);
if (!bytesRead) break;
length += bytesRead;
}
if (length !== stat.size) throw new Error('Controller evidence changed while being read');
return bytes.subarray(0, length);
} finally { await handle.close(); }
}
@@ -72,7 +72,7 @@ export function validateManifest(value, now = Date.now(), { admission = true } =
const provider = attempt.provider === 'codex' && attempt.acpxAgent === undefined ? 'openai'
: attempt.provider === 'acpx' && attempt.acpxAgent === 'claude' ? 'anthropic' : null;
const reservation = prices.get(`${provider}:${attempt.model}`);
if (!provider || !id(attempt.executionId) || !id(attempt.attemptId) || !hash(attempt.requestSha256)
if (!provider || !id(attempt.executionId) || !id(attempt.attemptId) || !/^[A-Za-z0-9_.-]+$/.test(attempt.attemptId) || !hash(attempt.requestSha256)
|| attempts.has(attempt.attemptId) || executions.has(attempt.executionId) || reservation === undefined
|| !Number.isSafeInteger(attempt.timeoutMs) || attempt.timeoutMs < 1 || attempt.timeoutMs > 4 * 3_600_000
|| !Number.isSafeInteger(attempt.maxRequests) || attempt.maxRequests < 1 || attempt.maxRequests > 1000
@@ -1,6 +1,7 @@
import { randomBytes } from 'node:crypto';
import { readFile, writeFile } from 'node:fs/promises';
import { writeFile } from 'node:fs/promises';
import { join } from 'node:path';
import { limitFor, readRegularFile } from './evidence-files.mjs';
import { canonical, costRuntime, ROOT_OTLP_LIMITS, sha256 } from './manifest.mjs';
function launchFor(manifest, request, identity, baseUrl) {
@@ -66,7 +67,7 @@ export async function startControlledObservability({ manifest, attempt, request,
await native.close();
signal?.throwIfAborted();
try {
const receipts = costRuntime.inspectEvalOtlpReceipts(await readFile(join(root, 'otlp-native.jsonl')),
const receipts = costRuntime.inspectEvalOtlpReceipts(await readRegularFile(join(root, 'otlp-native.jsonl'), limitFor('otlp-native.jsonl')),
policyFor(manifest, attempt, identity, deadlineAt, true));
if (receipts.status !== 'accepted' || !receipts.signals.logs.records || !receipts.signals.traces.records) traceFailure ??= 'native_export_incomplete';
} catch { traceFailure ??= 'native_evidence_incomplete'; }
@@ -79,7 +80,7 @@ export async function startControlledObservability({ manifest, attempt, request,
} finally { await close(); }
return inspectControlledObservability({ manifest, attempt, request, artifact: input.artifact,
report: { startedAt, traceFinishedAt: finishedAt, cleanupVerified: input.cleanupVerified, traceFailure },
deadlineAt, readEvidence: name => readFile(join(root, name)) });
deadlineAt, readEvidence: name => readRegularFile(join(root, name), limitFor(name)) });
})() };
} catch (error) { await close(); throw error; }
}
@@ -1,9 +1,9 @@
import { readFile } from 'node:fs/promises';
import { join, resolve } from 'node:path';
import { pathToFileURL } from 'node:url';
import { canonical, costRuntime, readJson, requestBindingSha256, sha256, units, usd, validateManifest } from './manifest.mjs';
import { inspectControlledObservability } from './observability.mjs';
import { attemptDirectory } from './run.mjs';
import { limitFor, readRegularFile } from './evidence-files.mjs';
export async function summarizeControlledCampaign(value, { filesByAttempt } = {}) {
const { manifest, manifestSha256, allocatedUsd, overheadUsd } = validateManifest(value, Date.now(), { admission: false });
@@ -12,9 +12,11 @@ export async function summarizeControlledCampaign(value, { filesByAttempt } = {}
for (const allocation of manifest.attempts) {
const root = attemptDirectory(manifest, allocation.attemptId);
const readEvidence = async name => {
if (filesByAttempt === undefined) return readFile(join(root, name));
const limit = limitFor(name);
if (!limit) throw new Error('Unexpected controller evidence path');
if (filesByAttempt === undefined) return readRegularFile(join(root, name), limit);
const bytes = filesByAttempt.get(allocation.attemptId)?.get(name);
if (!Buffer.isBuffer(bytes)) throw new Error('Missing archived controller evidence');
if (!Buffer.isBuffer(bytes) || bytes.length > limit) throw new Error('Missing archived controller evidence');
return bytes;
};
const readDocument = async (name, maximum = 1_048_576) => {
@@ -5,6 +5,7 @@ import { dirname, isAbsolute, join, relative, resolve, sep } from 'node:path';
import { pathToFileURL } from 'node:url';
import { parseArgs } from 'node:util';
import { startControlledObservability } from './observability.mjs';
import { limitFor, readRegularFile } from './evidence-files.mjs';
import { bindRequest, canonical, costRuntime, readJson, sha256, validateManifest } from './manifest.mjs';
async function atomicJson(file, value) {
@@ -146,8 +147,7 @@ registerHooks({ resolve(specifier, context, nextResolve) {
child = await runChild({ cli: stagedCli, loader, requestPath: retainedRequestPath, artifactPath, descriptorPath, observabilityPath: observability?.descriptorPath,
cwd: request.session.workingDirectory ?? root, environment: childEnvironment, deadlineAt, signal, closeGateway: () => Promise.all([gateway.close(), observability?.close()]) });
for (const stream of ['stdout', 'stderr']) await writeFile(join(root, `${stream}.log`), redact(child[stream]), { flag: 'wx', mode: 0o600 });
const raw = await readFile(artifactPath, 'utf8');
if (Buffer.byteLength(raw) > 16_777_216) throw new Error('Provider artifact exceeds its byte limit');
const raw = (await readRegularFile(artifactPath, limitFor('provider-artifact.json'))).toString('utf8');
secretExcluded = secrets.every(secret => !raw.includes(secret));
if (!secretExcluded) await writeFile(artifactPath, redact(raw), { mode: 0o600 });
providerArtifactSha256 = sha256(redact(raw));
@@ -178,9 +178,11 @@ registerHooks({ resolve(specifier, context, nextResolve) {
if (snapshot.overBudget || snapshot.conflictingRequests || snapshot.inflight || snapshot.unknownRequests !== snapshot.boundedUnresolvedRequests) failure ??= 'request_accounting_unbounded';
const receiptHashes = {};
try {
for (const request of snapshot.requests) receiptHashes[request.requestId] = sha256(await readFile(join(root, 'receipts', `${request.requestId}.json`)));
for (const request of snapshot.requests) receiptHashes[request.requestId] = sha256(await readRegularFile(join(root, 'receipts', `${request.requestId}.json`), limitFor(`receipts/${request.requestId}.json`)));
} catch { failure ??= 'request_receipt_missing'; }
const ledgerSha256 = sha256(await readFile(ledgerPath));
let ledgerSha256;
try { ledgerSha256 = sha256(await readRegularFile(ledgerPath, limitFor('ledger.jsonl'))); }
catch { failure ??= 'ledger_evidence_invalid'; }
if (observability) {
try {
observabilityEvidence = await observability.finish({ artifact, cleanupVerified, failure });
@@ -200,6 +202,7 @@ registerHooks({ resolve(specifier, context, nextResolve) {
await writeFile(outputPath, JSON.stringify(resultArtifact, null, 2) + '\n', { flag: 'wx', mode: 0o600 });
return { code: failure ? 2 : child?.code ?? 2, report };
} finally { await observability?.close(); }
}
if (process.argv[1] && import.meta.url === pathToFileURL(resolve(process.argv[1])).href) {
@@ -7,9 +7,9 @@ import { fileURLToPath } from "node:url";
import { resolve, dirname } from "node:path";
const root = resolve(dirname(fileURLToPath(import.meta.url)), "..");
const fixture = resolve(root, "test/fixtures/copilot-profile-v58-identity.json");
const fixture = resolve(root, "test/fixtures/copilot-profile-v59-identity.json");
const historical = JSON.parse(readFileSync(resolve(root, "test/fixtures/copilot-profile-v12-identity.json"), "utf8"));
const declaration = { ...historical.declaration, agentProfileVersion: 58 };
const declaration = { ...historical.declaration, agentProfileVersion: 59 };
delete declaration.permissionIdentitySourceSha256;
const sources = {
displayEventSourceSha256: "src/drivers/acpx/copilot-events.ts",
@@ -12,6 +12,7 @@ import { createHash, randomUUID } from "node:crypto";
import { spawn, type ChildProcessWithoutNullStreams } from "node:child_process";
import { readFileSync } from "node:fs";
import { fileURLToPath } from "node:url";
import { createAcpxSidecarHostEnvironment, createSanitizedAcpxSpawnInput } from "../drivers/acpx/environment.js";
import { afterEach, describe, expect, it, vi } from "vitest";
@@ -93,6 +94,22 @@ afterEach(async () => {
});
describe("qualified ACPX runtime sidecar", () => {
it("gives native Claude HTTP tools the Runner lease budget without changing sibling providers", () => {
const source = readFileSync(new URL("./acpx-runtime-sidecar.ts", import.meta.url), "utf8");
const start = source.indexOf(" environment:");
const end = source.indexOf(" expectedIdentity:", start);
expect(start).toBeGreaterThan(0); expect(end).toBeGreaterThan(start);
const create = new Function("process", "params", "createAcpxSidecarHostEnvironment", `return ({ ${source.slice(start, end)} }).environment;`);
const original = { MCP_TOOL_TIMEOUT: "1000", ANTHROPIC_API_KEY: "selected-key", UNRELATED_SECRET: "private" };
const claude = create({ env: original }, { agent: "claude", normalizedSessionId: "session" }, createAcpxSidecarHostEnvironment);
expect(createSanitizedAcpxSpawnInput(claude, "claude").env).toMatchObject({ MCP_TOOL_TIMEOUT: "3600000", ANTHROPIC_API_KEY: "selected-key" });
expect(createSanitizedAcpxSpawnInput(claude, "claude").env).not.toHaveProperty("UNRELATED_SECRET");
expect(original.MCP_TOOL_TIMEOUT).toBe("1000");
const codex = create({ env: original }, { agent: "codex", normalizedSessionId: "session" }, createAcpxSidecarHostEnvironment);
expect(codex).toBe(original);
expect(createSanitizedAcpxSpawnInput(codex, "codex").env).not.toHaveProperty("MCP_TOOL_TIMEOUT");
});
it("projects the actual sidecar terminal diagnostic block without authoritative accounting", () => {
const source = readFileSync(new URL("./acpx-runtime-sidecar.ts", import.meta.url), "utf8");
const start = source.indexOf(" try {", source.indexOf(" const usageAfter = await readSidecarHostStatusWithin(activeHost);"));
@@ -279,6 +296,42 @@ describe("qualified ACPX runtime sidecar", () => {
for (const agent of ["cursor", "codex", "pi"]) expect(create({ agent, tools: [] }, evidence, () => undefined).captureSemanticReceipt).toBeUndefined();
});
it("keeps a native tool pending through controller downtime without a duplicate side effect", async () => {
const source = readFileSync(new URL("./acpx-runtime-sidecar.ts", import.meta.url), "utf8");
const start = source.indexOf(" semanticTools: {");
const end = source.indexOf(" onGoalUpdate:", start);
expect(start).toBeGreaterThan(0); expect(end).toBeGreaterThan(start);
const create = new Function("params", "activeCopilotEvidence", "waitForTool", `return ({ ${source.slice(start, end)} }).semanticTools;`);
let admitted!: () => void, complete!: (value: unknown) => void;
const started = new Promise<void>(resolve => { admitted = resolve; });
const result = new Promise(resolve => { complete = resolve; });
let signal: AbortSignal | undefined;
const handler = vi.fn((call: RunnerToolCall) => { signal = call.signal; admitted(); return result; });
const bridge = await startRunnerToolBridge(create({ agent: "claude", tools: [{ name: "register_deliverable", inputSchema: { type: "object" } }] }, undefined, handler));
const call = () => fetch(bridge.url, { method: "POST", headers: { Authorization: `Bearer ${bridge.secret}`, "Content-Type": "application/json" },
body: JSON.stringify({ jsonrpc: "2.0", id: "saved-call", method: "tools/call", params: { name: "register_deliverable", arguments: {} } }) }).then(response => response.json());
let first: Promise<any> | undefined;
try {
vi.useFakeTimers({ toFake: ["setTimeout", "clearTimeout"] });
first = call();
await started;
await vi.advanceTimersByTimeAsync(125_000);
expect(signal?.aborted).toBe(false);
// The same pending invocation can be read again after reconnect. It must
// share the original result rather than create another external write.
const duplicate = call();
complete({ artifactId: "saved-artifact" });
expect(await first).toMatchObject({ result: { content: [{ text: '{"artifactId":"saved-artifact"}' }] } });
expect(await duplicate).toEqual(await first);
expect(handler).toHaveBeenCalledOnce();
} finally {
complete({ artifactId: "saved-artifact" });
await first?.catch(() => undefined);
await bridge.close();
vi.useRealTimers();
}
});
it.each(["forwarded", "emit-failed"])("binds native v2 receipts to actual sidecar-normalized input: %s", async mode => {
const raw = { reportedWorkDisposition: "done", summary: "Complete", evidence: [], verification: [],
completionClaim: { contractRevision: "1", objectiveSatisfied: true, criteria: [], remainingWork: [] } };
@@ -305,12 +305,22 @@ async function dispatch(
providerPolicy: params.providerPolicy,
systemInstructions: params.systemInstructions,
runtimeContext: params.runtimeContext,
environment: createAcpxSidecarHostEnvironment(process.env, params.agent, params.normalizedSessionId),
environment: params.agent === "claude" ? {
...createAcpxSidecarHostEnvironment(process.env, params.agent, params.normalizedSessionId),
// Claude's HTTP MCP client otherwise aborts at 60s, before the
// native Runner connection lease. Cancellation and Runner shutdown
// still revoke the pending call; this is not a new authority lease.
MCP_TOOL_TIMEOUT: String(60 * 60 * 1_000),
} : createAcpxSidecarHostEnvironment(process.env, params.agent, params.normalizedSessionId),
expectedIdentity: params.expectedIdentity,
assignedMcpTransport: "prp",
semanticTools: {
tools: params.tools,
handler: waitForTool,
// These calls are durable PRP requests. A controller outage must not
// turn a queued write into a provider-visible failure and retry.
// Runner lease expiry, turn cancellation and bridge close own them.
timeoutMs: null,
...(params.agent === "copilot" ? { captureSemanticReceipt: () => activeCopilotEvidence?.captureSemanticReceipt() } : {}),
},
onGoalUpdate: (goal) => {
@@ -47,6 +47,10 @@ export interface RunnerExecutionDriver {
}) => Promise<Record<string, unknown>>;
/** Active-connection recovery budget. Omitted for the existing local mode. */
runnerReconnectGraceMs?: number;
/** Process lifetime while disconnected. Defaults to the active-connection
* recovery budget for compatibility. Null uses the authenticated connection
* lease as the lifetime bound, so controller downtime cannot kill its owner. */
runnerProcessReconnectGraceMs?: number | null;
/**
* A verified Runner that outlived its controller. Adoption registers
* the durable authority and waits for this exact process to reconnect; it
@@ -87,9 +87,19 @@ vi.mock("./installation-integrity.js", () => ({ verifyNativeAcpxInstallation: vi
describe("Copilot build-owned installation", () => {
it("admits the current profile with environment launch preparation", () => {
expect(QUALIFIED_ACPX_PROFILES.copilot.agentProfileVersion).toBe(58);
expect(QUALIFIED_ACPX_PROFILES.copilot.agentProfileVersion).toBe(59);
expect(QUALIFIED_ACPX_PROFILES.copilot.qualificationStatus).toBeUndefined();
expect(QUALIFIED_ACPX_PROFILES.copilot.commandDigest).toBe("sha256:d490dacdab02051d24c84aae49ac6fb12b8226cf8a47b014ec53f3d3a8410a94");
expect(QUALIFIED_ACPX_PROFILES.copilot.commandDigest).toBe("sha256:2f39ffe54a6af832d044b5098c31df9e08d4d27aee0ac675ba4b0c9da7445c40");
});
it("preserves v58 while binding the separate remote process reconnect policy in v59", () => {
const prior = JSON.parse(readFileSync(new URL("../../../test/fixtures/copilot-profile-v58-identity.json", import.meta.url), "utf8"));
const current = JSON.parse(readFileSync(new URL("../../../test/fixtures/copilot-profile-v59-identity.json", import.meta.url), "utf8"));
expect(Object.keys(current.declaration).filter(key => current.declaration[key] !== prior.declaration[key]).sort()).toEqual([
"agentProfileVersion", "credentialBindingSourceSha256", "nativeInstructionsSourceSha256", "nativeSessionRuntimeSourceSha256", "profileCompatibilitySourceSha256", "runtimeRequestBridgeSourceSha256", "semanticBridgeSourceSha256", "semanticSidecarSourceSha256", "transportContractSourceSha256",
]);
expect(isSupportedAcpxProfileVersion("copilot", 58)).toBe(true);
expect(current.commandDigest).not.toBe(prior.commandDigest);
});
it("preserves v57 while binding the shared Pi setup and launch repairs in v58", () => {
@@ -6,6 +6,13 @@ afterEach(() => vi.unstubAllEnvs());
import { ACPX_CREDENTIAL_BINDING_ENV, createAcpxCredentialBinding, createAcpxSidecarHostEnvironment, createSanitizedAcpxSpawnInput } from "./environment.js";
describe("ACPX launch environment", () => {
it("retains an explicit Claude MCP timeout without inheriting an ambient or sibling setting", () => {
vi.stubEnv("MCP_TOOL_TIMEOUT", "999999");
expect(createSanitizedAcpxSpawnInput(undefined, "claude").env).not.toHaveProperty("MCP_TOOL_TIMEOUT");
expect(createSanitizedAcpxSpawnInput({ MCP_TOOL_TIMEOUT: "120000" }, "claude").env.MCP_TOOL_TIMEOUT).toBe("120000");
expect(createSanitizedAcpxSpawnInput({ MCP_TOOL_TIMEOUT: "120000" }, "codex").env).not.toHaveProperty("MCP_TOOL_TIMEOUT");
});
it("excludes general IAM from the Pi sidecar and launch, including selected task values", () => {
const iam = { AWS_ACCESS_KEY_ID: "general-key", AWS_SECRET_ACCESS_KEY: "general-secret", AWS_SESSION_TOKEN: "general-session" };
const source = { ...iam, ...configuredEnvironmentProjection({ ...iam, CUSTOM_SETTING: "allowed" }), AWS_BEARER_TOKEN_BEDROCK: "provider-key" };
@@ -131,6 +131,7 @@ export function createSanitizedAcpxSpawnInput(
...(environment === undefined ? [] : ["PAPERCLIP_AGENT_KEY_ID", "PAPERCLIP_AGENT_PUBLIC_KEY", "PAPERCLIP_AGENT_PRIVATE_KEY"]),
...credentialNames,
...(agent === "claude" ? CLAUDE_ROUTING_ENV_KEYS : []),
...(agent === "claude" && environment !== undefined ? ["MCP_TOOL_TIMEOUT"] : []),
...(usesPreconfiguredHome(source) ? [...PRECONFIGURED_HOME_KEYS, "PAPERCLIP_RUNNER_HOME_MODE", "ACPX_CLAUDE_INCLUDE_USER_SETTINGS"] : []),
]);
let retainedBytes = 0;
@@ -22,12 +22,12 @@ export const QUALIFIED_ACPX_PROFILE_DATA = {
"protocolVersion": 1,
"acpxVersion": "0.13.1",
"agent": "pi",
"agentProfileVersion": 28,
"agentProfileVersion": 29,
"agentServerPackage": "pi-acp",
"agentServerVersion": "0.0.33",
"agentRuntimePackage": "@earendil-works/pi-coding-agent",
"agentRuntimeVersion": "1.0.0",
"commandDigest": "sha256:4918a4d5472d66b22b1ec642b68b5c60a8092a39c84ec1ab1c9ad0289cb8c68c",
"commandDigest": "sha256:e80c6dc7bde12790371345ee5a12f0201d30941d26f1a780aec3d090545b2a06",
"permissionPolicy": "interactive"
},
"cursor": {
@@ -48,12 +48,12 @@ export const QUALIFIED_ACPX_PROFILE_DATA = {
"protocolVersion": 1,
"acpxVersion": "0.13.1",
"agent": "copilot",
"agentProfileVersion": 58,
"agentProfileVersion": 59,
"agentServerPackage": "@github/copilot",
"agentServerVersion": "1.0.88",
"agentRuntimePackage": null,
"agentRuntimeVersion": null,
"commandDigest": "sha256:d490dacdab02051d24c84aae49ac6fb12b8226cf8a47b014ec53f3d3a8410a94",
"commandDigest": "sha256:2f39ffe54a6af832d044b5098c31df9e08d4d27aee0ac675ba4b0c9da7445c40",
"permissionPolicy": "interactive"
},
"claude": {
@@ -155,11 +155,20 @@ describe("Pi installation factory", () => {
expect(current.commandDigest).not.toBe(prior.commandDigest);
});
it("preserves profile 28 while binding the Claude-only explicit MCP client budget source in profile 29", async () => {
const read = async (version: number) => JSON.parse(await readFile(new URL(`../../../test-fixtures/pi-acp/profile-v${version}-identity.json`, import.meta.url), "utf8"));
const prior = await read(28), current = await read(29);
expect(Object.keys(current.declaration).filter(key => JSON.stringify(current.declaration[key]) !== JSON.stringify(prior.declaration[key])).sort())
.toEqual(["agentProfileVersion", "credentialEnvironmentSourceSha256"]);
expect(current.declaration.closure).toEqual(prior.declaration.closure);
expect(current.commandDigest).not.toBe(prior.commandDigest);
});
it("binds the profile declaration to the reviewed patch and platform closure pins", async () => {
const hash = (bytes: string | Uint8Array) => createHash("sha256").update(bytes).digest("hex");
const canonical = (value: any): string => Array.isArray(value) ? `[${value.map(canonical).join(",")}]`
: value && typeof value === "object" ? `{${Object.keys(value).sort().map(key => `${JSON.stringify(key)}:${canonical(value[key])}`).join(",")}}` : JSON.stringify(value);
const identity = JSON.parse(await readFile(new URL("../../../test-fixtures/pi-acp/profile-v28-identity.json", import.meta.url), "utf8"));
const identity = JSON.parse(await readFile(new URL("../../../test-fixtures/pi-acp/profile-v29-identity.json", import.meta.url), "utf8"));
const declaration = identity.declaration;
expect(declaration.sharedRuntimeContract).toBe("paperclip.acpx-runtime-contract.v1");
expect(identity.commandDigest).toBe(`sha256:${hash(canonical(declaration))}`);
@@ -190,21 +199,21 @@ describe("Pi installation factory", () => {
it("rejects legacy profiles and tampered executable identities", () => {
expect(() => assertPiInstallationProfile(candidate())).not.toThrow();
expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 1 })).toThrow("version 28");
expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 2 })).toThrow("version 28");
expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 3 })).toThrow("version 28");
expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 4 })).toThrow("version 28");
expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 5 })).toThrow("version 28");
expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 6 })).toThrow("version 28");
expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 7 })).toThrow("version 28");
expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 8 })).toThrow("version 28");
expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 9 })).toThrow("version 28");
expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 10 })).toThrow("version 28");
expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 11 })).toThrow("version 28");
expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 12 })).toThrow("version 28");
expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 13 })).toThrow("version 28");
expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 14 })).toThrow("version 28");
for (const priorVersion of [15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27]) expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: priorVersion })).toThrow("version 28");
expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 1 })).toThrow("version 29");
expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 2 })).toThrow("version 29");
expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 3 })).toThrow("version 29");
expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 4 })).toThrow("version 29");
expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 5 })).toThrow("version 29");
expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 6 })).toThrow("version 29");
expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 7 })).toThrow("version 29");
expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 8 })).toThrow("version 29");
expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 9 })).toThrow("version 29");
expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 10 })).toThrow("version 29");
expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 11 })).toThrow("version 29");
expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 12 })).toThrow("version 29");
expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 13 })).toThrow("version 29");
expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 14 })).toThrow("version 29");
for (const priorVersion of [15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28]) expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: priorVersion })).toThrow("version 29");
for (const changed of [{ agentServerVersion: "latest" }, { commandDigest: `sha256:${"0".repeat(64)}` }, { agentRuntimePackage: "ambient-pi" }]) {
expect(() => assertPiInstallationProfile({ ...candidate(), ...changed })).toThrow("trusted declaration");
}
@@ -2,13 +2,13 @@ import { QUALIFIED_ACPX_PROFILE_DATA } from "./generated-profiles.js";
/** Decodable historical profile revisions, not permission to launch them.
* Exact current profile/package/digest admission is checked separately. */
type HistoricalAcpxProfileVersion = 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57;
type HistoricalAcpxProfileVersion = 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58;
type CurrentAcpxProfileVersion = typeof QUALIFIED_ACPX_PROFILE_DATA[keyof typeof QUALIFIED_ACPX_PROFILE_DATA]["agentProfileVersion"];
export type AcpxProfileVersion = HistoricalAcpxProfileVersion | CurrentAcpxProfileVersion;
const historicalVersions: Readonly<Record<string, readonly HistoricalAcpxProfileVersion[]>> = {
pi: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27], claude: [1, 2, 3, 4, 5], codex: [1, 2, 3, 4, 5],
grok: [1, 2, 3, 4, 5], copilot: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57],
pi: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28], claude: [1, 2, 3, 4, 5], codex: [1, 2, 3, 4, 5],
grok: [1, 2, 3, 4, 5], copilot: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58],
cursor: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15],
};
@@ -1513,7 +1513,7 @@ describe("OpenCodeServerDriver", () => {
);
const isolatedHome = runtimeEnvironment.home;
expect(startupFailurePending).toBe(false);
expect(dirname(isolatedHome)).toBe(sessionRoot);
expect(dirname(isolatedHome)).toBe(join(sessionRoot, "homes"));
expect(basename(isolatedHome)).toMatch(/^home-/);
await expect(
readFile(
@@ -1553,6 +1553,59 @@ describe("OpenCodeServerDriver", () => {
readFile(join(instructionRoot, "sibling.md"), "utf8"),
).resolves.toBe("instruction sibling\n");
await session.close({ reason: "test" });
await expect(readdir(join(sessionRoot, "homes"))).resolves.toEqual([]);
});
it("removes isolated homes left by earlier launches when a session reopens", async () => {
await chmod(fixture, 0o755);
const root = await mkdtemp(join(tmpdir(), "paperclip-opencode-legacy-home-"));
const workspace = await mkdtemp(
join(tmpdir(), "paperclip-opencode-legacy-home-workspace-"),
);
const skillRoot = join(root, "skill-source");
const instructionRoot = join(root, "instruction-source");
roots.push(root, workspace);
await Promise.all([
mkdir(skillRoot, { recursive: true }),
mkdir(instructionRoot, { recursive: true }),
]);
await writeFile(join(skillRoot, "SKILL.md"), "# Assigned\n");
await writeFile(join(instructionRoot, "AGENTS.md"), "instructions\n");
const sessionRoot = join(root, "legacy");
const legacyHome = join(sessionRoot, "home-AbC123");
const sealedSkill = join(legacyHome, ".claude", "skills", "assigned");
await mkdir(sealedSkill, { recursive: true });
await mkdir(join(legacyHome, ".npm", "_cacache"), { recursive: true });
await writeFile(join(legacyHome, ".npm", "_cacache", "entry"), "cache");
await writeFile(join(sealedSkill, "SKILL.md"), "# Assigned\n");
await chmod(join(sealedSkill, "SKILL.md"), 0o444);
await chmod(sealedSkill, 0o555);
await mkdir(join(sessionRoot, "data", "opencode"), { recursive: true });
await writeFile(join(sessionRoot, "data", "opencode", "opencode.db"), "history");
const driver = new OpenCodeServerDriver({
model: "openrouter/deepseek/deepseek-v4-flash-0731",
runtimeDirectory: root,
command: fixture,
runtimeContext: runtimeContext(skillRoot, instructionRoot),
environment: {
PATH: process.env.PATH,
OPENROUTER_API_KEY: "fixture-key",
},
});
const session = await driver.openSession({
runId: "run-legacy-home",
normalizedSessionId: "legacy",
workingDirectory: workspace,
});
await expect(lstat(legacyHome)).rejects.toMatchObject({ code: "ENOENT" });
await session.close({ reason: "test" });
expect(
(await readdir(sessionRoot)).filter((entry) => entry.startsWith("home-")),
).toEqual([]);
await expect(readdir(join(sessionRoot, "homes"))).resolves.toEqual([]);
await expect(
readFile(join(sessionRoot, "data", "opencode", "opencode.db"), "utf8"),
).resolves.toBe("history");
});
it("resends system instructions without repeating the task envelope on recovery", async () => {
@@ -6,6 +6,7 @@ import { accessSync, constants, readFileSync, realpathSync, statSync } from "nod
import { createRequire } from "node:module";
import {
chmod,
lstat,
mkdir,
mkdtemp,
readFile,
@@ -85,7 +86,10 @@ import {
type OpenCodeMcpBridge,
} from "./mcp-bridge.js";
import { nativeMcpLaunchBinding } from "../native-mcp.js";
import { materializeNativeRuntimeSkills } from "../runtime-context-materializer.js";
import {
materializeNativeRuntimeSkills,
releaseMaterializedNativeRuntimeSkills,
} from "../runtime-context-materializer.js";
export const OPENCODE_SERVER_DRIVER_KIND = "opencode_server" as const;
export const QUALIFIED_OPENCODE_VERSION = "1.18.34" as const;
@@ -369,6 +373,7 @@ export class OpenCodeServerDriver implements HarnessDriver {
);
await mkdir(root, { recursive: true, mode: 0o700 });
await writeFile(join(root, "workspace"), `${cwd}\n`, { mode: 0o600 });
await removeLegacyIsolatedHomes(root);
const trace = await createProviderTraceFileSink({
path: this.#options.environment?.PAPERCLIP_PROVIDER_TRACE_PATH,
provider: "opencode",
@@ -2281,7 +2286,9 @@ async function startRuntime(input: {
// HOME contains a read-only skill snapshot, whose destination must be fresh.
// Keep OpenCode's XDG data stable for provider-session recovery while giving
// every launch (including retries and recovery) a new isolated HOME.
const isolatedHome = await mkdtemp(join(input.root, "home-"));
const homesRoot = join(input.root, OPENCODE_ISOLATED_HOMES_DIRECTORY);
await mkdir(homesRoot, { recursive: true, mode: 0o700 });
const isolatedHome = await mkdtemp(join(homesRoot, "home-"));
await chmod(isolatedHome, 0o700);
try {
await materializeNativeRuntimeSkills(
@@ -2289,18 +2296,14 @@ async function startRuntime(input: {
join(isolatedHome, ".claude", "skills"),
);
} catch (error) {
await rm(isolatedHome, { recursive: true, force: true }).catch(
() => undefined,
);
await removeIsolatedHome(isolatedHome).catch(() => undefined);
throw error;
}
const bridge = await startOpenCodeMcpBridge({
tools: input.options.dynamicTools,
handler: input.dispatch,
}).catch(async (error) => {
await rm(isolatedHome, { recursive: true, force: true }).catch(
() => undefined,
);
await removeIsolatedHome(isolatedHome).catch(() => undefined);
throw error;
});
const assignedMcp = nativeMcpLaunchBinding(
@@ -2335,7 +2338,7 @@ async function startRuntime(input: {
const providerProxy = modelProvider === "paperclip" && input.options.environment?.PAPERCLIP_AI_PROVIDER_URL
? await startOpenCodeProviderProxy(input.options.environment.PAPERCLIP_AI_PROVIDER_URL, input.options.environment.PAPERCLIP_AI_PROVIDER_KEY ?? "", providerModelId, input.options.environment).catch(async error => {
await bridge.close().catch(() => {});
await rm(isolatedHome, { recursive: true, force: true }).catch(() => {});
await removeIsolatedHome(isolatedHome).catch(() => {});
throw error;
})
: null;
@@ -2568,9 +2571,7 @@ async function startRuntime(input: {
await rm(join(configHome, "opencode", "opencode.json"), {
force: true,
}).catch(() => undefined);
await rm(isolatedHome, { recursive: true, force: true }).catch(
() => undefined,
);
await removeIsolatedHome(isolatedHome).catch(() => undefined);
if (closeInput.finalizeTrace !== false) {
await input.trace?.finish({ reason: closeInput.reason ?? null });
}
@@ -2585,9 +2586,7 @@ async function startRuntime(input: {
await rm(join(configHome, "opencode", "opencode.json"), {
force: true,
}).catch(() => undefined);
await rm(isolatedHome, { recursive: true, force: true }).catch(
() => undefined,
);
await removeIsolatedHome(isolatedHome).catch(() => undefined);
throw error;
}
}
@@ -2962,8 +2961,9 @@ function sanitizedEnvironmentKeys(): string[] {
];
}
function sessionRoot(
runtimeDirectory: string,
export const OPENCODE_ISOLATED_HOMES_DIRECTORY = "homes";
export function opencodeRuntimeSessionDirectoryName(
normalizedSessionId: string,
): string {
const safe = normalizedSessionId
@@ -2971,7 +2971,35 @@ function sessionRoot(
.slice(0, 120);
if (!safe || safe === "." || safe === "..")
throw new Error("Invalid normalized OpenCode session id");
return join(resolve(runtimeDirectory), safe);
return safe;
}
function sessionRoot(
runtimeDirectory: string,
normalizedSessionId: string,
): string {
return join(
resolve(runtimeDirectory),
opencodeRuntimeSessionDirectoryName(normalizedSessionId),
);
}
async function removeIsolatedHome(home: string): Promise<void> {
const skillsParent = await lstat(join(home, ".claude")).catch(() => null);
if (skillsParent?.isDirectory()) {
await releaseMaterializedNativeRuntimeSkills(
join(home, ".claude", "skills"),
).catch(() => undefined);
}
await rm(home, { recursive: true, force: true });
}
async function removeLegacyIsolatedHomes(root: string): Promise<void> {
const entries = await readdir(root, { withFileTypes: true }).catch(() => []);
for (const entry of entries) {
if (!entry.isDirectory() || !entry.name.startsWith("home-")) continue;
await removeIsolatedHome(join(root, entry.name)).catch(() => undefined);
}
}
function validateWorkspace(value: string): string {
@@ -246,7 +246,7 @@ describe("runner semantic MCP bridge", () => {
});
});
it("honors MCP cancellation after the handler starts", async () => {
it.each([60_000, null])("honors MCP cancellation after the handler starts (timeout=%s)", async timeoutMs => {
let markStarted!: () => void;
const started = new Promise<void>((resolve) => {
markStarted = resolve;
@@ -254,7 +254,7 @@ describe("runner semantic MCP bridge", () => {
const aborted = vi.fn();
const bridge = await startRunnerToolBridge({
tools: [tool("documents.read")],
timeoutMs: 60_000,
timeoutMs,
handler: ({ signal }) => new Promise((_resolve, reject) => {
signal.addEventListener("abort", () => {
aborted();
@@ -291,6 +291,23 @@ describe("runner semantic MCP bridge", () => {
expect(aborted).toHaveBeenCalledOnce();
});
it("cancels a lease-owned native tool when its bridge closes", async () => {
let admitted!: () => void;
const started = new Promise<void>(resolve => { admitted = resolve; });
let signal: AbortSignal | undefined;
const bridge = await startRunnerToolBridge({ tools: [tool("documents.read")], timeoutMs: null,
handler: call => { signal = call.signal; admitted(); return new Promise(() => {}); },
});
bridges.push(bridge);
const pending = rpc(bridge, { id: "close-me", method: "tools/call", params: { name: "documents.read", arguments: {} } });
await started;
await bridge.close();
expect(signal?.aborted).toBe(true);
expect(await (await pending).json()).toMatchObject({ result: { isError: true,
content: [{ text: "Paperclip tool call cancelled" }],
} });
});
it("preserves successful mutation identity when the result is oversized", async () => {
const result = { text: `snowman-${"☃".repeat(65 * 1024)}` };
const handler = vi.fn(async () => result);
@@ -45,7 +45,9 @@ export interface RunnerToolBridgeOptions {
handler(call: RunnerToolCall): Promise<unknown>;
/** Copilot-only opt-in; capture the active invocation scope before dispatch. */
captureSemanticReceipt?: () => ((receipt: SemanticToolReceipt) => void) | undefined;
timeoutMs?: number;
/** Null delegates the wait to an owning durable runtime's lease and close.
* Ordinary callers retain the bounded request timeout. */
timeoutMs?: number | null;
privateToolTimeoutMs?: number;
maxBodyBytes?: number;
/** Internal test seam; production bounds headers and body reads to 30 seconds. */
@@ -143,7 +145,7 @@ export async function startRunnerToolBridge(
controllers,
handler: options.handler,
captureSemanticReceipt: options.captureSemanticReceipt,
timeoutMs: positiveBoundedInteger(
timeoutMs: options.timeoutMs === null ? null : positiveBoundedInteger(
options.timeoutMs,
DEFAULT_TIMEOUT_MS,
1,
@@ -214,7 +216,7 @@ async function handleRequest(
controllers: Map<string, AbortController>;
handler: RunnerToolBridgeOptions["handler"];
captureSemanticReceipt: RunnerToolBridgeOptions["captureSemanticReceipt"];
timeoutMs: number;
timeoutMs: number | null;
privateToolTimeoutMs: number;
privateToolNames: ReadonlySet<string>;
maxBodyBytes: number;
@@ -599,24 +601,24 @@ function closeServer(server: Server): Promise<void> {
function withCancellationAndTimeout<T>(
promise: Promise<T>,
controller: AbortController,
timeoutMs: number,
timeoutMs: number | null,
): Promise<T> {
return new Promise((resolve, reject) => {
let settled = false;
const finish = (callback: () => void) => {
if (settled) return;
settled = true;
clearTimeout(timer);
if (timer !== undefined) clearTimeout(timer);
controller.signal.removeEventListener("abort", onAbort);
callback();
};
const onAbort = () =>
finish(() => reject(new Error("Paperclip tool call cancelled")));
const timer = setTimeout(() => {
const timer = timeoutMs === null ? undefined : setTimeout(() => {
finish(() => reject(new Error("Paperclip tool call timed out")));
controller.abort();
}, timeoutMs);
timer.unref();
timer?.unref();
controller.signal.addEventListener("abort", onAbort, { once: true });
promise.then(
(value) => finish(() => resolve(value)),
@@ -857,6 +857,11 @@ function runnerCloseGraceMs(options: Pick<CapabilityRunnerdCodexTransportOptions
return options.closeGraceMs ?? (options.provider === "acpx" && options.acpxAgent === "pi" ? 45_000 : 10_000);
}
function runnerProcessReconnectGraceMs(options: CapabilityRunnerdCodexTransportOptions): number | undefined {
if (options.runnerProcessReconnectGraceMs === null) return undefined;
return options.runnerProcessReconnectGraceMs ?? options.runnerReconnectGraceMs;
}
function runnerCloseDeadlines(
startedAtMs: number,
graceMs: number,
@@ -3494,6 +3499,9 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
},
],
});
#handlerInstalled = false;
readonly #handlerReady: Promise<void>;
#resolveHandlerReady!: () => void;
#core: DurablePrpControlPlane | null = null;
#handle: RunnerProcessHandle | null = null;
#adoptedRunnerMonitor: NodeJS.Timeout | null = null;
@@ -3622,6 +3630,9 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
this.#failureSignal = new Promise<never>((_resolve, reject) => {
this.#rejectFailureSignal = reject;
});
this.#handlerReady = new Promise<void>(resolve => {
this.#resolveHandlerReady = resolve;
});
this.#closedSignal = new Promise<void>(resolve => {
this.#resolveClosedSignal = resolve;
});
@@ -3997,6 +4008,8 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
setServerRequestHandler(handler: CodexServerRequestHandler): void {
this.#handler = handler;
this.#handlerInstalled = true;
this.#resolveHandlerReady();
}
async #awaitWarmRunAttachmentReady(): Promise<void> {
@@ -5074,7 +5087,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
maxOutboxBytes: RUNNERD_MAX_OUTBOX_BYTES,
p0ReserveBytes: RUNNERD_P0_RESERVE_BYTES,
maxRuntimeMs: 0,
reconnectGraceMs: this.options.runnerReconnectGraceMs,
reconnectGraceMs: runnerProcessReconnectGraceMs(this.options),
lifecyclePolicy: this.options.lifecyclePolicy,
runnerBinaryPath,
runnerVersion: runnerArtifact.version,
@@ -5759,7 +5772,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
maxOutboxBytes: RUNNERD_MAX_OUTBOX_BYTES,
p0ReserveBytes: RUNNERD_P0_RESERVE_BYTES,
maxRuntimeMs: 0,
reconnectGraceMs: this.options.runnerReconnectGraceMs,
reconnectGraceMs: runnerProcessReconnectGraceMs(this.options),
lifecyclePolicy: this.options.lifecyclePolicy,
runnerBinaryPath,
runnerVersion: runnerArtifact.version,
@@ -5915,6 +5928,18 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
: await Promise.race([admission.settled, this.#failureSignal]).catch(() => {
throw new SemanticToolNotDispatchedError();
});
// An adopted Runner can authenticate and deliver queued tools before the
// driver has finished reconstructing its session. Handler publication is
// the dispatch barrier; the default RPC diagnostic is not a tool outcome.
// The owning startup/transport lifetime bounds this wait, and the original
// authority/epoch checks below still apply after it becomes ready.
if (!this.#handlerInstalled) {
await Promise.race([
this.#handlerReady,
this.#failureSignal,
this.#closedSignal.then(() => { throw new SemanticToolNotDispatchedError(); }),
]).catch(() => { throw new SemanticToolNotDispatchedError(); });
}
try {
this.#throwIfFailed();
} catch {
@@ -0,0 +1,110 @@
import { createServer } from "node:http";
import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join, resolve } from "node:path";
import { expect, it, vi } from "vitest";
import type { DurablePrpControlPlane } from "../control-plane/durable-prp-control-plane.js";
import { codexSemanticToolSpecs } from "../drivers/codex/codex-app-server-driver.js";
import {
createCapabilityRunnerdCodexTransport,
defaultCapabilityRunnerdBinary,
} from "./runnerd-codex-transport.js";
it.each([
{ policy: "connection lease", processGraceMs: null, survives: true },
{ policy: "explicit process deadline", processGraceMs: 500, survives: false },
])("preserves the $policy policy across controller downtime beyond the connection recovery budget", async ({ processGraceMs, survives }) => {
const root = await mkdtemp(join(tmpdir(), "runner-controller-outage-"));
const server = createServer();
let authority: DurablePrpControlPlane | null = null;
server.on("upgrade", (request, socket, head) => {
if (!authority) { socket.destroy(); return; }
authority.handleUpgrade(request, socket, "/runner", head);
});
await new Promise<void>(resolveListen => server.listen(0, "127.0.0.1", resolveListen));
const address = server.address();
if (!address || typeof address === "string") throw new Error("Missing outage test listener");
const registration = async (next: DurablePrpControlPlane) => {
authority = next;
return {
connectUrl: `ws://127.0.0.1:${address.port}/runner`,
release: () => { if (authority === next) authority = null; },
};
};
const shared = {
runnerBinary: defaultCapabilityRunnerdBinary(),
codexCommand: resolve(import.meta.dirname, "../../runner/target/debug/fake-codex-app-server"),
codexArgs: ["--state-file", join(root, "provider-state.json"), "--hold-first-durable-turn"],
stateDirectory: root,
prpIdentity: {
runnerInstanceId: "runner-outage", environmentLeaseId: "lease-outage", runId: "run-outage",
normalizedSessionId: "session-outage", turnId: "turn-outage", itemId: "item-outage",
},
// Scale the production 120-second reconnect budget down. The live process
// must be bounded by its authenticated connection lease, independently of
// how long a controller spends retrying a connection or booting its server.
runnerReconnectGraceMs: 500,
runnerProcessReconnectGraceMs: processGraceMs,
controlPlaneRegistration: registration,
};
const first = createCapabilityRunnerdCodexTransport(shared);
let adopted: ReturnType<typeof createCapabilityRunnerdCodexTransport> | undefined;
let pid: number | null = null;
try {
const opened = await first.transport.request("thread/start", {
cwd: root, dynamicTools: codexSemanticToolSpecs(),
});
await first.transport.request("turn/start", { input: [{ type: "text", text: "Keep this task active." }] });
for await (const notification of first.transport.notifications()) {
if (notification.method === "turn/started") break;
}
pid = first.evidence().runnerPid;
const providerPid = first.evidence().codexPid;
expect(pid).toEqual(expect.any(Number));
expect(providerPid).toEqual(expect.any(Number));
await writeFile(join(root, "saved-work.txt"), "completed work survives controller replacement");
await first.detachControllerForRestart();
// The deliberate fault is an offline controller, not a slow test assertion.
await new Promise(resolveWait => setTimeout(resolveWait, 1_250));
const runnerState = JSON.parse(await readFile(join(root, "runner", "runner-state.json"), "utf8"));
if (!survives) {
expect(runnerState.recoverableFailure).toBe("transport_reconnect_grace_exceeded");
await vi.waitFor(() => expect(() => process.kill(pid!, 0)).toThrow());
return;
}
expect(runnerState.recoverableFailure).not.toBe("transport_reconnect_grace_exceeded");
expect(() => process.kill(pid!, 0)).not.toThrow();
expect(() => process.kill(providerPid!, 0)).not.toThrow();
const duplicateLaunch = vi.fn(() => { throw new Error("Must adopt the existing runner"); });
const thread = opened.thread as Record<string, unknown>;
adopted = createCapabilityRunnerdCodexTransport({
...shared,
runnerReconnectGraceMs: 5_000,
resumeDynamicTools: codexSemanticToolSpecs(),
resumeProviderSession: { driverSessionId: String(thread.id), providerSessionId: String(thread.sessionId) },
runnerProcessLauncher: duplicateLaunch,
adoptExistingRunner: {
pid: pid!, processGroupId: pid, startedAt: new Date().toISOString(),
isAlive: () => { try { process.kill(pid!, 0); return true; } catch { return false; } },
signal: signal => { try { process.kill(-pid!, signal); return true; } catch { return false; } },
},
});
adopted.transport.setServerRequestHandler(async () => ({ success: true, contentItems: [] }));
const restored = await adopted.transport.request("thread/read", {});
expect(restored.thread).toMatchObject({ id: thread.id, sessionId: thread.sessionId });
expect(adopted.evidence()).toMatchObject({ runnerPid: pid, codexPid: providerPid, runnerExited: false });
expect(duplicateLaunch).not.toHaveBeenCalled();
expect(await readFile(join(root, "saved-work.txt"), "utf8")).toBe("completed work survives controller replacement");
// A real Stop still reaches the adopted owner; the lease-only disconnect
// policy cannot turn a detached process into an uncancellable executor.
await adopted.transport.close();
expect(adopted.evidence().runnerExited).toBe(true);
} finally {
await first.transport.close().catch(() => undefined);
await adopted?.transport.close().catch(() => undefined);
if (pid) { try { process.kill(-pid, "SIGKILL"); } catch { /* already stopped */ } }
server.closeAllConnections();
if (server.listening) await new Promise<void>(resolveClose => server.close(() => resolveClose()));
await rm(root, { recursive: true, force: true });
}
}, 30_000);
@@ -0,0 +1,54 @@
import { mkdtemp, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join, resolve } from "node:path";
import { expect, it, vi } from "vitest";
import type { DurablePrpControlPlane } from "../control-plane/durable-prp-control-plane.js";
import type { CodexRpcServerRequest } from "../drivers/codex/app-server-transport.js";
import { createCapabilityRunnerdCodexTransport, defaultCapabilityRunnerdBinary } from "./runnerd-codex-transport.js";
it.each(["install", "close", "detach"] as const)("holds an authenticated semantic call until handler readiness or %s", async mode => {
const root = await mkdtemp(join(tmpdir(), "runner-handler-readiness-"));
let core!: DurablePrpControlPlane;
const bundle = createCapabilityRunnerdCodexTransport({
runnerBinary: defaultCapabilityRunnerdBinary(),
codexCommand: resolve(import.meta.dirname, "../../runner/target/debug/fake-codex-app-server"),
codexArgs: ["--state-file", join(root, "provider.json"), "--emit-tool-call"],
stateDirectory: root,
controlPlaneRegistration: async authority => {
core = authority;
await authority.start();
return { release: () => undefined };
},
});
const handler = vi.fn(async (_request: CodexRpcServerRequest) => ({ success: true, contentItems: [{ type: "inputText", text: "one retained result" }] }));
let pid: number | null = null;
try {
await bundle.transport.request("thread/start", { cwd: root, dynamicTools: [{ name: "get_task_context", description: "Read the task.", inputSchema: { type: "object", properties: {}, additionalProperties: false } }] });
await bundle.transport.request("turn/start", { input: [{ type: "text", text: "Read the task." }] });
pid = bundle.evidence().runnerPid;
expect(bundle.evidence().runnerProcessGroupId).toBe(pid);
await vi.waitFor(() => expect(core.store.state.committedEvents.some(event => event.eventType === "semantic_tool.input")).toBe(true));
// The authenticated provider is already asking for a tool, but the driver
// has not published its recovered session/handler yet. No error result may
// be manufactured in this gap, and no external effect has been dispatched.
await new Promise(resolveWait => setTimeout(resolveWait, 50));
expect(core.store.state.commands.filter(command => command.type === "semantic_tool.result")).toHaveLength(0);
expect(core.semanticToolResultsSettled()).toBe(false);
if (mode === "close") await bundle.transport.close();
if (mode === "detach") await bundle.detachControllerForRestart();
bundle.transport.setServerRequestHandler(handler);
if (mode === "install") {
await vi.waitFor(() => expect(handler).toHaveBeenCalledOnce());
await vi.waitFor(() => expect(core.store.state.commands.filter(command => command.type === "semantic_tool.result")).toHaveLength(1));
expect(core.store.state.commands.find(command => command.type === "semantic_tool.result")?.payload).toMatchObject({ isError: false });
expect(handler.mock.calls[0]?.[0]).toMatchObject({ method: "item/tool/call", params: { tool: "get_task_context" } });
} else {
await new Promise(resolveWait => setTimeout(resolveWait, 50));
expect(handler).not.toHaveBeenCalled();
}
} finally {
await bundle.transport.close().catch(() => undefined);
if (pid) { try { process.kill(-pid, "SIGKILL"); } catch { /* owned child already stopped */ } }
await rm(root, { recursive: true, force: true });
}
}, 20_000);
@@ -0,0 +1,51 @@
{
"commandDigest": "sha256:e80c6dc7bde12790371345ee5a12f0201d30941d26f1a780aec3d090545b2a06",
"declaration": {
"schema": "paperclip.acpx_profile_declaration.v1",
"agent": "pi",
"agentProfileVersion": 29,
"acpxVersion": "0.13.1",
"agentServerVersion": "0.0.33",
"agentRuntimeVersion": "1.0.0",
"nodeVersion": "24.21.0",
"closure": {
"darwin-arm64": "282022db10150c6632b3444df421342e7d534bdf5d5fb1097a2e79d0625a2bcf",
"darwin-x64": "64e251e19009f755c0b04f73ce2138246faab71a961b0f13d75ebfcc34bef12e",
"linux-x64": "713b1fdff42fb56a1518bdc084f181d70bee8ebadc3e4b1d76321ed9108c8410"
},
"wrapperSha256": "9d129b3d38772e93e97080aa6c4e574ac5df4e47ce5bc331a484a9fbf8188b36",
"helperSha256": "41e0490b617da0d60e0c8ec58ef311236f945129d99f816cff6897f78da7f91d",
"extensionSha256": "f5818573a355702388072b28926897db546a5cf346b5171f646582ce5f9102a4",
"agentFilesBinding": "registered-runtime-context-v1",
"nativeQuestions": "select-confirm-input-editor-v4-root-field-types",
"pathPolicy": "sdk-normalization-v1",
"mcpToolNames": "provider-safe-exact-source-v1",
"acpxPatchSha256": "00d6af17216cb0cad902b48633eb19bf0ec6b05837a984cb24661fca593f09c1",
"nativeAssistantMessages": "pi1-rpc-deltas-v3-bound-message-and-tool-receipts",
"nativeNotices": "session-extension-v2-bounded-runtime-failure",
"messageProjectionSha256": "17d1e91b948dfa6a56e1905645add52b502041ad843c8d50a13b07956f22aadd",
"noticeProjectionSha256": "014485adc690998e395d334ea3ae67f72be6060a2fc78a1185df7b8d1da28fbf",
"sharedRuntimeContract": "paperclip.acpx-runtime-contract.v1",
"nativeInputDisposition": "queued-only-v1",
"cacheWarming": "disabled-owned-decision-v1",
"thinkingLevel": {
"supported": [
"off",
"low",
"high",
"max"
],
"newConfigurationDefault": "low",
"admission": "explicit-native-effective-before-prompt-v1",
"recovery": "mode-bound-v1"
},
"modelSelection": "caller-selected-native-verified-v1",
"providerConfigurationSourceSha256": "75a9a13f13f224dc092bd8ebf49e8747162172943761e44476c8eb18d80c52d4",
"credentialEnvironmentSourceSha256": "7586d44b47c83395498dc8806287cd3538ef58177ba7c5949f5766ed105a0161",
"runtimeSandboxSourceSha256": "92dfa53f5a3ed0062913ae417eeccec243b430d444495efde2a8e9c4c02e78ea",
"recoveryIdentitySourceSha256": "d2c9fc161fcc135fb9760110bfa3d984ee2bb38d87fa5ce7cb7418a49dd89e66",
"nativeCancellation": "acknowledged-cancelled-terminal-preserves-usage-v1",
"dependencySecurityPatchSha256": "5273a195b952f233336122842faaf650193c04f36b9359d20d4d63079b0ce814",
"openRouterAttributionSourceSha256": "1211191529d6073b733b311c3d69cada4973eede0f3e540c46e261f78e0b90f9"
}
}
@@ -0,0 +1,85 @@
{
"commandDigest": "sha256:2f39ffe54a6af832d044b5098c31df9e08d4d27aee0ac675ba4b0c9da7445c40",
"declaration": {
"schema": "paperclip.acpx_profile_declaration.v1",
"agent": "copilot",
"agentProfileVersion": 59,
"acpxVersion": "0.13.1",
"agentServerVersion": "1.0.88",
"protocolPolicyRevision": "copilot-agent-manual-v1",
"acpxPatchSha256": "00d6af17216cb0cad902b48633eb19bf0ec6b05837a984cb24661fca593f09c1",
"policySha256": "a4be69be43db84052560420a3ca546a665c9165573c24d543eb33d144ba138a3",
"distributionSourceSha256": "c0a65e247fc47b263bd369a3d5ab505c4b323d4e6493be8a9ecbeb0e9dbb9107",
"agentFilesBinding": "registered-runtime-context-v1",
"systemInstructionDelivery": "COPILOT_HOME/copilot-instructions.md:replace-under-lifetime-lease-before-launch:v1",
"sharedRuntimeContract": "paperclip.acpx-runtime-contract.v1",
"innerDistributionSourceSha256": "9bb08347eed01f1ab44e065c8b06ead9be3a25973d6e6329dc4c0a099096279a",
"upstreamAppSha256": "7b48282a19b5b0814a0c96ad5e3a125173d792cc55f9525b2effea962f6063c0",
"patchedAppSha256": "61f8eebe0c30cc03b311c6818b2acb158a6d16ca58c8bfac5ebd42db095c2679",
"messageIdentityContract": "copilot-native-message-id-v1",
"ownedDistributionDelivery": "COPILOT_CLI_DIST_DIR:lease-owned-guarded-inner-distribution:v1",
"permissionContextContract": "copilot-edit-permission-context-v1",
"permissionContextSourceSha256": "de2d38d702b3c574161af0c0c04790fb6f6347916eb3916ce7a131a8c229121b",
"permissionAdapterSourceSha256": "ee5929ed2ee647ad0369465dee9d51a00b2dd28e1ff735b43e969ddc9c164668",
"permissionLocationsSourceSha256": "f39a81da395b6e23aa2a1a45572a28846e5e81bfd3d764d9a456f74a79c752e9",
"permissionRedactionSourceSha256": "ad6770c69269087951790e2e87e22ac088e3d432769adf17ebc865ed512e6c4f",
"permissionClassifierSourceSha256": "1161eabd82e26133e3b3db3ab567f65d52808b03f176e93497e02c8330349fb4",
"semanticToolReceiptContract": "paperclip.semantic_tool_receipt.v2",
"semanticReceiptSourceSha256": "b2795726b2234a663fec1a6ca11f0551ae69882fe9e82aa39686e952d036e791",
"semanticBridgeSourceSha256": "8764b8c9032fdc7d1415b01e75a9fc57c550e7a5e11da4bed7d9100082872ae5",
"toolEvidenceSourceSha256": "6eef5039afc75bb851eaf9cd82a6f099e4bd398ca2deb59dd66f5b2a81a7a706",
"semanticNormalizedInputContract": "validated-forwarded-input-commit-v1",
"semanticSidecarSourceSha256": "17361d1782c0dcebb2b7972635f45c890f2a4081f8bc0ef2d2276550e0b91629",
"semanticDirectDriverSourceSha256": "ff572638f2cb5d9f8249ff8e2f412311c1e3af209ef0bf7e5ea9f10a21d91103",
"semanticValidationSourceSha256": "02fbb92681c78c5de5d2f1aae25aabc25a0e5df6b0ff4baccac652192a8da975",
"semanticNormalizationSourceSha256": "8b628b82e8eacc91cdcc6fdca2210ba4cf282f01e9c6f74e95ff271655b5047e",
"semanticCompletionContractSourceSha256": "3343df6362252d44d12b079ff78c8f6539d1544f3ce5ec36ff36db70ab0fac31",
"semanticValidatorsSourceSha256": "f22d9caf2035b600abc1a6625ae23ca97f1409d7e29ed5b7264bb8f9784c7dc9",
"semanticSchemaBundleSourceSha256": "016391a087cc217aafed4ba1f4efafc6e4881a7525952dce824b7f4d24530603",
"semanticSidecarProtocolSourceSha256": "03c0c95b900f9dc18f29f6be27b8c79bc0bcab29780190d46a74e95328081236",
"semanticSidecarReceiptCommitContract": "correlated-tool-resolve-v1",
"displayEventSourceSha256": "d72f9a6035c247bc6a42c067c05753076c54322c4b58984e93922f3325dc6877",
"displayProjectionSourceSha256": "75952606b26dd93bbc582c1f0a4424487377272ea745fb3a27ceae031fd5dc7e",
"configuredEnvironmentSourceSha256": "5c9008d121979a3a8987e6b8a5752172eafdc4a2b5d725a73f3bbb8fb83be8ff",
"configuredEnvironmentRustSourceSha256": "85c87fae62b8a9377e4f0a4c00302c7519fae4397b1c48253f8f2b7ba7faf994",
"profileActivitySourceSha256": "4fdb6b0f9125fd8372ba6f27945fa5c8a3d3b2ec891df544092e3dc7a75d5282",
"profileExtensionsSourceSha256": "a80ddc95eb4505095bfdbbf85c758199949ab20be9c84b34c77f762eb7180571",
"profileInstallationSourceSha256": "3e61c4cdd4b78e38fcb16eaadae4f1fd65ce3c7127184ea7a80948c8e5d77f88",
"attachmentPolicySourceSha256": "4f7638b7cc64e18f6a4ba33fd46de2b3d49225e32aeab41f5c695b268fa4f8e2",
"singleReadEvidenceSourceSha256": "4069393405089bde613cecec04f5123c426594811da4c668bd96f0e7e283550f",
"nativeSessionRuntimeSourceSha256": "1fe6fa9e6b96d84d70432b85a57c473f833606d4a521563d5022b4d3c41576ce",
"nativeRuntimeContextSourceSha256": "71607f80dbb8dc5cc6c123d0b8f3995d6c5407277695a8c82cbedc90fb8f4842",
"nativeRuntimePromptsSourceSha256": "103b83066da72434602e9e461e6278dfb40bb993d2f3b49b2e93df81bbb08b72",
"nativeSessionContractSourceSha256": "22a8c96ce41ba4f03e976fc37f0cc7c5d05cee99fa789941da31ee1dab440b90",
"harnessSessionContractSourceSha256": "03f4cd7f76f34f6d82e009b66aa51f18f6e23dc235b660778060e35b6ea07412",
"harnessBackendSourceSha256": "dc827e281ae6731545b166ee4d709a4f67f594a556dcbe63c70fe8f1aca26249",
"harnessSessionSourceSha256": "81361733e86b56dd5a66415aefc6f72e69be24acc8d3417370154ebb7fef6e94",
"transportContractSourceSha256": "03fca93d36a89b0cc0917335e293ad097fdf04356d21b307827bb0ea4c13f523",
"nativeBackendSourceSha256": "972c08e6e320d1384221e180391fe817ea1ba632db687201e457d64e68ef4360",
"nativeBackendFactorySourceSha256": "7e431c926b76d53c4c08e3a4105353dcd34bd940f6588a1a68f79631f6feeb44",
"nativeInstructionsSourceSha256": "9173ebccf09cb56a781d2bf5c7eb27fdafb13d3450d99342c6aff86b42d6b43a",
"providerLaunchSourceSha256": "d4f4a69baeeb2f3b2f350840f81f711e40f08588c28ea28039748dde90302a51",
"launchPreparationStoreSourceSha256": "ff6025568bbc0aa6ce545772d73dfca2d3da226740bd3a7b386b069e76cff296",
"runtimeRequestBridgeSourceSha256": "7497083b6cf1fc0268187aff948352f3bce6e054ac6e845005846473efdaff15",
"runtimeRequestFacadeSourceSha256": "d0ed28e0c5ddad34db8f437e704c8a755eb0b47f27e08e44185f66d1d2205295",
"runtimeRequestNotificationSourceSha256": "6196c6175f11c69295dc00fc773201f4fdeca26c500653870253c43b81e65211",
"guardedInstallationSourceSha256": "56a177d3e702914818dca4ca50e061f0d25f988bfc4c0582c36ce23c1696d82f",
"nativeExecutionSourceSha256": "fc7c6568feb0d8d709e9baa51839420c9b90336b4865a9b8405aba06e4d1e9c4",
"profileCompatibilitySourceSha256": "6a26b102b78f282137ff93b0f75eb8788c603fc177802c58715c5ad7b2f3b9e3",
"installationSourceSha256": "a1c7c6f103b898216973e77ffbac0ef7572e250aa574d1a4065270e9dfa0dc2d",
"assetOwnershipSourceSha256": "1785e3f31e5dad5a13b53e25d8cf9482a9bd89a27b6c127830aed6b81acc605c",
"assetInstallerSourceSha256": "d3bd9514e5920d0cc3567aa50487fb534c57a6f66b8da0d3f22c5f6e499c841f",
"credentialBindingSourceSha256": "7586d44b47c83395498dc8806287cd3538ef58177ba7c5949f5766ed105a0161",
"sidecarErrorSourceSha256": "b09af4577f8bf42eee7036b2ad9842202cfc763e72b53c3287f82c6090cdad80",
"profileSourceSha256": "f5538b3dbefa7634a332b87d5ededcefb56aa7e5790cede1c7a0d948a548a456",
"metadataProbeSourceSha256": "be4c8416edd27061109bc1f33d0c26071b4facbb3f651beaae6b3c0275cce727",
"permissionToolIdentitySourceSha256": "770b47305e571a81f344883057d0e4a482a1cd59ac77d045836593a8e739deb1",
"semanticContinuationSourceSha256": "d248e47a0fe7f2536d1e64cea0b0b410f34104800cd643d995b4473104ecc8e0",
"runtimeHostSourceSha256": "94a73749267f5a80b5df19413a65b4b9263c00a9bf7e98fda00f00b8f9fd32ec",
"runtimeAdapterSourceSha256": "e4c764c2c3c7dd655a965b39472bd1ad8d577d1e7258b25594b171c9429752b8",
"runtimeSandboxSourceSha256": "92dfa53f5a3ed0062913ae417eeccec243b430d444495efde2a8e9c4c02e78ea",
"cancellationSourceSha256": "a07d1a9e982dec0ae130c1ad1efa5bc75c6a39e013b730ba3ee45256f09ceeca",
"agentFilesSourceSha256": "fb32f6badc13dfa22f231479509b631b2f0eba0c99b1ab9f3cd5c0c601aa03ba",
"nativeDistributionSourceSha256": "aed327c389e957483a503cb1bb8b247ee7cd7ff4b528fb4567c39bcb4577690b"
}
}
+3
View File
@@ -889,6 +889,7 @@ export type {
AgentDetail,
ClearAgentErrorResponse,
AgentPermissions,
AgentProjectAccess,
AgentPublicIdentity,
AgentInstructionsBundleMode,
AgentInstructionsFileSummary,
@@ -1130,6 +1131,7 @@ export type {
IssueExecutionStagePrincipal,
IssueExecutionDecision,
IssueComment,
IssueRecentComments,
IssueQueuedCommentEntry,
IssueQueuedCommentProtocol,
IssueQueuedCommentQueue,
@@ -1882,6 +1884,7 @@ export {
qualifyAgentRunnerSchema,
agentPermissionsSchema,
updateAgentPermissionsSchema,
addAgentProjectAccessSchema,
type CreateAgent,
type BuiltInAgentProvision,
type BuiltInAgentReset,
+5
View File
@@ -25,6 +25,11 @@ export interface AgentPermissions extends Record<string, unknown> {
authorizationPolicy?: TrustAuthorizationPolicy;
}
export interface AgentProjectAccess {
projectIds: string[];
canManage: boolean;
}
export type AgentRuntimeConfig = Record<string, unknown> & {
aiConnection?: import("../ai-connection-router.js").AiRuntimeConnectionBinding;
};
@@ -104,7 +104,9 @@ export interface TaskInterruptionPresentation {
state: "working" | "retry_scheduled" | "failed" | "waiting_for_user" | "resolved";
nextRetryAt: string | null;
canRetry: boolean;
userAction: "approval" | "reconnect" | "assign_agent" | "choose_model" | "resume" | "budget" | null;
userAction: "approval" | "reconnect" | "assign_agent" | "choose_model" | "resume" | "budget" | "project_access" | null;
/** Only the task's own project id is exposed. Other source projects stay unnamed. */
projectAccess?: { agentId: string; projectId: string | null; ready: boolean };
sourceRunIds: string[];
observedRunId: string | null;
}
+2
View File
@@ -280,6 +280,7 @@ export type {
AgentDetail,
ClearAgentErrorResponse,
AgentPermissions,
AgentProjectAccess,
AgentPublicIdentity,
AgentRuntimeConfig,
AgentInstructionsBundleMode,
@@ -731,6 +732,7 @@ export type {
IssueReviewRequest,
IssueExecutionDecision,
IssueComment,
IssueRecentComments,
IssueQueuedCommentEntry,
IssueQueuedCommentProtocol,
IssueQueuedCommentQueue,
+7
View File
@@ -1720,3 +1720,10 @@ export interface IssueAttachment {
openPath?: string;
downloadPath?: string;
}
/** Bounded cold-open window; the complete chronological feed stays paginated. */
export interface IssueRecentComments {
latestCommentId: string | null;
comments: IssueComment[];
hasMore: boolean;
}
+4
View File
@@ -322,6 +322,10 @@ export const updateAgentPermissionsSchema = z.object({
export type UpdateAgentPermissions = z.infer<typeof updateAgentPermissionsSchema>;
export const addAgentProjectAccessSchema = z.object({
projectIds: z.array(z.string().uuid()).min(1).max(100),
}).strict();
export const resolveAgentInstructionCandidateSchema = z.object({
baseRevisionId: z.string().uuid().nullable(),
content: z.string().max(1024 * 1024),
+1
View File
@@ -370,6 +370,7 @@ export {
qualifyAgentRunnerSchema,
agentPermissionsSchema,
updateAgentPermissionsSchema,
addAgentProjectAccessSchema,
type CreateAgent,
type BuiltInAgentProvision,
type BuiltInAgentReset,
+2 -2
View File
@@ -52,8 +52,8 @@ assert.equal(JSON.parse(await readFile(join(server, 'package.json'), 'utf8')).na
const { resolvePaperclipRunnerBinary } = await import(pathToFileURL(join(server, 'dist/services/native-runtime/native-codex-runner.js')));
const { createCapabilityRunnerdCodexTransport } = await import(pathToFileURL(join(server, 'dist/vendor/paperclip-runner/live/runnerd-codex-transport.js')));
const { QUALIFIED_ACPX_PROFILES } = await import(pathToFileURL(join(server, 'dist/vendor/paperclip-runner/drivers/acpx/qualified-profiles.js')));
assert.equal(QUALIFIED_ACPX_PROFILES.pi.agentProfileVersion, 28);
assert.equal(QUALIFIED_ACPX_PROFILES.pi.commandDigest, 'sha256:4918a4d5472d66b22b1ec642b68b5c60a8092a39c84ec1ab1c9ad0289cb8c68c');
assert.equal(QUALIFIED_ACPX_PROFILES.pi.agentProfileVersion, 29);
assert.equal(QUALIFIED_ACPX_PROFILES.pi.commandDigest, 'sha256:e80c6dc7bde12790371345ee5a12f0201d30941d26f1a780aec3d090545b2a06');
assert.equal(Object.hasOwn(QUALIFIED_ACPX_PROFILES.pi, 'qualificationModel'), false);
assert.equal(Object.hasOwn(QUALIFIED_ACPX_PROFILES.pi, 'reportedModelId'), false);
const daemon = resolvePaperclipRunnerBinary();
+28 -1
View File
@@ -2,7 +2,7 @@ import { execFile } from "node:child_process";
import { mkdirSync, mkdtempSync, realpathSync, rmSync, symlinkSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import path from "node:path";
import { pathToFileURL } from "node:url";
import { fileURLToPath, pathToFileURL } from "node:url";
import { promisify } from "node:util";
import { afterEach, beforeEach, describe, expect, it } from "vitest";
@@ -85,6 +85,33 @@ describe("bundleRuntime", () => {
expect(output.entryDir).toBe(path.join(root, "server", "dist"));
});
it("keeps database caller locations public in a source-mapped runtime bundle", async () => {
const captureModule = fileURLToPath(new URL("../../packages/db/src/response-capture.ts", import.meta.url));
write(path.join(root, "server", "dist", "services", "response-caller.js"), [
`import { publicDatabaseCallers } from ${JSON.stringify(captureModule)};`,
'export function callers() { return publicDatabaseCallers(new Error("private-capture-message").stack); }',
].join("\n"));
write(path.join(root, "server", "dist", "index.js"), [
'import { callers } from "./services/response-caller.js";',
'console.log(JSON.stringify({ callers: callers() }));',
].join("\n"));
const summary = await bundleRuntime({ entry: path.join(root, "server", "dist", "index.js"), outdir: path.join(root, "server", "dist-runtime") });
const env = { ...process.env };
delete env.NODE_OPTIONS;
delete env.PAPERCLIP_DB_RESPONSE_CAPTURE_WINDOW;
const options = { cwd: path.join(root, "server"), env, timeout: 30_000 };
const plain = JSON.parse((await execFileAsync(process.execPath, [summary.outfile], options)).stdout);
const mapped = JSON.parse((await execFileAsync(process.execPath, ["--enable-source-maps", summary.outfile], options)).stdout);
expect(plain.callers).toEqual([]);
expect(mapped.callers).toEqual([expect.stringMatching(/^services\/response-caller\.js:\d+:\d+$/)]);
for (const result of [plain, mapped]) {
const output = JSON.stringify(result);
expect(output).not.toContain(root);
expect(output).not.toContain(captureModule);
expect(output).not.toContain("private-capture-message");
}
});
it("emits a server module that loads without starting the server", async () => {
const outdir = path.join(root, "server", "dist-runtime");
const summary = await bundleRuntime({ entry: path.join(root, "server", "dist", "index.js"), outdir });
@@ -5,6 +5,7 @@ import { beforeEach, describe, expect, it, vi } from "vitest";
const mockActivityService = vi.hoisted(() => ({
list: vi.fn(),
forIssue: vi.fn(),
inputFactsForIssue: vi.fn(),
runsForIssue: vi.fn(),
issuesForRun: vi.fn(),
create: vi.fn(),
@@ -152,6 +153,22 @@ describe("activity routes", () => {
expect(res.body).toEqual([{ action: "issue.recovery_action_resolved", actorType: "system" }]);
});
it("reads bounded input facts with the same issue access checks", async () => {
const commentId = "11111111-1111-4111-8111-111111111111";
mockIssueService.getByIdentifier.mockResolvedValue({ id: "issue-1", companyId: "company-1" });
mockActivityService.inputFactsForIssue.mockResolvedValue([{ action: "issue.queued_comment_steered" }]);
const app = await createApp({ type: "agent", companyId: "company-1", agentId: "agent-1" });
const res = await requestApp(app, baseUrl => request(baseUrl).get(`/api/issues/TEST-1/activity?view=comment-inputs&commentIds=${commentId}`));
expect(res.status).toBe(200);
expect(mockActivityService.inputFactsForIssue).toHaveBeenCalledWith("company-1", "issue-1", [commentId]);
expect(mockAccessService.decide).toHaveBeenCalledWith(expect.objectContaining({ action: "issue:read" }));
const invalid = await requestApp(app, baseUrl => request(baseUrl).get("/api/issues/TEST-1/activity?view=comment-inputs&commentIds=invalid"));
expect(invalid.status).toBe(400);
const oversized = await requestApp(app, baseUrl => request(baseUrl).get(`/api/issues/TEST-1/activity?view=comment-inputs&commentIds=${Array(101).fill(commentId).join(",")}`));
expect(oversized.status).toBe(400);
expect(mockActivityService.inputFactsForIssue).toHaveBeenCalledTimes(1);
});
it.each(["boundary", "company"])("denies agent activity reads outside the %s scope", async (kind) => {
const issue = { id: "issue-1", companyId: kind === "company" ? "company-2" : "company-1" };
mockIssueService.getById.mockResolvedValue(issue);
@@ -161,6 +178,9 @@ describe("activity routes", () => {
const res = await requestApp(app, baseUrl => request(baseUrl).get("/api/issues/issue-1/activity"));
expect(res.status).toBe(kind === "company" ? 404 : 403);
expect(mockActivityService.forIssue).not.toHaveBeenCalled();
const inputs = await requestApp(app, baseUrl => request(baseUrl).get("/api/issues/issue-1/activity?view=comment-inputs"));
expect(inputs.status).toBe(kind === "company" ? 404 : 403);
expect(mockActivityService.inputFactsForIssue).not.toHaveBeenCalled();
});
it("returns redacted all-actors rows to a basic company reader", async () => {
@@ -0,0 +1,102 @@
import { randomUUID } from "node:crypto";
import express from "express";
import request from "supertest";
import { eq } from "drizzle-orm";
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
import { agents, companies, createDb, heartbeatRunEvents, heartbeatRuns, issueComments, issues } from "@paperclipai/db";
import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { createLocalAgentJwt } from "../agent-auth-jwt.js";
import { actorMiddleware } from "../middleware/auth.js";
import { errorHandler } from "../middleware/error-handler.js";
import { captureRunIdentity, initializeRunIdentity, reserveSteeredIdentity } from "../services/run-identity.js";
const support = await getEmbeddedPostgresTestSupport();
(support.supported ? describe : describe.skip)("agent authentication identity transfer", () => {
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let db: ReturnType<typeof createDb>;
beforeAll(async () => {
vi.stubEnv("PAPERCLIP_AGENT_JWT_SECRET", "identity-transfer-test-secret");
vi.stubEnv("PAPERCLIP_INSTANCE_ID", "default");
database = await startEmbeddedPostgresTestDatabase("paperclip-auth-identity-transfer-");
db = createDb(database.connectionString);
}, 30_000);
afterAll(async () => {
vi.unstubAllEnvs();
await db?.$client.end({ timeout: 5 });
await database?.cleanup();
}, 60_000);
async function seed() {
const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), runId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Authentication transfer", issuePrefix: companyId.slice(0, 8) });
await db.insert(agents).values({ id: agentId, companyId, name: "Reader", role: "engineer", adapterType: "codex_local", status: "idle" });
await db.insert(issues).values({ id: issueId, companyId, title: "Authentication task", responsibleUserId: "owner" });
const padding = "retained input ".repeat(32_768);
const payloads = {
contextSnapshot: { issueId, launchInput: padding },
runnerProfileJson: { nativeExecutionInput: { savedInput: padding } },
resultJson: { savedResult: padding }, stdoutExcerpt: padding, stderrExcerpt: padding,
};
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running", ...payloads });
const identity = await initializeRunIdentity(db, { companyId, runId, issueId, messageIds: [], responsibleUserId: "owner", cause: "instruction" });
const app = express();
app.use(actorMiddleware(db, { deploymentMode: "authenticated", resolveSession: async () => null }));
app.get("/actor", (req, res) => res.json(req.actor));
app.patch("/actor", (req, res) => res.json(req.actor));
app.use(errorHandler);
const token = createLocalAgentJwt(agentId, companyId, "codex_local", runId, "stale-user");
expect(token).not.toBeNull();
return { companyId, agentId, issueId, runId, app, token: token!, identity, payloads };
}
it("bounds actual middleware responses and preserves the complete stored run", async () => {
const fixture = await seed();
const events: Array<Record<string, unknown>> = [], now = Date.now();
const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event));
installDatabaseResponseCapture(db, { heartbeatRuns }, capture);
try {
const response = await request(fixture.app).get("/actor").set("Authorization", `Bearer ${fixture.token}`);
expect(response.status).toBe(200);
expect(response.body).toMatchObject({ type: "agent", runId: fixture.runId,
identityContextId: fixture.identity.id, onBehalfOfUserId: "owner" });
} finally { capture.flush(); }
const reads = events.filter(event => event.event === "paperclip.db_response_capture.entry.v1"
&& event.table === "heartbeat_runs" && (event.operation === "select" || event.operation === "execute"));
expect(reads.reduce((sum, event) => sum + Number(event.calls), 0)).toBeGreaterThanOrEqual(3);
expect(reads.reduce((sum, event) => sum + Number(event.responseValueBytes), 0)).toBeLessThan(16_384);
expect((await captureRunIdentity(db, fixture)).run).toMatchObject(fixture.payloads);
});
it("blocks uncertain steering, accepts its durable acknowledgement, and rechecks cancellation", async () => {
const fixture = await seed(), messageId = randomUUID();
await db.insert(issueComments).values({ id: messageId, companyId: fixture.companyId, issueId: fixture.issueId,
authorUserId: "next-user", body: "Next instruction" });
const pending = await reserveSteeredIdentity(db, { ...fixture, messageId });
expect(pending).not.toBeNull();
const blocked = await request(fixture.app).get("/actor").set("Authorization", `Bearer ${fixture.token}`);
expect(blocked.status).toBe(409);
await db.insert(heartbeatRunEvents).values({ companyId: fixture.companyId, agentId: fixture.agentId, runId: fixture.runId,
seq: 1, eventType: "item.completed", sourceEventId: randomUUID(),
payload: { prpEvent: { turnId: "auth-turn", itemId: `auth-turn:steer:${messageId}`,
payload: { kind: "steering_acknowledgement" } } } });
const accepted = await request(fixture.app).get("/actor").set("Authorization", `Bearer ${fixture.token}`);
expect(accepted.status).toBe(200);
expect(accepted.body).toMatchObject({ identityContextId: pending!.id, onBehalfOfUserId: "next-user" });
await db.update(heartbeatRuns).set({ status: "cancelled" }).where(eq(heartbeatRuns.id, fixture.runId));
const revoked = await request(fixture.app).patch("/actor").set("Authorization", `Bearer ${fixture.token}`);
expect(revoked.status).toBe(403);
expect(revoked.body.code).toBe("agent_run_cancelled");
const [stored] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId));
expect(stored).toMatchObject(fixture.payloads);
});
it.each(["company", "agent", "missing", "finished"] as const)("rejects %s run identity for the compact scope", async (fault) => {
const fixture = await seed();
if (fault === "finished") await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, fixture.runId));
const input = { companyId: fault === "company" ? randomUUID() : fixture.companyId,
agentId: fault === "agent" ? randomUUID() : fixture.agentId,
runId: fault === "missing" ? randomUUID() : fixture.runId };
await expect(captureRunIdentity(db, input, { readScope: "request_authentication" })).rejects.toThrow(/active run/);
});
});
@@ -16,6 +16,7 @@ const mockHeartbeatService = vi.hoisted(() => ({
getActiveRunIssueSummaryForAgent: vi.fn(),
getRunReadAccess: vi.fn(),
getRunLogAccess: vi.fn(),
getRunWorkspaceAccess: vi.fn(),
readLog: vi.fn(),
wakeup: vi.fn(),
getRun: vi.fn(),
@@ -378,6 +379,10 @@ describe("agent live run routes", () => {
logStore: "local_file",
logRef: "logs/aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa.ndjson",
});
mockHeartbeatService.getRunWorkspaceAccess.mockResolvedValue({
id: "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa", companyId: "company-1", scopeKind: "company", issueId: null,
executionWorkspaceId: null,
});
mockHeartbeatService.readLog.mockResolvedValue({
runId: "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa",
store: "local_file",
@@ -609,10 +614,10 @@ describe("agent live run routes", () => {
});
});
it("uses narrow run log metadata lookups for log polling", async () => {
it.each([false, true])("uses narrow run log metadata lookups (aligned tail: %s)", async (aligned) => {
const res = await requestApp(await createApp(), (baseUrl) =>
request(baseUrl).get(
"/api/heartbeat-runs/aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa/log?offset=12&limitBytes=64",
`/api/heartbeat-runs/aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa/log?offset=12&limitBytes=64${aligned ? "&alignStartToRecord=true" : ""}`,
),
);
@@ -630,6 +635,8 @@ describe("agent live run routes", () => {
{
offset: 12,
limitBytes: 64,
signal: expect.any(AbortSignal),
...(aligned ? { alignStartToRecord: true } : {}),
},
);
expect(res.body).toEqual({
@@ -687,6 +694,21 @@ describe("agent live run routes", () => {
},
);
it("lists workspace operations using narrow current run access", async () => {
mockHeartbeatService.getRunWorkspaceAccess.mockResolvedValue({
id: "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa", companyId: "company-1", scopeKind: "company", issueId: null,
executionWorkspaceId: " workspace-one ",
});
mockWorkspaceOperationService.listForRun.mockResolvedValue([]);
const response = await requestApp(await createApp(), baseUrl => request(baseUrl)
.get("/api/heartbeat-runs/aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa/workspace-operations"));
expect(response.status).toBe(200);
expect(response.body).toEqual([]);
expect(mockHeartbeatService.getRunWorkspaceAccess).toHaveBeenCalledWith("aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa");
expect(mockHeartbeatService.getRun).not.toHaveBeenCalled();
expect(mockWorkspaceOperationService.listForRun).toHaveBeenCalledWith("aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa", "workspace-one");
});
it("caps company live run polling by default", async () => {
const rows = Array.from({ length: 75 }, (_, index) => ({
id: `run-${index}`,
@@ -0,0 +1,197 @@
import { randomUUID } from "node:crypto";
import express from "express";
import request from "supertest";
import { and, eq } from "drizzle-orm";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import { agents, agentWakeupRequests, activityLog, authUsers, companies, companyMemberships, createDb,
executionWorkspaces, heartbeatRuns, issues, projects, projectWorkspaces, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS } from "@paperclipai/db";
import { registerAgentProjectAccessRoutes } from "../routes/agent-project-access.js";
import { errorHandler } from "../middleware/error-handler.js";
import { assertTaskWorkspaceExecutionAccess, assertTaskWorkspaceSourceProjectAccess } from "../services/task-workspace-access.js";
import { taskInterruptionPresentation } from "../services/task-interruption-presentation.js";
import { ProjectAccessDeniedError } from "../services/project-access-denial.js";
import { issueService } from "../services/issues.js";
import { executionWorkspaceService } from "../services/execution-workspaces.js";
import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
describe("additive project access", () => {
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
let db: ReturnType<typeof createDb>;
beforeAll(async () => {
const external = process.env.PAPERCLIP_TEST_DATABASE_URL;
if (external) db = createDb(external);
else { database = await startEmbeddedPostgresTestDatabase("agent-project-access-"); db = createDb(database.connectionString); }
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
afterAll(async () => { await db?.$client.end(); await database?.cleanup(); });
async function seed() {
const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), runId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Project access", issuePrefix: `A${companyId.slice(0, 6)}` });
const selected = await db.insert(projects).values(["Existing", "Needed", "Third"].map(name => ({ companyId, name }))).returning();
const permissions = { canCreateAgents: false, canCreateSkills: false, trustPreset: "low_trust_review",
authorizationPolicy: { reviewPreset: { id: "low_trust_review", version: 1, rawOutputDisposition: "quarantine" },
customPolicy: { preserve: true }, trustBoundary: { mode: "low_trust_review", companyId,
projectIds: [selected[0]!.id], issueIds: [issueId], allowedToolClasses: ["git.read"], allowedSecretBindingIds: [] } } };
await db.insert(agents).values({ id: agentId, companyId, name: "Reviewer", adapterType: "process", status: "idle", permissions });
await db.insert(companyMemberships).values({ companyId, principalType: "agent", principalId: agentId, membershipRole: "member", status: "active" });
const [issue] = await db.insert(issues).values({ id: issueId, companyId, title: "Saved message", status: "blocked", assigneeAgentId: agentId, projectId: selected[1]!.id }).returning();
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, issueId, status: "failed", errorCode: "setup_failed",
error: "Project is outside this low-trust boundary.", contextSnapshot: { issueId } });
return { companyId, agentId, issue: issue!, runId, selected, permissions };
}
function app(actor: express.Request["actor"]) {
const app = express(); app.use(express.json()); app.use((req, _res, next) => { req.actor = actor; next(); });
registerAgentProjectAccessRoutes(app, db); app.use(errorHandler); return app;
}
const board = { type: "board" as const, source: "local_implicit" as const };
it("keeps a historical denial blocked after the task moves to an allowed project but retains its source", async () => {
const f = await seed();
const [source] = await db.insert(projectWorkspaces).values({ companyId: f.companyId, projectId: f.selected[1]!.id,
name: "Original repository", cwd: "/tmp/project-access-original" }).returning();
await db.update(issues).set({ projectWorkspaceId: source!.id,
workspaceSelection: { version: 1, source: "project", selection: {
kind: "configured_source", projectWorkspaceId: source!.id, mode: "managed_isolated",
} } }).where(eq(issues.id, f.issue.id));
const moved = await issueService(db).update(f.issue.id, { projectId: f.selected[0]!.id });
expect(moved).toMatchObject({ projectId: f.selected[0]!.id, projectWorkspaceId: source!.id });
const actor = { type: "agent" as const, source: "agent_key" as const, agentId: f.agentId, companyId: f.companyId };
await expect(assertTaskWorkspaceExecutionAccess(db, actor, f.companyId, { issueId: f.issue.id }))
.rejects.toBeInstanceOf(ProjectAccessDeniedError);
expect(await taskInterruptionPresentation(db, moved!)).toMatchObject({
canRetry: false, userAction: "project_access", projectAccess: { ready: false, projectId: null },
});
expect((await request(app(board)).post(`/agents/${f.agentId}/project-access`)
.send({ projectIds: [f.selected[1]!.id] })).status).toBe(200);
expect(await taskInterruptionPresentation(db, moved!)).toMatchObject({ canRetry: true, projectAccess: { ready: true } });
await expect(assertTaskWorkspaceExecutionAccess(db, actor, f.companyId, { issueId: f.issue.id })).resolves.toBeDefined();
});
it.each(["applied", "pending"] as const)("rechecks the selected source after an %s repair instead of the old denied project", async repair => {
const f = await seed();
const [source] = await db.insert(projectWorkspaces).values({ companyId: f.companyId, projectId: f.selected[0]!.id,
name: "Allowed repository", cwd: "/tmp/project-access-allowed" }).returning();
await db.update(heartbeatRuns).set({ errorCode: "project_access_denied", error: null,
resultJson: { projectAccessDenied: { projectId: f.selected[1]!.id } } }).where(eq(heartbeatRuns.id, f.runId));
const moved = await issueService(db).update(f.issue.id, { projectId: f.selected[0]!.id });
if (repair === "applied") {
await issueService(db).update(f.issue.id, { projectWorkspaceId: source!.id, workspaceSelectionActor: board });
} else {
await executionWorkspaceService(db).selectTaskWorkspace({ companyId: f.companyId, issueId: f.issue.id, actor: board,
selection: { kind: "configured_source", projectWorkspaceId: source!.id, mode: "managed_isolated" },
expectedBindingRevision: 0, requestKey: randomUUID() });
}
expect(await taskInterruptionPresentation(db, moved!)).toMatchObject({ canRetry: true, projectAccess: { ready: true } });
// The original denied source was not granted, and presentation creates no execution.
expect((await db.select().from(agents).where(eq(agents.id, f.agentId)))[0]!.permissions).toEqual(f.permissions);
expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId))).toHaveLength(0);
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId))).toHaveLength(1);
if (repair === "pending") {
await db.update(issues).set({ workspaceBindingRevision: 2 }).where(eq(issues.id, f.issue.id));
const stale = await taskInterruptionPresentation(db, moved!);
expect(stale).toMatchObject({ state: "failed", canRetry: false, userAction: null });
expect(stale).not.toHaveProperty("projectAccess");
}
});
it("keeps a pending existing selection blocked if its workspace is archived before retry", async () => {
const f = await seed();
const task = await issueService(db).update(f.issue.id, { projectId: f.selected[0]!.id });
const [workspace] = await db.insert(executionWorkspaces).values({ companyId: f.companyId,
projectId: f.selected[0]!.id, name: "Selected files", cwd: "/tmp/project-access-selected",
mode: "isolated_workspace", strategyType: "git_worktree" }).returning();
const service = executionWorkspaceService(db);
await service.selectTaskWorkspace({ companyId: f.companyId, issueId: f.issue.id, actor: board,
selection: { kind: "existing", workspaceId: workspace!.id }, expectedBindingRevision: 0, requestKey: randomUUID() });
expect(await taskInterruptionPresentation(db, task!)).toMatchObject({ canRetry: true, projectAccess: { ready: true } });
expect(await service.archiveWorkspaceUnderLifecycleLock({ id: workspace!.id, patch: {}, closedAt: new Date() }))
.toMatchObject({ outcome: "archived" });
await expect(service.applyPendingTaskWorkspaceSelection({ companyId: f.companyId, issueId: f.issue.id,
runId: f.runId, actor: { type: "agent", source: "agent_key", agentId: f.agentId, companyId: f.companyId } }))
.rejects.toThrow("Workspace is unavailable or inaccessible");
const archived = await taskInterruptionPresentation(db, task!);
expect(archived).toMatchObject({ state: "failed", canRetry: false, userAction: null });
expect(archived).not.toHaveProperty("projectAccess");
});
it.each(["denied_project", "foreign_company"] as const)("retains the %s gate on a bound workspace even when its source is allowed", async boundary => {
const f = await seed(), foreign = boundary === "foreign_company" ? await seed() : null;
const [source] = await db.insert(projectWorkspaces).values({ companyId: f.companyId, projectId: f.selected[0]!.id,
name: "Allowed source", cwd: "/tmp/project-access-source" }).returning();
const [workspace] = await db.insert(executionWorkspaces).values({ companyId: foreign?.companyId ?? f.companyId,
projectId: foreign?.selected[0]!.id ?? f.selected[1]!.id, projectWorkspaceId: foreign ? null : source!.id,
name: "Retained files", cwd: "/tmp/project-access-bound", mode: "isolated_workspace", strategyType: "git_worktree" }).returning();
const [task] = await db.update(issues).set({ projectId: f.selected[0]!.id, projectWorkspaceId: source!.id,
executionWorkspaceId: workspace!.id }).where(eq(issues.id, f.issue.id)).returning();
const blocked = await taskInterruptionPresentation(db, task!);
if (boundary === "denied_project") {
expect(blocked).toMatchObject({ canRetry: false,
userAction: "project_access", projectAccess: { ready: false, projectId: null } });
} else {
expect(blocked).toMatchObject({ state: "failed", canRetry: false, userAction: null });
expect(blocked).not.toHaveProperty("projectAccess");
}
});
it("preserves containment, audits concurrent additions, and changes the persisted denial into an explicit retry", async () => {
const f = await seed();
const actor = { type: "agent" as const, source: "agent_key" as const, agentId: f.agentId, companyId: f.companyId };
await expect(assertTaskWorkspaceSourceProjectAccess(db, actor, f.companyId, f.selected[1]!.id)).rejects.toBeInstanceOf(ProjectAccessDeniedError);
expect(await taskInterruptionPresentation(db, f.issue)).toMatchObject({ userAction: "project_access", canRetry: false });
const api = app(board);
const results = await Promise.all(f.selected.slice(1).map(project => request(api).post(`/agents/${f.agentId}/project-access`).send({ projectIds: [project.id] })));
expect(results.map(result => result.status)).toEqual([200, 200]);
const [saved] = await db.select().from(agents).where(eq(agents.id, f.agentId));
expect(saved!.permissions).toEqual({ ...f.permissions, authorizationPolicy: { ...f.permissions.authorizationPolicy,
trustBoundary: { ...f.permissions.authorizationPolicy.trustBoundary, projectIds: expect.arrayContaining(f.selected.map(project => project.id)) } } });
expect((saved!.permissions as typeof f.permissions).authorizationPolicy.trustBoundary.projectIds).toHaveLength(3);
await expect(assertTaskWorkspaceSourceProjectAccess(db, actor, f.companyId, f.selected[1]!.id)).resolves.toBeUndefined();
expect(await taskInterruptionPresentation(db, f.issue)).toMatchObject({ canRetry: true, projectAccess: { ready: true } });
const again = await request(api).post(`/agents/${f.agentId}/project-access`).send({ projectIds: [f.selected[1]!.id] });
expect(again.status).toBe(200);
expect(await db.select().from(activityLog).where(and(eq(activityLog.companyId, f.companyId), eq(activityLog.action, "agent.project_access_added")))).toHaveLength(2);
expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId))).toHaveLength(0);
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId))).toHaveLength(1);
});
it("rejects foreign projects, agent self-grants, unknown policy fields, and pending-approval agents", async () => {
const f = await seed(), other = await seed(), api = app(board);
expect((await request(api).post(`/agents/${f.agentId}/project-access`).send({ projectIds: [other.selected[1]!.id] })).status).toBe(404);
expect((await request(api).post(`/agents/${f.agentId}/project-access`).send({ projectIds: [f.selected[1]!.id], trustPreset: "standard" })).status).toBe(400);
expect((await request(app({ type: "agent", agentId: f.agentId, companyId: f.companyId, source: "agent_key" }))
.post(`/agents/${f.agentId}/project-access`).send({ projectIds: [f.selected[1]!.id] })).status).toBe(403);
await db.update(agents).set({ status: "pending_approval" }).where(eq(agents.id, f.agentId));
expect((await request(api).post(`/agents/${f.agentId}/project-access`).send({ projectIds: [f.selected[1]!.id] })).status).toBe(409);
const [saved] = await db.select().from(agents).where(eq(agents.id, f.agentId)); expect(saved!.permissions).toEqual(f.permissions);
});
it.each(["standard", "root_issue"])("rechecks actual project authority after a %s repair and acknowledges an accepted retry", async repair => {
const f = await seed();
const permissions = repair === "standard" ? { canCreateAgents: false, trustPreset: "standard", authorizationPolicy: {} }
: { ...f.permissions, authorizationPolicy: { ...f.permissions.authorizationPolicy,
trustBoundary: { ...f.permissions.authorizationPolicy.trustBoundary, rootIssueId: f.issue.id } } };
await db.update(agents).set({ permissions }).where(eq(agents.id, f.agentId));
expect(await taskInterruptionPresentation(db, f.issue)).toMatchObject({ canRetry: true, projectAccess: { ready: true } });
await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agentId,
source: "on_demand", status: "deferred_issue_execution", requestedByActorType: "user", payload: { issueId: f.issue.id } });
const accepted = await taskInterruptionPresentation(db, f.issue);
expect(accepted).toMatchObject({ state: "working", canRetry: false });
expect(accepted).not.toHaveProperty("projectAccess");
// Current project policy remains authoritative even after agent access changes.
await db.update(projects).set({ executionWorkspacePolicy: { trustPreset: "low_trust_review", authorizationPolicy: {
trustBoundary: { mode: "low_trust_review", companyId: f.companyId, projectIds: [f.selected[0]!.id] },
} } }).where(eq(projects.id, f.selected[1]!.id));
if (repair === "standard") expect(await taskInterruptionPresentation(db, f.issue))
.toMatchObject({ canRetry: false, userAction: "project_access" });
});
it("shows read-only access to viewers and denies their saves and cross-company reads", async () => {
const f = await seed(), other = await seed(), userId = randomUUID();
await db.insert(authUsers).values({ id: userId, name: "Viewer", email: `${userId}@example.test`, emailVerified: true, createdAt: new Date(), updatedAt: new Date() });
await db.insert(companyMemberships).values({ companyId: f.companyId, principalType: "user", principalId: userId, membershipRole: "viewer", status: "active" });
const api = app({ type: "board", source: "session", userId, companyIds: [f.companyId], memberships: [{ companyId: f.companyId, membershipRole: "viewer", status: "active" }] });
expect((await request(api).get(`/agents/${f.agentId}/project-access`)).body).toMatchObject({ canManage: false });
expect((await request(api).post(`/agents/${f.agentId}/project-access`).send({ projectIds: [f.selected[1]!.id] })).status).toBe(403);
expect((await request(api).get(`/agents/${other.agentId}/project-access`)).status).toBe(404);
});
});
@@ -0,0 +1,132 @@
import { randomUUID } from "node:crypto";
import { agents, companies, costEvents, createDb } from "@paperclipai/db";
import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture";
import { eq } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { agentRecordQueries } from "../lib/agent-records.js";
import { agentService } from "../services/agents.js";
const support = await getEmbeddedPostgresTestSupport();
describe.skipIf(!support.supported)("agent roster read projections", () => {
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let db: ReturnType<typeof createDb>;
beforeAll(async () => {
database = await startEmbeddedPostgresTestDatabase("agent-roster-read-");
db = createDb(database.connectionString);
}, 60_000);
beforeEach(() => {
vi.useFakeTimers({ toFake: ["Date"] });
vi.setSystemTime(new Date("2020-06-15T12:00:00Z"));
});
afterEach(() => { vi.useRealTimers(); });
afterAll(async () => {
await db?.$client.end();
await database?.cleanup();
});
async function fixture() {
const companyId = randomUUID(), foreignCompanyId = randomUUID();
const managerId = randomUUID(), workerId = randomUUID(), unrelatedId = randomUUID(), archivedId = randomUUID(), foreignId = randomUUID();
await db.insert(companies).values([
{ id: companyId, name: "Roster fixture", issuePrefix: `R${companyId.slice(0, 7).toUpperCase()}` },
{ id: foreignCompanyId, name: "Foreign roster", issuePrefix: `F${foreignCompanyId.slice(0, 7).toUpperCase()}` },
]);
const bulk = "retained-other-agent-configuration🙂".repeat(32_768);
await db.insert(agents).values([
{ id: managerId, companyId, name: "Manager", status: "terminated", adapterType: "process", adapterConfig: { instructions: bulk } },
{ id: unrelatedId, companyId, name: "Other worker", status: "idle", adapterType: "process", adapterConfig: { instructions: bulk } },
{ id: archivedId, companyId, name: "Archived worker", status: "terminated", adapterType: "process", runtimeConfig: { retained: bulk } },
{ id: foreignId, companyId: foreignCompanyId, name: "Foreign manager", status: "idle", adapterType: "process", adapterConfig: { instructions: bulk } },
]);
const [worker] = await db.insert(agents).values({ id: workerId, companyId, name: "Selected worker", status: "idle",
reportsTo: managerId, adapterType: "process", adapterConfig: { instructions: "selected instructions", model: "fixture-model" },
runtimeConfig: { heartbeat: { enabled: false } }, metadata: { retained: "selected metadata" },
permissions: { canCreateAgents: true }, spentMonthlyCents: 999,
}).returning();
const now = new Date();
await db.insert(costEvents).values([
{ companyId, agentId: workerId, provider: "fixture", model: "fixture-model", costCents: 19.5, occurredAt: now },
{ companyId, agentId: workerId, provider: "fixture", model: "fixture-model", costCents: 50,
occurredAt: new Date(Date.UTC(now.getUTCFullYear(), now.getUTCMonth() - 1, 15)) },
{ companyId: foreignCompanyId, agentId: foreignId, provider: "fixture", model: "fixture-model", costCents: 100, occurredAt: now },
]);
return { companyId, foreignCompanyId, managerId, workerId, unrelatedId, archivedId, foreignId, worker, bulk };
}
async function observe<T>(run: (observed: ReturnType<typeof createDb>) => Promise<T>) {
const observed = createDb(database.connectionString), events: Array<Record<string, unknown>> = [], now = Date.now();
const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event));
installDatabaseResponseCapture(observed, { agents, costEvents }, capture);
try {
const result = await run(observed);
capture.flush();
const reads = events.filter(event => event.table === "agents" && event.operation === "select");
return { result, calls: reads.reduce((n, event) => n + Number(event.calls), 0),
bytes: reads.reduce((n, event) => n + Number(event.responseValueBytes), 0), events };
} finally {
capture.flush();
await observed.$client.end();
}
}
it.each([false, true])("reads one full roster for list, preserving terminated-manager evidence and configurations (includeTerminated=%s)", async includeTerminated => {
const f = await fixture();
const captured = await observe(observed => agentService(observed).list(f.companyId, { includeTerminated }));
expect(captured.calls).toBe(1);
expect(captured.result.map(row => row.id).sort()).toEqual((includeTerminated
? [f.workerId, f.managerId, f.archivedId, f.unrelatedId] : [f.workerId, f.unrelatedId]).sort());
const selected = captured.result.find(row => row.id === f.workerId)!;
expect(selected).toMatchObject({ adapterConfig: f.worker.adapterConfig, runtimeConfig: f.worker.runtimeConfig,
metadata: f.worker.metadata, permissions: { canCreateAgents: true }, spentMonthlyCents: 19.5,
orgChainHealth: { status: "invalid_org_chain", reason: "terminated_ancestor", firstInvalidAncestor: { id: f.managerId } } });
expect(captured.result.find(row => row.id === f.unrelatedId)?.adapterConfig.instructions).toBe(f.bulk);
if (includeTerminated) expect(captured.result.find(row => row.id === f.archivedId)?.runtimeConfig.retained).toBe(f.bulk);
expect((await db.select().from(agents).where(eq(agents.id, f.workerId)))[0]).toEqual(f.worker);
expect(JSON.stringify(captured.events)).not.toContain("retained-other-agent-configuration");
});
it("keeps detail reads small while returning the selected agent's complete current configuration", async () => {
const f = await fixture();
const captured = await observe(observed => agentRecordQueries(observed).getById(f.workerId));
expect(captured.calls).toBe(2);
expect(captured.bytes).toBeLessThan(8192);
expect(captured.result).toMatchObject({ id: f.workerId, companyId: f.companyId, adapterConfig: f.worker.adapterConfig,
runtimeConfig: f.worker.runtimeConfig, metadata: f.worker.metadata, permissions: { canCreateAgents: true }, spentMonthlyCents: 19.5,
orgChainHealth: { reason: "terminated_ancestor", firstInvalidAncestor: { id: f.managerId, name: "Manager" } } });
expect((await db.select().from(agents).where(eq(agents.id, f.workerId)))[0]).toEqual(f.worker);
});
it("projects only current company chain fields for plugin normalization", async () => {
const f = await fixture();
const captured = await observe(observed => agentRecordQueries(observed).listCompanyAgentEligibilityRows(f.companyId));
expect(captured.calls).toBe(1);
expect(captured.bytes).toBeLessThan(4096);
expect(captured.result.map(row => row.id).sort()).toEqual([f.workerId, f.managerId, f.unrelatedId, f.archivedId].sort());
for (const row of captured.result) expect(Object.keys(row).sort()).toEqual(["companyId", "id", "name", "reportsTo", "status"]);
const normalized = agentRecordQueries(db).normalizeAgentRow(f.worker, captured.result);
expect(normalized).toMatchObject({ adapterConfig: f.worker.adapterConfig, metadata: f.worker.metadata,
orgChainHealth: { reason: "terminated_ancestor", firstInvalidAncestor: { id: f.managerId } } });
expect(await agentRecordQueries(db).listCompanyAgentEligibilityRows(randomUUID())).toEqual([]);
expect(await agentRecordQueries(db).getById(randomUUID())).toBeNull();
});
it("rechecks manager status, name, selected configuration, missing managers, and cycles on later reads", async () => {
const f = await fixture(), records = agentRecordQueries(db);
expect((await records.getById(f.workerId))?.orgChainHealth.reason).toBe("terminated_ancestor");
await db.update(agents).set({ status: "paused", name: "Current manager" }).where(eq(agents.id, f.managerId));
await db.update(agents).set({ adapterConfig: { instructions: "changed instructions" } }).where(eq(agents.id, f.workerId));
expect(await records.getById(f.workerId)).toMatchObject({ adapterConfig: { instructions: "changed instructions" },
orgChainHealth: { status: "healthy", pausedAncestors: [{ id: f.managerId, name: "Current manager", status: "paused" }] } });
await db.update(agents).set({ reportsTo: f.foreignId }).where(eq(agents.id, f.workerId));
expect((await records.getById(f.workerId))?.orgChainHealth.reason).toBe("missing_manager");
await db.update(agents).set({ reportsTo: f.managerId }).where(eq(agents.id, f.workerId));
await db.update(agents).set({ reportsTo: f.workerId }).where(eq(agents.id, f.managerId));
expect((await records.getById(f.workerId))?.orgChainHealth.reason).toBe("cycle");
});
});
+173 -1
View File
@@ -2,7 +2,8 @@ import { createAgentLifecycle } from "../services/agent-lifecycle.js";
import { randomUUID } from "node:crypto";
import express from "express";
import request from "supertest";
import { eq } from "drizzle-orm";
import { eq, sql } from "drizzle-orm";
import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture";
import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest";
import {
activityLog,
@@ -1034,6 +1035,177 @@ describeEmbeddedPostgres("attention service", () => {
expect(feed.items.filter((item) => item.sourceKind === "failed_run")).toEqual([]);
});
it("hides a failed run only after a newer run of the same agent on the same issue or task", async () => {
const { companyId, workerId, reviewerId } = await seedCompany("ATK");
const taskIssueId = await insertIssue({ companyId, identifier: "ATK-1", title: "Task-keyed", status: "in_progress" });
const issueKeyedId = await insertIssue({ companyId, identifier: "ATK-2", title: "Issue-keyed", status: "in_progress" });
const otherIssueId = await insertIssue({ companyId, identifier: "ATK-3", title: "Other work", status: "in_progress" });
const failedAt = new Date("2026-07-09T12:00:00.000Z");
const laterAt = new Date("2026-07-09T12:01:00.000Z");
const taskKeyedFailureId = randomUUID();
const issueKeyedFailureId = randomUUID();
const unkeyedHiddenFailureId = randomUUID();
const unkeyedKeptFailureId = randomUUID();
const failureIds = [taskKeyedFailureId, issueKeyedFailureId, unkeyedHiddenFailureId, unkeyedKeptFailureId];
const sameMillisecondRunId = randomUUID();
const run = (id: string, agentId: string, status: string, contextSnapshot: Record<string, unknown>, at: Date) => ({
id, companyId, agentId, invocationSource: "automation", status, contextSnapshot,
createdAt: at, updatedAt: at, finishedAt: at, ...(status === "failed" ? { error: "adapter failed" } : {}),
});
await db.insert(heartbeatRuns).values([
run(taskKeyedFailureId, workerId, "failed", { taskId: taskIssueId }, failedAt),
run(issueKeyedFailureId, workerId, "failed", { issueId: issueKeyedId }, failedAt),
run(unkeyedHiddenFailureId, workerId, "failed", {}, failedAt),
run(unkeyedKeptFailureId, reviewerId, "failed", {}, failedAt),
run(randomUUID(), workerId, "succeeded", { taskId: taskIssueId }, laterAt),
run(randomUUID(), workerId, "succeeded", { wakeReason: "timer" }, laterAt),
run(randomUUID(), reviewerId, "succeeded", { issueId: issueKeyedId }, laterAt),
run(randomUUID(), reviewerId, "succeeded", { issueId: otherIssueId }, laterAt),
run(randomUUID(), workerId, "succeeded", { issueId: otherIssueId }, laterAt),
run(randomUUID(), workerId, "succeeded", { issueId: otherIssueId, taskId: issueKeyedId }, laterAt),
run(randomUUID(), workerId, "succeeded", { issueId: "", taskId: issueKeyedId }, laterAt),
run(sameMillisecondRunId, workerId, "succeeded", { issueId: issueKeyedId }, failedAt),
run(randomUUID(), workerId, "succeeded", { issueId: issueKeyedId }, new Date("2026-07-09T11:59:00.000Z")),
]);
await db.execute(sql`update heartbeat_runs set created_at = created_at + interval '400 microseconds'
where id in (${sql.join(failureIds.map((id) => sql`${id}::uuid`), sql`, `)})`);
await db.execute(sql`update heartbeat_runs set created_at = created_at + interval '600 microseconds'
where id = ${sameMillisecondRunId}::uuid`);
await db.insert(heartbeatRunEvents).values(
failureIds.map((runId) => ({
companyId,
runId,
agentId: runId === unkeyedKeptFailureId ? reviewerId : workerId,
seq: 1,
eventType: "lifecycle",
message: "Bounded retry exhausted after 4 scheduled attempts; no further automatic retry will be queued",
createdAt: new Date("2026-07-09T12:00:01.000Z"),
})),
);
const feed = await attentionService(db).list(companyId, { userId: "board-user" });
expect(feed.items.filter((item) => item.sourceKind === "failed_run").map((item) => item.subject.id).sort())
.toEqual([issueKeyedFailureId, unkeyedKeptFailureId].sort());
});
it("keeps newer-run response size bounded despite unrelated retained history", async () => {
const { companyId, workerId } = await seedCompany("ATW");
const issueId = await insertIssue({ companyId, identifier: "ATW-1", title: "Exhausted task", status: "in_progress" });
const otherIssueId = await insertIssue({ companyId, identifier: "ATW-2", title: "Other task", status: "done" });
const failedId = randomUUID(), failedAt = new Date("2026-07-09T12:00:00.000Z");
await db.insert(heartbeatRuns).values({
id: failedId, companyId, agentId: workerId, status: "failed", contextSnapshot: { issueId },
createdAt: failedAt, updatedAt: failedAt, finishedAt: failedAt,
});
await db.insert(heartbeatRunEvents).values({
companyId, agentId: workerId, runId: failedId, seq: 1, eventType: "lifecycle",
message: "Bounded retry exhausted after scheduled attempts",
});
for (let batch = 0; batch < 4; batch += 1) {
await db.insert(heartbeatRuns).values(Array.from({ length: 500 }, (_, index) => ({
id: randomUUID(), companyId, agentId: workerId, status: "succeeded",
contextSnapshot: { issueId: otherIssueId, prompt: "x".repeat(8_000) },
createdAt: new Date(failedAt.getTime() + batch * 500 + index + 1),
updatedAt: failedAt, finishedAt: failedAt,
})));
}
const events: Array<Record<string, unknown>> = [], now = Date.now();
const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event));
installDatabaseResponseCapture(db, { heartbeatRuns }, capture);
try {
const feed = await attentionService(db).list(companyId, { userId: "board-user" });
expect(feed.items.filter(item => item.sourceKind === "failed_run").map(item => item.subject.id)).toEqual([failedId]);
} finally { capture.flush(); }
const reads = events.filter(event => event.event === "paperclip.db_response_capture.entry.v1"
&& event.table === "heartbeat_runs" && (event.operation === "select" || event.operation === "execute"));
expect(reads.reduce((sum, event) => sum + Number(event.calls), 0)).toBeGreaterThan(0);
expect(reads.reduce((sum, event) => sum + Number(event.rows), 0)).toBeLessThan(64);
expect(reads.reduce((sum, event) => sum + Number(event.responseValueBytes), 0)).toBeLessThan(4096);
expect(reads.filter(event => event.operation === "execute")
.reduce((sum, event) => sum + Number(event.calls), 0)).toBeGreaterThan(0);
const [remaining] = await db.select({ count: sql<number>`count(*)::int` }).from(heartbeatRuns)
.where(eq(heartbeatRuns.companyId, companyId));
expect(remaining?.count).toBe(2001);
});
it("reads keyless history once per agent despite thousands of failures", async () => {
const { companyId, workerId, reviewerId } = await seedCompany("ATB");
const failedAt = new Date("2026-07-09T12:00:00.000Z");
const laterAt = new Date("2026-07-09T12:01:00.000Z");
const hiddenIds = Array.from({ length: 16_390 }, () => randomUUID());
const keptIds = Array.from({ length: 10 }, () => randomUUID());
const failure = (id: string, agentId: string) => ({
id, companyId, agentId, invocationSource: "automation", status: "failed", error: "adapter failed",
contextSnapshot: {}, createdAt: failedAt, updatedAt: failedAt, finishedAt: failedAt,
});
const failures = hiddenIds.map((id) => failure(id, workerId));
keptIds.forEach((id, index) => failures.splice(index * 1_700, 0, failure(id, reviewerId)));
for (let start = 0; start < failures.length; start += 1_000) {
const chunk = failures.slice(start, start + 1_000);
await db.insert(heartbeatRuns).values(chunk);
await db.insert(heartbeatRunEvents).values(chunk.map((run) => ({
companyId, runId: run.id, agentId: run.agentId, seq: 1, eventType: "lifecycle",
message: "Bounded retry exhausted after 4 scheduled attempts; no further automatic retry will be queued",
createdAt: new Date("2026-07-09T12:00:01.000Z"),
})));
}
await db.insert(heartbeatRuns).values({
id: randomUUID(), companyId, agentId: workerId, invocationSource: "automation", status: "succeeded",
contextSnapshot: { wakeReason: "timer" }, createdAt: laterAt, updatedAt: laterAt, finishedAt: laterAt,
});
const events: Array<Record<string, unknown>> = [], now = Date.now();
const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event));
installDatabaseResponseCapture(db, { heartbeatRuns }, capture);
try {
const feed = await attentionService(db).list(companyId, {
includeDismissed: true, all: true, allowUnscopedAll: true,
});
expect(feed.items.filter((item) => item.sourceKind === "failed_run").map((item) => item.subject.id).sort())
.toEqual([...keptIds].sort());
} finally { capture.flush(); }
const reads = events.filter(event => event.event === "paperclip.db_response_capture.entry.v1"
&& event.table === "heartbeat_runs" && (event.operation === "select" || event.operation === "execute"));
expect(reads.reduce((sum, event) => sum + Number(event.calls), 0)).toBeGreaterThan(0);
expect(reads.reduce((sum, event) => sum + Number(event.calls), 0)).toBeLessThan(4);
}, 180_000);
it("checks more failed runs for a newer run than one statement can bind", async () => {
const { companyId, workerId, reviewerId } = await seedCompany("ATQ");
const issueId = await insertIssue({ companyId, identifier: "ATQ-1", title: "Keyed retained failures", status: "in_progress" });
const failedAt = new Date("2026-07-09T12:00:00.000Z");
const laterAt = new Date("2026-07-09T12:01:00.000Z");
const hiddenIds = Array.from({ length: 16_390 }, () => randomUUID());
const keptIds = Array.from({ length: 10 }, () => randomUUID());
const failure = (id: string, agentId: string) => ({
id, companyId, agentId, invocationSource: "automation", status: "failed", error: "adapter failed",
contextSnapshot: { issueId }, createdAt: failedAt, updatedAt: failedAt, finishedAt: failedAt,
});
const failures = hiddenIds.map((id) => failure(id, workerId));
keptIds.forEach((id, index) => failures.splice(index * 1_700, 0, failure(id, reviewerId)));
for (let start = 0; start < failures.length; start += 1_000) {
const chunk = failures.slice(start, start + 1_000);
await db.insert(heartbeatRuns).values(chunk);
await db.insert(heartbeatRunEvents).values(chunk.map((run) => ({
companyId, runId: run.id, agentId: run.agentId, seq: 1, eventType: "lifecycle",
message: "Bounded retry exhausted after 4 scheduled attempts; no further automatic retry will be queued",
createdAt: new Date("2026-07-09T12:00:01.000Z"),
})));
}
await db.insert(heartbeatRuns).values({
id: randomUUID(), companyId, agentId: workerId, invocationSource: "automation", status: "succeeded",
contextSnapshot: { issueId }, createdAt: laterAt, updatedAt: laterAt, finishedAt: laterAt,
});
const feed = await attentionService(db).list(companyId, {
includeDismissed: true, all: true, allowUnscopedAll: true,
});
expect(feed.items.filter(item => item.sourceKind === "failed_run").map(item => item.subject.id).sort())
.toEqual([...keptIds].sort());
}, 180_000);
it("enriches interaction details with project, workspace, plan metadata, and images", async () => {
const { companyId, workerId } = await seedCompany("ATE");
const projectId = randomUUID();
@@ -5,6 +5,8 @@ import { createSystemRecovery } from "../services/system-recovery-state.js";
import { remoteTerminationReceipt } from "../services/remote-execution-termination.js";
import { stageReceiptReactionRemovals } from "../services/chat-receipt-reactions.js";
import type { Db } from "@paperclipai/db";
import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture";
import { githubResponseAuthorityFields } from "../services/chat-channels/github/response-authority-read.js";
import * as activityLogModule from "../services/activity-log.js";
import * as connectionCredentialBindings from "../services/connection-credential-bindings.js";
import { chatSlackManagerGrants, chatSlackRegistrations, toolOauthStates } from "@paperclipai/db";
@@ -24,7 +26,7 @@ import { toolActionRequests, toolInvocations, toolPolicies } from "@paperclipai/
import { GitHubPublicationLeaseLost, withGitHubPublicationLease } from "../services/chat-github-publication-lease.js";
import { secretService } from "../services/secrets.js";
import { githubChatWizardService } from "../services/chat-github-wizard.js";
import { dispatchGitHubBotCloudEvent } from "../services/chat-github-cloud-ingress.js";
import { dispatchGitHubBotCloudEvent, registerGitHubBotCloudSetupPoller } from "../services/chat-github-cloud-ingress.js";
import { githubChatManagementService } from "../services/chat-github-management.js";
import { githubChatReviewService } from "../services/chat-github-reviews.js";
import { resolveGitHubReviewInstructions } from "../services/chat-channels/github/review-instructions.js";
@@ -2585,6 +2587,81 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
.rejects.toMatchObject({ status: 403 });
});
it("bounds GitHub response authorization reads with large saved chat content", async () => {
const f = await toolReplyFixture();
const [delivery] = await db.select().from(chatDeliveries).where(eq(chatDeliveries.endpointId, f.endpoint.id));
const [resource] = await db.select().from(chatEndpointResources).where(eq(chatEndpointResources.id, f.conversation.resourceId!));
const [configuration] = await db.select().from(chatGitHubConfigurations).where(eq(chatGitHubConfigurations.endpointId, f.endpoint.id));
const bulk = "unused-content-for-response-authorization".repeat(8_192);
const setup = { ...f.endpoint.setup, unusedContent: bulk };
await db.update(chatEndpoints).set({ communicationInstructions: bulk, setup }).where(eq(chatEndpoints.id, f.endpoint.id));
await db.update(chatConversations).set({ communicationGuidance: bulk }).where(eq(chatConversations.id, f.conversation.id));
await db.update(chatEndpointResources).set({ metadata: { ...resource.metadata, unusedContent: bulk } })
.where(eq(chatEndpointResources.id, resource.id));
await db.update(chatGitHubConfigurations).set({ configuration: { ...configuration.configuration,
defaults: { ...configuration.configuration.defaults, instructions: bulk } } })
.where(eq(chatGitHubConfigurations.endpointId, f.endpoint.id));
const normalizedEvent = { ...delivery.normalizedEvent, unusedContent: bulk };
await db.update(chatDeliveries).set({ normalizedEvent }).where(eq(chatDeliveries.id, delivery.id));
const events: Array<Record<string, unknown>> = [], now = Date.now();
const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event));
installDatabaseResponseCapture(db, { chatMessageLinks, chatDeliveries, chatEndpoints, chatConversations,
chatEndpointResources, chatGitHubConfigurations, chatExternalPrincipals, chatIdentityLinks,
companyMemberships, chatActions, heartbeatRuns }, capture);
try {
const receipt = await githubResponseCommentService(db, f.providerFetch).tryAcknowledge(delivery.id);
expect(receipt?.id).toBeTruthy();
} finally { capture.flush(); }
expect(Number(events.at(-1)?.responseValueBytes)).toBeLessThan(24_000);
expect(events.filter(event => event.table === "chat_message_links").length).toBeGreaterThan(0);
expect((await db.select({ normalizedEvent: chatDeliveries.normalizedEvent }).from(chatDeliveries)
.where(eq(chatDeliveries.id, delivery.id)))[0].normalizedEvent).toEqual(normalizedEvent);
expect((await db.select({ setup: chatEndpoints.setup }).from(chatEndpoints)
.where(eq(chatEndpoints.id, f.endpoint.id)))[0].setup).toEqual(setup);
});
it.each(["githubAutomatic", "githubIssue"])("preserves %s policy context while omitting body content", async key => {
const f = await toolReplyFixture();
const [delivery] = await db.select().from(chatDeliveries).where(eq(chatDeliveries.endpointId, f.endpoint.id));
const context = { event: key === "githubIssue" ? "issue_opened" : "opened", repositoryId: "418", repository: "acme/repo",
author: { id: "42", login: "octocat", isBot: false }, sender: { id: "42", login: "octocat" },
labels: ["review"], draft: false, baseBranch: "master", headSha: "a".repeat(40),
body: "unused-body".repeat(32_768), title: "unused-title".repeat(8_192) };
const normalizedEvent = { text: "unrelated message text", [key]: { context, revision: 42,
policy: { instructions: "unused-policy-instructions".repeat(32_768),
prompts: { opened: "unused-policy-prompt".repeat(32_768) } },
} };
await db.update(chatDeliveries).set({ normalizedEvent }).where(eq(chatDeliveries.id, delivery.id));
const events: Array<Record<string, unknown>> = [], now = Date.now();
const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event));
installDatabaseResponseCapture(db, { chatDeliveries }, capture);
let projected: { event: Record<string, unknown> };
try {
[projected] = await db.select({ event: githubResponseAuthorityFields.delivery.normalizedEvent })
.from(chatDeliveries).where(eq(chatDeliveries.id, delivery.id));
} finally { capture.flush(); }
const { body: _body, title: _title, ...policy } = context;
expect(Number(events.at(-1)?.responseValueBytes)).toBeLessThan(2_048);
expect(projected.event).toEqual({ [key]: { context: policy } });
expect((await db.select({ event: chatDeliveries.normalizedEvent }).from(chatDeliveries)
.where(eq(chatDeliveries.id, delivery.id)))[0].event).toEqual(normalizedEvent);
});
it.each([null, false, 7, "invalid-event", []])("preserves malformed GitHub authority input %j", async value => {
const f = await toolReplyFixture();
const [delivery] = await db.select().from(chatDeliveries).where(eq(chatDeliveries.endpointId, f.endpoint.id));
await db.update(chatDeliveries).set({ normalizedEvent: sql`${JSON.stringify(value)}::jsonb` })
.where(eq(chatDeliveries.id, delivery.id));
await db.update(chatEndpoints).set({ setup: sql`${JSON.stringify(value)}::jsonb` })
.where(eq(chatEndpoints.id, f.endpoint.id));
const [projected] = await db.select({ event: githubResponseAuthorityFields.delivery.normalizedEvent })
.from(chatDeliveries).where(eq(chatDeliveries.id, delivery.id));
const [endpoint] = await db.select({ setup: githubResponseAuthorityFields.endpoint.setup })
.from(chatEndpoints).where(eq(chatEndpoints.id, f.endpoint.id));
expect(projected.event).toEqual(value);
expect(endpoint.setup).toEqual(value);
});
async function publicationRetryFixture() {
const f = await toolReplyFixture("github:paperclipai/paperclip:418", true);
const head = "b".repeat(40);
@@ -5135,13 +5212,22 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
(await f.service.get(f.endpoint.id)).setup.webhookVerifiedAt,
).toBeNull();
expect(f.wakeup).not.toHaveBeenCalled();
event.payload.githubApp.headers["x-hub-signature-256"] =
`sha256=${createHmac("sha256", f.webhookSecret).update(body).digest("hex")}`;
expect(await dispatchGitHubBotCloudEvent(db, event)).toBe(true);
expect(
(await f.service.get(f.endpoint.id)).setup.webhookVerifiedAt,
).toBeTruthy();
expect(f.wakeup).not.toHaveBeenCalled();
const poll = vi.fn(async () => { await dispatchGitHubBotCloudEvent(db, event); });
const unregister = registerGitHubBotCloudSetupPoller(db, poll);
try {
const waiting = await f.service.githubWizard.advance(f.endpoint.id, "owner-user");
expect(waiting.state).toBe("verify");
expect(poll).toHaveBeenCalledOnce();
expect((await f.service.get(f.endpoint.id)).setup.webhookVerifiedAt).toBeNull();
event.payload.githubApp.headers["x-hub-signature-256"] =
`sha256=${createHmac("sha256", f.webhookSecret).update(body).digest("hex")}`;
const ready = await f.service.githubWizard.advance(f.endpoint.id, "owner-user");
expect(ready.state).not.toBe("verify");
expect((await f.service.get(f.endpoint.id)).setup.webhookVerifiedAt).toBeTruthy();
expect(f.wakeup).not.toHaveBeenCalled();
await f.service.githubWizard.advance(f.endpoint.id, "owner-user");
expect(poll).toHaveBeenCalledTimes(2);
} finally { unregister(); }
});
it("legacy name-only registration preserves an explicit empty repository selection", async () => {
const fixture = await seedCompany();
@@ -16,6 +16,7 @@ import {
projectWorkspaces,
} from "@paperclipai/db";
import { parseFrontmatterMarkdown } from "@paperclipai/shared";
import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture";
import {
getEmbeddedPostgresTestSupport,
startEmbeddedPostgresTestDatabase,
@@ -722,6 +723,73 @@ describeEmbeddedPostgres("companySkillService.list", () => {
expect(refreshed.filter((skill) => skill.key === key)).toHaveLength(1);
});
it("validates warm bundled releases without returning their stored file bodies", async () => {
const companyId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Release read checks", issuePrefix: "RRC" });
await svc.list(companyId);
const expected = await svc.list(companyId);
const before = await db.select().from(companySkillVersions).where(eq(companySkillVersions.companyId, companyId))
.orderBy(companySkillVersions.id);
expect(before.length).toBeGreaterThan(0);
const events: Array<Record<string, unknown>> = [], now = Date.now();
const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event));
installDatabaseResponseCapture(db, { companySkillVersions }, capture);
try { expect(await companySkillService(db).list(companyId)).toEqual(expected); }
finally { capture.flush(); }
const reads = events.filter(event => event.event === "paperclip.db_response_capture.entry.v1"
&& event.table === "company_skill_versions" && event.operation === "select");
expect(reads.reduce((sum, event) => sum + Number(event.calls), 0)).toBeGreaterThan(0);
expect(reads.reduce((sum, event) => sum + Number(event.responseValueBytes), 0)).toBeLessThan(4096);
expect(await db.select().from(companySkillVersions).where(eq(companySkillVersions.companyId, companyId))
.orderBy(companySkillVersions.id)).toEqual(before);
});
it.each(["agent", "board"] as const)("checks %s company access before using a warm release validation", async type => {
const companyId = randomUUID(), otherCompanyId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Release access checks", issuePrefix: "RAC" });
await svc.list(companyId);
await svc.list(companyId);
const actor = type === "agent"
? { type, companyId: otherCompanyId, agentId: randomUUID() }
: { type, companyIds: [otherCompanyId] };
const events: Array<Record<string, unknown>> = [], now = Date.now();
const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event));
installDatabaseResponseCapture(db, { companySkillVersions }, capture);
try { expect(await companySkillService(db, { actor }).list(companyId)).toEqual([]); }
finally { capture.flush(); }
expect(events.find(event => event.event === "paperclip.db_response_capture.summary.v1")?.completedQueries).toBe(0);
});
it("restores a deleted release after warm validation", async () => {
const companyId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Release deletion checks", issuePrefix: "RDC" });
await svc.list(companyId);
await svc.list(companyId);
const before = await db.select().from(companySkillVersions).where(eq(companySkillVersions.companyId, companyId));
const original = before[0]!;
await db.delete(companySkillVersions).where(eq(companySkillVersions.id, original.id));
await companySkillService(db).list(companyId);
const after = await db.select().from(companySkillVersions).where(eq(companySkillVersions.companyId, companyId));
expect(after).toHaveLength(before.length);
expect(after.find(version => version.releaseId === original.releaseId)).toMatchObject({
companyId, companySkillId: original.companySkillId, fileInventory: original.fileInventory,
});
expect(after.some(version => version.id === original.id)).toBe(false);
});
it("rejects a changed bundled snapshot after warm validation without overwriting it", async () => {
const companyId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Release integrity checks", issuePrefix: "RIC" });
await svc.list(companyId);
await svc.list(companyId);
const [version] = await db.select().from(companySkillVersions).where(eq(companySkillVersions.companyId, companyId));
const changed = [{ path: "SKILL.md", kind: "skill" as const, content: "Changed stored snapshot" }];
await db.update(companySkillVersions).set({ fileInventory: changed }).where(eq(companySkillVersions.id, version!.id));
await expect(companySkillService(db).list(companyId)).rejects.toThrow("does not match its seeded snapshot");
expect((await db.select().from(companySkillVersions).where(eq(companySkillVersions.id, version!.id)))[0]!.fileInventory)
.toEqual(changed);
});
it("seeds bundled skill releases idempotently and materializes the frozen champion snapshot", async () => {
const companyId = randomUUID();
await db.insert(companies).values({
@@ -0,0 +1,105 @@
import { randomUUID } from "node:crypto";
import { and, eq, sql } from "drizzle-orm";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import { agents, companies, createDb, heartbeatRuns, issueComments, issueThreadInteractions, issues } from "@paperclipai/db";
import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { settleConversationTurnForRun } from "../services/agent-conversations.js";
const support = await getEmbeddedPostgresTestSupport();
(support.supported ? describe : describe.skip)("conversation settlement transfer", () => {
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let db: ReturnType<typeof createDb>;
beforeAll(async () => {
database = await startEmbeddedPostgresTestDatabase("paperclip-conversation-settlement-");
db = createDb(database.connectionString);
}, 30_000);
afterAll(async () => { await db?.$client.end({ timeout: 5 }); await database?.cleanup(); }, 60_000);
async function seed(reply: "response" | "interaction" | "reset" | "none" = "response") {
const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), runId = randomUUID(), wakeId = randomUUID();
const padding = "saved conversation ".repeat(32_768);
await db.insert(companies).values({ id: companyId, name: "Settlement transfer", issuePrefix: companyId.slice(0, 8) });
await db.insert(agents).values({ id: agentId, companyId, name: "Conversation agent", role: "engineer" });
await db.insert(issues).values({ id: issueId, companyId, title: "Conversation", description: padding,
status: "in_progress", conversationState: "active", conversationUserId: "conversation-owner",
conversationAgentId: agentId, assigneeAgentId: agentId, conversationSessionGeneration: 3 });
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "succeeded",
contextSnapshot: { issueId, conversationSessionGeneration: 3, wakeCommentId: wakeId,
conversationReset: reply === "reset", savedInput: padding },
runnerProfileJson: { savedProfile: padding }, resultJson: { savedResult: padding },
stdoutExcerpt: padding, stderrExcerpt: padding });
await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId));
const wakeTime = new Date(Date.now() - 10_000);
await db.insert(issueComments).values({ id: wakeId, companyId, issueId, authorUserId: "conversation-owner",
body: padding, createdAt: wakeTime });
if (reply === "response") await db.insert(issueComments).values({ companyId, issueId,
createdByRunId: runId, authorAgentId: agentId, origin: "comment", body: padding });
if (reply === "interaction") await db.insert(issueThreadInteractions).values({ companyId, issueId,
sourceRunId: runId, createdByAgentId: agentId, status: "pending", kind: "request_confirmation", payload: { version: 1 } });
return { companyId, agentId, issueId, runId, wakeId, wakeTime };
}
async function sourceDigests(input: Awaited<ReturnType<typeof seed>>) {
return db.execute(sql`select
(select encode(sha256(convert_to(jsonb_build_array(context_snapshot,runner_profile_json,result_json,stdout_excerpt,stderr_excerpt)::text,'UTF8')),'hex')
from heartbeat_runs where id=${input.runId}::uuid) as run_digest,
(select encode(sha256(convert_to(description,'UTF8')),'hex') from issues where id=${input.issueId}::uuid) as description_digest,
(select encode(sha256(convert_to(jsonb_agg(jsonb_build_array(id,body) order by id)::text,'UTF8')),'hex')
from issue_comments where issue_id=${input.issueId}::uuid) as comments_digest`);
}
it.each(["response", "interaction", "reset"] as const)("bounds %s settlement reads and retains all saved text and run payloads", async reply => {
const input = await seed(reply), before = await sourceDigests(input);
const events: Array<Record<string, unknown>> = [], now = Date.now();
const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event));
installDatabaseResponseCapture(db, { heartbeatRuns, issues, issueComments }, capture);
try { expect(await settleConversationTurnForRun(db, input.companyId, input.runId)).toBe(true); }
finally { capture.flush(); }
const reads = events.filter(event => event.event === "paperclip.db_response_capture.entry.v1"
&& ["heartbeat_runs", "issues", "issue_comments"].includes(String(event.table))
&& (event.operation === "select" || event.operation === "execute"));
expect(reads.reduce((sum, event) => sum + Number(event.calls), 0)).toBeGreaterThanOrEqual(4);
expect(reads.reduce((sum, event) => sum + Number(event.responseValueBytes), 0)).toBeLessThan(16_384);
const [issue] = await db.select({ status: issues.status, state: issues.conversationState }).from(issues).where(eq(issues.id, input.issueId));
expect(issue).toEqual({ status: "in_review", state: "waiting" });
expect(await sourceDigests(input)).toEqual(before);
});
it.each(["conversationReplyBoundaryCommentId", "wakeCommentId", "commentId"] as const)("keeps later user messages active through %s", async key => {
const input = await seed();
await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: input.issueId,
conversationSessionGeneration: 3, [key]: input.wakeId } }).where(eq(heartbeatRuns.id, input.runId));
await db.insert(issueComments).values({ companyId: input.companyId, issueId: input.issueId,
authorUserId: "conversation-owner", body: "A new instruction", createdAt: new Date(input.wakeTime.getTime() + 1000) });
expect(await settleConversationTurnForRun(db, input.companyId, input.runId)).toBe(true);
const [issue] = await db.select({ status: issues.status, state: issues.conversationState }).from(issues).where(eq(issues.id, input.issueId));
expect(issue).toEqual({ status: "in_progress", state: "active" });
});
it.each(["missing", "company", "issue-company", "generation", "owner", "failed", "no-reply"] as const)("refuses to settle a %s mismatch", async fault => {
const input = await seed(fault === "no-reply" ? "none" : "response");
if (fault === "generation") await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: input.issueId,
conversationSessionGeneration: 2, wakeCommentId: input.wakeId } }).where(eq(heartbeatRuns.id, input.runId));
if (fault === "owner") {
const otherRun = randomUUID();
await db.insert(heartbeatRuns).values({ id: otherRun, companyId: input.companyId, agentId: input.agentId });
await db.update(issues).set({ executionRunId: otherRun }).where(eq(issues.id, input.issueId));
}
if (fault === "failed") await db.update(heartbeatRuns).set({ status: "failed" }).where(eq(heartbeatRuns.id, input.runId));
if (fault === "issue-company") {
const otherCompany = randomUUID(), otherIssue = randomUUID(), otherAgent = randomUUID();
await db.insert(companies).values({ id: otherCompany, name: "Other company", issuePrefix: otherCompany.slice(0, 8) });
await db.insert(agents).values({ id: otherAgent, companyId: otherCompany, name: "Other agent" });
await db.insert(issues).values({ id: otherIssue, companyId: otherCompany, title: "Other conversation",
conversationUserId: "other-owner", conversationAgentId: otherAgent, assigneeAgentId: otherAgent,
conversationState: "active", conversationSessionGeneration: 3 });
await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: otherIssue, conversationSessionGeneration: 3,
conversationReset: true } }).where(eq(heartbeatRuns.id, input.runId));
}
expect(await settleConversationTurnForRun(db, fault === "company" ? randomUUID() : input.companyId,
fault === "missing" ? randomUUID() : input.runId)).toBe(false);
const [issue] = await db.select({ status: issues.status }).from(issues).where(and(eq(issues.id, input.issueId), eq(issues.companyId, input.companyId)));
expect(issue.status).toBe("in_progress");
});
});
@@ -92,6 +92,7 @@ vi.mock("../services/plugin-environment-driver.js", () => ({
REUSABLE_LEASE_WORKER_METHODS: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"],
listReadyPluginEnvironmentDrivers: vi.fn(async () => []),
resolvePluginSandboxProviderDriverByKey: vi.fn(async () => null),
resolvePluginSandboxProviderConfigSchema: vi.fn(async () => null),
validatePluginEnvironmentDriverConfig: vi.fn(async ({ config }) => config),
validatePluginSandboxProviderConfig: vi.fn(async ({ provider, config }) => ({
normalizedConfig: config,
@@ -79,6 +79,7 @@ vi.mock("../services/plugin-environment-driver.js", () => ({
REUSABLE_LEASE_WORKER_METHODS: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"],
listReadyPluginEnvironmentDrivers: vi.fn(async () => []),
resolvePluginSandboxProviderDriverByKey: vi.fn(async () => null),
resolvePluginSandboxProviderConfigSchema: vi.fn(async () => null),
validatePluginEnvironmentDriverConfig: vi.fn(async ({ config }) => config),
validatePluginSandboxProviderConfig: vi.fn(async ({ provider, config }) => ({
normalizedConfig: config,
@@ -25,6 +25,8 @@ vi.mock("../services/plugin-environment-driver.js", () => ({
probePluginEnvironmentDriver: mockProbePluginEnvironmentDriver,
probePluginSandboxProviderDriver: mockProbePluginSandboxProviderDriver,
resolvePluginSandboxProviderDriverByKey: mockResolvePluginSandboxProviderDriverByKey,
resolvePluginSandboxProviderConfigSchema: async (input: { db: unknown; driverKey: string }) =>
(await mockResolvePluginSandboxProviderDriverByKey(input))?.driver?.configSchema ?? null,
}));
vi.mock("../services/environment-runtime.js", () => ({
@@ -129,6 +129,8 @@ vi.mock("../services/plugin-environment-driver.js", () => ({
REUSABLE_LEASE_WORKER_METHODS: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"],
listReadyPluginEnvironmentDrivers: mockListReadyPluginEnvironmentDrivers,
resolvePluginSandboxProviderDriverByKey: mockResolvePluginSandboxProviderDriverByKey,
resolvePluginSandboxProviderConfigSchema: async (input: { db: unknown; driverKey: string }) =>
(await mockResolvePluginSandboxProviderDriverByKey(input))?.driver?.configSchema ?? null,
startPluginEnvironmentInteractiveSetup: mockStartPluginEnvironmentInteractiveSetup,
getPluginEnvironmentInteractiveSetup: mockGetPluginEnvironmentInteractiveSetup,
capturePluginEnvironmentTemplate: mockCapturePluginEnvironmentTemplate,
@@ -9,6 +9,7 @@ import {
GIT_CREDENTIAL_TOKEN_ENV_KEY,
buildGitAuthInvocation,
createGitRemoteAuthProvider,
createIssueGitRemoteAuthProvider,
describeGitAuthFailure,
isGitHubHttpsRemoteUrl,
scrubGitCredentialText,
@@ -379,3 +380,39 @@ describe("DEFAULT_GITHUB_TOKEN_SECRET_NAMES", () => {
]);
});
});
describe("createIssueGitRemoteAuthProvider", () => {
const githubUrl = "https://github.com/paperclipai/private-repo.git";
function issueDb(rows: unknown[]) {
const where = vi.fn(async () => rows);
return { db: { select: vi.fn(() => ({ from: () => ({ where }) })) } as unknown as Db, where };
}
it("resolves credentials with the issue's responsible user and assigned agent", async () => {
const { db } = issueDb([{ responsibleUserId: "user-1", assigneeAgentId: "agent-1" }]);
const inner = vi.fn(async () => ({ env: {}, configArgs: [], source: "managed_connection" }));
const create = vi.fn(() => inner) as unknown as typeof createGitRemoteAuthProvider;
const provider = createIssueGitRemoteAuthProvider(db, "company-1", "issue-1", create);
await expect(provider(githubUrl)).resolves.toMatchObject({ source: "managed_connection" });
expect(create).toHaveBeenCalledWith(db, "company-1", { issueId: "issue-1", responsibleUserId: "user-1", agentId: "agent-1" });
expect(inner).toHaveBeenCalledWith(githubUrl);
});
it("looks the issue up once across calls", async () => {
const { db } = issueDb([{ responsibleUserId: "user-1", assigneeAgentId: null }]);
const inner = vi.fn(async () => null);
const create = vi.fn(() => inner) as unknown as typeof createGitRemoteAuthProvider;
const provider = createIssueGitRemoteAuthProvider(db, "company-1", "issue-1", create);
await provider(githubUrl); await provider(githubUrl);
expect(db.select).toHaveBeenCalledTimes(1);
expect(create).toHaveBeenCalledTimes(1);
expect(inner).toHaveBeenCalledTimes(2);
});
it("falls back to the issue-only context when the issue is missing", async () => {
const { db } = issueDb([]);
const create = vi.fn(() => vi.fn(async () => null)) as unknown as typeof createGitRemoteAuthProvider;
await createIssueGitRemoteAuthProvider(db, "company-1", "issue-1", create)(githubUrl);
expect(create).toHaveBeenCalledWith(db, "company-1", { issueId: "issue-1", responsibleUserId: null, agentId: null });
});
});
@@ -7,6 +7,7 @@ import {
connectionGrants,
createDb,
externalObjects,
EMBEDDED_POSTGRES_TEST_TIMEOUT_MS,
toolApplications,
toolConnections,
} from "@paperclipai/db";
@@ -30,7 +31,7 @@ describeEmbeddedPostgres("GitHub connection event delivery", () => {
beforeAll(async () => {
tempDb = await startEmbeddedPostgresTestDatabase("paperclip-github-events-");
db = createDb(tempDb.connectionString);
}, 20_000);
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
afterEach(async () => {
await db.delete(activityLog);
@@ -46,7 +47,7 @@ describeEmbeddedPostgres("GitHub connection event delivery", () => {
await tempDb?.cleanup();
});
it("applies a normalized merged PR once while retaining undecryptable siblings for retry", async () => {
it.each([false, true])("applies a normalized merged PR once while retaining undecryptable siblings for retry (active setup=%s)", async activeSetup => {
const companyId = randomUUID();
const userId = `github-owner-${randomUUID()}`;
const applicationId = randomUUID();
@@ -151,8 +152,21 @@ describeEmbeddedPostgres("GitHub connection event delivery", () => {
let currentTime = new Date("2026-09-04T12:00:05.000Z");
const service = githubConnectionEventService(db, { connector, now: () => currentTime });
if (activeSetup) {
// An existing personal connection can leave a long empty-queue backoff
// before a new dedicated App starts its signed-ping verification.
for (const delay of [0, 10_000, 20_000, 40_000, 80_000]) {
currentTime = new Date(currentTime.getTime() + delay);
vi.mocked(connector.leaseEvents).mockResolvedValueOnce(null);
await service.pollOnce();
}
currentTime = new Date(currentTime.getTime() + 5_000);
await expect(service.pollOnce()).resolves.toMatchObject({ leased: 0 });
expect(connector.leaseEvents).toHaveBeenCalledTimes(5);
}
const warning = vi.spyOn(logger, "warn").mockImplementation(() => undefined);
await expect(service.pollOnce()).resolves.toMatchObject({ leased: 2, processed: 1, duplicate: 0, failed: 1 });
await expect(service.pollOnce({ forSetup: activeSetup })).resolves.toMatchObject({ leased: 2, processed: 1, duplicate: 0, failed: 1 });
expect(warning.mock.calls).toEqual([[{
deliveryId: "undecryptable-delivery", reason: "inner_event_authentication_failed",
}, "GitHub connector event retained for retry"]]);
@@ -6,6 +6,7 @@ import { createRuntimeToolsToken } from "../runtime-tools-token.js";
import { errorHandler } from "../middleware/index.js";
import { randomUUID } from "node:crypto";
import { eq } from "drizzle-orm";
import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture";
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
import {
agents,
@@ -31,6 +32,7 @@ import {
} from "./helpers/embedded-postgres.js";
import {
initializeRunIdentity,
captureRunIdentity,
reserveSteeredIdentity,
acceptSteeredIdentity,
} from "../services/run-identity.js";
@@ -192,6 +194,72 @@ const support = await getEmbeddedPostgresTestSupport();
});
return { id, connectionId, secretId, definitionId };
}
it.each(["config", "transportConfig"] as const)("reads only current GitHub identity connections via %s", async location => {
const input = await seed(), selected = await grant(input, "A");
const [connection] = await db.select().from(toolConnections).where(eq(toolConnections.id, selected.connectionId));
await db.update(toolConnections).set({ config: {}, transportConfig: {}, [location]: { sourceTemplateKey: "github" } })
.where(eq(toolConnections.id, selected.connectionId));
const payload = "unrelated-connection-config🙂".repeat(10_000);
const irrelevant = [null, true, 42, ["github"], { name: "github" }, "GitHub", " github ", "other"].map(sourceTemplateKey => {
const id = randomUUID();
return { id, companyId: input.companyId, applicationId: connection.applicationId, name: id, uid: id,
transport: "mcp_remote" as const, config: { sourceTemplateKey, payload }, transportConfig: { sourceTemplateKey } };
});
await db.insert(toolConnections).values(irrelevant);
const observed = createDb(database.connectionString), events: Array<Record<string, unknown>> = [], now = Date.now();
const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event));
installDatabaseResponseCapture(observed, { toolConnections }, capture);
try {
expect(await resolveManagedGitHubIdentitySelection(observed, input.companyId, { agentId: input.agentId, responsibleUserId: "A" }))
.toMatchObject({ configured: true, grant: { id: selected.id } });
capture.flush();
const reads = events.filter(event => event.table === "tool_connections" && event.operation === "select");
expect(reads).toHaveLength(1);
expect(reads[0].rows).toBe(1);
expect(reads[0].responseValueBytes).toBeLessThan(8192);
await db.update(toolConnections).set({ enabled: false }).where(eq(toolConnections.id, selected.connectionId));
expect((await resolveManagedGitHubIdentitySelection(observed, input.companyId, { agentId: input.agentId, responsibleUserId: "A" })).grant)
.toBeUndefined();
await db.update(toolConnections).set({ [location]: { sourceTemplateKey: "other" } }).where(eq(toolConnections.id, selected.connectionId));
expect(await resolveManagedGitHubIdentitySelection(observed, input.companyId, { agentId: input.agentId, responsibleUserId: "A" }))
.toEqual({ configured: false });
expect(await resolveManagedGitHubIdentitySelection(observed, randomUUID(), { agentId: input.agentId, responsibleUserId: "A" }))
.toEqual({ configured: false });
const saved = await db.select().from(toolConnections).where(eq(toolConnections.companyId, input.companyId));
for (const row of irrelevant) expect(saved.find(candidate => candidate.id === row.id)?.config).toEqual(row.config);
} finally { capture.flush(); await observed.$client.end(); }
});
it("resolves current GitHub credentials without transferring execution snapshots", async () => {
const input = await seed(), selected = await grant(input, "A");
const payload = "github-credential-saved-input🙂".repeat(25_000);
await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: input.issueId, prompt: payload },
runnerProfileJson: { nativeExecutionInput: { savedInput: payload }, savedInput: payload },
resultJson: { stdout: payload }, stdoutExcerpt: payload, stderrExcerpt: payload,
}).where(eq(heartbeatRuns.id, input.runId));
const [before] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, input.runId));
const observed = createDb(database.connectionString), events: Array<Record<string, unknown>> = [], now = Date.now();
const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event));
installDatabaseResponseCapture(observed, { heartbeatRuns }, capture);
try {
expect(await resolveGitHubOperationCredentials(observed, input)).toMatchObject({
status: "available", login: "A", grantId: selected.id,
});
capture.flush();
const reads = events.filter(event => event.table === "heartbeat_runs" && event.operation === "select");
expect(reads.length).toBeGreaterThan(0);
expect(reads.reduce((bytes, event) => bytes + Number(event.responseValueBytes), 0)).toBeLessThan(8192);
expect(JSON.stringify(events)).not.toContain("github-credential-saved-input");
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, input.runId)))[0]).toEqual(before);
const full = await captureRunIdentity(observed, input);
expect(full.run.runnerProfileJson).toEqual(before.runnerProfileJson);
expect(full.run.resultJson).toEqual(before.resultJson);
expect(full.run.contextSnapshot).toEqual(before.contextSnapshot);
await db.update(connectionGrants).set({ status: "revoked" }).where(eq(connectionGrants.id, selected.id));
expect(await resolveGitHubOperationCredentials(observed, input)).toMatchObject({ status: "unavailable" });
} finally { capture.flush(); await observed.$client.end(); }
});
async function switchTo(
input: Awaited<ReturnType<typeof seed>>,
user: string,
@@ -1,5 +1,6 @@
import { describe, expect, it } from "vitest";
import { eq } from "drizzle-orm";
import { eq, sql } from "drizzle-orm";
import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture";
import {
agents,
companies,
@@ -15,6 +16,62 @@ import {
import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
describe("P6-11..13 / P6-17 canonical event allocator", () => {
it("keeps native cursor responses bounded with long histories and a late gap fill", async () => {
const temporary = await startEmbeddedPostgresTestDatabase("paperclip-native-cursor-budget-");
const db = createDb(temporary.connectionString);
const companyId = "20000000-0000-4000-8000-000000000001";
const agentId = "20000000-0000-4000-8000-000000000002";
const runId = "20000000-0000-4000-8000-000000000003";
const otherRunId = "20000000-0000-4000-8000-000000000004";
try {
await db.insert(companies).values({ id: companyId, name: "Cursor budget", issuePrefix: "CBG" });
await db.insert(agents).values({ id: agentId, companyId, name: "Cursor agent" });
await db.insert(heartbeatRuns).values([
{ id: runId, companyId, agentId, status: "running", nextEventSeq: 5005 },
{ id: otherRunId, companyId, agentId, status: "running" },
]);
for (let batch = 0; batch < 10; batch += 1) {
await db.insert(heartbeatRunEvents).values(Array.from({ length: 500 }, (_, index) => {
const seq = batch * 500 + index + 1;
return { companyId, agentId, runId, seq, eventType: "item.delta", sourceInstanceId: "runner-a",
sourceSeq: seq, sourceEventId: `seed:${seq}` };
}));
}
await db.insert(heartbeatRunEvents).values([
...[-1, 0, null].map((sourceSeq, index) => ({
companyId, agentId, runId, seq: 5001 + index, eventType: "item.delta",
sourceInstanceId: "runner-a", sourceSeq, sourceEventId: `legacy:${index}`,
})),
{ companyId, agentId, runId, seq: 5004, eventType: "item.delta", sourceInstanceId: "runner-b",
sourceSeq: 5001, sourceEventId: "other-source:5001" },
{ companyId, agentId, runId: otherRunId, seq: 1, eventType: "item.delta", sourceInstanceId: "runner-a",
sourceSeq: 5001, sourceEventId: "other-run:5001" },
]);
const append = (sourceSeq: number) => appendHeartbeatRunEvent(db, {
companyId, agentId, runId, eventType: "item.delta", payload: { sourceSeq },
nativeSource: { sourceInstanceId: "runner-a", sourceSeq, sourceEventId: `runner-a:${sourceSeq}`,
protocolSchemaVersion: 1, canonicalPayload: { sourceSeq } },
});
const events: Array<Record<string, unknown>> = [], now = Date.now();
const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event));
installDatabaseResponseCapture(db, { heartbeatRunEvents }, capture);
try { expect((await append(5002)).highestContiguousSourceSeq).toBe(5000); }
finally { capture.flush(); }
const reads = events.filter(event => event.event === "paperclip.db_response_capture.entry.v1"
&& event.table === "heartbeat_run_events" && (event.operation === "select" || event.operation === "execute"));
expect(reads.reduce((sum, event) => sum + Number(event.calls), 0)).toBeGreaterThan(0);
expect(reads.reduce((sum, event) => sum + Number(event.rows), 0)).toBeLessThan(16);
expect(reads.reduce((sum, event) => sum + Number(event.responseValueBytes), 0)).toBeLessThan(2048);
expect((await append(5001)).highestContiguousSourceSeq).toBe(5002);
const replay = await append(5002);
expect(replay.disposition).toBe("duplicate");
expect(replay.highestContiguousSourceSeq).toBe(5002);
const [remaining] = await db.select({ count: sql<number>`count(*)::int` }).from(heartbeatRunEvents)
.where(eq(heartbeatRunEvents.runId, runId));
expect(remaining?.count).toBe(5006);
} finally { await temporary.cleanup(); }
}, 60_000);
it.each([false, true])("deduplicates concurrent retry exhaustion (historical receipt: %s)", async (historical) => {
const temporary = await startEmbeddedPostgresTestDatabase("paperclip-exhaustion-events-");
const db = createDb(temporary.connectionString);
@@ -165,4 +222,46 @@ describe("P6-11..13 / P6-17 canonical event allocator", () => {
await temporary.cleanup();
}
}, 60_000);
it("reports the highest contiguous source seq per source instance across gaps and late arrivals", async () => {
const temporary = await startEmbeddedPostgresTestDatabase("paperclip-native-cursor-");
const db = createDb(temporary.connectionString);
const companyId = "20000000-0000-4000-8000-000000000001";
const agentId = "20000000-0000-4000-8000-000000000002";
const runId = "20000000-0000-4000-8000-000000000003";
try {
await db.insert(companies).values({ id: companyId, name: "Cursor fixture", issuePrefix: "CUR" });
await db.insert(agents).values({ id: agentId, companyId, name: "Cursor agent" });
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running" });
const append = async (sourceInstanceId: string, sourceSeq: number) => (await appendHeartbeatRunEvent(db, {
companyId,
runId,
agentId,
eventType: "item.delta",
payload: { sourceInstanceId, sourceSeq },
nativeSource: {
sourceInstanceId,
sourceEventId: `${sourceInstanceId}:${sourceSeq}`,
sourceSeq,
protocolSchemaVersion: 1,
canonicalPayload: { sourceInstanceId, sourceSeq },
},
})).highestContiguousSourceSeq;
expect(await append("runner-a", 2)).toBe(0);
expect(await append("runner-a", 1)).toBe(2);
expect(await append("runner-a", 3)).toBe(3);
expect(await append("runner-a", 5)).toBe(3);
expect(await append("runner-a", 6)).toBe(3);
expect(await append("runner-b", 1)).toBe(1);
expect(await append("runner-a", 4)).toBe(6);
for (let sourceSeq = 7; sourceSeq <= 200; sourceSeq += 1) {
expect(await append("runner-a", sourceSeq)).toBe(sourceSeq);
}
expect(await append("runner-b", 3)).toBe(1);
expect(await append("runner-b", 2)).toBe(3);
} finally {
await temporary.cleanup();
}
}, 60_000);
});
@@ -1,6 +1,7 @@
import { randomUUID } from "node:crypto";
import { eq } from "drizzle-orm";
import { drizzle } from "drizzle-orm/postgres-js";
import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture";
import express from "express";
import request from "supertest";
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
@@ -257,6 +258,29 @@ describeEmbeddedPostgres("heartbeat run privacy routes", { concurrent: false },
return app;
}
it("lists workspace operations with small run reads and checks a revoked issue grant again", async () => {
const f = await seedFixture(), payload = "workspace-operation-saved-input🙂".repeat(25_000);
await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: f.issueId, prompt: payload },
runnerProfileJson: { instructions: payload }, stdoutExcerpt: payload, stderrExcerpt: payload,
}).where(eq(heartbeatRuns.id, f.privateRunId));
const [before] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.privateRunId));
const observed = createDb(tempDb!.connectionString), events: Array<Record<string, unknown>> = [], now = Date.now();
const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event));
installDatabaseResponseCapture(observed, { heartbeatRuns }, capture);
try {
const app = createApp(f.companyId, f.grantedAgentId, observed);
const response = await request(app).get(`/api/heartbeat-runs/${f.privateRunId}/workspace-operations`).expect(200);
expect(response.body).toEqual([expect.objectContaining({ id: f.operationId })]);
capture.flush();
const reads = events.filter(event => event.table === "heartbeat_runs" && event.operation === "select");
expect(reads.length).toBeGreaterThan(0);
expect(reads.reduce((bytes, event) => bytes + Number(event.responseValueBytes), 0)).toBeLessThan(2048);
await db.delete(issueAccessGrants).where(eq(issueAccessGrants.subjectId, f.grantedAgentId));
await request(app).get(`/api/heartbeat-runs/${f.privateRunId}/workspace-operations`).expect(404);
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.privateRunId)))[0]).toEqual(before);
} finally { capture.flush(); await observed.$client.end(); }
});
it("returns 404 for direct private-run content and metadata-only list stubs to a non-member", async () => {
const fixture = await seedFixture();
const app = createApp(fixture.companyId, fixture.otherAgentId);
@@ -459,6 +483,105 @@ describeEmbeddedPostgres("heartbeat run privacy routes", { concurrent: false },
process.env.PAPERCLIP_ISSUE_PRIVACY_MODE = "enforce";
}
});
it("batches distinct issue bindings without changing current authorization decisions", async () => {
const fixture = await seedFixture();
const ids = Array.from({ length: 64 }, () => randomUUID());
await db.insert(issues).values(ids.map((id, index) => ({
id, companyId: fixture.companyId, title: `History ${index}`, status: "in_progress",
visibility: index % 2 ? "private" : "open", privacyRootIssueId: index % 2 ? id : null,
assigneeAgentId: fixture.ownerAgentId, responsibleUserId: fixture.ownerUserId,
})));
const queries: string[] = [];
const observedDb = drizzle(db.$client, { schema, logger: { logQuery(query) { queries.push(query); } } }) as typeof db;
const access = authorizationService(observedDb);
const actor = { type: "agent" as const, companyId: fixture.companyId, agentId: fixture.grantedAgentId };
const bindings = [...ids, fixture.issueId].map(issueId => ({ companyId: fixture.companyId, scopeKind: "issue" as const, issueId }));
const runs = [...bindings, ...bindings, { ...bindings[0], companyId: randomUUID() },
{ ...bindings[0], issueId: randomUUID() }, { companyId: fixture.companyId, scopeKind: "company" as const },
{ companyId: fixture.companyId, scopeKind: "issue" as const }];
const individual = await getHeartbeatRunReadDecisions(observedDb, { decide: access.decide }, actor, runs);
const individualCount = queries.length;
queries.length = 0;
const batched = await getHeartbeatRunReadDecisions(observedDb, access, actor, runs);
expect(batched).toEqual(individual);
expect(batched.slice(0, 64)).toEqual(ids.map((_, index) => index % 2 === 0));
expect(batched.at(64)).toBe(true);
expect(batched.slice(-4)).toEqual([false, false, true, false]);
expect(queries.length).toBeLessThan(25);
expect(queries.length).toBeLessThan(individualCount / 4);
await db.update(issueAccessGrants).set({ revokedAt: new Date() }).where(eq(issueAccessGrants.subjectId, fixture.grantedAgentId));
expect((await getHeartbeatRunReadDecisions(observedDb, access, actor, runs)).at(64)).toBe(false);
});
it.each(["enforce", "shadow", "off"])("keeps missing and foreign issue bindings closed in %s mode", async mode => {
const fixture = await seedFixture();
process.env.PAPERCLIP_ISSUE_PRIVACY_MODE = mode;
try {
const access = authorizationService(db);
const actor = { type: "board" as const, userId: fixture.ownerUserId };
const binding = { companyId: fixture.companyId, scopeKind: "issue" as const, issueId: fixture.issueId };
const runs = [binding, { ...binding, issueId: randomUUID() }, { ...binding, companyId: randomUUID() }];
const expected = await getHeartbeatRunReadDecisions(db, { decide: access.decide }, actor, runs);
expect(await getHeartbeatRunReadDecisions(db, access, actor, runs)).toEqual(expected);
expect(expected).toEqual([true, false, false]);
await db.delete(companyMemberships).where(eq(companyMemberships.principalId, fixture.ownerUserId));
expect(await getHeartbeatRunReadDecisions(db, access, actor, [binding])).toEqual(
await getHeartbeatRunReadDecisions(db, { decide: access.decide }, actor, [binding]),
);
} finally { process.env.PAPERCLIP_ISSUE_PRIVACY_MODE = "enforce"; }
});
it("checks more than 500 task bindings with bounded company-scoped queries", async () => {
const fixture = await seedFixture();
const ids = Array.from({ length: 501 }, () => randomUUID());
await db.insert(issues).values(ids.map((id, index) => ({
id, companyId: fixture.companyId, title: `Batch ${index}`, visibility: "open", status: "in_progress",
})));
const queries: { query: string; params: unknown[] }[] = [];
const observedDb = drizzle(db.$client, { schema, logger: { logQuery(query, params) { queries.push({ query, params }); } } }) as typeof db;
const access = authorizationService(observedDb);
const actor = { type: "agent" as const, companyId: fixture.companyId, agentId: fixture.ownerAgentId };
const runs = ids.map(issueId => ({ companyId: fixture.companyId, scopeKind: "issue" as const, issueId }));
expect(await getHeartbeatRunReadDecisions(observedDb, access, actor, runs)).toEqual(ids.map(() => true));
const existence = queries.filter(({ query }) => query.startsWith('select "id" from "issues"'));
expect(existence.map(({ params }) => params.length).sort((a, b) => a - b)).toEqual([2, 501]);
expect(existence.every(({ query, params }) => query.includes('"company_id"') && params[0] === fixture.companyId)).toBe(true);
});
it("captures batch inputs before awaiting reads and rejects a failed batch", async () => {
const fixture = await seedFixture();
const access = authorizationService(db);
const actor = { type: "agent" as const, companyId: fixture.companyId, agentId: fixture.ownerAgentId };
const binding = { companyId: fixture.companyId, scopeKind: "issue" as const, issueId: fixture.issueId };
const decideReads = vi.spyOn(access, "decideReads");
const pending = getHeartbeatRunReadDecisions(db, access, actor, [binding, binding]);
binding.issueId = randomUUID();
expect(await pending).toEqual([true, true]);
expect(decideReads).toHaveBeenCalledExactlyOnceWith(actor, [expect.objectContaining({ issueId: fixture.issueId })]);
const error = new Error("authorization unavailable");
decideReads.mockRejectedValueOnce(error);
const current = { ...binding, issueId: fixture.issueId };
await expect(getHeartbeatRunReadDecisions(db, access, actor, [current])).rejects.toBe(error);
const transient = Object.assign(new Error("temporary disconnect"), { code: "CONNECTION_CLOSED" });
decideReads.mockRejectedValueOnce(transient);
expect(await getHeartbeatRunReadDecisions(db, access, actor, [current])).toEqual([true]);
});
it("does no database or authorization work for company-only or empty batches", async () => {
const fixture = await seedFixture();
const queries: string[] = [];
const observedDb = drizzle(db.$client, { schema, logger: { logQuery(query) { queries.push(query); } } }) as typeof db;
const access = authorizationService(observedDb);
const decideReads = vi.spyOn(access, "decideReads");
const actor = { type: "agent" as const, companyId: fixture.companyId, agentId: fixture.ownerAgentId };
expect(await getHeartbeatRunReadDecisions(observedDb, access, actor, [])).toEqual([]);
expect(await getHeartbeatRunReadDecisions(observedDb, access, actor, [
{ companyId: fixture.companyId, scopeKind: "company" }, { companyId: fixture.companyId, scopeKind: "issue" },
])).toEqual([true, false]);
expect(queries).toEqual([]);
expect(decideReads).not.toHaveBeenCalled();
});
it.each(["owner", "outsider"])("checks concurrent duplicate issue runs once for the %s", async (viewer) => {
const fixture = await seedFixture();
const access = authorizationService(db);
@@ -469,7 +592,7 @@ describeEmbeddedPostgres("heartbeat run privacy routes", { concurrent: false },
agentId: viewer === "owner" ? fixture.ownerAgentId : fixture.otherAgentId,
};
const binding = { companyId: fixture.companyId, scopeKind: "issue" as const, issueId: fixture.issueId };
const decisions = await getHeartbeatRunReadDecisions(db, access, actor, Array.from({ length: 200 }, () => ({ ...binding })));
const decisions = await getHeartbeatRunReadDecisions(db, { decide: access.decide }, actor, Array.from({ length: 200 }, () => ({ ...binding })));
expect(decisions).toEqual(Array(200).fill(viewer === "owner"));
expect(decide).toHaveBeenCalledTimes(1);
decide.mockRestore();
@@ -493,7 +616,7 @@ describeEmbeddedPostgres("heartbeat run privacy routes", { concurrent: false },
const decide = vi.spyOn(access, "decide");
const actor = { type: "agent" as const, companyId: fixture.companyId, agentId: fixture.ownerAgentId };
const binding = { companyId: fixture.companyId, scopeKind: "issue" as const, issueId: fixture.issueId };
const pending = getHeartbeatRunReadDecisions(db, access, actor, [binding, binding]);
const pending = getHeartbeatRunReadDecisions(db, { decide: access.decide }, actor, [binding, binding]);
binding.issueId = randomUUID();
expect(await pending).toEqual([true, true]);
expect(decide).toHaveBeenCalledExactlyOnceWith(expect.objectContaining({
@@ -567,7 +690,7 @@ describeEmbeddedPostgres("heartbeat run privacy routes", { concurrent: false },
const decide = vi.spyOn(access, "decide").mockRejectedValue(error);
const actor = { type: "agent" as const, companyId: fixture.companyId, agentId: fixture.ownerAgentId };
const binding = { companyId: fixture.companyId, scopeKind: "issue" as const, issueId: fixture.issueId };
await expect(getHeartbeatRunReadDecisions(db, access, actor, Array(20).fill(binding))).rejects.toBe(error);
await expect(getHeartbeatRunReadDecisions(db, { decide: access.decide }, actor, Array(20).fill(binding))).rejects.toBe(error);
expect(decide).toHaveBeenCalledTimes(1);
decide.mockRestore();
expect(await getHeartbeatRunReadDecisions(db, access, actor, [binding])).toEqual([true]);
@@ -25,10 +25,10 @@ describe("model-facing run errors", () => {
});
});
describe("scheduled continuation presentation", () => {
describe.each(["scheduled_monitor_waiting", "scheduled_monitor_dispatched", "scheduled_monitor_preserved"])("scheduled continuation presentation (%s)", finalizationReasonCode => {
const resultJson = {
finalizationPhase: "committed",
finalizationReasonCode: "scheduled_monitor_waiting",
finalizationReasonCode,
nativeResult: {
schema: "paperclip.run_result.v1",
reportedWorkDisposition: "yielded",
@@ -0,0 +1,109 @@
import { randomUUID } from "node:crypto";
import { activityLog, agents, companies, createDb, heartbeatRuns, issues } from "@paperclipai/db";
import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture";
import { eq } from "drizzle-orm";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { resolveIssueCreationSource, runBoundIssueReference } from "../services/issue-creation-source.js";
const support = await getEmbeddedPostgresTestSupport();
describe.skipIf(!support.supported)("creation source run projections", () => {
const companyId = randomUUID(), otherCompanyId = randomUUID(), agentId = randomUUID();
const sourceId = randomUUID(), secondId = randomUUID(), foreignId = randomUUID();
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let db: ReturnType<typeof createDb>;
let childNumber = 100;
beforeAll(async () => {
database = await startEmbeddedPostgresTestDatabase("creation-source-read-");
db = createDb(database.connectionString);
await db.insert(companies).values([
{ id: companyId, name: "Creation source fixture", issuePrefix: "CSR" },
{ id: otherCompanyId, name: "Foreign source fixture", issuePrefix: "FSR" },
]);
await db.insert(agents).values({ id: agentId, companyId, name: "Source agent", adapterType: "process" });
await db.insert(issues).values([
{ id: sourceId, companyId, title: "Source task", identifier: "CSR-1", issueNumber: 1 },
{ id: secondId, companyId, title: "Second source task", identifier: "CSR-2", issueNumber: 2 },
{ id: foreignId, companyId: otherCompanyId, title: "Foreign task", identifier: "FSR-1", issueNumber: 1 },
]);
}, 60_000);
afterAll(async () => {
await db?.$client.end();
await database?.cleanup();
});
async function fixture(values: Partial<typeof heartbeatRuns.$inferInsert>) {
const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId, status: "succeeded", ...values }).returning();
const number = childNumber++;
const [child] = await db.insert(issues).values({ companyId, title: "Created task", identifier: `CSR-${number}`,
issueNumber: number, originRunId: run.id }).returning();
return { run, child };
}
it.each(["scoped", "legacy task key", "legacy nested identifier"])("omits saved run payloads for %s bindings while preserving provenance and storage", async (binding) => {
const bulk = "retained-source-context🙂".repeat(32_768);
const contextSnapshot = binding === "scoped"
? { issueId: sourceId, paperclipIssue: { id: sourceId, retainedDescription: bulk }, retainedLaunchInput: bulk }
: binding === "legacy task key"
? { taskKey: "CSR-1", paperclipIssue: { retainedDescription: bulk }, retainedLaunchInput: bulk }
: { paperclipIssue: { id: "CSR-1", retainedDescription: bulk }, retainedLaunchInput: bulk };
const { run, child } = await fixture({ contextSnapshot,
runnerProfileJson: { retainedInput: bulk }, stdoutExcerpt: bulk, stderrExcerpt: bulk, resultJson: { retainedReceipt: bulk } });
expect(run.issueId).toBe(binding === "scoped" ? sourceId : null);
expect(run.nativeIssueId).toBeNull();
const observed = createDb(database.connectionString), events: Array<Record<string, unknown>> = [], now = Date.now();
const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event));
installDatabaseResponseCapture(observed, { heartbeatRuns, issues }, capture);
try {
const result = await resolveIssueCreationSource(observed, child);
capture.flush();
const reads = events.filter(event => event.table === "heartbeat_runs" && event.operation === "select");
expect(reads.reduce((count, event) => count + Number(event.calls), 0)).toBe(1);
expect(reads.reduce((bytes, event) => bytes + Number(event.responseValueBytes), 0)).toBeLessThan(8192);
expect(result).toMatchObject({ run: { id: run.id, companyId, agentId }, sourceIssue: { id: sourceId } });
expect(runBoundIssueReference(result!.run)).toBe(binding === "scoped" ? sourceId : "CSR-1");
for (const key of ["runnerProfileJson", "stdoutExcerpt", "stderrExcerpt", "resultJson"]) expect(result!.run).not.toHaveProperty(key);
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0]).toEqual(run);
expect(JSON.stringify(events)).not.toContain("retained-source-context");
} finally {
capture.flush();
await observed.$client.end();
}
});
it.each([
{ label: "native identity", nativeIssueId: sourceId, issueId: secondId, contextSnapshot: { issueId: secondId }, expected: sourceId },
{ label: "issue scope", issueId: sourceId, contextSnapshot: { issueId: secondId }, expected: sourceId },
{ label: "legacy issue id", contextSnapshot: { issueId: sourceId, taskId: secondId }, expected: sourceId },
{ label: "typed task id fallback", contextSnapshot: { issueId: 42, taskId: sourceId }, expected: sourceId },
{ label: "nested issue identifier", contextSnapshot: { issueId: " ", paperclipIssue: { id: "CSR-1", ignored: "payload" }, taskKey: "CSR-2" }, expected: sourceId },
{ label: "typed nested id fallback", contextSnapshot: { paperclipIssue: { id: 42 }, taskKey: "CSR-1" }, expected: sourceId },
{ label: "foreign context", contextSnapshot: { taskId: foreignId }, expected: null },
{ label: "null context", contextSnapshot: null, expected: null },
])("preserves binding precedence for $label", async ({ label: _label, expected, ...values }) => {
const { child } = await fixture(values);
expect((await resolveIssueCreationSource(db, child))?.sourceIssue.id ?? null).toBe(expected);
});
it.each([42, "scalar", ["issueId", sourceId]])("does not invent a task binding from malformed context %j", async (context) => {
const { child } = await fixture({ contextSnapshot: context as unknown as Record<string, unknown> });
expect(await resolveIssueCreationSource(db, child)).toBeNull();
});
it("preserves historical activity lookup, current bindings, and missing or foreign source denial", async () => {
const { run, child } = await fixture({ contextSnapshot: { taskKey: "CSR-1" } });
await db.update(issues).set({ originRunId: null }).where(eq(issues.id, child.id));
await db.insert(activityLog).values({ companyId, actorType: "system", actorId: "fixture", action: "issue.created",
entityType: "issue", entityId: child.id, runId: run.id });
expect((await resolveIssueCreationSource(db, { ...child, originRunId: null }))?.sourceIssue.id).toBe(sourceId);
await db.update(heartbeatRuns).set({ contextSnapshot: { taskKey: "CSR-2" } }).where(eq(heartbeatRuns.id, run.id));
expect((await resolveIssueCreationSource(db, child))?.sourceIssue.id).toBe(secondId);
expect(await resolveIssueCreationSource(db, { ...child, companyId: otherCompanyId })).toBeNull();
expect(await resolveIssueCreationSource(db, { ...child, originRunId: randomUUID() })).toBeNull();
await db.update(heartbeatRuns).set({ contextSnapshot: { taskId: child.id } }).where(eq(heartbeatRuns.id, run.id));
expect(await resolveIssueCreationSource(db, child)).toBeNull();
});
});
@@ -791,6 +791,62 @@ describeEmbeddedPostgres("issue list routes assigneeAgentId filter", () => {
});
});
it.each([
"projectId",
"parentId",
"parentIssueId",
"workspaceId",
"executionWorkspaceId",
"descendantOf",
"labelId",
"participantAgentId",
])("returns 422 instead of a DB error for malformed %s filters", async (param) => {
const companyId = randomUUID();
await db.insert(companies).values({
id: companyId,
name: "Paperclip",
issuePrefix: uniqueIssuePrefix(),
requireBoardApprovalForNewAgents: false,
});
await seedCloudTenantMember(companyId);
const app = createApp(companyId);
const res = await request(app)
.get(`/api/companies/${companyId}/issues`)
.query({ status: "todo", [param]: "d", limit: "20" });
expect(res.status, JSON.stringify(res.body)).toBe(422);
expect(res.body).toMatchObject({ error: `${param} must be a UUID` });
});
it.each([
"projectId",
"parentId",
"parentIssueId",
"workspaceId",
"executionWorkspaceId",
"descendantOf",
"labelId",
"participantAgentId",
])("returns 422 from issues/count for malformed %s filters", async (param) => {
const companyId = randomUUID();
await db.insert(companies).values({
id: companyId,
name: "Paperclip",
issuePrefix: uniqueIssuePrefix(),
requireBoardApprovalForNewAgents: false,
});
await seedCloudTenantMember(companyId);
const app = createApp(companyId);
const res = await request(app)
.get(`/api/companies/${companyId}/issues/count`)
.query({ attention: "blocked", [param]: "d" });
expect(res.status, JSON.stringify(res.body)).toBe(422);
expect(res.body).toMatchObject({ error: `${param} must be a UUID` });
});
it("returns opt-in live descendant counts for offscreen live descendants only", async () => {
const companyId = randomUUID();
const otherCompanyId = randomUUID();
@@ -4,6 +4,7 @@ import { randomUUID } from "node:crypto";
import express from "express";
import request from "supertest";
import { eq, sql } from "drizzle-orm";
import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture";
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
import {
agentRuntimeState,
@@ -79,7 +80,7 @@ describeEmbeddedPostgres("issue queued-comment routes", () => {
await tempDb?.cleanup();
});
function app(companyId: string, userId = "queue-owner", agentActor?: { agentId: string; runId: string }) {
function app(companyId: string, userId = "queue-owner", agentActor?: { agentId: string; runId: string }, database = db) {
const testApp = express();
testApp.use(express.json());
testApp.use((req, _res, next) => {
@@ -94,7 +95,7 @@ describeEmbeddedPostgres("issue queued-comment routes", () => {
};
next();
});
testApp.use("/api", issueRoutes(db, {} as any, {}));
testApp.use("/api", issueRoutes(database, {} as any, {}));
testApp.use(errorHandler);
return testApp;
}
@@ -192,6 +193,104 @@ describeEmbeddedPostgres("issue queued-comment routes", () => {
return { companyId, agentId, issueId, runId, wakeId, commentIds };
}
it.each([false, true])("polls queued comments without transferring run snapshots (agent fallback: %s)", async fallback => {
const f = await seedQueue(), payload = "saved-execution-input🙂".repeat(25_000);
await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: f.issueId, prompt: payload },
runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner", adapterConfig: { instructions: payload } } },
stdoutExcerpt: payload, stderrExcerpt: payload,
}).where(eq(heartbeatRuns.id, f.runId));
if (fallback) await db.update(issues).set({ executionRunId: null }).where(eq(issues.id, f.issueId));
const [before] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
const observed = createDb(tempDb!.connectionString), events: Array<Record<string, unknown>> = [], now = Date.now();
const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event));
installDatabaseResponseCapture(observed, { heartbeatRuns }, capture);
try {
const response = await request(app(f.companyId, "queue-owner", undefined, observed))
.get(`/api/issues/${f.issueId}/queued-comments`).expect(200);
expect(response.body.queueId).toBe(f.wakeId);
expect(response.body.targetRunId).toBe(f.runId);
expect(response.body.entries.map((entry: { comment: { id: string } }) => entry.comment.id)).toEqual(f.commentIds);
capture.flush();
const reads = events.filter(event => event.table === "heartbeat_runs" && event.operation === "select");
expect(reads.length).toBeGreaterThan(0);
expect(reads.reduce((bytes, event) => bytes + Number(event.responseValueBytes), 0)).toBeLessThan(2048);
await request(app(randomUUID(), "queue-owner", undefined, observed))
.get(`/api/issues/${f.issueId}/queued-comments`).expect(404);
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0]).toEqual(before);
} finally { capture.flush(); await observed.$client.end(); }
});
it.each(["deferred", "queued"])("reads only queue display inputs from large %s wake payloads", async state => {
const f = await seedQueue();
if (state === "queued") await promoteQueue(f);
const client = app(f.companyId);
const path = `/api/issues/${f.issueId}/queued-comments`;
const expected = await request(client).get(path).expect(200);
const [wake] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, f.wakeId));
const payload = "saved-wake-input🙂".repeat(30_000);
const context = wake.payload?._paperclipWakeContext as Record<string, unknown>;
await db.update(agentWakeupRequests).set({ payload: {
...wake.payload, prompt: payload,
_paperclipWakeContext: { ...context, prompt: payload, nativeExecutionInput: { instructions: payload } },
} }).where(eq(agentWakeupRequests.id, f.wakeId));
await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: f.issueId, prompt: payload },
runnerProfileJson: { instructions: payload }, resultJson: { output: payload },
}).where(eq(heartbeatRuns.companyId, f.companyId));
const [before] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, f.wakeId));
const observed = createDb(tempDb!.connectionString), events: Array<Record<string, unknown>> = [], now = Date.now();
const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event));
installDatabaseResponseCapture(observed, { agentWakeupRequests, heartbeatRuns }, capture);
try {
const response = await request(app(f.companyId, "queue-owner", undefined, observed)).get(path).expect(200);
expect(response.body).toEqual(expected.body);
capture.flush();
const reads = events.filter(event => ["agent_wakeup_requests", "heartbeat_runs"].includes(String(event.table)) && event.operation === "select");
expect(reads.some(event => event.table === "agent_wakeup_requests")).toBe(true);
expect(reads.reduce((bytes, event) => bytes + Number(event.responseValueBytes), 0)).toBeLessThan(4096);
expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, f.wakeId)))[0]).toEqual(before);
await request(app(randomUUID(), "queue-owner", undefined, observed)).get(path).expect(404);
} finally { capture.flush(); await observed.$client.end(); }
});
it.each(["deferred", "queued"])("filters unrelated task wakes before reading the %s comment queue", async state => {
const f = await seedQueue();
if (state === "queued") await promoteQueue(f);
const path = `/api/issues/${f.issueId}/queued-comments`;
const expected = await request(app(f.companyId)).get(path).expect(200);
const unrelated = Array.from({ length: 80 }, (_, index) => ({
companyId: f.companyId, agentId: f.agentId, source: "issue_comment",
status: index % 2 ? "queued" : "deferred_issue_execution",
requestedAt: new Date("2020-01-01T00:00:00Z"),
payload: { issueId: randomUUID(), _paperclipWakeContext: {
wakeCommentIds: Array.from({ length: 60 }, () => randomUUID()),
} },
}));
const malformed = [null, [f.issueId], { id: f.issueId }, f.issueId.toUpperCase()];
await db.insert(agentWakeupRequests).values([
...unrelated,
...malformed.map(issueId => ({ companyId: f.companyId, agentId: f.agentId,
source: "issue_comment", status: "deferred_issue_execution",
requestedAt: new Date("2020-01-01T00:00:00Z"),
payload: { issueId, _paperclipWakeContext: { wakeCommentIds: f.commentIds } },
})),
]);
const observed = createDb(tempDb!.connectionString), events: Array<Record<string, unknown>> = [], now = Date.now();
const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event));
installDatabaseResponseCapture(observed, { agentWakeupRequests }, capture);
try {
const response = await request(app(f.companyId, "queue-owner", undefined, observed)).get(path).expect(200);
expect(response.body).toEqual(expected.body);
capture.flush();
const reads = events.filter(event => event.table === "agent_wakeup_requests" && event.operation === "select");
expect(reads.length).toBeGreaterThan(0);
expect(reads.reduce((bytes, event) => bytes + Number(event.responseValueBytes), 0)).toBeLessThan(4096);
expect(reads.reduce((rows, event) => rows + Number(event.rows), 0)).toBe(1);
expect(await db.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests)
.where(eq(agentWakeupRequests.companyId, f.companyId))).toHaveLength(85);
await request(app(randomUUID(), "queue-owner", undefined, observed)).get(path).expect(404);
} finally { capture.flush(); await observed.$client.end(); }
});
it.each(["cancelled", "running"])("rejects a late Done from an interrupted %s task run at the write boundary", async status => {
const seeded = await seedQueue();
await db.update(heartbeatRuns).set({ status, resultJson: { executionCancellation: { state: "requested" } } })
@@ -14,6 +14,7 @@ const mockTreeControlService = vi.hoisted(() => ({
cancelIssueStatusesForHold: vi.fn(),
restoreIssueStatusesForHold: vi.fn(),
getHold: vi.fn(),
getActivePauseHoldGate: vi.fn(),
releaseHold: vi.fn(),
cancelUnclaimedWakeupsForTree: vi.fn(),
}));
@@ -72,6 +73,7 @@ describe("issue tree control routes", () => {
mockReplayBlocks.mockResolvedValue([]);
mockExecutionBlocker.mockResolvedValue(null);
mockTreeControlService.getHold.mockResolvedValue(null);
mockTreeControlService.getActivePauseHoldGate.mockResolvedValue(null);
mockIssueService.getById.mockResolvedValue({
id: "11111111-1111-4111-8111-111111111111",
companyId: "company-2",
@@ -213,6 +215,36 @@ describe("issue tree control routes", () => {
},
);
it.each([
{ skipReason: "terminal_status", stillPaused: false },
{ skipReason: "already_held", stillPaused: false },
{ skipReason: "already_held", stillPaused: true },
])("rechecks skipped pause members with $skipReason and remaining pause=$stillPaused", async ({ skipReason, stillPaused }) => {
const rootId = "11111111-1111-4111-8111-111111111111";
const holdId = "33333333-3333-4333-8333-333333333333";
const childId = "44444444-4444-4444-8444-444444444444";
const root = { id: rootId, companyId: "company-2", status: "todo", assigneeAgentId: null };
const child = { id: childId, companyId: "company-2", status: "todo", assigneeAgentId: "current-agent" };
mockIssueService.getById.mockImplementation(async (id: string) => id === rootId ? root : child);
const hold = { id: holdId, rootIssueId: rootId, mode: "pause", members: [
{ issueId: childId, skipped: true, skipReason, issueStatus: "done" },
] };
mockTreeControlService.getHold.mockResolvedValue(hold);
mockTreeControlService.releaseHold.mockResolvedValue({ ...hold, status: "released" });
mockTreeControlService.getActivePauseHoldGate.mockResolvedValue(stillPaused ? { holdId: "remaining-hold" } : null);
mockReplayBlocks.mockResolvedValue([{ id: childId, identifier: "TEST-2" }]);
const app = await createApp({ type: "board", userId: "user-1", companyIds: ["company-2"], source: "session" });
const response = await request(app)
.post(`/api/issues/${rootId}/tree-holds/${holdId}/release`)
.send({ metadata: { wakeAgents: true } });
expect(response.status).toBe(200);
expect(mockExecutionBlocker).toHaveBeenCalledWith(expect.anything(), "company-2", childId);
expect(mockHeartbeatService.wakeup).toHaveBeenCalledTimes(stillPaused ? 0 : 1);
if (!stillPaused) expect(mockHeartbeatService.wakeup).toHaveBeenCalledWith("current-agent", expect.objectContaining({
payload: expect.objectContaining({ issueId: childId }),
}));
});
it("resumes sandbox-backed work through the shared plugin worker manager", async () => {
const rootId = "11111111-1111-4111-8111-111111111111";
const holdId = "33333333-3333-4333-8333-333333333333";
@@ -284,14 +316,14 @@ describe("issue tree control routes", () => {
).toEqual([rootId, "child"]);
});
it("keeps the hold active when resume requests an unsafe execution replay", async () => {
it.each([false, true])("keeps the hold active when resume requests an unsafe execution replay (previously skipped=%s)", async (skipped) => {
const rootId = "11111111-1111-4111-8111-111111111111";
const holdId = "33333333-3333-4333-8333-333333333333";
mockTreeControlService.getHold.mockResolvedValue({
id: holdId,
rootIssueId: rootId,
mode: "pause",
members: [{ issueId: rootId }],
members: [{ issueId: rootId, skipped, skipReason: skipped ? "terminal_status" : null }],
});
mockReplayBlocks.mockResolvedValue([{ id: rootId, identifier: "TEST-1" }]);
mockExecutionBlocker.mockResolvedValue({ nextAction: "Resume without waking agents until the previous execution stops.",
@@ -6,6 +6,8 @@ import {
agents,
agentWakeupRequests,
companies,
applyPendingMigrations,
closeRegisteredClients,
createDb,
heartbeatRuns,
issueComments,
@@ -20,8 +22,10 @@ import {
} from "./helpers/embedded-postgres.js";
import { issueTreeControlService } from "../services/issue-tree-control.js";
import { issueService } from "../services/issues.js";
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_ISSUE_TREE_TEST_DATABASE_URL");
const embeddedPostgresSupport = externalUrl ? { supported: true } : await getEmbeddedPostgresTestSupport();
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
if (!embeddedPostgresSupport.supported) {
@@ -35,6 +39,11 @@ describeEmbeddedPostgres("issueTreeControlService", () => {
let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
beforeAll(async () => {
if (externalUrl) {
await applyPendingMigrations(externalUrl);
db = createDb(externalUrl);
return;
}
tempDb = await startEmbeddedPostgresTestDatabase("paperclip-issue-tree-control-");
db = createDb(tempDb.connectionString);
}, 20_000);
@@ -53,6 +62,7 @@ describeEmbeddedPostgres("issueTreeControlService", () => {
});
afterAll(async () => {
if (externalUrl) await closeRegisteredClients(externalUrl);
await tempDb?.cleanup();
});
@@ -623,6 +633,58 @@ describeEmbeddedPostgres("issueTreeControlService", () => {
expect(checkedOutLegacyFullPauseRoot.checkoutRunId).toBe(rootRunId);
});
it("previews only executable current assignees as resume wake targets", async () => {
const companyId = randomUUID();
const rootId = randomUUID();
const readyAgentId = randomUUID();
const parkedAgentId = randomUUID();
const previousAgentId = randomUUID();
const runId = randomUUID();
await db.insert(companies).values({
id: companyId, name: "Resume preview", issuePrefix: "RESUME",
});
await db.insert(agents).values([readyAgentId, parkedAgentId, previousAgentId].map((id) => ({
id, companyId, name: id, role: "engineer", status: "idle", adapterType: "process",
})));
const children = ["todo", "in_progress", "in_review", "blocked", "backlog", "done"].map((status) => ({
id: randomUUID(), companyId, parentId: rootId, title: status, status,
assigneeAgentId: status === "blocked" || status === "backlog" ? parkedAgentId : readyAgentId,
}));
await db.insert(issues).values([
{ id: rootId, companyId, title: "Unassigned root", status: "todo" },
...children,
]);
const svc = issueTreeControlService(db);
const pause = await svc.createHold(companyId, rootId, {
mode: "pause",
actor: { actorType: "user", actorId: "board-user", userId: "board-user" },
});
expect(pause.hold.members?.find((member) => member.issueId === children[5].id)).toMatchObject({
skipped: true, skipReason: "terminal_status",
});
await db.update(issues).set({ status: "todo" }).where(eq(issues.id, children[5].id));
// A previous executor is observable while stopping, but is not the current assignee to wake.
await db.insert(heartbeatRuns).values({
id: runId, companyId, agentId: previousAgentId, invocationSource: "assignment",
status: "running", contextSnapshot: { issueId: children[0].id },
});
await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, children[0].id));
const preview = await svc.preview(companyId, rootId, { mode: "resume" });
expect(preview.totals.affectedIssues).toBe(7);
expect(preview.affectedAgents).toEqual([{ agentId: readyAgentId, issueCount: 4, activeRunCount: 0 }]);
expect(preview.totals.affectedAgents).toBe(1);
expect(preview.warnings).toContainEqual({
code: "resume_tasks_remain_idle",
message: "Blocked, backlog, and unassigned tasks will be unpaused but will not start.",
issueIds: [rootId, children[3].id, children[4].id].sort(),
});
const rows = await db.select().from(issues).where(eq(issues.companyId, companyId));
expect(rows.find((row) => row.id === children[3].id)?.status).toBe("blocked");
expect(rows.find((row) => row.id === children[4].id)?.status).toBe("backlog");
expect((await svc.getHold(companyId, pause.hold.id))?.status).toBe("active");
});
it("resumes subtree pauses by releasing matching pause holds", async () => {
const companyId = randomUUID();
const rootIssueId = randomUUID();
@@ -1,6 +1,7 @@
import { randomUUID } from "node:crypto";
import { randomBytes, randomUUID } from "node:crypto";
import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture";
import { and, eq, sql } from "drizzle-orm";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
import { agents, companies, createDb, environmentLeases, environments, executionWorkspaces, heartbeatRuns, issueRecoveryActions, issues, projects } from "@paperclipai/db";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { recordLegacyWorkspaceRestoreFailure, legacyExecutionNeedsReconciliationWithEvidence, terminalizeLegacyExecution } from "../services/legacy-execution-recovery.js";
@@ -8,6 +9,7 @@ import { hasConversationContinuationPolicy, conversationRecoveryActionPredicate
import { preserveWorkspaceRestoreRecoveryMetadataSql } from "../services/legacy-workspace-restore-recovery.js";
import { hasConfirmedSandboxStopAndRetain, prepareSandboxStopAndRetain, settleStopOnlyCleanup } from "../services/sandbox-stop-and-retain.js";
import { environmentService } from "../services/environments.js";
import { deliverExecutionStatuses } from "../services/execution-status-delivery.js";
import { hasRequiredWorkspaceRecovery } from "../services/workspace-restore-recovery-state.js";
const externalTestDatabaseUrl = process.env.PAPERCLIP_TEST_DATABASE_URL?.trim();
@@ -45,6 +47,127 @@ const support = externalTestDatabaseUrl ? { supported: true } : await getEmbedde
} };
const readLease = async (id: string) => (await db.select().from(environmentLeases).where(eq(environmentLeases.id, id)))[0];
it("locks only terminal control state while returning and retaining the complete run", async () => {
const f = await seed();
const bulk = randomBytes(128 * 1024).toString("base64");
const [saved] = await db.update(heartbeatRuns).set({ contextSnapshot: { ...f.run.contextSnapshot, savedInput: bulk },
runnerProfileJson: { savedProfile: bulk }, stdoutExcerpt: bulk, stderrExcerpt: bulk,
}).where(eq(heartbeatRuns.id, f.runId)).returning();
const events: Array<Record<string, unknown>> = [], now = Date.now();
const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event));
installDatabaseResponseCapture(db, { heartbeatRuns }, capture);
let returned: Awaited<ReturnType<typeof terminalizeLegacyExecution>>;
try { returned = await terminalizeLegacyExecution({ db, run: saved!, status: "failed", patch }); }
finally { capture.flush(); }
const groups = events.filter(event => event.event === "paperclip.db_response_capture.entry.v1");
const reads = groups.filter(group => group.table === "heartbeat_runs" && group.operation === "select");
expect(reads.reduce((sum, group) => sum + Number(group.calls), 0)).toBe(1);
expect(reads.reduce((sum, group) => sum + Number(group.responseValueBytes), 0)).toBeLessThan(4096);
expect(returned).toMatchObject({ id: f.runId, status: "failed", contextSnapshot: saved!.contextSnapshot,
runnerProfileJson: saved!.runnerProfileJson, stdoutExcerpt: bulk, stderrExcerpt: bulk });
expect(hasRequiredWorkspaceRecovery(returned!.resultJson)).toBe(true);
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0]).toEqual(returned);
expect(await readLease(f.lease.id)).toMatchObject({ status: "pending_cleanup", leasePolicy: "retain_on_failure" });
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId)))
.toEqual([expect.objectContaining({ evidence: expect.objectContaining({ runId: f.runId }), ownerType: "board" })]);
expect(JSON.stringify(events)).not.toContain(bulk.slice(0, 64));
});
it.each(["legacy", "native"])("returns one complete run when retained workspace evidence is unchanged for %s", async (runtimeMode) => {
const f = await seed();
const bulk = randomBytes(128 * 1024).toString("base64");
const [saved] = await db.update(heartbeatRuns).set({ runtimeMode, status: "failed", finishedAt: new Date(),
contextSnapshot: { ...f.run.contextSnapshot, savedInput: bulk }, runnerProfileJson: { savedProfile: bulk },
stdoutExcerpt: bulk, stderrExcerpt: bulk,
}).where(eq(heartbeatRuns.id, f.runId)).returning();
const recordRestoreFailureOnly = runtimeMode === "native";
const failed = (await terminalizeLegacyExecution({ db, run: saved!, status: "failed", recordRestoreFailureOnly,
workspaceRestoreSource: f.lease,
patch: { resultJson: { workspaceRestoreFailure: "restore_failed", futureResult: bulk } },
}))!;
const events: Array<Record<string, unknown>> = [], now = Date.now();
const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event));
installDatabaseResponseCapture(db, { heartbeatRuns }, capture);
let returned: Awaited<ReturnType<typeof terminalizeLegacyExecution>>;
try { returned = await terminalizeLegacyExecution({ db, run: failed, status: "failed", recordRestoreFailureOnly,
workspaceRestoreSource: f.lease,
...(recordRestoreFailureOnly ? { patch: { resultJson: failed.resultJson } } : {}),
}); }
finally { capture.flush(); }
const writes = events.filter(event => event.event === "paperclip.db_response_capture.entry.v1"
&& event.table === "heartbeat_runs" && event.operation === "update");
expect(writes.reduce((sum, group) => sum + Number(group.rows), 0)).toBe(1);
expect(returned).toMatchObject({ id: f.runId, status: "failed", finishedAt: failed.finishedAt,
contextSnapshot: failed.contextSnapshot, runnerProfileJson: failed.runnerProfileJson, resultJson: failed.resultJson,
stdoutExcerpt: bulk, stderrExcerpt: bulk });
if (!recordRestoreFailureOnly) expect(returned!.executionStatusDeliveryId).toBe(failed.executionStatusDeliveryId);
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0]).toEqual(returned);
expect(await readLease(f.lease.id)).toMatchObject({ status: "pending_cleanup", leasePolicy: "retain_on_failure" });
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId)))
.toEqual([expect.objectContaining({ evidence: expect.objectContaining({ runId: f.runId }), ownerType: "board" })]);
expect(JSON.stringify(events)).not.toContain(bulk.slice(0, 64));
});
it("preserves pending and acknowledged status delivery on unpatched legacy recovery retries", async () => {
await deliverExecutionStatuses(db, { publish: vi.fn() });
const f = await seed();
const failed = (await terminalizeLegacyExecution({ db, run: f.run, status: "failed", patch }))!;
const holds = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
expect(holds).toHaveLength(1);
expect(hasRequiredWorkspaceRecovery(failed.resultJson)).toBe(true);
const repeated = (await terminalizeLegacyExecution({ db, run: failed, status: "failed" }))!;
expect(repeated.executionStatusDeliveryId).toBe(failed.executionStatusDeliveryId);
const publish = vi.fn();
await deliverExecutionStatuses(db, { publish });
await terminalizeLegacyExecution({ db, run: failed, status: "failed" });
await deliverExecutionStatuses(db, { publish });
expect(publish.mock.calls.filter(([event]) => event.payload.runId === f.runId)).toHaveLength(1);
const after = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
expect(after).toHaveLength(1);
expect(after[0]).toMatchObject({ id: holds[0].id, status: holds[0].status, outcome: holds[0].outcome,
ownerType: holds[0].ownerType, wakePolicy: null, evidence: { ...holds[0].evidence,
automaticRecovery: expect.objectContaining({ policy: "preserve_without_replay_v1", replay: "blocked",
runId: f.runId, actionOutcome: "unknown" }),
} });
expect(await readLease(f.lease.id)).toMatchObject({ status: "pending_cleanup", leasePolicy: "retain_on_failure" });
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0].resultJson).toEqual(failed.resultJson);
});
it("delivers new retained workspace evidence even when legacy run status is unchanged", async () => {
await deliverExecutionStatuses(db, { publish: vi.fn() });
const f = await seed();
const failed = (await terminalizeLegacyExecution({ db, run: f.run, status: "failed", patch }))!;
const publish = vi.fn();
await deliverExecutionStatuses(db, { publish });
const [added] = await db.insert(environmentLeases).values({ ...f.lease, id: randomUUID(), providerLeaseId: randomUUID() }).returning();
const updated = (await terminalizeLegacyExecution({ db, run: failed, status: "failed" }))!;
expect(updated.executionStatusDeliveryId).toBeTruthy();
expect(updated.executionStatusDeliveryId).not.toBe(failed.executionStatusDeliveryId);
expect(updated.resultJson?.workspaceRestoreRecovery).toMatchObject({ leaseIds: expect.arrayContaining([f.lease.id, added.id]) });
await deliverExecutionStatuses(db, { publish });
expect(publish.mock.calls.filter(([event]) => event.payload.runId === f.runId)).toHaveLength(2);
await terminalizeLegacyExecution({ db, run: updated, status: "failed" });
await deliverExecutionStatuses(db, { publish });
expect(publish.mock.calls.filter(([event]) => event.payload.runId === f.runId)).toHaveLength(2);
});
it("retains write conditions and delivers an explicit patch on unchanged legacy status", async () => {
await deliverExecutionStatuses(db, { publish: vi.fn() });
const f = await seed();
const failed = (await terminalizeLegacyExecution({ db, run: f.run, status: "failed", patch }))!;
const publish = vi.fn();
await deliverExecutionStatuses(db, { publish });
const holds = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
expect(await terminalizeLegacyExecution({ db, run: failed, status: "failed", writeConditions: [sql`false`] })).toBeNull();
await deliverExecutionStatuses(db, { publish });
expect(publish.mock.calls.filter(([event]) => event.payload.runId === f.runId)).toHaveLength(1);
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId))).toEqual(holds);
const patched = (await terminalizeLegacyExecution({ db, run: failed, status: "failed", patch: { error: "Retained workspace still needs repair" } }))!;
expect(patched.error).toBe("Retained workspace still needs repair");
await deliverExecutionStatuses(db, { publish });
expect(publish.mock.calls.filter(([event]) => event.payload.runId === f.runId)).toHaveLength(2);
});
async function stopAndRetain(f: Awaited<ReturnType<typeof seed>>, retry = false) {
let stopping = (await prepareSandboxStopAndRetain(db, f.lease))!;
expect(stopping.status).toBe("pending_cleanup");
@@ -58,9 +58,19 @@ describe("live-event run read boundaries", () => {
}).returning({ id: heartbeatRuns.id });
const [maintenance] = await db.insert(heartbeatRuns).values({ companyId, agentId: agent.id, scopeKind: "company" }).returning({ id: heartbeatRuns.id });
const [foreign] = await db.insert(heartbeatRuns).values({ companyId: foreignCompanyId, agentId: foreignAgent.id, scopeKind: "company" }).returning({ id: heartbeatRuns.id });
const hiddenIssueId = randomUUID();
await db.insert(issues).values({ id: hiddenIssueId, companyId, title: "Other private task", visibility: "private", privacyRootIssueId: hiddenIssueId, responsibleUserId: owner });
const [hiddenRun] = await db.insert(heartbeatRuns).values({ companyId, agentId: agent.id, scopeKind: "issue", issueId: hiddenIssueId }).returning({ id: heartbeatRuns.id });
const reads: string[] = [];
const queries: string[] = [];
let failNextRunRead = false;
let failNextIssueRead = false;
const observedDb = drizzle(db.$client, { schema, logger: { logQuery(query) {
queries.push(query);
if (query.startsWith("select ") && query.includes('from "issues"') && failNextIssueRead) {
failNextIssueRead = false;
throw new Error("fixture issue lookup unavailable");
}
if (query.startsWith("select ") && query.includes('from "heartbeat_runs"')) {
reads.push(query);
if (failNextRunRead) { failNextRunRead = false; throw new Error("fixture run lookup unavailable"); }
@@ -78,17 +88,32 @@ describe("live-event run read boundaries", () => {
socket.on("message", bytes => messages.push(JSON.parse(bytes.toString())));
try {
await once(socket, "open");
async function deliver(runId: string) {
let deliveries = 0;
async function deliver(runId: string, payloadIssueId?: string) {
deliveries += 1;
const start = messages.length;
const barrier = randomUUID();
publishLiveEvent({ companyId, type: "heartbeat.run.status", payload: { runId, status: "running" } });
publishLiveEvent({ companyId, type: "heartbeat.run.status", payload: {
runId, status: "running", ...(payloadIssueId ? { issueId: payloadIssueId } : {}),
} });
publishLiveEvent({ companyId, type: "agent.status", payload: { agentId: barrier, status: "idle" } });
await expect.poll(() => messages.some(message => message.payload.agentId === barrier)).toBe(true);
return messages.slice(start).filter(message => message.type === "heartbeat.run.status");
}
expect(await deliver(run.id)).toMatchObject([{ payload: { runId: run.id, status: "running" } }]);
const batchStart = queries.length;
expect(await deliver(run.id, issueId)).toMatchObject([{ payload: { runId: run.id, issueId, status: "running" } }]);
expect(queries.length - batchStart).toBeLessThanOrEqual(6);
expect(queries.slice(batchStart).filter(query => query.startsWith("select ") && query.includes('from "issues"')).length).toBeLessThanOrEqual(2);
expect(await deliver(hiddenRun.id, issueId)).toEqual([]);
expect(await deliver(run.id, hiddenIssueId)).toEqual([]);
expect(await deliver(run.id, randomUUID())).toEqual([]);
failNextIssueRead = true;
expect(await deliver(run.id, issueId)).toEqual([]);
expect(await deliver(run.id, issueId)).toHaveLength(1);
await db.update(issueAccessGrants).set({ revokedAt: new Date() }).where(eq(issueAccessGrants.id, grant.id));
expect(await deliver(run.id, issueId)).toEqual([]);
expect(await deliver(run.id)).toEqual([]);
expect(await deliver(maintenance.id)).toHaveLength(1);
expect(await deliver(foreign.id)).toEqual([]);
@@ -101,7 +126,7 @@ describe("live-event run read boundaries", () => {
await db.delete(issues).where(eq(issues.id, issueId));
expect(await deliver(run.id)).toEqual([]);
expect(reads).toHaveLength(9);
expect(reads).toHaveLength(deliveries);
for (const query of reads) {
expect(query).toMatch(/^select "company_id", "scope_kind", "issue_id" from "heartbeat_runs"/);
expect(query).not.toMatch(/context_snapshot|runner_profile_json|stdout_excerpt|result_json/);
@@ -14,7 +14,10 @@ vi.mock("../realtime/task-toast-eligibility.js", () => ({ taskToastEligible: vi.
vi.mock("../services/authorization.js", () => ({
authorizationService: () => ({ decide: async ({ resource }: { resource: { issueId: string } }) => ({ allowed: resource.issueId !== "hidden" }) }),
}));
vi.mock("../services/heartbeat-run-privacy.js", () => ({ canActorReadHeartbeatRun: async () => false }));
vi.mock("../services/heartbeat-run-privacy.js", () => ({
getHeartbeatRunReadDecisions: async (_db: unknown, _access: unknown, _actor: unknown, runs: { issueId?: string }[]) =>
runs.map(run => run.issueId !== "hidden"),
}));
vi.mock("../services/case-access.js", () => ({
canActorReadCase: async (_db: unknown, _actor: unknown, record: { id: string }) => record.id !== "hidden-case",
}));
@@ -37,4 +37,15 @@ describe("native harness backup expiration config parsing", () => {
vi.stubEnv("PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_DAYS", "0");
expect(loadConfig()).toMatchObject({ nativeHarnessBackupExpirationDays: 1 });
});
it("keeps native runner evidence for seven days by default and at least one day", () => {
vi.stubEnv("PAPERCLIP_NATIVE_RUNNER_EVIDENCE_RETENTION_DAYS", undefined);
expect(loadConfig()).toMatchObject({ nativeRunnerEvidenceRetentionDays: 7 });
vi.stubEnv("PAPERCLIP_NATIVE_RUNNER_EVIDENCE_RETENTION_DAYS", "14");
expect(loadConfig()).toMatchObject({ nativeRunnerEvidenceRetentionDays: 14 });
vi.stubEnv("PAPERCLIP_NATIVE_RUNNER_EVIDENCE_RETENTION_DAYS", "0");
expect(loadConfig()).toMatchObject({ nativeRunnerEvidenceRetentionDays: 1 });
vi.stubEnv("PAPERCLIP_NATIVE_RUNNER_EVIDENCE_RETENTION_DAYS", "many");
expect(loadConfig()).toMatchObject({ nativeRunnerEvidenceRetentionDays: 7 });
});
});
@@ -301,4 +301,378 @@ describeEmbeddedPostgres("native harness backup expiration", () => {
expect(await exists(leftover)).toBe(false);
expect(await exists(session.current)).toBe(true);
});
async function seedArchive(root: string, input: { stampedDaysAgo?: number; mtimeDaysAgo?: number; name?: string } = {}) {
const stampedAt = now.getTime() - (input.stampedDaysAgo ?? 10) * DAY_MS;
const archive = path.join(root, "continuity-breaks", input.name ?? `${stampedAt}-${randomUUID()}`);
await fs.mkdir(path.join(archive, "codex-home"), { recursive: true });
await fs.writeFile(path.join(archive, "codex-home", "history.jsonl"), "x".repeat(64 * 1024));
const mtime = new Date(now.getTime() - (input.mtimeDaysAgo ?? input.stampedDaysAgo ?? 10) * DAY_MS);
await fs.utimes(archive, mtime, mtime);
return archive;
}
async function seedQuarantine(
scope: string,
input: { stampedDaysAgo?: number; name?: string } = {},
) {
if (!stateBase) stateBase = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-durable-sessions-"));
const stampedAt = now.getTime() - (input.stampedDaysAgo ?? 10) * DAY_MS;
const entry = path.join(
stateBase,
"quarantine",
input.name ?? `${scope}.identity_indeterminate.${stampedAt}.4242.${randomUUID()}`,
);
await fs.mkdir(path.join(entry, "control-plane"), { recursive: true });
await fs.writeFile(path.join(entry, "history.jsonl"), "x".repeat(64 * 1024));
return entry;
}
function evidenceService(overrides: Parameters<typeof nativeHarnessBackupExpirationService>[1] = {}) {
return nativeHarnessBackupExpirationService(db, {
stateBase,
now,
evidenceRetentionDays: 7,
isSessionBusy: () => false,
...overrides,
});
}
async function emptyStateBase() {
if (!stateBase) stateBase = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-durable-sessions-"));
return createHash("sha256").update(randomUUID()).digest("hex");
}
it("removes continuity archives older than the retention window and keeps recent ones", async () => {
const session = await seedSession({ issueStatus: "in_progress" });
const old = await seedArchive(session.root, { stampedDaysAgo: 9 });
const recent = await seedArchive(session.root, { stampedDaysAgo: 2 });
const result = await evidenceService().tick();
expect(result).toMatchObject({ evidenceCandidates: 1, evidenceRemoved: 1, failed: 0 });
expect(result.evidenceReclaimedBytes).toBeGreaterThan(0);
expect(result.skipped.recent_evidence).toBe(1);
expect(await exists(old)).toBe(false);
expect(await exists(recent)).toBe(true);
expect(await exists(session.current)).toBe(true);
});
it("removes the continuity-breaks folder once its archives are gone", async () => {
const session = await seedSession({ issueStatus: "in_progress" });
await seedArchive(session.root);
expect(await evidenceService().tick()).toMatchObject({ evidenceRemoved: 1 });
expect(await exists(path.join(session.root, "continuity-breaks"))).toBe(false);
});
it("ages a continuity archive by its name and falls back to mtime for a future stamp", async () => {
const session = await seedSession({ issueStatus: "in_progress" });
const stampedRecently = await seedArchive(session.root, { stampedDaysAgo: 1, mtimeDaysAgo: 30 });
const futureStamp = await seedArchive(session.root, {
name: `${now.getTime() + DAY_MS}-${randomUUID()}`,
mtimeDaysAgo: 30,
});
const futureRecent = await seedArchive(session.root, {
name: `${now.getTime() + 2 * DAY_MS}-${randomUUID()}`,
mtimeDaysAgo: 1,
});
expect(await evidenceService().tick()).toMatchObject({ evidenceRemoved: 1 });
expect(await exists(stampedRecently)).toBe(true);
expect(await exists(futureStamp)).toBe(false);
expect(await exists(futureRecent)).toBe(true);
});
it("ignores folders that are not continuity archives", async () => {
const session = await seedSession({ issueStatus: "in_progress" });
const other = await seedArchive(session.root, { name: "notes", stampedDaysAgo: 30 });
expect(await evidenceService().tick()).toMatchObject({ evidenceCandidates: 0 });
expect(await exists(other)).toBe(true);
});
it("keeps the evidence of a session that is executing in this process", async () => {
const session = await seedSession({ issueStatus: "in_progress" });
const archive = await seedArchive(session.root);
const quarantine = await seedQuarantine(session.scope);
const result = await evidenceService({ isSessionBusy: (scope) => scope === session.scope }).tick();
expect(result).toMatchObject({ evidenceCandidates: 0, evidenceRemoved: 0 });
expect(result.skipped.session_active).toBe(3);
expect(await exists(archive)).toBe(true);
expect(await exists(quarantine)).toBe(true);
});
it("keeps evidence when its session starts while the removal is being prepared", async () => {
const scope = await emptyStateBase();
const root = path.join(stateBase, scope);
const archive = await seedArchive(root);
let checks = 0;
const result = await evidenceService({ isSessionBusy: () => ++checks > 1 }).tick();
expect(result).toMatchObject({ evidenceCandidates: 1, evidenceRemoved: 0 });
expect(result.skipped.session_active).toBe(1);
expect(await exists(archive)).toBe(true);
});
it("reports evidence without deleting it in dry run", async () => {
const session = await seedSession({ issueStatus: "in_progress" });
const archive = await seedArchive(session.root);
const quarantine = await seedQuarantine(session.scope);
const leftover = path.join(session.root, "continuity-breaks", `.expired-${randomUUID()}`);
await fs.mkdir(leftover);
const result = await evidenceService({ dryRun: true }).tick();
expect(result).toMatchObject({ evidenceCandidates: 2, evidenceRemoved: 0, evidenceReclaimedBytes: 0 });
expect(result.evidenceCandidateBytes).toBeGreaterThan(0);
expect(await exists(archive)).toBe(true);
expect(await exists(quarantine)).toBe(true);
expect(await exists(leftover)).toBe(true);
});
it("skips symlinked evidence without following it", async () => {
const session = await seedSession({ issueStatus: "in_progress" });
const outside = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-evidence-outside-"));
try {
await fs.writeFile(path.join(outside, "keep.txt"), "keep");
const stampedAt = now.getTime() - 10 * DAY_MS;
await fs.mkdir(path.join(session.root, "continuity-breaks"), { recursive: true });
const archiveLink = path.join(session.root, "continuity-breaks", `${stampedAt}-${randomUUID()}`);
await fs.symlink(outside, archiveLink);
await fs.mkdir(path.join(stateBase, "quarantine"), { recursive: true });
const quarantineLink = path.join(
stateBase,
"quarantine",
`${session.scope}.identity_indeterminate.${stampedAt}.4242.${randomUUID()}`,
);
await fs.symlink(outside, quarantineLink);
const result = await evidenceService().tick();
expect(result).toMatchObject({ evidenceCandidates: 0, evidenceRemoved: 0 });
expect(result.skipped.unsafe_path).toBe(2);
expect(await exists(archiveLink)).toBe(true);
expect(await exists(quarantineLink)).toBe(true);
expect(await exists(path.join(outside, "keep.txt"))).toBe(true);
} finally {
await fs.rm(outside, { recursive: true, force: true });
}
});
it("skips symlinked evidence folders without following them", async () => {
const session = await seedSession({ issueStatus: "in_progress" });
const outside = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-evidence-outside-"));
try {
const stampedAt = now.getTime() - 10 * DAY_MS;
const inside = path.join(outside, `${stampedAt}-${randomUUID()}`);
await fs.mkdir(inside);
await fs.utimes(inside, new Date(stampedAt), new Date(stampedAt));
await fs.symlink(outside, path.join(session.root, "continuity-breaks"));
await fs.symlink(outside, path.join(stateBase, "quarantine"));
const result = await evidenceService().tick();
expect(result).toMatchObject({ evidenceCandidates: 0, evidenceRemoved: 0 });
expect(result.skipped.unsafe_path).toBe(2);
expect(await exists(inside)).toBe(true);
} finally {
await fs.rm(outside, { recursive: true, force: true });
}
});
it("removes quarantine entries older than the retention window and keeps recent ones", async () => {
const scope = await emptyStateBase();
const old = await seedQuarantine(scope, { stampedDaysAgo: 9 });
const recent = await seedQuarantine(scope, { stampedDaysAgo: 2 });
const result = await evidenceService().tick();
expect(result).toMatchObject({ evidenceCandidates: 1, evidenceRemoved: 1, failed: 0 });
expect(result.skipped.recent_evidence).toBe(1);
expect(await exists(old)).toBe(false);
expect(await exists(recent)).toBe(true);
expect(await fs.readdir(path.join(stateBase, "quarantine"))).toEqual([path.basename(recent)]);
});
it("keeps cleanup source archives in quarantine", async () => {
const scope = await emptyStateBase();
const entry = await seedQuarantine(scope, { name: `${scope}.identity_indeterminate.cleanup.${randomUUID()}` });
const old = new Date(now.getTime() - 30 * DAY_MS);
await fs.utimes(entry, old, old);
const result = await evidenceService().tick();
expect(result.evidenceCandidates).toBe(0);
expect(result.skipped.cleanup_archive).toBe(1);
expect(await exists(entry)).toBe(true);
});
it("finishes removing evidence left over from an interrupted removal", async () => {
const session = await seedSession({ issueStatus: "in_progress" });
const archive = await seedArchive(session.root, { stampedDaysAgo: 1 });
const leftover = path.join(session.root, "continuity-breaks", `.expired-${randomUUID()}`);
await fs.mkdir(path.join(leftover, "codex-home"), { recursive: true });
const quarantineLeftover = path.join(stateBase, "quarantine", `.expired-${randomUUID()}`);
await fs.mkdir(quarantineLeftover, { recursive: true });
await evidenceService().tick();
expect(await exists(leftover)).toBe(false);
expect(await exists(quarantineLeftover)).toBe(false);
expect(await exists(archive)).toBe(true);
});
it.each([false, true])("pages evidence removals across ticks (dry run %s)", async (dryRun) => {
const session = await seedSession({ issueStatus: "in_progress" });
const archives = [await seedArchive(session.root), await seedArchive(session.root), await seedArchive(session.root)];
const quarantine = await seedQuarantine(session.scope);
const retention = evidenceService({ dryRun, maxRemovals: 2 });
const results = [];
for (let i = 0; i < 4; i++) {
const result = await retention.tick();
results.push(result);
if (!result.truncated) break;
}
expect(results.map((result) => result.evidenceCandidates)).toEqual([2, 2]);
expect(results.map((result) => result.truncated)).toEqual([true, false]);
for (const archive of archives) expect(await exists(archive)).toBe(dryRun);
expect(await exists(quarantine)).toBe(dryRun);
});
it("does not recount a session's backup when its archives resume on the next tick", async () => {
const session = await seedSession();
await seedArchive(session.root);
await seedArchive(session.root);
const retention = evidenceService({ dryRun: true, maxRemovals: 2 });
const first = await retention.tick();
const second = await retention.tick();
expect(first).toMatchObject({ candidates: 1, evidenceCandidates: 1, truncated: true });
expect(second).toMatchObject({ candidates: 0, evidenceCandidates: 1, truncated: false });
});
it("removes old quarantine entries even when their session's issue is still open", async () => {
const session = await seedSession({ issueStatus: "in_progress", backupAgeDays: 1 });
const entry = await seedQuarantine(session.scope);
await fs.cp(path.join(session.root, "control-plane"), path.join(entry, "control-plane"), { recursive: true });
expect(await evidenceService().tick()).toMatchObject({ evidenceRemoved: 1 });
expect(await exists(entry)).toBe(false);
});
it("stays quiet until retained evidence is due and wakes when new evidence is written", async () => {
const session = await seedSession({ issueStatus: "in_progress", backupAgeDays: 1 });
const recent = await seedArchive(session.root, { stampedDaysAgo: 2 });
const retention = evidenceService();
const at = (days: number) => ({ now: new Date(now.getTime() + days * DAY_MS) });
expect((await retention.tick()).skipped.recent_evidence).toBe(1);
expect((await retention.tick(at(4))).skipped.recent_evidence).toBeUndefined();
expect(await retention.tick(at(6))).toMatchObject({ evidenceRemoved: 1 });
expect(await exists(recent)).toBe(false);
const written = await seedArchive(session.root, { stampedDaysAgo: 0 });
expect(await retention.tick(at(8))).toMatchObject({ evidenceCandidates: 0 });
retention.noteEvidence(now.getTime());
expect(await retention.tick(at(8))).toMatchObject({ evidenceRemoved: 1 });
expect(await exists(written)).toBe(false);
});
it("retries evidence of a busy session on the next tick", async () => {
const session = await seedSession({ issueStatus: "in_progress", backupAgeDays: 1 });
const archive = await seedArchive(session.root);
let busy = true;
const retention = evidenceService({ isSessionBusy: () => busy });
expect(await retention.tick()).toMatchObject({ evidenceRemoved: 0 });
busy = false;
expect(await retention.tick()).toMatchObject({ evidenceRemoved: 1 });
expect(await exists(archive)).toBe(false);
});
it("reports dry-run evidence once until new evidence is written", async () => {
const session = await seedSession({ issueStatus: "in_progress", backupAgeDays: 1 });
await seedArchive(session.root);
const retention = evidenceService({ dryRun: true });
expect(await retention.tick()).toMatchObject({ evidenceCandidates: 1 });
expect(await retention.tick()).toMatchObject({ evidenceCandidates: 0 });
retention.noteEvidence(now.getTime());
expect(await retention.tick({ now: new Date(now.getTime() + 7 * DAY_MS) })).toMatchObject({ evidenceCandidates: 1 });
});
it("counts inspected archives and leftovers against the entry limit", async () => {
const session = await seedSession({ issueStatus: "in_progress", backupAgeDays: 1 });
const archives = [
await seedArchive(session.root, { stampedDaysAgo: 1 }),
await seedArchive(session.root, { stampedDaysAgo: 2 }),
await seedArchive(session.root, { stampedDaysAgo: 3 }),
];
const leftovers = [randomUUID(), randomUUID()].map((id) => path.join(session.root, "continuity-breaks", `.expired-${id}`));
for (const leftover of leftovers) await fs.mkdir(leftover);
const retention = evidenceService({ maxEntries: 2 });
const first = await retention.tick();
expect(first).toMatchObject({ checked: 2, truncated: true });
expect(first.skipped.recent_evidence).toBeUndefined();
expect((await fs.readdir(path.join(session.root, "continuity-breaks"))).filter((name) => name.startsWith(".expired-"))).toHaveLength(1);
const results = [first];
for (let i = 0; i < 6 && results.at(-1)!.truncated; i++) results.push(await retention.tick());
expect(results.at(-1)!.truncated).toBe(false);
expect(results.every((result) => result.checked <= 2)).toBe(true);
expect(results.every((result) => (result.skipped.recent_evidence ?? 0) <= 2)).toBe(true);
expect(results.reduce((sum, result) => sum + (result.skipped.recent_evidence ?? 0), 0)).toBe(3);
for (const leftover of leftovers) expect(await exists(leftover)).toBe(false);
for (const archive of archives) expect(await exists(archive)).toBe(true);
});
it("stops removing leftovers once the sweep must stop", async () => {
const session = await seedSession({ issueStatus: "in_progress", backupAgeDays: 1 });
await fs.mkdir(path.join(session.root, "continuity-breaks"));
const leftover = path.join(session.root, "continuity-breaks", `.expired-${randomUUID()}`);
await fs.mkdir(leftover);
let calls = 0;
const retention = evidenceService({ shouldContinue: () => ++calls !== 2 });
await retention.tick();
expect(await exists(leftover)).toBe(true);
await retention.tick();
expect(await exists(leftover)).toBe(false);
});
it("keeps a due evidence pass pending when drain stops it before any session", async () => {
const session = await seedSession({ issueStatus: "in_progress", backupAgeDays: 1 });
const archive = await seedArchive(session.root);
let draining = true;
const retention = evidenceService({ shouldContinue: () => !draining });
expect(await retention.tick()).toMatchObject({ checked: 0, truncated: true });
draining = false;
expect(await retention.tick()).toMatchObject({ evidenceRemoved: 1 });
expect(await exists(archive)).toBe(false);
});
it("finishes one-entry batches without recounting a resumed session", async () => {
const session = await seedSession({ issueStatus: "in_progress", backupAgeDays: 1 });
const archives = [await seedArchive(session.root), await seedArchive(session.root)];
const quarantine = await seedQuarantine(session.scope);
const retention = evidenceService({ maxEntries: 1 });
const results = [];
for (let i = 0; i < 8; i++) {
results.push(await retention.tick());
if (!results.at(-1)!.truncated) break;
}
expect(results.at(-1)!.truncated).toBe(false);
for (const archive of archives) expect(await exists(archive)).toBe(false);
expect(await exists(quarantine)).toBe(false);
});
});
@@ -1,5 +1,6 @@
import { EMBEDDED_POSTGRES_TEST_TIMEOUT_MS } from "@paperclipai/db";
import { randomUUID } from "node:crypto";
import { randomBytes, randomUUID } from "node:crypto";
import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture";
import { spawn } from "node:child_process";
import os from "node:os";
import { and, eq } from "drizzle-orm";
@@ -207,6 +208,40 @@ suite("settled native run process identity", () => {
expect((await readRun(runId)).processPid).toBeNull();
});
it("releases the stopped identity without reading saved execution payloads", async () => {
const { runId } = await seed();
const bulk = randomBytes(128 * 1024).toString("base64");
const before = await readRun(runId);
await db.update(heartbeatRuns).set({ contextSnapshot: { ...before.contextSnapshot, savedTask: bulk },
runnerProfileJson: { ...before.runnerProfileJson, sessionCheckpoint: { providerSessionId: "thread", savedState: bulk },
nativeExecutionInput: { task: bulk } }, resultJson: { savedResult: bulk }, stdoutExcerpt: bulk,
}).where(eq(heartbeatRuns.id, runId));
const saved = await readRun(runId);
const events: Array<Record<string, unknown>> = [];
const now = Date.now();
const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event));
installDatabaseResponseCapture(db, { heartbeatRuns }, capture);
try { expect(await releaseSettledNativeRunProcess(db, runId)).toBe("released"); }
finally { capture.flush(); }
const summary = events.find(event => event.event === "paperclip.db_response_capture.summary.v1")!;
expect(summary.responseValueBytes).toBeLessThan(8192);
const after = await readRun(runId);
expect(after).toEqual({ ...saved, processPid: null, processGroupId: null, processStartedAt: null,
nextEventSeq: saved.nextEventSeq + 1, updatedAt: after.updatedAt });
expect(await hasNativeLocalProcessStop(db, companyId, runId)).toBe(true);
expect(JSON.stringify(events)).not.toContain(bulk.slice(0, 64));
});
it.each([null, false, 0, "", {}, []].map(checkpoint => ({ checkpoint })))(
"preserves checkpoint presence semantics when releasing an identity ($checkpoint)", async ({ checkpoint }) => {
const { runId, pid } = await seed({ checkpoint: false });
await db.update(heartbeatRuns).set({ runnerProfileJson: { sessionCheckpoint: checkpoint } })
.where(eq(heartbeatRuns.id, runId));
expect(await releaseSettledNativeRunProcess(db, runId)).toBe(checkpoint === null ? "skipped" : "released");
expect((await readRun(runId)).processPid).toBe(checkpoint === null ? pid : null);
},
);
it("keeps the identity while the process is still alive, then releases it after exit", async () => {
const child = startDetached();
children.push(child);
@@ -14,7 +14,7 @@ import { realizeExecutionWorkspace } from "../services/workspace-runtime.js";
import { executionWorkspaceService } from "../services/execution-workspaces.js";
import { prepareProjectRepositoryWorkspaces } from "../services/heartbeat/workspaces.js";
import { expireReconstructableGitWorkspace } from "../services/git-workspace-expiration.js";
import { currentRestorePointSnapshot } from "../services/git-restore-point.js";
import { currentRestorePointSnapshot, restorePointHistoryDigest } from "../services/git-restore-point.js";
import { nestedRepositoryBaseCandidates, nestedWorktreeBranchPrefix, readLinkedNestedCheckout, withNestedRepositoryBaseLock } from "../services/nested-repository-worktrees.js";
import { listNestedCloneMigrationNames, migrateNestedRepositoryClone, nextNestedCloneMigrationBatch, type NestedCloneMigrationOptions } from "../services/nested-clone-migration.js";
import { getStorageService } from "../storage/index.js";
@@ -225,6 +225,41 @@ describeEmbeddedPostgres("nested repository clone migration", () => {
expect(git(f.checkout, "cat-file", "-t", feature)).toBe("commit");
});
it("expires a converted checkout whose only extra history is remote-tracking and restores that history", async () => {
const f = await fixture({ issueStatus: "done" });
git(f.nested.author, "checkout", "-q", "-b", "squashed-away");
await fs.writeFile(path.join(f.nested.author, "feature.txt"), "feature\n");
git(f.nested.author, "add", "."); git(f.nested.author, "commit", "-qm", "Feature");
const feature = git(f.nested.author, "rev-parse", "HEAD");
git(f.nested.author, "push", "-q", f.nested.remote, "squashed-away");
git(f.checkout, "fetch", "-q", "origin");
git(f.nested.remote, "branch", "-D", "squashed-away");
git(f.nested.remote, "reflog", "expire", "--expire=now", "--all");
git(f.nested.remote, "gc", "-q", "--prune=now");
expect(await f.migrate()).toMatchObject({ outcome: "migrated" });
expect(await f.expire()).toMatchObject({ expired: true, restorePoint: true, skipReason: null });
for (const line of ownedRefs(f.base, f.prefix).split("\n").filter(Boolean)) git(f.base, "update-ref", "-d", line.split(" ")[0]!);
git(f.base, "reflog", "expire", "--expire=now", "--all");
git(f.base, "gc", "-q", "--prune=now");
expect(() => git(f.base, "cat-file", "-e", `${feature}^{commit}`)).toThrow();
expect(await f.reopen()).toMatchObject({ ok: true, reopened: true });
expect(git(f.checkout, "cat-file", "-t", feature)).toBe("commit");
});
it("fingerprints a converted checkout's own history rather than the shared clone's remote refs", async () => {
const f = await fixture();
expect(await f.migrate()).toMatchObject({ outcome: "migrated" });
const head = git(f.checkout, "rev-parse", "HEAD");
const before = await restorePointHistoryDigest(f.checkout);
git(f.base, "update-ref", "refs/remotes/origin/another-task", head);
expect(await restorePointHistoryDigest(f.checkout)).toBe(before);
git(f.base, "update-ref", `refs/heads/${f.prefix}remotes/origin/extra`, head);
expect(await restorePointHistoryDigest(f.checkout)).not.toBe(before);
for (let index = 0; index < 64; index++) git(f.base, "update-ref", `refs/heads/${f.prefix}remotes/origin/bulk-${index}`, head);
await expect(restorePointHistoryDigest(f.checkout)).rejects.toThrow("managed_checkout_local_tip_budget");
});
it("finishes an interrupted conversion after the workspace is archived and forgets one whose workspace is gone", async () => {
const f = await fixture();
const { unpushed } = await nestedWork(f.checkout);
Loaded 100 of 315 files, more files were not shown because too many files have changed in this diff. Show more