diff --git a/.dockerignore b/.dockerignore index 83ef4daeb0d..3bf8e79b7b1 100644 --- a/.dockerignore +++ b/.dockerignore @@ -88,6 +88,7 @@ packages/grok-acp/bin !packages/paperclip-runner/test-fixtures/pi-acp/profile-v27-identity.json !packages/paperclip-runner/test-fixtures/pi-acp/profile-v28-identity.json +!packages/paperclip-runner/test-fixtures/pi-acp/profile-v29-identity.json !packages/paperclip-runner/test/fixtures/copilot-profile-v23-identity.json diff --git a/DESIGN.md b/DESIGN.md index a9baddf65b4..a4804cb0e27 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -63,13 +63,16 @@ reversals and Safari's edge bounce. It glides out and eases back in with shared motion tokens; the task composer follows the same motion. Keep page padding stable while the navigation moves, and honor reduced-motion preferences. -Task conversations reveal their initial comments, interaction cards, plan, and -relevant run history together after positioning the latest message. Keep the -mobile loading surface at a stable viewport height while that history loads; -concealed content must not stretch the document. Background refreshes keep an -already revealed conversation and composer mounted and visible. -Bound the initial wait to 15 seconds. If a request stalls, reveal the available -conversation and composer with a notice that some history is still loading. +Cold task opens reveal the instruction and recent conversation independently of +plans, related metadata, and historical run activity. Show the latest human +follow-up, recent saved replies, and text or pending requests from the latest run. +Put older messages and diagnostic activity behind **More messages**. Fetch those +only when expanded; never download all history just to calculate a count. +Retain messages and runs observed while the task stays open, preserve expansion +on return within the session, and load linked older comments when requested. +Background refreshes and history expansion keep the conversation and composer +mounted. A failed history request shows an inline retry without hiding readable +content. Full-history and classic views retain the bounded 15-second initial wait. Only the viewer's own human comments sit on the right in blue bubbles. Agents and other humans sit on the left on the page surface, with their own name and diff --git a/doc/DATABASE.md b/doc/DATABASE.md index 1ff88e3368d..2032268da2d 100644 --- a/doc/DATABASE.md +++ b/doc/DATABASE.md @@ -49,11 +49,42 @@ and leaves all existing cases passive. Source identities are not grants. ### Task query performance +Agent request authentication uses a compact run-identity read. It omits saved +execution payloads, results, and log excerpts from the query response. The read +still locks the task before the run and reconciles pending steering from durable +acknowledgements on every request. Default identity reads retain the full run. +Stored payloads and separate native tool authorization checks remain unchanged. + +Native event acknowledgements read a count and maximum through the existing +source-sequence index. A stream with gaps uses indexed lookups for its first +missing successor. The acknowledgement keeps its contiguous-prefix rule without +returning every historical sequence number. All events stay stored. Counting the +index entries still grows with the stream length. + +Conversation recovery reads only the saved run fields used to settle a turn. +Settlement locks and reads current conversation state without loading the task +description or comment bodies. It still checks the durable reply or pending +interaction, session generation, owning run, and newer user messages. Saved +execution payloads, task text, and comments remain stored in full. + +The attention feed checks for newer runs with company-scoped existence probes. +The probes match the failed run's agent and issue or task context. They preserve +the existing millisecond comparison and empty-context rules. Each batch holds +at most 1,000 failed runs. Retained run history stays in the database; this check +returns only the identifiers of failures that have a newer matching run. +For failures without a task key, it reads the latest keyless timestamp once per +agent. Unique agents are batched separately, so repeated failures do not repeat +the same history scan. Both paths retain the current millisecond comparison. + The issue-lock adapter's release read omits `result_json.stdout` and -`result_json.stderr` before transferring a run to the controller. Every other result key, context field, and -stop, retry, lease, and workspace-recovery check stays intact. Full output remains -stored for execution history and detail reads. Terminal telemetry loads only its -run identity, timestamps, usage, and typed task identifier. +`result_json.stderr` before transferring a run to the controller. Native context +also omits saved launch text, execution-continuation bodies, workspace and +environment payloads, and the launch wake envelope. Task identity, retry and +continuation controls, interaction and comment references, execution policy, unknown context fields, and every other +result key stay intact. Legacy reads retain full context. Stop, retry, lease, +and workspace-recovery checks do not change. Full inputs and output remain stored +for execution history and detail reads. Terminal telemetry loads only its run +identity, timestamps, usage, and typed task identifier. Recovery's latest-run queries return the context fields used by recovery decisions. Prompts, execution continuation bodies, and workspace setup payloads @@ -62,11 +93,48 @@ the PostgreSQL row version. Native blocked-wait repair compares that version against a fresh full source read before applying changes. Any intervening row update, including an edit outside the projection, defers repair to a fresh sweep. +The two preliminary reconciliation reads omit named launch renderings and +preparation payloads. Native profiles omit unused native execution input. +They retain all other context, full results, legacy profiles, task identity, +retry accounting, and unknown controls. Native runs still reach their native +recovery guard. Terminalization locks its current control state and returns +full rows from its writes. Stored input remains intact. + Execution-blocker reads select process, lifecycle, and task-binding fields rather than full execution rows. Context retains typed task and pre-dispatch evidence; -Runner profiles retain the adapter type and complete native execution input and -workspace-sync receipts. Full result evidence stays available to the existing -retry and workspace checks. Stored context, profiles, and output are unchanged. +Runner profiles retain the adapter type, native input schema, binding, provider, +workspace-access evidence, and complete workspace-sync receipts. Native task and +runtime prompt payloads are not needed for these checks. Missing fields and +malformed input values retain their original shape. Full result evidence stays +available to the existing retry and workspace checks. Stored context, profiles, +and output are unchanged. + +Committed native finalization reads run identity and ownership state before it +claims the coordinator. It loads complete saved input only when finalization +still needs to process an accepted result. Review presentation retains the full +result. Muse settlement reads its exact provider binding. Cleanup reads fresh +status and ownership metadata; cancelled native workspace retention and legacy +Stop settlement still load complete run evidence. Workspace settlement reads the +saved sync descriptor, export retry, and first-heartbeat flag. These projections +leave stored prompts, logs, results, and recovery state unchanged. + +Settled native process release locks run identity, terminal state, process +identity, and checkpoint presence without loading the checkpoint or execution +payloads. It retains current coordinator checks, provider evidence, local-host +identity checks, and every lease's termination receipt. The stop receipt and +process-field update remain in the same transaction. + +Legacy terminalization locks runtime mode, status, full result evidence, and the +status-delivery token without loading unused input, profile, or log bodies. +It keeps the existing lock, full returned run, workspace-retention +receipts, and source-bound recovery holds. Stored payloads remain unchanged. + +Conversation ownership checks select process identity and full recovery result +evidence without execution prompts or profiles. Committed native reply repair +selects its persisted result and identity bindings plus typed presentation flags. +It keeps its existing row locks, accepted-result digest checks, publication +authorization, and selected-answer protections. Full execution history stays in +storage. Run-context lookups use expression indexes for both `context_snapshot.issueId` and `context_snapshot.paperclipIssue.id`. Task creation provenance has its own @@ -107,11 +175,77 @@ launch context does not travel with each policy or parent-report check. JSON types, task-key precedence, and company and agent identity checks are preserved. Each decision still reads current data; this is not an authorization cache. +Restoring a saved monitor's receipt or completion proof creates a new native +assessment identity even when its scheduled deadline is unchanged. The saved +proof is part of the assessment input. Prior decisions and accepted results stay +intact, and an unchanged replay reuses the same assessment and decision. + +Native tool authority lookups select current run identity, status, invocation, +responsible user, context, and receipt data. Saved profiles and output excerpts +stay in storage and are omitted from that read. Task and agent binding, current +status, budget, and task privacy checks retain their existing predicates. +Mutation transactions still lock and read complete run records. The unlocked +authority read also omits named launch renderings and preparation data that its +tools do not consume. Ordinary tool checks also omit the wake payload. Mail +checks load the current full wake to enforce external-chat restrictions. Other +context and result receipts remain available. The complete saved context stays +in storage for execution. + +WebSocket delivery uses one fresh authorization batch for an event's task and +run bindings. When both bindings name the same task, the batch reads its current +authority once. Different task bindings must both pass. The batch is discarded +after that delivery; later events still recheck grants, membership, task +existence, and company boundaries. + +Browser progress events update cached live run status directly. A successful +patch does not refetch task details, activity, or run history. Unknown runs +retain the existing refresh fallback. Run transitions, activity events, and +reconnect recovery still refresh their affected data. + +Task run queries stop interval polling only after a successful empty response +and an open shared live connection for the current company and session. Initial +reads, active work, errors, connection setup, and disconnections retain their +existing polling cadence. Company or session changes require a new connection. +Views without an assigned agent retain polling. Matching run lifecycle events +refresh the displayed task's run queries independently of route and window +focus. Events that omit task identity conservatively refresh these queries, +including reviewer runs from another agent. One parent view owns both query +observers and reuses refreshes already covered by the focused-route handler. A +matching event that overlaps an older request queues a fresh read after that +request settles. Connection readiness also reconciles the displayed run +queries after any older request. Follow-up reads belong to the current mounted +scope and stop when it changes or unmounts. The first open and reconnect still +reconcile missed events. + +Bundled skill release checks read the current stored version ID and a SHA-256 +fingerprint of its file inventory. A per-database memo keeps at most 64 verified +snapshots. It reuses a verification only when the current database fingerprint +and current on-disk inventory hash both match. A changed or missing snapshot +still follows full validation or seeding. Company access and local source +reconciliation still run for each request; authorization is not cached. + +Connection-assignment checks select profile bindings for the current company or +agent and read rules that can match the selected connection's catalog rows. +All five selector kinds, null application matching, exclusions, and profile +precedence retain their existing evaluator. Rule filtering uses the catalog +values already read by that check, so a later catalog change cannot drop an +exclusion from the decision being evaluated. There is no cross-call cache. + Filesystem preparation uses current task/run and selected-configuration checks. Historical root-reader authorization has been retired; no retained-reader batch is loaded at admission or publication. Workspace-record disclosure still uses task privacy checks, independently of filesystem access. +Saved-message execution-wait scans read current run state, process ownership, +startup and cancellation receipts, task binding, and the recorded adapter kind. +Saved profiles, output excerpts, and unrelated launch context stay in storage. +Continuation admission and its locked mutation reads retain their existing gates. + +Task creation provenance reads current run identity, scope, and task binding. +Legacy task references retain their typed values and existing precedence. Saved +profiles, output, and unrelated context stay in storage. Viewer access to the +source task and run is still checked before provenance is returned. + The initial agent JWT run lookup reads only the cancellation state and conversation flag from saved run JSON. It does not transfer the full context or result. Run status, responsible-user identity, company and agent boundaries, @@ -176,6 +310,32 @@ The benchmark loads the baseline authorization module with current dependencies, compares complete workspace rows, and prints five warm samples and query plans. It measures SQL execution, not the full repository admission or agent startup. +Native finalization retries preserve a run's terminal completion time when its +status is unchanged. A retry mints another status delivery only when the current +run's status, finalization metadata, phase, error fields, or missing completion +time changes. An unchanged projection retains its existing pending delivery ID, +including after a publish failure. Status delivery stays at least once; retry +deadlines, recovery, accepted results, and workspace cleanup still run normally. +Legacy recovery also retains the pending delivery when status is unchanged and +no patch was supplied. Explicit patches, new terminal statuses, and newly +retained workspace evidence still create a delivery. File retention, ownership +checks, failed write conditions, lock cleanup, and recovery holds still apply. +The retention-evidence write returns another full run only when that evidence +changes. An unchanged retry keeps the full run returned by the first write and +still processes retained leases and recovery holds. + +Agent lists read the complete company roster once and use that same snapshot for +termination filtering and organization-chain normalization. Detail and plugin runtime reads +load only identity, company, name, status, and manager from other company agents +for chain checks. The selected agent's full configuration, current monthly spend, +and existing response redaction remain available. Every call reads current data; +these lookups do not cache authorization or agent configuration. + +Queued-comment display filters wake requests by the current company, assigned +agent, status, and exact JSON string task identity in SQL. Other tasks' comment +identifiers stay in the database. The display projection, ordering, current run +ownership checks, and persisted wake payloads retain their existing contract. + ## 1. Embedded PostgreSQL — zero config If you don't set `DATABASE_URL`, the server automatically starts an embedded PostgreSQL instance and manages a local data directory. @@ -298,14 +458,14 @@ DATABASE_URL=postgres://postgres.[PROJECT-REF]:[PASSWORD]@aws-0-[REGION].pooler. DATABASE_MIGRATION_URL=postgres://postgres.[PROJECT-REF]:[PASSWORD]@aws-0-[REGION].pooler.supabase.com:5432/postgres ``` -If your hosted database requires transaction-pooling-only connections (pgbouncer transaction mode, Supavisor port 6543, Neon `-pooler` endpoints), set `DATABASE_PREPARED_STATEMENTS=false` so the client does not rely on session-scoped prepared statements, and keep `DATABASE_MIGRATION_URL` on a direct connection. Do not edit database client source files as part of deployment setup. +If your transaction-mode pooler does not support protocol-level named prepared statements, set `DATABASE_PREPARED_STATEMENTS=false`. If your pooler supports them, you can opt in with `DATABASE_PREPARED_STATEMENTS=true`. Check your provider's pooler configuration before you enable this setting. Keep `DATABASE_MIGRATION_URL` on a direct connection. Do not edit database client source files as part of deployment setup. ### Client tuning (optional) All of these are optional; when unset, the driver defaults apply and behavior is unchanged — typical self-hosted setups need none of them: ```sh -DATABASE_PREPARED_STATEMENTS=false # required for transaction-mode poolers; default: enabled +DATABASE_PREPARED_STATEMENTS=false # use false when the pooler does not support named prepared statements DATABASE_POOL_MAX=25 # connection pool size; default: 10 DATABASE_IDLE_TIMEOUT_SECONDS=60 # close idle pooled connections; default: 60 (0 = keep open) DATABASE_CONNECT_TIMEOUT_SECONDS=10 # default: 30 @@ -313,6 +473,70 @@ DATABASE_MAX_LIFETIME_SECONDS=1800 # recycle a pooled connection after this lo DATABASE_APPLICATION_NAME=paperclip # application_name in pg_stat_activity; default: paperclip ``` +An explicit `DATABASE_PREPARED_STATEMENTS=true` enables named prepared statements +for parameterized ORM queries, transactions, reserved connections, and dedicated +clients. Postgres.js manages the statement cache. This does not cache query +results. Bound values and returned data stay fresh. Explicit per-query options +still apply, including `prepare: false` and simple-protocol execution. With the +setting unset, ORM queries keep their existing unprepared behavior and tagged +SQL uses the driver default. With `false`, preparation stays disabled. + +### Bounded response diagnostics + +`PAPERCLIP_DB_RESPONSE_CAPTURE_WINDOW=/` enables one explicit UTC +window. Both timestamps use ISO format ending in `Z`. The window lasts at most +60 seconds and starts within 30 minutes. Unset, invalid, and expired windows +leave database construction unchanged. An expired window stays inactive after a +restart; remove the setting after the investigation. + +The diagnostic measures awaited Postgres.js `unsafe` responses before Drizzle +column decoding. It retains the Drizzle builder caller for normal, prepared, +transaction, and dedicated-pool execution. It issues no queries. +It logs the public server caller locations, static schema table, operation, +query/row counts, and response value sizes. It emits at most 2,048 aggregate +groups and a summary. Measurement stops at the deadline. Environment-enabled +captures emit at most 64 records per second and can take another 33 seconds to +finish writing the bounded output. Wait for the summary before collecting it. +It never logs SQL, parameters, result +values, connection strings, or query errors. In-flight queries continue normally. + +For a bundled server, keep the linked source maps and enable Node's +`--enable-source-maps` option so the diagnostic can resolve public caller +locations. Restore the prior Node options after capture. Sizes still count when +caller locations are unavailable. + +Look for `paperclip.db_response_capture.entry.v1` and +`paperclip.db_response_capture.summary.v1` in the process logs. A log provider +can store `event` as a structured field instead of message text. Filter by that +field or read the bounded window from the exact deployment. Check the summary +and group totals, including the unattributed overflow group. + +Fetch a complete capture as follows: + +1. Read the summary events from the exact deployment and window. Keep each + distinct `captureId` separately. +2. Each entry has a unique `groupIndex` within that capture and a `groupPage`. + The summary gives `groupPageSize` (128) and `groupPages`. Fetch every page + from zero through `groupPages - 1`, filtering on the entry event, `captureId`, + and `groupPage`. Request at least `groupPageSize` matching records per page. + A structured query can use + `@event:paperclip.db_response_capture.entry.v1 AND @captureId: AND @groupPage:0`. +3. If the provider's record limit is smaller than a page, fetch each + `groupIndex` separately with the same capture ID. Each index matches one + entry. Do not split by timestamp: entries can share a timestamp. +4. Deduplicate by `(captureId, groupIndex)`. Require every index from zero + through `groups - 1`, with no gaps. Entry count must equal `groups`, summed + `calls` must equal `completedQueries`, and summed `responseValueBytes` must + equal the summary total. A missing page is an incomplete measurement. + +Tagged raw-driver calls, streams, unawaited raw-driver +dispatch, and protocol overhead are not measured. Relational queries use a driver-level caller fallback. +Queries still in flight at the deadline are reported separately. Sizes describe +driver values before column decoding, not billed network bytes. Complex parsed +values use JSON size estimates, counted in `estimatedValues`. Reconcile them with +provider metering and socket measurements. Serialization adds temporary CPU and memory work; +measure the final transfer rate after the window expires. + ### Push the schema ```sh @@ -402,6 +626,13 @@ bindings, selectors, exclusions, and installed connections on every call. The public effective-profile response continues to include complete tool schemas and annotations. +Connection readiness checks use a scoped assignment reader. It applies the same +current profile precedence, selectors, and exclusions to the candidate connection +IDs. The reader selects rule match fields and catalog identities. It does not +load rule conditions, catalog schemas, profile summaries, or the company-agent +list. Installed connections and later identity and gateway checks remain fresh. +Full effective-profile and effective-tool responses keep their complete shape. + ## Execution identity row locks Identity initialization, credential acquisition, and steering reconciliation lock @@ -741,6 +972,19 @@ pnpm secrets:migrate-inline-env --apply Hosted AWS provider notes live in [SECRETS-AWS-PROVIDER.md](./SECRETS-AWS-PROVIDER.md). +Sandbox runtime configuration resolves a provider's secret-reference fields +through its schema-only lookup. Each lookup reads the selected plugin ID +and a SHA-256 hash of its current schema in one database statement. A matching +retained schema is omitted from the query result. Selection order, schema +changes, and deletion remain visible on the next lookup, without a time-based +cache. A database error still fails the lookup. + +The retained schemas are private to one `Db` object, with at most 32 entries of +64 KiB each in serialized form. Larger schemas still work and are read in full. +Callers receive independent schema objects. This reuse stores neither resolved +secrets nor authorization decisions. Default driver lookups still read complete +current plugin records and enforce their existing readiness and worker checks. + ### Persistent agent conversations Migration `0274_agent_chat.sql` adds conversation identity/state and session generation/boundary columns to `issues`, plus idempotent client request IDs and processed session-boundary generations to `issue_comments`. The company/agent/user unique index resolves concurrent first writes to one issue. A check constraint preserves the assigned-agent identity and prevents terminal conversation status. Comment request IDs are unique per issue and user. There is no separate chat/message store. Provider sessions continue to use `agent_task_sessions`; `/new` removes only the matching conversation session, and session writers fence stale generations against the issue row. diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index e399790fbbb..f8d408cdff9 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -907,6 +907,15 @@ can update an entry only after the current account's summary confirms access. The first healthy live subscription reconciles domain data without re-fetching session and health; reconnects and visibility recovery retain full reconciliation. +### Run history authorization reads + +Run history reuses the authorization service's batch read API for distinct task +bindings. Existence checks remain company-scoped and use batches of at most 500 +IDs. Missing or foreign tasks stay unreadable in every privacy mode. The batch +snapshot lasts for one request only; the next request checks current grants, +membership, and run restrictions again. Adapters that expose only `decide` +retain the individual-read path with duplicate bindings shared within the request. + ### Local Linux runner qualification Before serving requests, the server reads `getconf GNU_LIBC_VERSION` once with @@ -1807,11 +1816,21 @@ dispatch with cancellation and execution, and re-exports the existing public helpers and configuration-incomplete error class. Keep preparation policy changes in this module and its tests. +Queued-comment display reads select only wake identity, queue state, typed comment +and interaction bindings, the displayed wait reason, and fresh-session flags. +They read only the queued run's runtime mode. Mutation locks retain complete +wake and run rows, so edits, ordering, interruption, and saved execution inputs +keep their existing write contract. + Run retrieval and session state are in `server/src/services/heartbeat/run-state.ts`. It owns bounded run projections, database encoding checks, task session reads and writes, explicit resumes, session compaction, and usage/billing helpers. `createHeartbeatRunState(db)` binds these operations without doing database work during construction. The encoding-check cache belongs to each factory instance. +Queued-comment polling reads current issue control fields, the typed issue binding, +and the selected adapter type. Mutation and execution paths retain their full run reads. +Workspace-operation polling reads the current run access scope and typed workspace +binding, then applies the existing run and operation permission checks. `heartbeat.ts` keeps run execution and re-exports the existing public helpers. Keep session policy changes separate from run orchestration changes. @@ -2608,6 +2627,11 @@ Environment overrides: negative, empty, or non-numeric value falls back to the default. `PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_ENABLED=false` turns expiry off, and `PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_DRY_RUN=true` only reports candidates. +- `PAPERCLIP_NATIVE_RUNNER_EVIDENCE_RETENTION_DAYS=` sets how long native + session continuity-break archives and quarantined session state are kept (see + `doc/recoverable-disk-cleanup.md`). The default is `7` and the minimum is `1`. A + negative, empty, or non-numeric value falls back to the default. The backup + expiration enabled and dry-run switches apply to it as well. - `PAPERCLIP_NATIVE_WORKSPACE_SYNC_RETENTION_DAYS=` sets how long a settled run's native workspace sync folder (descriptor, manifest and seed archives under `/native-workspace-sync/`) is kept (see diff --git a/doc/SPEC-implementation.md b/doc/SPEC-implementation.md index 13067391ef2..948d3fd20c2 100644 --- a/doc/SPEC-implementation.md +++ b/doc/SPEC-implementation.md @@ -891,6 +891,21 @@ grant privileged tools. Backlog assignments retain the existing human requester without starting a run. Reassignment transactionally cancels prior human requests, including service/plugin writes; cancelled runs cannot authorize later retries. See `doc/LOW-TRUST-PRESETS.md` for containment details. +Project or repository access is a separate boundary from exact-task direction. +A workspace project denial appears on the task with a link to the agent's +Permissions / Trust settings. Board operators with `agents:create` can add +same-company projects through `POST /api/agents/:id/project-access`; the operation +locks the agent, preserves existing permissions and audits added projects. +Saving does not retry work. The task offers explicit Retry after the required +projects are allowed, and dispatch rechecks authorization. Readiness checks the +task's current organizational project and selected workspace source, including +a valid pending selection. Moving the task to an allowed project does not clear +a retained source's access requirement. Existing setup failures with the exact +project-boundary denial also receive the explanation on reload, without assuming +that the task's current project caused the historical failure. A stale or +unavailable workspace selection stays in the non-retryable failure state; it +does not ask the user to grant project access that cannot repair the selection. + Cross-issue writes are contained per heartbeat run. An agent-authored comment may wake the target assignee, including an explicit `resume: true` comment on a `done` or `cancelled` issue, but the wake remains agent-class and is subject to @@ -1887,6 +1902,13 @@ Required UX behaviors: - conflict toasts on atomic checkout failure - no silent background failures; every failed run visible in UI +Cold opens of ordinary tasks show the instruction and recent conversation before +related metadata and historical run activity. Earlier messages and diagnostics +load through **More messages**. Messages observed while the task stays open remain +visible; deep links still load their target. Compact reads retain the full reads' +authorization and redaction rules. Agent Chat and the classic task interface keep +their existing history behavior. + Regular task chats and Agent Chat keep requests as compact, reopenable entries at their original position in the feed. Dismissing any composer request clears its form and pending reminder, including completion reviews, confirmations, diff --git a/doc/architecture/native-status-arbitration.md b/doc/architecture/native-status-arbitration.md index f56014fa5b0..7fa6d5782ed 100644 --- a/doc/architecture/native-status-arbitration.md +++ b/doc/architecture/native-status-arbitration.md @@ -374,3 +374,27 @@ and records a recovery action for the reviewer without an automatic retry loop. Both Codex and ACPX providers wait for server completion feedback before their terminal tool call succeeds. A rejected completion report stays correctable in the same provider turn. The runner does not latch it as an accepted result. + +## Completed runs with saved monitors + +A successful accepted result can report a monitor wait after the run has +already released execution and checkout. Finalization can preserve that wait +when the same agent still owns a valid ordinary task monitor and the completed +run has a server-recorded `set_task_monitor` receipt for the task. The current +monitor can have a later deadline; finalization does not restore an older timer. + +The server rechecks the accepted source, current monitor, empty execution and +checkout slots, and governance under the coordinator and task locks. The +`scheduled_monitor_preserved` decision keeps task status and has no task effects. +It records the completed result without dispatching another provider turn. +The existing monitor scheduler receives a transaction-aware notification when +this finalization blocker clears. Replaying the committed decision sends no +second notification. +Live response selection and committed response repair retain the accepted +monitor explanation under the existing reply and publication authorization rules. + +Missing receipts, changed ownership, cancellation intents, cleared or quota +monitors, and unfinished workspace finalization retain their recovery paths. +Due and exhausted monitors still belong to the scheduler. This does not change +their deadlines or attempts, clear unrelated recovery holds, or relax the +Cloud sleep and wake contract. diff --git a/doc/connections/GITHUB-REVIEW-BOT.md b/doc/connections/GITHUB-REVIEW-BOT.md index 2d9dc55197b..f63732c56a4 100644 --- a/doc/connections/GITHUB-REVIEW-BOT.md +++ b/doc/connections/GITHUB-REVIEW-BOT.md @@ -40,6 +40,13 @@ Installation may return before GitHub's separately delivered signed webhook ping. Paperclip keeps the same draft and credentials, shows a waiting state, and continues automatically after authenticating the ping. Refreshing or resuming that draft must not create another App or bypass signature verification. +While this Cloud setup screen is waiting, its authenticated advance request also +drains the existing sealed event queue. It can bypass an older idle-connection +backoff, at most once every five seconds. Setup and background requests share +one in-flight lease. The normal stack webhook handler still authenticates the +original GitHub bytes before setup can continue. This adds no background timer; +closing the screen stops these readiness requests. Existing background recovery +and its sleep restrictions remain unchanged. New connections default to authorized mentions, advisory reviews, linked-member access, guests off, and enabled bot GitHub tools. Existing instructions, explicit diff --git a/doc/evals.md b/doc/evals.md index 2c0af7e5ec7..78304500307 100644 --- a/doc/evals.md +++ b/doc/evals.md @@ -1,5 +1,11 @@ # Paperclip evaluation guide +The explicit-only [Daytona restart continuity journey](../tests/runner-e2e/README.md) +extends the existing native restart oracle to a 125-second controller outage, +using the recorded Claude and Codex models in disposable development fixtures. +It requires same-process adoption, saved files and finalization without replay; +local regression checks and catalog discovery are separate from live acceptance. + The explicit-only [GitHub review bot journey](../tests/runner-e2e/GITHUB-REVIEW-JOURNEY.md) converts the Ponytail walkthrough into attended App setup, scoped installation, automatic PR review, and independent two-service checks. It supports attached diff --git a/doc/execution-github-identity.md b/doc/execution-github-identity.md index 662f20ef11d..3d179337f17 100644 --- a/doc/execution-github-identity.md +++ b/doc/execution-github-identity.md @@ -14,6 +14,12 @@ Delegated work and interactions persist their originating context. Retries retai Executions with managed GitHub configured receive token-free `git` and `gh` launchers. Each launcher invocation resolves one eligible credential from the current run's accepted identity at operation start. A `gh` command's child Git processes inherit that command's captured identity; later steering does not change already-started operations. +GitHub credential capture reads the current task, project, and retained trust +policy bindings without transferring saved prompts, runner profiles, results, +or log excerpts. It keeps the existing task/run lock order and accepted-identity +reconciliation. Other identity callers retain their complete execution input, +including the native input required by project creation authorization. + Native runners retain a session-owned broker and launcher path across warm turns. The provider keeps an opaque transport token, not a GitHub credential or the previous run's signed capability. After acquiring exclusive session ownership, the controller binds the broker to the current company, agent, task, and run. Requests cannot choose another run or responsible person. The broker rejects requests while idle and discards credential responses if their run binding changed during acquisition. Each operation still rechecks the live run, accepted identity, grants, and trust policy through the shared credential resolver. Changing run IDs alone no longer replaces the provider process; changes to authentication mode, provider credentials, permissions, or other session configuration retain their existing retirement rules. The session broker listens only on controller loopback and accepts only its authenticated GitHub credential operation. Remote executions reach it through the existing authenticated callback bridge. Its transport and launchers are retired with the provider session. If remote bridge startup fails, anonymous launchers keep ordinary work available; the next run retries setup with a fresh session. Controller restart/cold recovery still uses the existing checkpoint and process-recovery rules; the broker's in-memory authority is not persisted for adoption. diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index 4b2df094f3a..52dbcac821e 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -1676,6 +1676,30 @@ waiting for provider startup. An abrupt controller exit during session opening can then recover through the same exact process-identity checks as an active turn. Both graceful and hot restarts detach the old controller from native sessions. +Remote Runner processes use their authenticated connection lease as the bound +on disconnected lifetime. The controller's shorter connection-recovery budget +limits its own attempt; it must not terminate the remote owner while another +controller is starting or classifying runs. Lease expiry still shuts down the +provider, and exact identity, authentication, Stop and revocation checks remain +required before any recovery or new execution. +Native ACPX semantic tool calls wait on their durable PRP invocation across a +controller outage. Their provider-side MCP bridge does not impose a separate +request deadline that could make the provider retry an already queued write. +Runner lease expiry, turn cancellation and bridge closure still end those waits. +Ordinary bridge callers retain their request deadline. Repeating the same call +identity returns its one retained result; it does not execute the operation again. +The native Claude sidecar also sets its HTTP MCP client budget to one hour, +matching the Runner connection lease budget. This prevents Claude's default +60-second HTTP deadline from aborting an otherwise pending durable request. +The setting applies to that native Claude session's MCP client; other provider +sessions and ordinary bridge deadlines remain unchanged. Explicit cancellation +and Runner shutdown still take precedence over that client budget. +An adopted transport can receive authenticated queued semantic calls before +the driver publishes its recovered session and tool handler. It holds those +calls until the handler is installed, then rechecks the original authority and +turn epoch before dispatch. Transport failure, close or controller detachment +ends that wait without invoking a late handler. Missing startup wiring is not +reported to the provider as the result of an external operation. If shutdown begins while a provider session is opening, its eventual publication honors the pending detachment before dispatching a turn. Once detached, an old execution finalizer cannot suspend or signal the durable runner: the next @@ -1683,6 +1707,18 @@ controller must recover it through the authenticated ownership checks. This preserves active work and queued messages without treating a server restart as user cancellation. +During controller shutdown, already registered provider startups share one +30-second sweep deadline to finish read-only run admission and publish their exact +sessions for detachment. Its company, run, session, runner and completion-contract binding +must still match persisted authority. The draining controller cannot checkpoint, +publish events, submit a turn or settle the run. A new startup after the shutdown +fence, or admission after the window expires, remains rejected. A Stop pending +before or after session publication keeps cancellation and cleanup with the +current owner. Shutdown waits for that owner only until the same sweep deadline; +it does not detach the session or submit a prompt while Stop is pending. An +unsettled owner remains explicitly pending for reconciliation. Its timeout does +not abort detachment of healthy sessions or trigger a global graceful drain. + Recovery restores a structured result proposal from the durable run events when its append survived but the active turn's session checkpoint did not. Advancing the replay cursor must preserve that proposal for the exact run, session, source, @@ -1807,7 +1843,11 @@ experimental isolated-workspaces UI. Ordinary workspace updates remain gated; the runtime can bind only a validated shared workspace in the issue's company and project. Follow-ups can therefore reuse the same sandbox and provider session. A staged provider package is reused only after the complete expected -manifest and artifact hashes verify. A missing, changed, or incompatible package +manifest and artifact hashes verify. Exact live-process reattachment uses the +controller-owned pack and runner identities recorded after the original session +passed authentication. It preserves protocol and capability checks. Fresh or +replacement launches use current release authority. A missing historical receipt +retains the current release gate; guest files cannot establish old authority. A missing, changed, or incompatible package must be replaced and verified before launch. Warm attachment requires two consecutive authenticated readiness snapshots. diff --git a/doc/native-workspace-finalization-recovery.md b/doc/native-workspace-finalization-recovery.md index f6ba0ee73ea..5b1ffb5ce98 100644 --- a/doc/native-workspace-finalization-recovery.md +++ b/doc/native-workspace-finalization-recovery.md @@ -1,5 +1,15 @@ # Native workspace finalization ownership and recovery +Periodic heartbeat recovery runs one pass at a time in each server process. +A tick that arrives during a slow pass reports its age and starts no duplicate. +A later tick can start a fresh pass after the current work settles. Shutdown +and idle sleep still track the actual recovery promise through cleanup. Direct +run operations and scheduled work delivery keep their existing paths. A pass +that remains active for three scheduler intervals produces an error log with its +age. A waiting timeout does not admit another pass: the actual work must settle +before ownership is released. A stuck operation requires investigation and +recovery at the operation or provider boundary. + A native provider can fail before it submits a result while its workspace still contains unpublished edits. A prepared workspace reference is not an export receipt. Terminal provider failure pins the exact bound sandbox and records a @@ -9,6 +19,17 @@ cleanup run. This includes descriptor reads that fail inside runtime selection. preparation failure; another owner or a stale claim prevents cleanup. Settling that claim and recording file retention share a transaction. +The reconciliation candidate scan selects the complete workspace-sync record and +the native status wake identifier from each saved run. It keeps their JSON types +and reads full evidence later where finalization or recovery needs it. Saved +execution input, runner profiles, and logs stay in the stored run. + +Wake-queue release reads keep complete context and result receipts. Native runs +omit runner profiles and log excerpts from this read; cleanup and execution +read their own current evidence. Legacy runs retain the full release row so +workspace-restore and pre-dispatch proofs keep their existing inputs. The +selection is one scoped database query for either runtime. + The recovery card names the original run and requires evidence that its exact allocation stopped and its files were recovered. Recording that evidence does not retry the old turn, certify unknown external actions, change the current task @@ -371,6 +392,26 @@ and operator guidance remain under Recovery details. Source-only repair remains available through the board-authorized recovery resolution API and its original stop-receipt and evidence checks. +After a remote provider pack passes full integrity verification and the session +passes authenticated admission, the controller saves its manifest and runner +binary identity in `provider-pack-authority.json`. The receipt binds the company, +agent, workspace, session, runner instance, environment lease, remote working +directory, and provider descriptor. It stays in private controller state, outside +guest workspace and harness backup directories. Projectless runs use their +original environment lease as the workspace identity because each run otherwise +has a new workspace placeholder. Real workspace identities still require an exact +match. A receipt write failure warns and returns the admitted session to its +normal owner; it does not discard the live session handle. Future recovery still +fails closed without matching authority. + +Exact process reattachment uses that saved release authority. It still verifies +the full remote pack, runner protocol and capabilities, exact process identity, +and authenticated PRP handshake. A new controller release alone does not require +a live provider to change its executable files. Fresh launches and explicit +session replacement select current release artifacts. Invalid or mismatched +receipts fail closed. Older sessions without a receipt retain the strict current +release gate; a sandbox's own manifest cannot supply missing launch authority. + Remote provider-pack verification retries an interrupted read at most once. Each command retains its 30-second transport timeout. A timeout, signal, absent exit status, or transport exception does not prove artifact incompatibility. @@ -380,7 +421,9 @@ a new provider attempt, so it cannot renew this exhausted read budget. A completed nonzero verification, digest mismatch, or wrong OpenCode version remains incompatible and cannot authorize a new provider turn. Recovery retains the exact source workspace and existing ownership gates; -these read-only retries do not replay tools or alter a retained pack. +these read-only retries do not replay tools or alter a retained pack. Verification +errors include bounded, redacted output from both transport streams, including +providers that combine stderr into stdout. ### Continue after confirmed source destruction @@ -443,3 +486,33 @@ For a cancelled native run, the file-recovery form also offers an optional confirmation that no external actions were performed. Leave it unchecked unless the run history verifies that fact. The default records files without certifying action outcomes. Changing the selected source clears the evidence and confirmations. + +Committed chat response authorization reads current run identity, completion +binding, and chat source fields under the existing nonblocking share lock. +It excludes saved execution input, runner profiles, and log excerpts. Actual +retry admission still reads the complete run and its recovery evidence. These +reads preserve stored run data, source revocation checks, and JSON value types. +External review-wait discovery selects only chat authorization context fields. +It keeps the current endpoint, membership, resource, and question-receipt checks; +saved prompts remain stored and do not cross this discovery query. + +Legacy continuation recovery rejects native runs before it loads their saved +execution input. The persisted disposition check reads only the task binding +for a native run. It keeps the same task-binding and native-finalization skip +reasons, including the original nonblank binding values. Native input and history +stay stored. Real legacy recovery still reads +its complete context, profile, excerpts, and required failure evidence. + +The native Stop continuation recheck reads current run identity, the full Stop +acknowledgement receipt, and process identity. It does not load saved execution +input, runner profiles, excerpts, or unrelated result fields. Lease cleanup, +remote termination receipts, local process checks, task ownership, and admission +checks keep their existing behavior. Stored runs retain all of their data. + +One periodic stranded-message pass coalesces successful native release checks +for the same company, task, source run, and current source row version. The release +path already drains that +task's queue through current admission checks. Each saved request still advances +its recovery cursor. Rejected attempts, changed source runs, and updated source evidence remain eligible, +and the next pass checks current state again. Legacy per-request handling stays +unchanged. Pending work is awaited before a source is marked checked. diff --git a/doc/plans/2026-10-24-task-cold-open.md b/doc/plans/2026-10-24-task-cold-open.md new file mode 100644 index 00000000000..faa2ef48abf --- /dev/null +++ b/doc/plans/2026-10-24-task-cold-open.md @@ -0,0 +1,46 @@ +# Task cold-open performance + +Branch: `codex/task-cold-open`. Owner: task detail UI and existing issue/run read services. +Endpoint: tested, reviewable PR; deployment and staging acceptance are separate. + +Cold opens show the instruction and recent exchange without hydrating historical +run diagnostics. Full issue enrichment stays on its existing authorized route. +The core view skips enrichment. Recent comments retain the latest human input +and four recent replies, plus the first human instruction when no description +exists. Recent activity preserves queue/steering metadata. Recent run summaries +select the latest run; conversation event reads exclude tools and traces. +Original comment/steering facts are read separately for visible comments, with +at most two facts per comment and 100 comment IDs per request. A reconnect that +leaves a watched unfinished run outside the recent window reconciles run summaries. +Core snapshots share task-detail mutation updates, rollback and invalidation. + +The browser retains messages and runs observed while open. New native runs use +the normal event path. Expansion fetches full history, preserves existing rows, +and remains expanded on return in the session. Deep links and classic/Agent Chat +retain their full-history behavior. No database migration or permissions change. + +| Case | Evidence | +| --- | --- | +| Cold task | Page tests hold enrichment/history pending while recent messages render; desktop/mobile thread tests keep content and composer visible. | +| Long history | Disposable app with 80 comments and six large run histories; cold network uses recent comments, recent runs, and one conversation-events request. Expansion reveals older comments. | +| Active page | Burst catch-up tests retain 60 messages across cursor pages; new-run tests retain previously visible runs. | +| Native history | Real PostgreSQL test filters tool payloads, preserves request lifecycles and message deltas, and pages by sequence. Expansion test resets cursor without duplicates. | +| Access | Core and preview route tests reject a foreign authenticated tenant. | +| Compatibility | Existing task UI, full event reads, and full activity/run reads remain covered. | + +Measurements on the disposable local fixture: core read 50 ms; recent comments +24 ms; recent run summaries 17 ms. Latest native conversation events: 839 JSON +characters / 8 ms versus full latest-run events: 1,099,005 / 86 ms. These are local +HTTP samples, not staging latency promises. Browser navigation showed the initial +content paint at 1.34 seconds, with development assets already cached. + +Verification: affected UI/API tests, real PostgreSQL tests, workspace typecheck, +production build, token gates, and map checks. The full suite was attempted but +this host exhausts macOS PostgreSQL semaphores. Targeted database tests were run +against a separate disposable database on an existing local server; the temporary +test setup override was restored before typecheck/build. + +Related open PRs: #14667 changes the reveal gate and scroll anchors; #16229 adds +general history pagination. This change adds a core read and a folded cold-open +projection, while preserving the default full API contracts. Coordinate those +shared files if either PR lands first. diff --git a/doc/recoverable-disk-cleanup.md b/doc/recoverable-disk-cleanup.md index b33a0cc74fd..93f9ccb6fef 100644 --- a/doc/recoverable-disk-cleanup.md +++ b/doc/recoverable-disk-cleanup.md @@ -89,10 +89,43 @@ The rest of the session directory, including its control-plane state, is kept. | `PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_ENABLED` | `true` | Set `false` to stop expiring copies | | `PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_DRY_RUN` | `false` | Set `true` to report candidates without deleting copies or task sessions | | `PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_DAYS` | `7` | Days an issue must be closed before its sessions' copies expire (minimum `1`) | +| `PAPERCLIP_NATIVE_RUNNER_EVIDENCE_RETENTION_DAYS` | `7` | Days continuity-break archives and quarantined session state are kept (minimum `1`) | Each batch that finds candidates logs them with candidate and reclaimed allocated bytes, cleared task sessions, and skip reasons. +### Continuity-break archives and quarantined session state + +The same sweep also removes two kinds of evidence that native sessions leave behind +under the same root: + +- `/continuity-breaks/-/`, written when a session starts a + fresh provider session and moves the retired session's state aside. Nothing reads + these archives again. +- `quarantine/..../`, written when a session's + state loses authority. Recovery and failed-run retry still consult these entries. + +An entry's age comes from the epoch in its name, or from its modification time when +that epoch is in the future. An entry is removed once it is older than +`PAPERCLIP_NATIVE_RUNNER_EVIDENCE_RETENTION_DAYS` and its session is not executing, +initializing, warm, or closing in this process. Quarantine entries created by +native cleanup (`.identity_indeterminate.cleanup.`) are referenced +from recovery history and kept. Symlinks are never followed or removed. Once a +quarantined entry is removed, recovery and failed-run retry no longer see it, so a +later run of that session may start a fresh provider session instead of refusing. + +The evidence check runs on the sweep only when something is due: once at startup, +then when the oldest retained entry, or an entry the executor has written since, +reaches the window, and on the next sweep after a busy session, a failed removal, +or a full batch. It only reads the filesystem and never queries the +database. A dry run reports each candidate once until newly written evidence comes due. + +An entry is renamed to `.expired-` in its folder before it is deleted, and a +later sweep finishes an interrupted deletion. An empty `continuity-breaks` folder +is removed. Archives, quarantine entries, and leftovers count toward the batch +limits of 200 entries and 50 removals shared with backup copies, and +`PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_ENABLED` and `_DRY_RUN` apply to both. + ## Native workspace sync folders Each native run that syncs a workspace with a remote sandbox writes a folder at diff --git a/docs/deploy/database.md b/docs/deploy/database.md index 4f72b70c097..6e85cebce7e 100644 --- a/docs/deploy/database.md +++ b/docs/deploy/database.md @@ -56,12 +56,19 @@ For production, use a hosted provider like [Supabase](https://supabase.com/). Use the **direct connection** (port 5432) for migrations and the **pooled connection** (port 6543) for the application. -If using connection pooling (transaction mode), disable prepared statements via the environment — no source edits needed: +If your transaction-mode pooler does not support protocol-level named prepared statements, disable them via the environment: ```sh DATABASE_PREPARED_STATEMENTS=false ``` +For a direct connection or a pooler that supports named prepared statements, +`DATABASE_PREPARED_STATEMENTS=true` also prepares parameterized ORM queries and +queries inside transactions and reserved or dedicated connections. Check the +provider's configuration before you enable it. Preparation caches statement +metadata, not query results. Unset preserves the existing ORM behavior and the +driver default for tagged SQL. Explicit per-query options still apply. + Related optional client tuning: `DATABASE_POOL_MAX`, `DATABASE_IDLE_TIMEOUT_SECONDS`, `DATABASE_CONNECT_TIMEOUT_SECONDS`, `DATABASE_MAX_LIFETIME_SECONDS`, `DATABASE_APPLICATION_NAME`. Driver defaults apply when unset, except that idle pooled connections close after 60 seconds (`DATABASE_IDLE_TIMEOUT_SECONDS=0` keeps them open) and the pool reports `application_name=paperclip`. See [Connection pool settings](#connection-pool-settings). ## Connection Pool Settings @@ -75,7 +82,7 @@ The server opens one postgres.js pool for its own queries (and a second one when | `DATABASE_CONNECT_TIMEOUT_SECONDS` | `30` (driver) | Give up on a connection attempt after this long. | | `DATABASE_MAX_LIFETIME_SECONDS` | 30–60 min, randomized (driver) | Recycle a pooled connection once it is this old. | | `DATABASE_APPLICATION_NAME` | `paperclip` | Value of `application_name` in `pg_stat_activity`, so you can find Paperclip's backends: `SELECT * FROM pg_stat_activity WHERE application_name = 'paperclip';` | -| `DATABASE_PREPARED_STATEMENTS` | `true` (driver) | Set `false` behind a transaction-mode pooler (see above). | +| `DATABASE_PREPARED_STATEMENTS` | Unset | `true` opts parameterized ORM queries into preparation. `false` disables preparation. Use `false` when the pooler does not support named statements. Unset preserves existing behavior. | The server ends its pools during shutdown (SIGINT/SIGTERM) and when startup fails after the pool was opened, so a restarting server does not leave idle backends behind. Size `max_connections` on the PostgreSQL side for at least `DATABASE_POOL_MAX` per server process plus your other clients. Each server process also opens up to 2 connections (`application_name=paperclip-controller-lease`) for agent-run lease renewals, so they never queue behind slow application queries; count those too. diff --git a/maps/agent-access-map/agents.md b/maps/agent-access-map/agents.md index efc593ec743..1fa58307434 100644 --- a/maps/agent-access-map/agents.md +++ b/maps/agent-access-map/agents.md @@ -104,6 +104,12 @@ Review: `baseline`. API: `present` — representative operation IDs: - `GET /api/agents/{id}/instructions-bundle/file`. +- `GET /api/agents/{id}/project-access` and `POST /api/agents/{id}/project-access` + are board-only. The POST requires `agents:create`, checks selected projects in + the same company, and adds IDs under an agent row lock. It preserves existing + containment, audits new grants and rejects pending-approval agents. Agent keys + cannot use this route to grant themselves access; a read-only board viewer + receives `canManage: false`. Saving never creates a run or wakeup. - `PUT /api/agents/{id}/instructions-bundle/file`. - `PATCH /api/agents/{id}` with an explicit `runner: "paperclip"` upgrade. - `POST /api/agents/{id}/qualify-runner` to qualify the saved effective sandbox before upgrading. diff --git a/maps/agent-access-map/runs-adapters.md b/maps/agent-access-map/runs-adapters.md index 7f5552f9919..a873db39633 100644 --- a/maps/agent-access-map/runs-adapters.md +++ b/maps/agent-access-map/runs-adapters.md @@ -209,8 +209,8 @@ Live latency verification remains separate from the existing access baseline. API: `present` — representative operation IDs: - `GET /api/heartbeat-runs/{runId}`. -- `GET /api/heartbeat-runs/{runId}/log`. -- `GET /api/heartbeat-runs/{runId}/events`. +- `GET /api/heartbeat-runs/{runId}/log` (optional `alignStartToRecord=true` extends a tail backward to its first complete record; this can exceed `limitBytes` by that record's prefix and retains the same access/redaction checks). +- `GET /api/heartbeat-runs/{runId}/events` (optional `view=conversation` returns assistant messages and request lifecycles through the same access/redaction checks; `afterSeq` remains a sequence cursor). - `GET /api/heartbeat-runs/{runId}/workspace-operations`. - `GET /api/heartbeat-runs/{runId}/issues`. - `GET /api/companies/{companyId}/heartbeat-runs`. diff --git a/maps/agent-access-map/task-coordination.md b/maps/agent-access-map/task-coordination.md index a9bd1ddcc56..dda7994a89a 100644 --- a/maps/agent-access-map/task-coordination.md +++ b/maps/agent-access-map/task-coordination.md @@ -27,7 +27,11 @@ API: `present` — representative operation IDs: Agent access: `available` — Native create_task/reassign_task; REST / CLI / MCP. Policy: Task visibility/mutation, task-assignment and ownership policy; review audience -independent; tree controls board-only. +independent; tree controls board-only. Resume preview counts only current agent +assignees on todo, in-progress, and in-review tasks as wake targets. Release keeps +blocked and backlog statuses intact; existing execution gates still apply. +Reopened tasks use current eligibility rather than the pause snapshot's skipped +flag. Other active pause holds and execution reconciliation gates remain effective. Evidence: `source-only` — [implementation](../../server/src/routes/issue-tree-control.ts#L95) at the reviewed diff --git a/maps/agent-access-map/tasks.md b/maps/agent-access-map/tasks.md index 1d91ae0aa70..167a5d71ebd 100644 --- a/maps/agent-access-map/tasks.md +++ b/maps/agent-access-map/tasks.md @@ -84,7 +84,7 @@ Gaps: Bounded native search and one-page CLI local match are not complete compan ### `read-task-and-summaries` -API: `present` — GET /api/issues/{id}; GET /api/companies/{companyId}/issues/summaries?ids=... . +API: `present` — GET /api/issues/{id} (optional `view=core` omits related enrichment); GET /api/companies/{companyId}/issues/summaries?ids=... . Agent access: `partial` — REST/CLI issue get; native get_task_context reads CURRENT task only; call_api for eligible arbitrary task/summaries. No dedicated summaries CLI/tool. @@ -254,7 +254,7 @@ Read, append/retry, author-delete or queued-cancel messages, attach allowed file ### `read-history-comment` -API: `present` — GET /api/issues/{id}/comments and /comments/{commentId}. +API: `present` — GET /api/issues/{id}/comments and /comments/{commentId}; GET /api/issues/{id}/recent-comments returns four latest comments plus the latest human comment, a cursor, and `hasMore`. The compact read also includes the first human instruction when there is no description. Cold task views use `GET /api/issues/{id}/activity?view=recent` (50 rows) and `/runs?view=recent` (two summaries) without changing the default full-read contracts. It uses the same issue authorization and run/trust redaction. Agent access: `available` — REST/call_api, CLI issue comments/comment:get; native get_task_history CURRENT task only, declared limit 1..200 but production clamps to 1..100 (default 50). @@ -264,7 +264,9 @@ Evidence: `source-only` — reviewed at `3bbc2104b65caf9f7763938a01260a5673e12f7 Verification: `not-run` — Read multipage history and exact comment, reload same content; reject inaccessible task, foreign/mismatched comment and inspect trust redaction. Request native limit 200 and confirm at most 100 comments. -Gaps: Native bounded history is not an arbitrary-task/paginated API substitute. +The same authorized activity route supports `view=comment-inputs&commentIds=` for original comment/steering facts. Each request accepts at most 100 comment IDs and returns at most two facts per comment, scoped to the accessible issue and company. This preserves delivered-input state when its receipt is older than the recent activity window. + +Gaps: Native bounded history is not an arbitrary-task/paginated API substitute. The new UI preview route has no dedicated native tool or CLI command. Existing full comment reads remain available. ### `append-retry-comment` diff --git a/maps/feature-map/agents.md b/maps/feature-map/agents.md index 95fed092ec1..bb3b705cd1d 100644 --- a/maps/feature-map/agents.md +++ b/maps/feature-map/agents.md @@ -41,6 +41,12 @@ existing preparation and verification lifecycle. Open `/agents/:agentId` and the configuration/instruction surfaces it exposes. +For a low-trust agent, open **Permissions / Trust → Allowed projects**. An operator +with agent-management permission can select additional projects and explicitly +save them. Existing projects and other containment restrictions remain intact. +Cancel discards the draft; a failed save retains it. Reload to verify persistence. +This does not start or retry a task. + In `/agents/all`, select roster checkboxes and choose **Edit connection & model**. A mixed selection offers one action per harness. Each field is opt-in. The editor previews each changed value and excludes agents already at the target value. diff --git a/maps/feature-map/connection-setup.md b/maps/feature-map/connection-setup.md index 22306048e8a..d27fc36c657 100644 --- a/maps/feature-map/connection-setup.md +++ b/maps/feature-map/connection-setup.md @@ -205,6 +205,15 @@ configuration changes, saved retry deadlines, rollback, and parking pending repair while an endpoint or connection is inactive. The provider responses are fixtures. Settled maintenance leaves no discovery timer or database scans. +GitHub Cloud setup also covers an older empty-queue backoff, concurrent setup and +background leases, and retry after a Cloud error in the +[event poller regression](../../server/src/services/github-connection-events.test.ts). +Channel integration checks that active setup drains its signed ping through the +normal webhook handler; a forged ping still cannot verify the endpoint. These +fixtures do not establish live Cloud delivery. For live proof, start a disposable +App after an idle personal GitHub connection, complete installation, and confirm +that the same draft advances after authenticating the queued ping. + Manual: complete setup for one named provider, reload endpoint settings, link the intended sender, and send a disposable message. Verify one task and a useful reply in the originating conversation. Inspect an unlinked sender's denial and diff --git a/maps/feature-map/recovery.md b/maps/feature-map/recovery.md index 4294a1876a3..a6f49522fef 100644 --- a/maps/feature-map/recovery.md +++ b/maps/feature-map/recovery.md @@ -48,6 +48,18 @@ Open the affected task from Tasks, Inbox/Blocked, search, or a direct task link. Observe one working, backoff, failed or waiting-for-user state. Use Retry after an infrastructure failure. No process or file certification is required. +A workspace project denial names the blocked access and links to the agent's +**Permissions / Trust** settings. After an authorized person adds the required +project, return to the saved task for an explicit **Retry task** action. Saving +permissions alone does not wake the agent. Historical setup denials receive the +same notice on reload; unrelated errors and prior assignees do not. +Move a denied task to a project the agent can access while retaining a source +from another project: the notice must stay blocked until the selected source is +also allowed or an authorized workspace selection replaces it. A valid pending +selection is checked for the next admission; a stale or unavailable selection +must show a non-retryable failure without a project-permission repair action. +Project changes alone preserve selected files. + Thinking appears immediately after Retry, before the request completes. Technical source/run details remain in deliberate diagnostics. The [task recovery contract](../../doc/task-recovery.md) describes receipt identity, @@ -170,6 +182,14 @@ and rejected identity evidence. The Runner restart tests require positive exit evidence before replacement. This coverage does not prove recovery of a historical retained sandbox or behavior during a live provider outage. +For a release change during a remote provider session, verify that reattachment +uses the controller's admitted pack manifest and runner digest. Check useful +output from the same process without replaying a turn. Repeat with a modified +sidecar, mismatched receipt scope, replacement lease, and incompatible runner +protocol. The [authority tests](../../server/src/services/native-runtime/remote-provider-pack-authority.test.ts) +and executor wiring regressions cover selection and fail-closed scope checks. +They do not prove a historical retained sandbox was recovered. + For completion recovery, restart after a native completion proposal is appended but before its session checkpoint. Verify that the original result is finalized after provider termination, with no new provider turn or repeated external action. diff --git a/maps/feature-map/task-coordination.md b/maps/feature-map/task-coordination.md index 86a60dc5611..67f58c06244 100644 --- a/maps/feature-map/task-coordination.md +++ b/maps/feature-map/task-coordination.md @@ -49,6 +49,13 @@ Automated: [execution policy](../../server/src/__tests__/issue-execution-policy. Manual: Complete implementation under a review-required policy and confirm it waits for the required review outcome. Exercise rejection and later approval. Pause the parent while child work is pending, then verify resumption respects budget and ownership gates. +Resume removes the pause without changing task status. Its wake count includes only +current agent assignees on todo, in-progress, or in-review tasks. The dialog explains +that blocked, backlog, and unassigned tasks remain idle. Verify this in both task +interfaces with [the paused composer browser suite](../../tests/e2e/paused-composer.spec.ts). +Release rechecks tasks reopened after the pause; another active pause still prevents +their execution, and uncertain prior executions must be reconciled before waking. + ## Gotchas - Review completion and task completion are different events under some policies. diff --git a/maps/feature-map/tasks.md b/maps/feature-map/tasks.md index b9e747561eb..1108526dfb0 100644 --- a/maps/feature-map/tasks.md +++ b/maps/feature-map/tasks.md @@ -39,7 +39,7 @@ Open `/issues` or a project’s Issues view; legacy `/tasks` links redirect. ### `task-detail` -Open `/issues/:issueId` from a row, mention, or direct link. +Open `/issues/:issueId` from a row, mention, or direct link. Cold opens show the instruction and recent exchange first. **More messages** loads earlier messages and run activity; incoming messages remain visible while the task stays open. ### `task-sharing` @@ -87,7 +87,9 @@ Manual: Filter by status and assignee, change grouping/sort, open a task, and re ### `task-detail` -Automated: [task detail](../../ui/src/pages/IssueDetail.test.tsx) covers component behavior, while [issue service](../../server/src/__tests__/issues-service.test.ts) covers server contracts. +Manual: Open a long task with slow historical requests. Read the instruction and recent reply before those requests finish. Edit the title/status while enrichment is pending and verify saves and rollbacks. Expand **More messages**, verify scroll position and no duplicate replies, return to the task, and follow an older comment link. Leave a task open across new messages and runs and confirm none disappear. Reconnect after several turns and verify watched runs settle. Reopen after steering with more than 50 newer activity rows; the delivered message must not reappear in the queue. + +Automated: [cold-open reads](../../server/src/__tests__/task-cold-open.test.ts) cover bounded reads, event filtering and company denial. [Task detail](../../ui/src/pages/IssueDetail.test.tsx) covers component behavior, while [issue service](../../server/src/__tests__/issues-service.test.ts) covers server contracts. Failure markers have no dashed divider rules. An adjacent run error and no-live-execution-path notice show only the last error, with its details and diff --git a/packages/db/src/client.ts b/packages/db/src/client.ts index 5149c1c9b27..7f067cbfc1c 100644 --- a/packages/db/src/client.ts +++ b/packages/db/src/client.ts @@ -4,6 +4,8 @@ import { readFile, readdir } from "node:fs/promises"; import { fileURLToPath } from "node:url"; import postgres from "postgres"; import { installDatabaseWorkSignals } from "./work-signals.js"; +import { installDatabaseResponseCapture } from "./response-capture.js"; +import { installPreparedQueries } from "./prepared-queries.js"; import * as schema from "./schema/index.js"; const MIGRATIONS_FOLDER = fileURLToPath(new URL("./migrations", import.meta.url)); @@ -128,13 +130,6 @@ export type MigrationState = }; export interface DatabaseClientOptions { - /** - * postgres.js `prepare`. Set false when connecting through a - * transaction-mode pooler (pgbouncer / Neon `-pooler` endpoints / - * Supabase Supavisor transaction ports) so the client does not rely on - * session-scoped prepared statements. Defaults to the driver default - * (enabled), preserving existing behavior on direct connections. - */ prepare?: boolean; /** postgres.js `max` — connection pool size (driver default: 10). */ maxConnections?: number; @@ -300,6 +295,7 @@ export function createDb(url: string, options?: DatabaseClientOptions) { function createDbClient(url: string, resolved: DatabaseClientOptions, owner?: object) { const sql = postgres(url, postgresJsOptions(resolved)); + if (resolved.prepare === true) installPreparedQueries(sql); const key = hostPortKeyOrNull(url); if (key) registerClient(key, sql); // A disconnect can lose the response after a statement has committed. @@ -307,6 +303,7 @@ function createDbClient(url: string, resolved: DatabaseClientOptions, owner?: ob // message cannot establish that replay is safe. Leave retries to callers // that know the complete operation is idempotent. const db = installDatabaseWorkSignals(drizzlePg(sql, { schema }), owner); + installDatabaseResponseCapture(db, schema); dedicatedDbFactories.set(db, () => createDbClient(url, { ...resolved, maxConnections: 1, applicationName: "paperclip-workspace-finalization-lock", }, db)); diff --git a/packages/db/src/prepared-queries-postgres.test.ts b/packages/db/src/prepared-queries-postgres.test.ts new file mode 100644 index 00000000000..edef6dc060e --- /dev/null +++ b/packages/db/src/prepared-queries-postgres.test.ts @@ -0,0 +1,175 @@ +import { eq, sql } from "drizzle-orm"; +import { integer, jsonb, pgTable, text } from "drizzle-orm/pg-core"; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it } from "vitest"; +import { closeLivenessDb, createDb, databaseClientOptionsFromEnv, livenessDb, withDedicatedDbConnection, type DatabaseClientOptions, type Db } from "./client.js"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./test-embedded-postgres.js"; +import { runEventPayload } from "./run-event-payload.js"; + +const records = pgTable("prepared_query_fixture", { id: text("id").primaryKey(), companyId: text("company_id").notNull(), value: integer("value").notNull(), payload: jsonb("payload") }); +const encoded = pgTable("prepared_encoded_fixture", { id: text("id").primaryKey(), payload: runEventPayload("payload") }); +const support = await getEmbeddedPostgresTestSupport(); + +describe.skipIf(!support.supported)("explicit ORM prepared statements on PostgreSQL", () => { + let temporary: Awaited>; + let setup: Db; + const clients: Db[] = []; + beforeAll(async () => { + temporary = await startEmbeddedPostgresTestDatabase("prepared-orm-"); + setup = createDb(temporary.connectionString, { prepare: false, maxConnections: 1 }); + await setup.execute(sql`create table prepared_query_fixture(id text primary key, company_id text not null, value integer not null, payload jsonb)`); + await setup.execute(sql`create table prepared_encoded_fixture(id text primary key, payload jsonb)`); + await setup.execute(sql`create table prepared_schema_change_fixture(id text primary key)`); + await setup.execute(sql`insert into prepared_schema_change_fixture(id) values ('retained')`); + }, 90_000); + beforeEach(async () => { + await setup.execute(sql`truncate prepared_query_fixture, prepared_encoded_fixture`); + await setup.insert(records).values([ + { id: "a", companyId: "company-a", value: 1, payload: { owner: "a" } }, + { id: "b", companyId: "company-b", value: 2, payload: { owner: "b" } }, + ]); + }); + afterEach(async () => { + for (const db of clients.splice(0)) { + await closeLivenessDb(db, 2); + await db.$client.end(); + } + }); + afterAll(async () => { await setup?.$client.end(); await temporary?.cleanup(); }); + function fixture(options: DatabaseClientOptions = {}) { + const db = createDb(temporary.connectionString, { ...options, maxConnections: 1 }); + clients.push(db); return db; + } + async function statements(db: Db, pattern = '%"prepared_query_fixture"%') { + return db.$client.unsafe("select name,statement from pg_prepared_statements where statement like $1", [pattern]); + } + + it.each([undefined, false])("keeps the existing ORM mode when prepare is %s", async prepare => { + const db = fixture({ prepare }); + expect(await db.select().from(records).where(eq(records.companyId, "company-a"))).toEqual([ + { id: "a", companyId: "company-a", value: 1, payload: { owner: "a" } }, + ]); + expect(await statements(db)).toHaveLength(0); + }); + + it("honors the explicit environment opt-in for ORM queries with fresh company bindings", async () => { + const db = fixture(databaseClientOptionsFromEnv({ DATABASE_PREPARED_STATEMENTS: "true" })); + for (let i = 0; i < 12; i++) { + const companyId = i % 2 ? "company-b" : "company-a"; + const rows = await db.select().from(records).where(eq(records.companyId, companyId)); + expect(rows).toEqual([{ id: i % 2 ? "b" : "a", companyId, value: i % 2 ? 2 : 1, payload: { owner: i % 2 ? "b" : "a" } }]); + } + expect(await statements(db)).toHaveLength(1); + }); + + it("preserves explicit per-query opt-out", async () => { + const db = fixture({ prepare: true }); + expect((await db.$client.unsafe("select $1::integer as explicit_opt_out_marker", [7], { prepare: false }))[0].explicit_opt_out_marker).toBe(7); + expect(await statements(db, "%explicit_opt_out_marker%")).toHaveLength(0); + }); + + it("preserves simple queries and the two-argument options overload", async () => { + const db = fixture({ prepare: true }); + const simpleOptions = { prepare: true, simple: true }; + const simple = await db.$client.unsafe("select 1 as first; select 2 as second", [], simpleOptions); + expect(simple).toEqual([[{ first: 1 }], [{ second: 2 }]]); + const unsafe = db.$client.unsafe as unknown as (query: string, options: { simple: boolean }) => Promise; + expect(await unsafe("select 3 as third; select 4 as fourth", { simple: true })) + .toEqual([[{ third: 3 }], [{ fourth: 4 }]]); + }); + + it("does not replay a rejected prepared write", async () => { + const db = fixture({ prepare: true }); + await expect(db.insert(records).values({ id: "a", companyId: "company-b", value: 9 })) + .rejects.toMatchObject({ cause: { code: "23505" } }); + expect(await db.select({ companyId: records.companyId, value: records.value }).from(records).where(eq(records.id, "a"))) + .toEqual([{ companyId: "company-a", value: 1 }]); + }); + + it("keeps write queries lazy and repeated awaits from repeating writes", async () => { + const db = fixture({ prepare: true }); + const pending = db.$client.unsafe("insert into prepared_query_fixture(id,company_id,value) values ($1,$2,$3) returning id", ["lazy", "company-a", 3]); + expect(typeof pending.values).toBe("function"); + expect(await setup.select().from(records).where(eq(records.id, "lazy"))).toEqual([]); + expect(await pending).toEqual([{ id: "lazy" }]); + expect(await pending).toEqual([{ id: "lazy" }]); + expect(await setup.select({ id: records.id }).from(records).where(eq(records.id, "lazy"))).toEqual([{ id: "lazy" }]); + expect(await statements(db, "%insert into prepared_query_fixture%")).toHaveLength(1); + }); + + it("preserves transaction commits, nested savepoint rollback, and scoped reads", async () => { + const db = fixture({ prepare: true }); + await db.transaction(async tx => { + expect(await tx.select({ value: records.value }).from(records).where(eq(records.companyId, "company-a"))).toEqual([{ value: 1 }]); + await expect(tx.transaction(async nested => { + await nested.update(records).set({ value: 99 }).where(eq(records.companyId, "company-a")); + throw new Error("rollback savepoint"); + })).rejects.toThrow("rollback savepoint"); + expect(await tx.select({ value: records.value }).from(records).where(eq(records.companyId, "company-a"))).toEqual([{ value: 1 }]); + await tx.insert(records).values({ id: "committed", companyId: "company-b", value: 3 }); + const names = await tx.execute(sql`select statement from pg_prepared_statements`); + expect(names.some(row => String(row.statement).includes('"prepared_query_fixture"'))).toBe(true); + }, { isolationLevel: "serializable" }); + expect(await db.select({ value: records.value }).from(records).where(eq(records.id, "committed"))).toEqual([{ value: 3 }]); + }); + + it("rolls back failed outer transactions without replay", async () => { + const db = fixture({ prepare: true }); + await expect(db.transaction(async tx => { + await tx.update(records).set({ value: 42 }).where(eq(records.id, "a")); + throw new Error("rollback outer transaction"); + })).rejects.toThrow("rollback outer transaction"); + expect(await db.select({ value: records.value }).from(records).where(eq(records.id, "a"))).toEqual([{ value: 1 }]); + }); + + it("prepares reserved-client queries and keeps their bound values fresh", async () => { + const db = fixture({ prepare: true }); + const reserved = await db.$client.reserve(); + try { + for (const marker of [1, 2, 3]) { + expect((await reserved.unsafe("select $1::integer as reserved_prepared_marker", [marker]))[0].reserved_prepared_marker).toBe(marker); + } + const names = await reserved.unsafe("select statement from pg_prepared_statements where statement like $1", ["%reserved_prepared_marker%"]); + expect(names).toHaveLength(1); + } finally { reserved.release(); } + }); + + it("preserves object and array results for the same prepared shape", async () => { + const db = fixture({ prepare: true }); + expect(await db.$client.unsafe("select $1::integer as result_mode_marker", [41]).values()).toEqual([[41]]); + expect(await db.$client.unsafe("select $1::integer as result_mode_marker", [42])).toEqual([{ result_mode_marker: 42 }]); + }); + + it("keeps custom stored-payload decoding fresh", async () => { + const db = fixture({ prepare: true }); + const first = { text: "encoded first payload🙂".repeat(2048) }; + const second = { text: "encoded second payload🙂".repeat(2048) }; + await setup.insert(encoded).values({ id: "encoded", payload: first }); + expect((await db.select().from(encoded).where(eq(encoded.id, "encoded")))[0].payload).toEqual(first); + await setup.update(encoded).set({ payload: second }).where(eq(encoded.id, "encoded")); + expect((await db.select().from(encoded).where(eq(encoded.id, "encoded")))[0].payload).toEqual(second); + expect(await statements(db, '%"prepared_encoded_fixture"%')).toHaveLength(1); + }); + + it("inherits the opt-in in dedicated and liveness clients", async () => { + const db = fixture({ prepare: true }); + await withDedicatedDbConnection(db, async dedicated => { + expect(await dedicated.select({ value: records.value }).from(records).where(eq(records.companyId, "company-b"))).toEqual([{ value: 2 }]); + expect(await statements(dedicated)).toHaveLength(1); + }); + const live = livenessDb(db); + expect(await live.select({ value: records.value }).from(records).where(eq(records.companyId, "company-a"))).toEqual([{ value: 1 }]); + expect(await statements(live)).toHaveLength(1); + }); + + it("refreshes prepared result metadata after a schema change", async () => { + const db = fixture({ prepare: true }); + const query = "select * from prepared_schema_change_fixture where id=$1"; + expect(await db.$client.unsafe(query, ["retained"])).toEqual([{ id: "retained" }]); + try { + await setup.execute(sql`alter table prepared_schema_change_fixture add column extra text`); + expect(await db.$client.unsafe(query, ["retained"])).toEqual([{ id: "retained", extra: null }]); + } finally { + await setup.execute(sql`alter table prepared_schema_change_fixture drop column if exists extra`); + } + }); +}); diff --git a/packages/db/src/prepared-queries.ts b/packages/db/src/prepared-queries.ts new file mode 100644 index 00000000000..7622722027a --- /dev/null +++ b/packages/db/src/prepared-queries.ts @@ -0,0 +1,47 @@ +type Method = (this: unknown, ...args: unknown[]) => unknown; + +export function installPreparedQueries(client: unknown): void { + const visited = new WeakSet(); + + function visit(value: unknown): void { + if ((typeof value !== "object" || value === null) && typeof value !== "function") return; + if (visited.has(value)) return; + visited.add(value); + const target = value as Record; + const unsafe = target.unsafe; + if (typeof unsafe === "function") { + target.unsafe = function (this: unknown, ...args: unknown[]) { + const index = args.length === 2 && !Array.isArray(args[1]) ? 1 : 2; + const options = args[index]; + if (options !== undefined && (options === null || typeof options !== "object")) { + return Reflect.apply(unsafe, this, args); + } + if (index === 2 && args.length < 2) args[1] = []; + args[index] = { prepare: true, ...options }; + return Reflect.apply(unsafe, this, args); + }; + } + for (const method of ["begin", "savepoint"]) { + const original = target[method]; + if (typeof original !== "function") continue; + target[method] = function (this: unknown, ...args: unknown[]) { + const wrapped = args.map(arg => typeof arg !== "function" ? arg : function (this: unknown, ...inner: unknown[]) { + visit(inner[0]); + return Reflect.apply(arg, this, inner); + }); + return Reflect.apply(original, this, wrapped); + } satisfies Method; + } + const reserve = target.reserve; + if (typeof reserve === "function") { + target.reserve = function (this: unknown, ...args: unknown[]) { + return Promise.resolve(Reflect.apply(reserve, this, args)).then(reserved => { + visit(reserved); + return reserved; + }); + } satisfies Method; + } + } + + visit(client); +} diff --git a/packages/db/src/response-capture-postgres.test.ts b/packages/db/src/response-capture-postgres.test.ts new file mode 100644 index 00000000000..42d5d805a0c --- /dev/null +++ b/packages/db/src/response-capture-postgres.test.ts @@ -0,0 +1,204 @@ +import { randomUUID } from "node:crypto"; +import { compileFunction } from "node:vm"; +import { eq, sql } from "drizzle-orm"; +import { pgTable, text } from "drizzle-orm/pg-core"; +import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; +import { createDb, withDedicatedDbConnection } from "./client.js"; +import { createDatabaseResponseCapture, databaseResponseCaptureFromEnv, installDatabaseResponseCapture } from "./response-capture.js"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./test-embedded-postgres.js"; +import { encodeRunEventPayload, runEventPayload } from "./run-event-payload.js"; +import { databaseWorkPending, reconcileDatabaseWork, signalDatabaseWork, subscribeDatabaseWork } from "./work-signals.js"; + +const table = pgTable("response_capture_fixture", { id: text("id").primaryKey(), payload: text("payload").notNull() }); +const encodedTable = pgTable("response_capture_encoded_fixture", { id: text("id").primaryKey(), payload: runEventPayload("payload") }); +const support = await getEmbeddedPostgresTestSupport(); +describe.skipIf(!support.supported)("database response capture PostgreSQL behavior", () => { + let temporary: Awaited>; + const clients: Array> = []; + const captures: Array> = []; + beforeAll(async () => { + temporary = await startEmbeddedPostgresTestDatabase("response-capture-"); + const db = createDb(temporary.connectionString); + try { + await db.execute(sql`create table response_capture_fixture (id text primary key, payload text not null)`); + await db.execute(sql`create table response_capture_encoded_fixture (id text primary key, payload jsonb not null)`); + } + finally { await db.$client.end(); } + }, 90_000); + afterEach(async () => { + for (const capture of captures.splice(0)) capture.flush(); + for (const client of clients.splice(0)) await client.$client.end(); + vi.restoreAllMocks(); vi.unstubAllEnvs(); + }); + afterAll(async () => { await temporary?.cleanup(); }); + function fixture(prepare?: boolean) { + const events: Array> = []; + const now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + const db = createDb(temporary.connectionString, { prepare }); + clients.push(db); captures.push(capture); + expect(installDatabaseResponseCapture(db, { table, encodedTable }, capture)).toBe(db); + expect(installDatabaseResponseCapture(db, { table, encodedTable }, capture)).toBe(db); + return { db, capture, events }; + } + function summary(events: Array>) { return events.find(event => event.event === "paperclip.db_response_capture.summary.v1")!; } + + it("retains capture counts when ORM preparation is enabled", async () => { + const f = fixture(true), id = randomUUID(), payload = "prepared-capture-value"; + await f.db.insert(table).values({ id, payload }); + await f.db.transaction(async tx => { + expect(await tx.select().from(table).where(eq(table.id, id))).toEqual([{ id, payload }]); + }); + f.capture.flush(); + expect(summary(f.events)).toMatchObject({ startedQueries: 2, completedQueries: 2 }); + expect(JSON.stringify(f.events)).not.toContain(payload); + expect(JSON.stringify(f.events)).not.toContain(id); + }); + + it("keeps rows and builder identity through lazy select and write returning", async () => { + const f = fixture(), id = randomUUID(), payload = "sentinel-result-and-parameter-value🙂\n\""; + const insert = f.db.insert(table).values({ id, payload }); + expect(insert.returning()).toBe(insert); + expect(await insert).toEqual([{ id, payload }]); + const query = f.db.select().from(table); + expect(query.where(eq(table.id, id))).toBe(query); + expect(await query).toEqual([{ id, payload }]); + const updated = await f.db.update(table).set({ payload: "changed" }).where(eq(table.id, id)).returning(); + expect(updated).toEqual([{ id, payload: "changed" }]); + expect(await f.db.delete(table).where(eq(table.id, id)).returning()).toEqual([{ id, payload: "changed" }]); + f.capture.flush(); + expect(summary(f.events)).toMatchObject({ startedQueries: 4, completedQueries: 4 }); + expect(f.events.filter(event => event.event === "paperclip.db_response_capture.entry.v1").map(event => event.operation).sort()) + .toEqual(["delete", "insert", "select", "update"]); + expect(JSON.stringify(f.events)).not.toContain(payload); + expect(JSON.stringify(f.events)).not.toContain(id); + }); + it("counts the stored encoding before a custom column decoder changes the value", async () => { + const setup = createDb(temporary.connectionString); clients.push(setup); + const id = randomUUID(), payload = { text: "codec-payload-sentinel\u0000value".repeat(512) }; + await setup.insert(encodedTable).values({ id, payload }); + const f = fixture(); + expect(await f.db.select().from(encodedTable).where(eq(encodedTable.id, id))).toEqual([{ id, payload }]); + f.capture.flush(); + const entry = f.events.find(event => event.table === "response_capture_encoded_fixture")!; + expect(entry.responseValueBytes).toBe(Buffer.byteLength(id) + Buffer.byteLength(encodeRunEventPayload(payload))); + expect(entry.responseValueBytes).toBeGreaterThan(Buffer.byteLength(JSON.stringify({ id, payload }))); + expect(JSON.stringify(f.events)).not.toContain("codec-payload-sentinel"); + }); + it("keeps a raw driver query lazy and counts repeated awaits without replay", async () => { + const f = fixture(), id = randomUUID(); + const observer = createDb(temporary.connectionString); clients.push(observer); + const pending = f.db.$client.unsafe("insert into response_capture_fixture (id, payload) values ($1, $2) returning id", [id, "raw-lazy-sentinel"]); + expect(await observer.select().from(table).where(eq(table.id, id))).toEqual([]); + const first = await pending; + expect(await pending).toBe(first); + expect(first).toEqual([{ id }]); + expect(await observer.select().from(table).where(eq(table.id, id))).toEqual([{ id, payload: "raw-lazy-sentinel" }]); + f.capture.flush(); + expect(summary(f.events)).toMatchObject({ startedQueries: 1, completedQueries: 1 }); + expect(JSON.stringify(f.events)).not.toContain("raw-lazy-sentinel"); + expect(JSON.stringify(f.events)).not.toContain(id); + }); + it("counts prepared executions once and retains their parameter bindings", async () => { + const f = fixture(), id = randomUUID(); + await f.db.insert(table).values({ id, payload: "prepared-value" }); + const prepared = f.db.select().from(table).where(eq(table.id, sql.placeholder("id"))).prepare("response_capture_prepared"); + expect(await prepared.execute({ id })).toEqual([{ id, payload: "prepared-value" }]); + expect(await prepared.execute({ id: randomUUID() })).toEqual([]); + f.capture.flush(); + expect(summary(f.events)).toMatchObject({ startedQueries: 3, completedQueries: 3 }); + expect(JSON.stringify(f.events)).not.toContain("prepared-value"); + }); + it("retains distinct builder callers through normal, prepared, and transaction execution", async () => { + const f = fixture(), id = randomUUID(); + const setup = createDb(temporary.connectionString); clients.push(setup); + await setup.insert(table).values({ id, payload: "caller-sentinel" }); + const shape = f.db.select().from(table); + const caller = (name: string) => compileFunction( + "return db.select().from(table).where(eq(table.id, id))", ["db", "id"], + { filename: `/fixture/server/src/services/${name}.ts`, contextExtensions: [{ table, eq }] }, + ) as (db: Pick, id: string) => typeof shape; + const fromA = caller("capture-caller-a"), fromB = caller("capture-caller-b"); + const normalA = fromA(f.db, id), normalB = fromB(f.db, id); + expect(await normalA).toEqual([{ id, payload: "caller-sentinel" }]); + expect(await normalB).toEqual([{ id, payload: "caller-sentinel" }]); + const prepared = fromA(f.db, id).prepare("capture_caller_prepared"); + expect(await prepared.execute()).toEqual([{ id, payload: "caller-sentinel" }]); + await f.db.transaction(async tx => { + expect(await fromB(tx, id)).toEqual([{ id, payload: "caller-sentinel" }]); + }); + f.capture.flush(); + const entries = f.events.filter(event => event.event === "paperclip.db_response_capture.entry.v1"); + expect(entries).toHaveLength(2); + for (const name of ["a", "b"]) { + expect(entries.find(event => (event.callers as string[]).some(location => location.startsWith(`services/capture-caller-${name}.ts:`)))) + .toMatchObject({ operation: "select", table: "response_capture_fixture", calls: 2, rows: 2 }); + } + expect(summary(f.events)).toMatchObject({ startedQueries: 4, completedQueries: 4 }); + expect(JSON.stringify(f.events)).not.toContain("/fixture/"); + expect(JSON.stringify(f.events)).not.toContain("caller-sentinel"); + expect(JSON.stringify(f.events)).not.toContain(id); + }); + it("records the executed CTE rather than its unexecuted inner select", async () => { + const f = fixture(), id = randomUUID(); + await f.db.insert(table).values({ id, payload: "cte-value" }); + const cte = f.db.$with("capture_cte").as(f.db.select().from(table).where(eq(table.id, id))); + expect(await f.db.with(cte).select().from(cte)).toEqual([{ id, payload: "cte-value" }]); + f.capture.flush(); + expect(summary(f.events)).toMatchObject({ startedQueries: 2, completedQueries: 2 }); + }); + it("keeps raw execute results and rejection behavior without logging SQL or errors", async () => { + const f = fixture(); + expect(await f.db.execute(sql`select ${"sentinel-sql-parameter"}::text as value`)).toEqual([{ value: "sentinel-sql-parameter" }]); + await expect(f.db.execute(sql`select 1/0`)).rejects.toThrow(); + f.capture.flush(); + expect(summary(f.events)).toMatchObject({ startedQueries: 2, completedQueries: 2 }); + expect(f.events.find(event => event.operation === "execute")).toMatchObject({ queryErrors: 1 }); + expect(JSON.stringify(f.events)).not.toContain("sentinel-sql-parameter"); + expect(JSON.stringify(f.events)).not.toContain("select 1/0"); + }); + it("preserves nested commit, rollback, and durable work-signal boundaries", async () => { + const f = fixture(), id = randomUUID(), events: string[] = []; + const unsubscribe = subscribeDatabaseWork(f.db, "capture-topic", event => events.push(event)); + try { + expect(await f.db.transaction(async tx => { + await tx.transaction(async nested => { + await signalDatabaseWork(nested, "capture-topic"); + await nested.insert(table).values({ id, payload: "committed" }); + }); + expect(events).toEqual(["intent"]); expect(databaseWorkPending(f.db, "capture-topic")).toBe(true); + return 42; + })).toBe(42); + expect(events).toEqual(["intent", "settled"]); + expect(databaseWorkPending(f.db, "capture-topic")).toBe(false); + const rolledBack = randomUUID(); + await expect(f.db.transaction(async tx => { + await signalDatabaseWork(tx, "capture-topic"); + await tx.insert(table).values({ id: rolledBack, payload: "rolled-back" }); + throw new Error("rollback fixture"); + })).rejects.toThrow("rollback fixture"); + await reconcileDatabaseWork(f.db, "capture-topic"); + expect(databaseWorkPending(f.db, "capture-topic")).toBe(false); + expect(await f.db.select().from(table).where(eq(table.id, rolledBack))).toEqual([]); + expect(await f.db.select().from(table).where(eq(table.id, id))).toEqual([{ id, payload: "committed" }]); + } finally { unsubscribe(); } + f.capture.flush(); + expect(summary(f.events).completedQueries).toBeGreaterThan(4); + }); + it.each([undefined, true])("covers automatically created dedicated pools with prepare %s and retains owner identity", async prepare => { + const logs: string[] = [], now = Date.now(); + vi.spyOn(console, "info").mockImplementation(value => logs.push(String(value))); + vi.stubEnv("PAPERCLIP_DB_RESPONSE_CAPTURE_WINDOW", `${new Date(now - 100).toISOString()}/${new Date(now + 59_000).toISOString()}`); + const db = createDb(temporary.connectionString, { prepare }); clients.push(db); + const capture = databaseResponseCaptureFromEnv()!; captures.push(capture); + expect(await withDedicatedDbConnection(db, async dedicated => dedicated.execute(sql`select ${"dedicated-value"}::text as value`))) + .toEqual([{ value: "dedicated-value" }]); + capture.flush(); + await vi.waitFor(() => expect(logs.some(line => JSON.parse(line).event === "paperclip.db_response_capture.summary.v1")).toBe(true), + { timeout: 1_000, interval: 5 }); + const events = logs.map(line => JSON.parse(line) as Record); + expect(summary(events)).toMatchObject({ startedQueries: 1, completedQueries: 1 }); + expect(logs.join("\n")).not.toContain("dedicated-value"); + expect(logs.join("\n")).not.toContain(temporary.connectionString); + }); +}); diff --git a/packages/db/src/response-capture.test.ts b/packages/db/src/response-capture.test.ts new file mode 100644 index 00000000000..73d25d8fc3c --- /dev/null +++ b/packages/db/src/response-capture.test.ts @@ -0,0 +1,203 @@ +import { integer, pgTable } from "drizzle-orm/pg-core"; +import { afterEach, describe, expect, it, vi } from "vitest"; +import { createDatabaseResponseCapture, databaseResponseCaptureFromEnv, installDatabaseResponseCapture, publicDatabaseCallers, responseCaptureWindow } from "./response-capture.js"; + +const epoch = Date.parse("2020-01-01T00:00:00Z"); +const metadata = { operation: "select", table: "capture_fixture", callers: ["services/fixture.ts:4:2"] }; +const table = pgTable("capture_fixture", { id: integer("id") }); +const captures: Array> = []; +afterEach(() => { for (const capture of captures.splice(0)) capture.flush(); vi.unstubAllEnvs(); }); +function fixture(emitOverride?: (event: Record) => void) { + let clock = epoch; + const events: Array> = []; + const capture = createDatabaseResponseCapture({ start: epoch, end: epoch + 60_000 }, emitOverride ?? (event => events.push(event)), () => clock); + captures.push(capture); + return { capture, events, now: (value: number) => { clock = value; } }; +} + +describe("bounded database response capture", () => { + it("paces a complete environment capture without extending the measurement window or leaving timers", () => { + vi.useFakeTimers(); + vi.setSystemTime(epoch); + const emitted: Array<{ at: number; event: Record }> = []; + const log = vi.spyOn(console, "info").mockImplementation(value => emitted.push({ at: Date.now(), event: JSON.parse(value) })); + try { + const capture = databaseResponseCaptureFromEnv({ PAPERCLIP_DB_RESPONSE_CAPTURE_WINDOW: "2020-01-01T00:00:00Z/2020-01-01T00:01:00Z" })!; + captures.push(capture); + const rows = [{ secret: "capture-response-sentinel" }]; + for (let index = 0; index < 2200; index++) { + expect(capture.execute(() => rows, { ...metadata, callers: [`services/capture.ts:${index + 1}:1`] })).toBe(rows); + } + vi.advanceTimersByTime(60_001); + expect(emitted.length).toBeGreaterThan(0); + expect(emitted.length).toBeLessThanOrEqual(64); + expect(capture.active()).toBe(false); + expect(capture.execute(() => rows, metadata)).toBe(rows); + capture.flush(); + vi.advanceTimersByTime(32_000); + expect(emitted).toHaveLength(2049); + const groups = emitted.filter(item => item.event.event === "paperclip.db_response_capture.entry.v1"); + const summary = emitted.at(-1)!.event; + expect(groups).toHaveLength(2048); + expect(summary).toMatchObject({ event: "paperclip.db_response_capture.summary.v1", completedQueries: 2200, + groups: 2048, closedAt: "2020-01-01T00:01:00.000Z" }); + expect(groups.reduce((calls, item) => calls + Number(item.event.calls), 0)).toBe(summary.completedQueries); + expect(groups.reduce((bytes, item) => bytes + Number(item.event.responseValueBytes), 0)).toBe(summary.responseValueBytes); + expect(groups.map(item => item.event.groupIndex)).toEqual(Array.from({ length: 2048 }, (_, index) => index)); + for (const { at } of emitted) expect(emitted.filter(item => item.at >= at && item.at < at + 1000).length).toBeLessThanOrEqual(64); + expect(JSON.stringify(emitted)).not.toContain("capture-response-sentinel"); + expect(vi.getTimerCount()).toBe(0); + vi.advanceTimersByTime(60_000); + expect(emitted).toHaveLength(2049); + } finally { log.mockRestore(); vi.clearAllTimers(); vi.useRealTimers(); } + }); + + it("contains deferred logger failures and finishes its bounded output", () => { + vi.useFakeTimers(); + vi.setSystemTime(epoch + 120_000); + const log = vi.spyOn(console, "info").mockImplementation(() => { throw new Error("fixture logger failure"); }); + try { + const capture = databaseResponseCaptureFromEnv({ PAPERCLIP_DB_RESPONSE_CAPTURE_WINDOW: "2020-01-01T00:02:00Z/2020-01-01T00:03:00Z" })!; + captures.push(capture); + expect(capture.execute(() => 7, metadata)).toBe(7); + expect(() => vi.advanceTimersByTime(60_001)).not.toThrow(); + expect(log).toHaveBeenCalledTimes(2); + expect(vi.getTimerCount()).toBe(0); + } finally { log.mockRestore(); vi.clearAllTimers(); vi.useRealTimers(); } + }); + + it.each([undefined, "", "sentinel-invalid-input", "2020-01-01T00:00:00Z", "2020-01-01T00:00:00Z/2020-01-01T00:00:00Z", + "2020-01-01T00:00:00Z/2020-01-01T00:01:01Z", "2020-01-01T01:00:00Z/2020-01-01T01:01:00Z", + "2020-01-01T00:00:00Z/2020-01-01T00:01:00Z/extra", + ])("disables absent, malformed, expired, or unbounded windows: %s", value => { + expect(responseCaptureWindow(value, epoch + 60_000)).toBeUndefined(); + }); + it("accepts a fixed future window of at most one minute", () => { + expect(responseCaptureWindow("2020-01-01T00:01:00Z/2020-01-01T00:02:00Z", epoch)) + .toEqual({ start: epoch + 60_000, end: epoch + 120_000 }); + expect(() => createDatabaseResponseCapture({ start: epoch, end: epoch + 60_001 }, () => {})).toThrow("Invalid"); + }); + it("keeps only public server file locations from stacks", () => { + const stack = ["Error: sentinel-message", "at f (/private/fixture/server/src/services/example.ts:4:2)", + "at f (/private/fixture/server/src/services/example.ts:4:2)", "at f (/private/fixture/server/dist/routes/example.js:5:3)", + "at f (/private/fixture/customer-file.js:7:9)", "at f (/app/server/src/services/../../customer-file.ts:8:1)"].join("\n"); + expect(publicDatabaseCallers(stack)).toEqual(["services/example.ts:4:2", "routes/example.js:5:3"]); + }); + it("leaves methods and identity unchanged when disabled", () => { + vi.stubEnv("PAPERCLIP_DB_RESPONSE_CAPTURE_WINDOW", undefined); + const db = { select: vi.fn(), transaction: vi.fn() }, select = db.select, transaction = db.transaction; + expect(installDatabaseResponseCapture(db, { table })).toBe(db); + expect(db.select).toBe(select); expect(db.transaction).toBe(transaction); + }); + it("preserves exact values and only emits counts, sizes, and caller metadata", async () => { + const f = fixture(), rows = [{ text: "sentinel-result-value🙂\n\"", n: 4n }]; + expect(await f.capture.execute(() => Promise.resolve(rows), metadata)).toBe(rows); + f.capture.flush(); + const entry = f.events.find(event => event.event === "paperclip.db_response_capture.entry.v1")!; + expect(entry).toMatchObject({ calls: 1, rows: 1, queryErrors: 0, serializationErrors: 0, + responseValueBytes: Buffer.byteLength(JSON.stringify(rows, (_, value) => typeof value === "bigint" ? value.toString() : value)) }); + expect(JSON.stringify(f.events)).not.toContain("sentinel-result-value"); + expect(f.events.at(-1)).toMatchObject({ startedQueries: 1, completedQueries: 1, inFlightAtEnd: 0, + rowValuesLogged: false, sqlOrParametersLogged: false, wireBytesMeasured: false }); + }); + it("counts driver text values without adding JSON escaping or column names", async () => { + const f = fixture(), text = "sentinel-driver-text\n\""; + const rows = Object.assign([[text, true, 42n, null]], { columns: [ + { name: "sentinel-column-name" }, { name: "b" }, { name: "n" }, { name: "empty" } ] }); + expect(await f.capture.execute(() => Promise.resolve(rows), metadata)).toBe(rows); + f.capture.flush(); + expect(f.events[0]).toMatchObject({ responseValueBytes: Buffer.byteLength(text) + 1 + 2, estimatedValues: 0 }); + expect(JSON.stringify(f.events)).not.toContain("sentinel-driver-text"); + expect(JSON.stringify(f.events)).not.toContain("sentinel-column-name"); + }); + it("does not alter synchronous or asynchronous errors or retry them", async () => { + const f = fixture(), error = new Error("sentinel-private-error"); + const sync = vi.fn(() => { throw error; }), asyncFailure = vi.fn(() => Promise.reject(error)); + expect(() => f.capture.execute(sync, metadata)).toThrow(error); + await expect(f.capture.execute(asyncFailure, metadata)).rejects.toBe(error); + expect(sync).toHaveBeenCalledTimes(1); expect(asyncFailure).toHaveBeenCalledTimes(1); + f.capture.flush(); + expect(f.events[0]).toMatchObject({ calls: 2, queryErrors: 2, responseValueBytes: 0 }); + expect(JSON.stringify(f.events)).not.toContain("sentinel-private-error"); + }); + it("keeps unserializable results and logging failures out of the query outcome", () => { + const f = fixture(), circular: Record = {}; circular.self = circular; + expect(f.capture.execute(() => circular, metadata)).toBe(circular); + f.capture.flush(); + expect(f.events[0]).toMatchObject({ calls: 1, serializationErrors: 1 }); + const brokenLogger = fixture(() => { throw new Error("logger fixture"); }); + expect(brokenLogger.capture.execute(() => circular, metadata)).toBe(circular); + expect(() => brokenLogger.capture.flush()).not.toThrow(); + }); + it("stops accounting at the deadline and leaves in-flight work untouched", async () => { + const f = fixture(); + let resolve!: (rows: unknown[]) => void; + const pending = new Promise(done => { resolve = done; }); + const result = f.capture.execute(() => pending, metadata); + f.now(epoch + 60_000); f.capture.flush(); + expect(f.events.at(-1)).toMatchObject({ startedQueries: 1, completedQueries: 0, inFlightAtEnd: 1 }); + const rows = [{ marker: "late-result-value" }]; resolve(rows); + expect(await result).toBe(rows); + const direct = Promise.resolve(rows); + expect(f.capture.execute(() => direct, metadata)).toBe(direct); + expect(f.events).toHaveLength(1); + }); + it("retains the deadline count when a query resolves before a delayed flush", async () => { + const f = fixture(); + let resolve!: (rows: unknown[]) => void; + const pending = new Promise(done => { resolve = done; }); + const result = f.capture.execute(() => pending, metadata); + f.now(epoch + 60_001); + const rows = [{ marker: "late-before-flush-sentinel" }]; resolve(rows); + expect(await result).toBe(rows); + f.capture.flush(); + expect(f.events).toHaveLength(1); + expect(f.events[0]).toMatchObject({ startedQueries: 1, completedQueries: 0, inFlightAtEnd: 1, responseValueBytes: 0 }); + expect(JSON.stringify(f.events)).not.toContain("late-before-flush-sentinel"); + }); + it("bounds cardinality without dropping response-byte totals", () => { + const f = fixture(); + for (let i = 0; i < 2200; i += 1) f.capture.execute(() => [i], { ...metadata, callers: [`services/f${i}.ts:1:1`] }); + f.capture.flush(); + const summary = f.events.at(-1)!; + expect(summary).toMatchObject({ startedQueries: 2200, completedQueries: 2200, groups: 2048, overflowCalls: 153, + groupPageSize: 128, groupPages: 16 }); + expect(f.events).toHaveLength(2049); + const entries = f.events.slice(0, -1); + expect(entries.map(entry => entry.groupIndex)).toEqual(Array.from({ length: 2048 }, (_, index) => index)); + for (let page = 0; page < 16; page += 1) { + expect(entries.filter(entry => entry.groupPage === page)).toHaveLength(128); + } + expect(entries.reduce((sum, entry) => sum + Number(entry.calls), 0)).toBe(2200); + expect(summary.responseValueBytes).toBe(Array.from({ length: 2200 }, (_, i) => Buffer.byteLength(JSON.stringify([i]))).reduce((a, b) => a + b, 0)); + }); + it.each([0, 129])("numbers every page including an empty or partial final page (%s)", count => { + const f = fixture(); + for (let index = 0; index < count; index += 1) { + f.capture.execute(() => [index], { ...metadata, callers: [`services/page${index}.ts:1:1`] }); + } + f.capture.flush(); + const entries = f.events.slice(0, -1), summary = f.events.at(-1)!; + expect(summary).toMatchObject({ groups: count, groupPageSize: 128, groupPages: Math.ceil(count / 128) }); + expect(entries.map(entry => entry.groupIndex)).toEqual(Array.from({ length: count }, (_, index) => index)); + expect(entries.filter(entry => entry.groupPage === 0)).toHaveLength(Math.min(count, 128)); + expect(entries.filter(entry => entry.groupPage === 1)).toHaveLength(Math.max(0, count - 128)); + }); + it("does not execute builders or prepared statements until requested", async () => { + const f = fixture(), rows = [{ id: 1 }], execute = vi.fn(async () => rows); + const query = { execute, prepare: () => ({ execute }) }; + const builder = { from: () => query }; + const db = { select: () => builder }; + expect(installDatabaseResponseCapture(db, { table }, f.capture)).toBe(db); + expect(db.select()).toBe(builder); + expect(db.select().from()).toBe(query); + expect(execute).not.toHaveBeenCalled(); + const prepared = query.prepare(); + expect(execute).not.toHaveBeenCalled(); + expect(await prepared.execute()).toBe(rows); + expect(await query.execute()).toBe(rows); + expect(execute).toHaveBeenCalledTimes(2); + f.capture.flush(); + expect(f.events.at(-1)).toMatchObject({ completedQueries: 0 }); + }); +}); diff --git a/packages/db/src/response-capture.ts b/packages/db/src/response-capture.ts new file mode 100644 index 00000000000..d53b5fe0c07 --- /dev/null +++ b/packages/db/src/response-capture.ts @@ -0,0 +1,299 @@ +import { AsyncLocalStorage } from "node:async_hooks"; +import { randomUUID } from "node:crypto"; +import { getTableName, is, Table } from "drizzle-orm"; + +type Metadata = { operation: string; table: string; callers: string[] }; +type Entry = Metadata & { calls: number; rows: number; bytes: number; maxBytes: number; queryErrors: number; serializationErrors: number; estimatedValues: number }; +type Event = Record; +type Method = (this: unknown, ...args: unknown[]) => unknown; +export type ResponseCaptureWindow = { start: number; end: number }; +const GROUP_PAGE_SIZE = 128; + +export function responseCaptureWindow(value: string | undefined, now = Date.now()): ResponseCaptureWindow | undefined { + if (!value) return; + const parts = value.split("/"); + if (parts.length !== 2 || parts.some(part => !/^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d{3})?Z$/.test(part))) return; + const start = Date.parse(parts[0]!), end = Date.parse(parts[1]!); + if (!Number.isFinite(start) || !Number.isFinite(end) || end <= start || end - start > 60_000 || end <= now || start > now + 30 * 60_000) return; + return { start, end }; +} + +export function publicDatabaseCallers(stack: string | undefined): string[] { + const callers: string[] = []; + for (const line of (stack ?? "").split("\n")) { + const match = line.match(/[/\\]server[/\\](?:src|dist)[/\\]((?:services|routes|modules|middleware)[/\\][A-Za-z0-9_./\\-]+\.[cm]?[jt]s):(\d+):(\d+)/); + if (!match) continue; + const relative = match[1]!.replaceAll("\\", "/"); + if (relative.split("/").includes("..")) continue; + const caller = `${relative}:${match[2]}:${match[3]}`; + if (!callers.includes(caller)) callers.push(caller); + if (callers.length === 4) break; + } + return callers; +} + +function responseValueSize(value: unknown): { bytes: number; estimatedValues: number } { + const columns = value && typeof value === "object" ? (value as { columns?: unknown }).columns : undefined; + if (!Array.isArray(value) || !Array.isArray(columns)) { + const serialized = JSON.stringify(value, (_, item: unknown) => typeof item === "bigint" ? item.toString() : item); + return { bytes: serialized === undefined ? 0 : Buffer.byteLength(serialized), estimatedValues: 1 }; + } + let bytes = 0, estimatedValues = 0; + for (const row of value) { + const cells = Array.isArray(row) ? row : columns.map(column => row[column.name]); + for (const cell of cells) { + if (cell === null || cell === undefined) continue; + if (typeof cell === "string") bytes += Buffer.byteLength(cell); + else if (typeof cell === "boolean") bytes += 1; + else if (typeof cell === "number" || typeof cell === "bigint") bytes += Buffer.byteLength(String(cell)); + else { + estimatedValues += 1; + const serialized = JSON.stringify(cell, (_, item: unknown) => typeof item === "bigint" ? item.toString() : item); + if (serialized !== undefined) bytes += Buffer.byteLength(serialized); + } + } + } + return { bytes, estimatedValues }; +} + +export function createDatabaseResponseCapture( + window: ResponseCaptureWindow, + emit: (event: Event) => void, + now: () => number = Date.now, +) { + if (!Number.isFinite(new Date(window.start).getTime()) || !Number.isFinite(new Date(window.end).getTime()) || + window.end <= window.start || window.end - window.start > 60_000) throw new Error("Invalid database response capture window"); + const id = randomUUID(), entries = new Map(); + const roots = new WeakSet(), queries = new WeakSet(), factories = new WeakSet(); + const clients = new WeakSet(), driverQueries = new WeakSet(); + const scope = new AsyncLocalStorage(); + let closed = false, inFlight = 0, started = 0, completed = 0, overflowCalls = 0; + const send = (event: Event) => { try { emit(event); } catch {} }; + const expired = () => closed || now() >= window.end; + const active = () => !expired() && now() >= window.start; + const flush = () => { + if (closed) return; + closed = true; + clearTimeout(timer); + scope.disable(); + const ordered = [...entries.values()].sort((a, b) => b.bytes - a.bytes); + for (const [groupIndex, entry] of ordered.entries()) send({ event: "paperclip.db_response_capture.entry.v1", captureId: id, + groupIndex, groupPage: Math.floor(groupIndex / GROUP_PAGE_SIZE), + operation: entry.operation, table: entry.table, callers: entry.callers, calls: entry.calls, rows: entry.rows, + responseValueBytes: entry.bytes, maxResponseValueBytes: entry.maxBytes, estimatedValues: entry.estimatedValues, + queryErrors: entry.queryErrors, serializationErrors: entry.serializationErrors }); + send({ event: "paperclip.db_response_capture.summary.v1", captureId: id, + windowStart: new Date(window.start).toISOString(), windowEnd: new Date(window.end).toISOString(), + closedAt: new Date(now()).toISOString(), + startedQueries: started, completedQueries: completed, inFlightAtEnd: inFlight, groups: entries.size, + groupPageSize: GROUP_PAGE_SIZE, groupPages: Math.ceil(ordered.length / GROUP_PAGE_SIZE), + overflowCalls, responseValueBytes: ordered.reduce((sum, entry) => sum + entry.bytes, 0), + rowValuesLogged: false, sqlOrParametersLogged: false, wireBytesMeasured: false, + measurementBoundary: "postgres-js results before Drizzle column decoding", + relationalQueriesMeasured: true, unsafeDriverQueriesMeasured: true, + taggedDriverQueriesMeasured: false, streamingQueriesMeasured: false }); + entries.clear(); + }; + const timer = setTimeout(flush, Math.max(0, window.end - now())); + timer.unref(); + function finish(metadata: Metadata, value: unknown, failed: boolean) { + if (!active()) return; + inFlight -= 1; + completed += 1; + try { + let key = JSON.stringify(metadata); + if (!entries.has(key) && entries.size >= 2047) { + overflowCalls += 1; + metadata = { operation: "other", table: "other", callers: [] }; + key = "overflow"; + } + let entry = entries.get(key); + if (!entry) { + entry = { ...metadata, calls: 0, rows: 0, bytes: 0, maxBytes: 0, queryErrors: 0, serializationErrors: 0, estimatedValues: 0 }; + entries.set(key, entry); + } + entry.calls += 1; + if (failed) { entry.queryErrors += 1; return; } + entry.rows += Array.isArray(value) ? value.length : 0; + try { + const size = responseValueSize(value); + entry.bytes += size.bytes; + entry.maxBytes = Math.max(entry.maxBytes, size.bytes); + entry.estimatedValues += size.estimatedValues; + } catch { entry.serializationErrors += 1; } + } catch {} + } + function execute(action: () => unknown, metadata: Metadata) { + if (!active()) return action(); + inFlight += 1; started += 1; + let result: unknown; + try { result = action(); } + catch (error) { finish(metadata, undefined, true); throw error; } + let asynchronous = false; + try { asynchronous = Boolean(result && typeof (result as { then?: unknown }).then === "function"); } catch {} + if (asynchronous) { + return Promise.resolve(result).then(value => { finish(metadata, value, false); return value; }, + error => { finish(metadata, undefined, true); throw error; }); + } + finish(metadata, result, false); + return result; + } + return { window, roots, queries, factories, clients, driverQueries, scope, active, expired, execute, flush }; +} + +type Capture = ReturnType; +let environmentCapture: { key: string; capture: Capture } | undefined; + +function pacedCaptureLogger() { + const pending: Event[] = []; + let timer: ReturnType | undefined; + const drain = () => { + timer = undefined; + for (const event of pending.splice(0, 64)) { + try { console.info(JSON.stringify(event)); } catch {} + } + if (pending.length) { + timer = setTimeout(drain, 1_000); + timer.unref(); + } + }; + return (event: Event) => { + pending.push(event); + if (!timer) { + timer = setTimeout(drain, 0); + timer.unref(); + } + }; +} + +export function databaseResponseCaptureFromEnv(env: NodeJS.ProcessEnv = process.env): Capture | undefined { + const key = env.PAPERCLIP_DB_RESPONSE_CAPTURE_WINDOW; + const window = responseCaptureWindow(key); + if (!key || !window) return; + if (environmentCapture?.key === key) return environmentCapture.capture; + const capture = createDatabaseResponseCapture(window, pacedCaptureLogger()); + environmentCapture = { key, capture }; + return capture; +} + +export function installDatabaseResponseCapture(db: T, schema: Record, capture = databaseResponseCaptureFromEnv()): T { + if (!capture || capture.expired()) return db; + const tables = new Set(Object.values(schema).filter(value => is(value, Table)).map(table => getTableName(table))); + const object = (value: unknown): value is Record => value !== null && typeof value === "object"; + function replace(target: Record, name: string, replacement: (original: Method) => Method) { + const original = target[name]; + if (typeof original !== "function") return; + try { target[name] = replacement(original as Method); } catch {} + } + function tableName(value: unknown) { + try { if (is(value, Table)) { const name = getTableName(value); if (tables.has(name)) return name; } } catch {} + return "other"; + } + function query(value: unknown, metadata: Metadata) { + if (!object(value) || capture!.queries.has(value)) return; + capture!.queries.add(value); + replace(value, "execute", original => function (this: unknown, ...args: unknown[]) { + if (capture!.expired()) return Reflect.apply(original, this, args); + return capture!.scope.run(metadata, () => Reflect.apply(original, this, args)); + }); + replace(value, "prepare", original => function (this: unknown, ...args: unknown[]) { + const prepared = Reflect.apply(original, this, args); + try { query(prepared, metadata); } catch {} + return prepared; + }); + } + function factory(value: unknown, step: string, metadata: Metadata, from = false) { + if (!object(value) || capture!.factories.has(value)) return; + capture!.factories.add(value); + replace(value, step, original => function (this: unknown, ...args: unknown[]) { + const result = Reflect.apply(original, this, args); + try { query(result, { ...metadata, table: from ? tableName(args[0]) : metadata.table }); } catch {} + return result; + }); + } + function driverQuery(value: unknown, metadata: Metadata) { + if (!object(value) || capture!.driverQueries.has(value)) return; + capture!.driverQueries.add(value); + let consumed = false; + replace(value, "then", original => function (this: unknown, ...handlers: unknown[]) { + if (consumed || !capture!.active()) { + consumed = true; + return Reflect.apply(original, this, handlers); + } + consumed = true; + const result = capture!.execute(() => Reflect.apply(original, this, []), metadata) as Promise; + return Reflect.apply(result.then, result, handlers); + }); + for (const method of ["execute", "cursor", "forEach", "readable", "writable"]) { + replace(value, method, original => function (this: unknown, ...args: unknown[]) { + consumed = true; + return Reflect.apply(original, this, args); + }); + } + } + function sqlMetadata(value: unknown): Metadata { + const scoped = capture!.scope.getStore(); + const text = typeof value === "string" ? value : ""; + let table = "other"; + for (const match of text.matchAll(/\b(?:from|update|into)\s+"([A-Za-z0-9_]+)"/gi)) { + if (tables.has(match[1]!)) { table = match[1]!; break; } + } + return scoped ? { ...scoped, table: scoped.table === "other" ? table : scoped.table } + : { operation: text.match(/^\s*(select|insert|update|delete)\b/i)?.[1]?.toLowerCase() ?? "execute", + table, callers: publicDatabaseCallers(new Error().stack) }; + } + function client(value: unknown) { + if ((!object(value) && typeof value !== "function") || capture!.clients.has(value)) return; + capture!.clients.add(value); + const target = value as Record; + replace(target, "unsafe", original => function (this: unknown, ...args: unknown[]) { + const result = Reflect.apply(original, this, args); + if (!capture!.expired()) { try { driverQuery(result, sqlMetadata(args[0])); } catch {} } + return result; + }); + for (const method of ["begin", "savepoint"]) { + replace(target, method, original => function (this: unknown, ...args: unknown[]) { + const wrapped = args.map(arg => typeof arg !== "function" ? arg : function (this: unknown, ...inner: unknown[]) { + try { client(inner[0]); } catch {} + return Reflect.apply(arg, this, inner); + }); + return Reflect.apply(original, this, wrapped); + }); + } + replace(target, "reserve", original => function (this: unknown, ...args: unknown[]) { + return Promise.resolve(Reflect.apply(original, this, args)).then(reserved => { + try { client(reserved); } catch {} + return reserved; + }); + }); + } + function root(value: unknown) { + if (!object(value) || capture!.roots.has(value)) return; + capture!.roots.add(value); + for (const method of ["select", "selectDistinct", "selectDistinctOn", "insert", "update", "delete", "execute", "with"] as const) { + replace(value, method, original => function (this: unknown, ...args: unknown[]) { + const result = Reflect.apply(original, this, args); + if (capture!.expired()) return result; + try { + if (method === "with") { root(result); return result; } + const operation = method.startsWith("select") ? "select" : method; + const metadata = { operation, table: operation === "select" || method === "execute" ? "other" : tableName(args[0]), callers: publicDatabaseCallers(new Error().stack) }; + if (operation === "select") factory(result, "from", metadata, true); + else if (method === "insert") factory(result, "values", metadata); + else if (method === "update") factory(result, "set", metadata); + else query(result, metadata); + } catch {} + return result; + }); + } + replace(value, "transaction", original => function (this: unknown, callback: unknown, ...options: unknown[]) { + if (typeof callback !== "function" || capture!.expired()) return Reflect.apply(original, this, [callback, ...options]); + return Reflect.apply(original, this, [function (this: unknown, ...args: unknown[]) { + try { root(args[0]); } catch {} + return Reflect.apply(callback, this, args); + }, ...options]); + }); + } + try { client((db as Record).$client); root(db); } catch {} + return db; +} diff --git a/packages/paperclip-runner/acpx-profiles.json b/packages/paperclip-runner/acpx-profiles.json index 6b28b8dad51..fdf1b06255c 100644 --- a/packages/paperclip-runner/acpx-profiles.json +++ b/packages/paperclip-runner/acpx-profiles.json @@ -14,12 +14,12 @@ "requiresProviderPolicy": false }, "pi": { - "agentProfileVersion": 28, + "agentProfileVersion": 29, "agentServerPackage": "pi-acp", "agentServerVersion": "0.0.33", "agentRuntimePackage": "@earendil-works/pi-coding-agent", "agentRuntimeVersion": "1.0.0", - "commandDigest": "sha256:4918a4d5472d66b22b1ec642b68b5c60a8092a39c84ec1ab1c9ad0289cb8c68c", + "commandDigest": "sha256:e80c6dc7bde12790371345ee5a12f0201d30941d26f1a780aec3d090545b2a06", "requiresProviderPolicy": true }, "cursor": { @@ -32,12 +32,12 @@ "requiresProviderPolicy": true }, "copilot": { - "agentProfileVersion": 58, + "agentProfileVersion": 59, "agentServerPackage": "@github/copilot", "agentServerVersion": "1.0.88", "agentRuntimePackage": null, "agentRuntimeVersion": null, - "commandDigest": "sha256:d490dacdab02051d24c84aae49ac6fb12b8226cf8a47b014ec53f3d3a8410a94", + "commandDigest": "sha256:2f39ffe54a6af832d044b5098c31df9e08d4d27aee0ac675ba4b0c9da7445c40", "requiresProviderPolicy": true }, "claude": { diff --git a/packages/paperclip-runner/docs/runner-protocol-live-evals.md b/packages/paperclip-runner/docs/runner-protocol-live-evals.md index 4a87b8a6aa2..03a7d7788b3 100644 --- a/packages/paperclip-runner/docs/runner-protocol-live-evals.md +++ b/packages/paperclip-runner/docs/runner-protocol-live-evals.md @@ -159,6 +159,84 @@ change the hosted matrix. The controller must enforce request admission, deadlines, and limits before forwarding traffic. A local fake-API probe only proves routing; it does not qualify live provider or Braintrust behavior. + +## Controlled request budgets + +The source-checkout controller is +`packages/paperclip-runner/scripts/controlled-evals/run.mjs`. Runner/evals owns +this orchestration. It reuses the existing eval request proxy and ledger from +`tests/runner-e2e` through one import graph; it does not copy their reservation +or provider-usage rules. It requires the workspace's installed `tsx` runtime. +It is not a published Runner package entry point. + +The controller accepts the existing `--request` and `--output` contract. Set +`PAPERCLIP_RUNNER_EVAL_CONTROL_MANIFEST` to an explicit reviewed manifest and +supply the selected upstream credential only to this trusted controller. For +Runner Evals, pass this controller path as `eval_program.py run --runner-cli`; +the manifest identifies the actual built `eval-session.js`. The provider child +receives only its temporary proxy credential and private home. + +The manifest schema is `paperclip.runner-eval.controller/v1`. It contains: + +- `campaignId`, `maxUsd`, `deadlineAt`, `pricingVerifiedAt`, and + `pricingExpiresAt`; admission must fit a current four-hour window. +- `sourceRevision` and `definitionsRevision`, each an exact commit ID. + These are the trusted preparer's provenance declarations. +- `runnerCli`, `runnerPackage`, and `runnerd`, each an absolute `path` and + content `sha256`. The controller checks the three files before launch. +- An absolute durable `outputDirectory`, outside the disposable case workspace. +- `route` (`direct` or `braintrust`), plus `projectId` for Braintrust. +- `models`, using the existing exact-model price, request-limit, and reviewed + full-lifecycle `provisionalBound` contract. +- `overhead`, using the existing four-component operating allowance with a + current full-lifecycle bound for every component. This is reserved once. +- `attempts`, each with a unique `executionId` and `attemptId`, + `requestSha256`, `provider`, optional `acpxAgent`, exact `model`, `maxUsd`, + `maxRequests`, `maxConcurrent`, and `timeoutMs`. Native Codex and ACPX + Claude are supported. Each execution has one explicit attempt and no retry. + +Use `requestBindingSha256` from `controlled-evals/manifest.mjs` to bind the +complete compiled request. It excludes only the temporary working directory +and native executable path; the executable digest remains bound and its path +is checked separately. The controller gives the child a retained copy of the +validated request. Changing prompts, fixtures, models, limits, or attempts +requires a new reviewed binding. + +The sum of every cell's allocation and the once-only operating allowance must +fit `maxUsd` before any cell starts. A request reserves against its cell's +ledger before forwarding. Cells cannot borrow unused allocations. Unknown +requests keep their reservations, and failed or missing cell evidence keeps +that cell's full allocation in the campaign report. An existing cell directory +or output is never reset for a retry. Cancellation closes admission, drains +upstream receipts within the proxy limit, and stops the owned CLI process +group. Unverified provider cleanup remains explicit and retains the allocation. + +Each result retains `controllerAccounting` alongside the original provider +outcome and usage. Retain the durable controller directory as private evidence: +its admission, request, gateway descriptor, request receipts, ledger, original +provider artifact, and final result are required for reconciliation. The +private program's existing immutable attempt copy includes the result metadata; +its current default history publisher does not archive this separate directory. +Do not infer receipt retention from that metadata alone. + +```sh +node packages/paperclip-runner/scripts/controlled-evals/report.mjs \ + /private/path/controller-manifest.json > /private/path/controller-report.json +``` + +The report recomputes ledger costs, checks their attempt/manifest bindings, and +preserves known partial charges. It reports operating costs as held and total +costs as provisional when evidence is intact, or incomplete otherwise. It does +not release historical holds or claim invoice +settlement. Existing provider usage is retained separately; do not add the +provider and controller cost observations as if they were separate charges. + +This local controller is not yet wired into the hosted matrix. Its durable +attempt-directory guard applies to one shared filesystem, not independent +GitHub workers. Hosted integration must bind unique cells to the prepared +allocation, reject workflow/job reruns, and preserve the controller evidence +before enabling paid work. Native OTLP routing, report integration, vendor +metering bounds, and actual paid qualification remain separate gates. ## Controller-assigned native observability The same CLI accepts `--observability-config /absolute/path/observability.json` @@ -292,6 +370,12 @@ hashes. These detect corruption; they are not signatures or proof against a host that can rewrite all evidence. Known configured credential values are checked before encoding and after decoding. Keep bundles private. +Controller post-child reads, archive creation, and durable reports use the same +nonblocking, no-follow, byte-capped regular-file reader. Configured private/provider +keys and credentials inside JSON configuration contents are checked in both raw +and JSON-escaped form. Attempt IDs contain only letters, digits, `_`, `.`, and `-` +so each retained attempt has one directory name. + The reader accepts only bounded regular files and fixed evidence paths. A bundle is limited to 64 MiB of decoded content plus encoding overhead; this storage-format limit does not qualify a vendor charge or operating-cost bound. diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_backend.rs b/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_backend.rs index c3f53e20b84..212852476cd 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_backend.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_backend.rs @@ -3212,7 +3212,7 @@ mod tests { #[test] fn pi_profile_matches_published_identity_and_rejects_prior_profiles() { let published: Value = serde_json::from_str(include_str!( - "../../../../test-fixtures/pi-acp/profile-v28-identity.json" + "../../../../test-fixtures/pi-acp/profile-v29-identity.json" )) .unwrap(); let mut value = descriptor("codex"); @@ -3229,6 +3229,7 @@ mod tests { current.validate(&context()).unwrap(); for prior in [ + include_str!("../../../../test-fixtures/pi-acp/profile-v28-identity.json"), include_str!("../../../../test-fixtures/pi-acp/profile-v24-identity.json"), include_str!("../../../../test-fixtures/pi-acp/profile-v23-identity.json"), include_str!("../../../../test-fixtures/pi-acp/profile-v15-identity.json"), diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/generated_acpx_profiles.rs b/packages/paperclip-runner/runner/crates/runner-core/src/generated_acpx_profiles.rs index 128ec81306e..eb6c82cefcb 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/generated_acpx_profiles.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/generated_acpx_profiles.rs @@ -29,7 +29,7 @@ pub(crate) fn acpx_release_profile(agent: &str) -> Option { agent_runtime_package: Some("@earendil-works/pi-coding-agent"), agent_runtime_version: Some("1.0.0"), command_digest: - "sha256:4918a4d5472d66b22b1ec642b68b5c60a8092a39c84ec1ab1c9ad0289cb8c68c", + "sha256:e80c6dc7bde12790371345ee5a12f0201d30941d26f1a780aec3d090545b2a06", requires_provider_policy: true, }, "cursor" => AcpxReleaseProfile { @@ -47,7 +47,7 @@ pub(crate) fn acpx_release_profile(agent: &str) -> Option { agent_runtime_package: None, agent_runtime_version: None, command_digest: - "sha256:d490dacdab02051d24c84aae49ac6fb12b8226cf8a47b014ec53f3d3a8410a94", + "sha256:2f39ffe54a6af832d044b5098c31df9e08d4d27aee0ac675ba4b0c9da7445c40", requires_provider_policy: true, }, "claude" => AcpxReleaseProfile { diff --git a/packages/paperclip-runner/runner/crates/runner-core/tests/native_provider_backend.rs b/packages/paperclip-runner/runner/crates/runner-core/tests/native_provider_backend.rs index 872935a3a03..615a2e74839 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/tests/native_provider_backend.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/tests/native_provider_backend.rs @@ -17,7 +17,7 @@ use sha2::{Digest, Sha256}; const CODEX_ACPX_DIGEST: &str = "sha256:443491f4e814501a5dc47347e4510e84faeec9c9382e2d4e53bfe69386d5aabc"; const PI_ACPX_DIGEST: &str = - "sha256:4918a4d5472d66b22b1ec642b68b5c60a8092a39c84ec1ab1c9ad0289cb8c68c"; + "sha256:e80c6dc7bde12790371345ee5a12f0201d30941d26f1a780aec3d090545b2a06"; fn temporary_directory(label: &str) -> PathBuf { let nonce = SystemTime::now() diff --git a/packages/paperclip-runner/scripts/controlled-evals/archive.mjs b/packages/paperclip-runner/scripts/controlled-evals/archive.mjs index f17ef9b882c..3d8febd4708 100644 --- a/packages/paperclip-runner/scripts/controlled-evals/archive.mjs +++ b/packages/paperclip-runner/scripts/controlled-evals/archive.mjs @@ -1,47 +1,39 @@ -import { constants } from 'node:fs'; -import { lstat, open, readdir, writeFile } from 'node:fs/promises'; +import { lstat, readdir, writeFile } from 'node:fs/promises'; import { basename, join, resolve } from 'node:path'; import { pathToFileURL } from 'node:url'; import { bindRequest, canonical, fields, readJson, requestBindingSha256, sha256, units, usd, validateManifest } from './manifest.mjs'; import { attemptDirectory, controlledResultArtifact } from './run.mjs'; import { summarizeControlledCampaign } from './report.mjs'; +import { FILE_LIMITS, RECEIPT, limitFor, readRegularFile } from './evidence-files.mjs'; const MiB = 1_048_576; const MAX_RAW_BYTES = 64 * MiB; const MAX_ARCHIVE_BYTES = 96 * MiB; -const FILE_LIMITS = new Map([ - ['admission.json', MiB], ['request.json', 8 * MiB], ['gateway.json', 8192], - ['ledger.jsonl', 16 * MiB], ['result.json', MiB], ['provider-artifact.json', 16 * MiB], - ['observability.json', 131_072], ['otlp-native.jsonl', 2 * MiB], ['otlp-controller.jsonl', 2 * MiB], ['otlp-root.json', 16_384], - ['stdout.log', MiB], ['stderr.log', MiB], -]); -const RECEIPT = /^receipts\/[a-f0-9]{8}(?:-[a-f0-9]{4}){3}-[a-f0-9]{12}\.json$/; -const limitFor = name => FILE_LIMITS.get(name) ?? (RECEIPT.test(name) ? MiB : 0); + function secretValues({ environment = process.env, secrets = [] } = {}) { - return [...new Set([...secrets, ...Object.entries(environment) - .filter(([name]) => /(?:API_KEY|TOKEN|SECRET|PASSWORD|ACCESS_KEY_ID)$/.test(name)) - .map(([, value]) => value)].filter(value => typeof value === 'string' && value.length > 0))]; + const configured = [...secrets, ...Object.entries(environment) + .filter(([name]) => /(?:KEY|TOKEN|SECRET|PASSWORD|ACCESS_KEY_ID|CONFIG_CONTENT)$/i.test(name)) + .map(([, value]) => value)].filter(value => typeof value === 'string' && value.length > 0); + for (const [name, value] of Object.entries(environment)) { + if (!/CONFIG_CONTENT$/i.test(name) || typeof value !== 'string') continue; + try { + const visit = item => { + if (!item || typeof item !== 'object') return; + for (const [key, entry] of Object.entries(item)) { + if (/(?:key|token|secret|password)$/i.test(key) && typeof entry === 'string' && entry) configured.push(entry); + else visit(entry); + } + }; + visit(JSON.parse(value)); + } catch {} + } + return [...new Set(configured.flatMap(value => [value, JSON.stringify(value).slice(1, -1)]))]; } function excludeSecrets(bytes, secrets) { if (secrets.some(secret => bytes.includes(Buffer.from(secret)))) throw new Error('Controller evidence contains a configured credential'); } -async function readRegularFile(path, maximum) { - const handle = await open(path, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK); - try { - const stat = await handle.stat(); - if (!stat.isFile() || stat.size > maximum) throw new Error('Controller evidence is not a bounded regular file'); - const bytes = Buffer.alloc(stat.size + 1); - let length = 0; - while (length < bytes.length) { - const { bytesRead } = await handle.read(bytes, length, bytes.length - length, null); - if (!bytesRead) break; - length += bytesRead; - } - if (length !== stat.size) throw new Error('Controller evidence changed while being read'); - return bytes.subarray(0, length); - } finally { await handle.close(); } -} + async function realDirectory(path) { try { const stat = await lstat(path); diff --git a/packages/paperclip-runner/scripts/controlled-evals/controller.test.mjs b/packages/paperclip-runner/scripts/controlled-evals/controller.test.mjs index 7162d41663e..9c4f4936773 100644 --- a/packages/paperclip-runner/scripts/controlled-evals/controller.test.mjs +++ b/packages/paperclip-runner/scripts/controlled-evals/controller.test.mjs @@ -1,6 +1,7 @@ import assert from 'node:assert/strict'; import { test } from 'node:test'; -import { mkdtemp, mkdir, readFile, readdir, rm, symlink, writeFile } from 'node:fs/promises'; +import { execFileSync } from 'node:child_process'; +import { mkdtemp, mkdir, readFile, readdir, rm, symlink, truncate, writeFile } from 'node:fs/promises'; import { writeFileSync } from 'node:fs'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; @@ -15,7 +16,9 @@ import assert from 'node:assert/strict'; import {createHash} from 'node:crypto'; import {spawn} from 'node:child_process'; import {once} from 'node:events'; -import {readFile,writeFile} from 'node:fs/promises'; +import {readFile,writeFile,rm,readdir,symlink,truncate} from 'node:fs/promises'; +import {dirname,join} from 'node:path'; +import {execFileSync} from 'node:child_process'; const args=new Map();for(let i=2;i { const f = await fixture(t, { count: 2 }); for (const mutate of [ + value => { value.attempts[0].attemptId = 'nested/attempt'; }, value => { value.maxUsd = '0.01'; }, value => { value.attempts[1].executionId = value.attempts[0].executionId; }, value => { value.attempts[0].provider = 'acpx'; value.attempts[0].acpxAgent = 'codex'; }, value => { delete value.overhead.components[0].provisionalBound; }, value => { value.models[0].secret = 'unexpected'; }, @@ -395,6 +410,51 @@ test('configured credentials are excluded before encoding and after archive deco await assert.rejects(readControlledEvidence(file, { environment }), /configured credential/); }); +for (const name of ['provider-artifact.json', 'receipt', 'ledger.jsonl']) for (const kind of ['fifo', 'symlink', 'oversized']) { + test(`post-child ${name} rejects ${kind} without blocking or releasing its allocation`, { timeout: 20_000 }, async t => { + const f = await fixture(t, { requests: 1, invalidEvidence: name, invalidFileType: kind }); + const result = await f.run(); + assert.equal(result.report.status, 'incomplete'); + assert.equal(result.report.accounting.knownUsd, '0.000068000'); + const summary = await summarizeControlledCampaign(f.manifest); + assert.equal(summary.incompleteAttempts, 1); + assert.equal(summary.committedUsd, '0.014000000'); + }); +} + +test('durable reports bound every evidence file and retain full holds for invalid proof', { timeout: 20_000 }, async t => { + const f = await fixture(t, { requests: 1, observability: true }); await f.run(); + const root = attemptDirectory(f.manifest, 'attempt-0'); + const receipt = `receipts/${(await readdir(join(root, 'receipts')))[0]}`; + for (const name of ['admission.json', 'request.json', 'result.json', 'provider-artifact.json', 'gateway.json', 'ledger.jsonl', receipt, 'observability.json', 'otlp-native.jsonl', 'otlp-controller.jsonl', 'otlp-root.json']) { + const path = join(root, name), original = await readFile(path); + for (const kind of ['fifo', 'symlink', 'oversized']) { + await rm(path); + if (kind === 'fifo') execFileSync('mkfifo', [path]); + else if (kind === 'symlink') await symlink('/dev/zero', path); + else { await writeFile(path, ''); await truncate(path, 16 * 1024 * 1024 + 1); } + const summary = await summarizeControlledCampaign(f.manifest); + assert.equal(summary.incompleteAttempts, 1, `${name}:${kind}`); + assert.equal(summary.committedUsd, '0.014000000', `${name}:${kind}`); + } + await rm(path); await writeFile(path, original); + } + assert.equal((await summarizeControlledCampaign(f.manifest)).incompleteAttempts, 0); +}); + +for (const name of ['PAPERCLIP_AGENT_PRIVATE_KEY', 'PAPERCLIP_AI_PROVIDER_KEY', 'AWS_SECRET_ACCESS_KEY', 'OPENCODE_CONFIG_CONTENT']) { + test(`archive excludes configured ${name} before and after encoding`, async t => { + const f = await fixture(t); await f.run(); + const secret = name === 'PAPERCLIP_AGENT_PRIVATE_KEY' ? '-----BEGIN PRIVATE KEY-----\nsynthetic-private-material\n-----END PRIVATE KEY-----' : 'synthetic-extra-credential'; + const environment = { [name]: name === 'OPENCODE_CONFIG_CONTENT' ? JSON.stringify({ provider: { apiKey: secret } }) : secret }; + const root = attemptDirectory(f.manifest, 'attempt-0'); + await writeFile(join(root, 'stderr.log'), JSON.stringify({ retained: secret })); + await assert.rejects(packControlledAttempt({ manifest: f.manifest, attemptId: 'attempt-0', environment }), /configured credential/); + const unchecked = await packControlledAttempt({ manifest: f.manifest, attemptId: 'attempt-0', environment: {} }); + assert.throws(() => decodeControlledEvidence(unchecked, { environment }), /configured credential/); + }); +} + test('retention annotations preserve an already-failed controller result and its known costs', async t => { const f = await fixture(t, { requests: 1, missingRoute: true }); await f.run(); const bundle = await packControlledAttempt({ manifest: f.manifest, attemptId: 'attempt-0' }); @@ -579,7 +639,9 @@ test('cancellation aborts a pending native export and retains its uncertain rece test('lost ledger finalization still closes both OTLP relays', async t => { const f = await fixture(t, { observability: true, requests: 1 }); const root = attemptDirectory(f.manifest, 'attempt-0'); - await assert.rejects(f.run(0, async () => { await rm(join(root, 'ledger.jsonl')); return success(); }), { code: 'ENOENT' }); + const result = await f.run(0, async () => { await rm(join(root, 'ledger.jsonl')); return success(); }); + assert.equal(result.code, 2); assert.equal(result.report.failure, 'ledger_evidence_invalid'); + assert.equal((await summarizeControlledCampaign(f.manifest)).committedUsd, '0.014000000'); for (const file of ['otlp-native.jsonl', 'otlp-controller.jsonl']) { const rows = (await readFile(join(root, file), 'utf8')).trim().split('\n').map(line => JSON.parse(line)); assert.equal(rows.at(-1).type, 'closed'); @@ -619,3 +681,15 @@ test('cancellation during root export stops forwarding and cannot close the atte const summary = await summarizeControlledCampaign(f.manifest); assert.equal(summary.committedUsd, '0.014000000'); assert.equal(summary.knownUsd, '0.000068000'); }); + +for (const name of ['observability.json', 'otlp-native.jsonl', 'otlp-controller.jsonl']) for (const kind of ['fifo', 'symlink', 'oversized']) { + test(`native post-child ${name} rejects ${kind} and retains model cost plus the full hold`, { timeout: 20_000 }, async t => { + const f = await fixture(t, { requests: 1, observability: true, invalidEvidence: name, invalidFileType: kind }); + const result = await f.run(); + assert.equal(result.code, 2); + const summary = await summarizeControlledCampaign(f.manifest); + assert.equal(summary.incompleteAttempts, 1); + assert.equal(summary.knownUsd, '0.000068000'); + assert.equal(summary.committedUsd, '0.014000000'); + }); +} diff --git a/packages/paperclip-runner/scripts/controlled-evals/evidence-files.mjs b/packages/paperclip-runner/scripts/controlled-evals/evidence-files.mjs new file mode 100644 index 00000000000..d6a38aef844 --- /dev/null +++ b/packages/paperclip-runner/scripts/controlled-evals/evidence-files.mjs @@ -0,0 +1,29 @@ +import { constants } from 'node:fs'; +import { open } from 'node:fs/promises'; + +const MiB = 1_048_576; +export const FILE_LIMITS = new Map([ + ['admission.json', MiB], ['request.json', 8 * MiB], ['gateway.json', 8192], + ['ledger.jsonl', 16 * MiB], ['result.json', MiB], ['provider-artifact.json', 16 * MiB], + ['observability.json', 131_072], ['otlp-native.jsonl', 2 * MiB], ['otlp-controller.jsonl', 2 * MiB], ['otlp-root.json', 16_384], + ['stdout.log', MiB], ['stderr.log', MiB], +]); +export const RECEIPT = /^receipts\/[a-f0-9]{8}(?:-[a-f0-9]{4}){3}-[a-f0-9]{12}\.json$/; +export const limitFor = name => FILE_LIMITS.get(name) ?? (RECEIPT.test(name) ? MiB : 0); + +export async function readRegularFile(path, maximum) { + const handle = await open(path, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK); + try { + const stat = await handle.stat(); + if (!stat.isFile() || stat.size > maximum) throw new Error('Controller evidence is not a bounded regular file'); + const bytes = Buffer.alloc(stat.size + 1); + let length = 0; + while (length < bytes.length) { + const { bytesRead } = await handle.read(bytes, length, bytes.length - length, null); + if (!bytesRead) break; + length += bytesRead; + } + if (length !== stat.size) throw new Error('Controller evidence changed while being read'); + return bytes.subarray(0, length); + } finally { await handle.close(); } +} diff --git a/packages/paperclip-runner/scripts/controlled-evals/manifest.mjs b/packages/paperclip-runner/scripts/controlled-evals/manifest.mjs index 333e13cb81d..1c437cd0f1e 100644 --- a/packages/paperclip-runner/scripts/controlled-evals/manifest.mjs +++ b/packages/paperclip-runner/scripts/controlled-evals/manifest.mjs @@ -72,7 +72,7 @@ export function validateManifest(value, now = Date.now(), { admission = true } = const provider = attempt.provider === 'codex' && attempt.acpxAgent === undefined ? 'openai' : attempt.provider === 'acpx' && attempt.acpxAgent === 'claude' ? 'anthropic' : null; const reservation = prices.get(`${provider}:${attempt.model}`); - if (!provider || !id(attempt.executionId) || !id(attempt.attemptId) || !hash(attempt.requestSha256) + if (!provider || !id(attempt.executionId) || !id(attempt.attemptId) || !/^[A-Za-z0-9_.-]+$/.test(attempt.attemptId) || !hash(attempt.requestSha256) || attempts.has(attempt.attemptId) || executions.has(attempt.executionId) || reservation === undefined || !Number.isSafeInteger(attempt.timeoutMs) || attempt.timeoutMs < 1 || attempt.timeoutMs > 4 * 3_600_000 || !Number.isSafeInteger(attempt.maxRequests) || attempt.maxRequests < 1 || attempt.maxRequests > 1000 diff --git a/packages/paperclip-runner/scripts/controlled-evals/observability.mjs b/packages/paperclip-runner/scripts/controlled-evals/observability.mjs index 274a25eb438..0bb3d142fdd 100644 --- a/packages/paperclip-runner/scripts/controlled-evals/observability.mjs +++ b/packages/paperclip-runner/scripts/controlled-evals/observability.mjs @@ -1,6 +1,7 @@ import { randomBytes } from 'node:crypto'; -import { readFile, writeFile } from 'node:fs/promises'; +import { writeFile } from 'node:fs/promises'; import { join } from 'node:path'; +import { limitFor, readRegularFile } from './evidence-files.mjs'; import { canonical, costRuntime, ROOT_OTLP_LIMITS, sha256 } from './manifest.mjs'; function launchFor(manifest, request, identity, baseUrl) { @@ -66,7 +67,7 @@ export async function startControlledObservability({ manifest, attempt, request, await native.close(); signal?.throwIfAborted(); try { - const receipts = costRuntime.inspectEvalOtlpReceipts(await readFile(join(root, 'otlp-native.jsonl')), + const receipts = costRuntime.inspectEvalOtlpReceipts(await readRegularFile(join(root, 'otlp-native.jsonl'), limitFor('otlp-native.jsonl')), policyFor(manifest, attempt, identity, deadlineAt, true)); if (receipts.status !== 'accepted' || !receipts.signals.logs.records || !receipts.signals.traces.records) traceFailure ??= 'native_export_incomplete'; } catch { traceFailure ??= 'native_evidence_incomplete'; } @@ -79,7 +80,7 @@ export async function startControlledObservability({ manifest, attempt, request, } finally { await close(); } return inspectControlledObservability({ manifest, attempt, request, artifact: input.artifact, report: { startedAt, traceFinishedAt: finishedAt, cleanupVerified: input.cleanupVerified, traceFailure }, - deadlineAt, readEvidence: name => readFile(join(root, name)) }); + deadlineAt, readEvidence: name => readRegularFile(join(root, name), limitFor(name)) }); })() }; } catch (error) { await close(); throw error; } } diff --git a/packages/paperclip-runner/scripts/controlled-evals/report.mjs b/packages/paperclip-runner/scripts/controlled-evals/report.mjs index 5a30f01f298..b3614f1b72a 100644 --- a/packages/paperclip-runner/scripts/controlled-evals/report.mjs +++ b/packages/paperclip-runner/scripts/controlled-evals/report.mjs @@ -1,9 +1,9 @@ -import { readFile } from 'node:fs/promises'; import { join, resolve } from 'node:path'; import { pathToFileURL } from 'node:url'; import { canonical, costRuntime, readJson, requestBindingSha256, sha256, units, usd, validateManifest } from './manifest.mjs'; import { inspectControlledObservability } from './observability.mjs'; import { attemptDirectory } from './run.mjs'; +import { limitFor, readRegularFile } from './evidence-files.mjs'; export async function summarizeControlledCampaign(value, { filesByAttempt } = {}) { const { manifest, manifestSha256, allocatedUsd, overheadUsd } = validateManifest(value, Date.now(), { admission: false }); @@ -12,9 +12,11 @@ export async function summarizeControlledCampaign(value, { filesByAttempt } = {} for (const allocation of manifest.attempts) { const root = attemptDirectory(manifest, allocation.attemptId); const readEvidence = async name => { - if (filesByAttempt === undefined) return readFile(join(root, name)); + const limit = limitFor(name); + if (!limit) throw new Error('Unexpected controller evidence path'); + if (filesByAttempt === undefined) return readRegularFile(join(root, name), limit); const bytes = filesByAttempt.get(allocation.attemptId)?.get(name); - if (!Buffer.isBuffer(bytes)) throw new Error('Missing archived controller evidence'); + if (!Buffer.isBuffer(bytes) || bytes.length > limit) throw new Error('Missing archived controller evidence'); return bytes; }; const readDocument = async (name, maximum = 1_048_576) => { diff --git a/packages/paperclip-runner/scripts/controlled-evals/run.mjs b/packages/paperclip-runner/scripts/controlled-evals/run.mjs index 005afa43922..0bd508e7da7 100644 --- a/packages/paperclip-runner/scripts/controlled-evals/run.mjs +++ b/packages/paperclip-runner/scripts/controlled-evals/run.mjs @@ -5,6 +5,7 @@ import { dirname, isAbsolute, join, relative, resolve, sep } from 'node:path'; import { pathToFileURL } from 'node:url'; import { parseArgs } from 'node:util'; import { startControlledObservability } from './observability.mjs'; +import { limitFor, readRegularFile } from './evidence-files.mjs'; import { bindRequest, canonical, costRuntime, readJson, sha256, validateManifest } from './manifest.mjs'; async function atomicJson(file, value) { @@ -146,8 +147,7 @@ registerHooks({ resolve(specifier, context, nextResolve) { child = await runChild({ cli: stagedCli, loader, requestPath: retainedRequestPath, artifactPath, descriptorPath, observabilityPath: observability?.descriptorPath, cwd: request.session.workingDirectory ?? root, environment: childEnvironment, deadlineAt, signal, closeGateway: () => Promise.all([gateway.close(), observability?.close()]) }); for (const stream of ['stdout', 'stderr']) await writeFile(join(root, `${stream}.log`), redact(child[stream]), { flag: 'wx', mode: 0o600 }); - const raw = await readFile(artifactPath, 'utf8'); - if (Buffer.byteLength(raw) > 16_777_216) throw new Error('Provider artifact exceeds its byte limit'); + const raw = (await readRegularFile(artifactPath, limitFor('provider-artifact.json'))).toString('utf8'); secretExcluded = secrets.every(secret => !raw.includes(secret)); if (!secretExcluded) await writeFile(artifactPath, redact(raw), { mode: 0o600 }); providerArtifactSha256 = sha256(redact(raw)); @@ -178,9 +178,11 @@ registerHooks({ resolve(specifier, context, nextResolve) { if (snapshot.overBudget || snapshot.conflictingRequests || snapshot.inflight || snapshot.unknownRequests !== snapshot.boundedUnresolvedRequests) failure ??= 'request_accounting_unbounded'; const receiptHashes = {}; try { - for (const request of snapshot.requests) receiptHashes[request.requestId] = sha256(await readFile(join(root, 'receipts', `${request.requestId}.json`))); + for (const request of snapshot.requests) receiptHashes[request.requestId] = sha256(await readRegularFile(join(root, 'receipts', `${request.requestId}.json`), limitFor(`receipts/${request.requestId}.json`))); } catch { failure ??= 'request_receipt_missing'; } - const ledgerSha256 = sha256(await readFile(ledgerPath)); + let ledgerSha256; + try { ledgerSha256 = sha256(await readRegularFile(ledgerPath, limitFor('ledger.jsonl'))); } + catch { failure ??= 'ledger_evidence_invalid'; } if (observability) { try { observabilityEvidence = await observability.finish({ artifact, cleanupVerified, failure }); @@ -200,6 +202,7 @@ registerHooks({ resolve(specifier, context, nextResolve) { await writeFile(outputPath, JSON.stringify(resultArtifact, null, 2) + '\n', { flag: 'wx', mode: 0o600 }); return { code: failure ? 2 : child?.code ?? 2, report }; } finally { await observability?.close(); } + } if (process.argv[1] && import.meta.url === pathToFileURL(resolve(process.argv[1])).href) { diff --git a/packages/paperclip-runner/scripts/copilot-profile-identity.mjs b/packages/paperclip-runner/scripts/copilot-profile-identity.mjs index 4d55c9f8367..7f43d83e8af 100644 --- a/packages/paperclip-runner/scripts/copilot-profile-identity.mjs +++ b/packages/paperclip-runner/scripts/copilot-profile-identity.mjs @@ -7,9 +7,9 @@ import { fileURLToPath } from "node:url"; import { resolve, dirname } from "node:path"; const root = resolve(dirname(fileURLToPath(import.meta.url)), ".."); -const fixture = resolve(root, "test/fixtures/copilot-profile-v58-identity.json"); +const fixture = resolve(root, "test/fixtures/copilot-profile-v59-identity.json"); const historical = JSON.parse(readFileSync(resolve(root, "test/fixtures/copilot-profile-v12-identity.json"), "utf8")); -const declaration = { ...historical.declaration, agentProfileVersion: 58 }; +const declaration = { ...historical.declaration, agentProfileVersion: 59 }; delete declaration.permissionIdentitySourceSha256; const sources = { displayEventSourceSha256: "src/drivers/acpx/copilot-events.ts", diff --git a/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.test.ts b/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.test.ts index 62dc7e15321..28a0c5814b2 100644 --- a/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.test.ts +++ b/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.test.ts @@ -12,6 +12,7 @@ import { createHash, randomUUID } from "node:crypto"; import { spawn, type ChildProcessWithoutNullStreams } from "node:child_process"; import { readFileSync } from "node:fs"; import { fileURLToPath } from "node:url"; +import { createAcpxSidecarHostEnvironment, createSanitizedAcpxSpawnInput } from "../drivers/acpx/environment.js"; import { afterEach, describe, expect, it, vi } from "vitest"; @@ -93,6 +94,22 @@ afterEach(async () => { }); describe("qualified ACPX runtime sidecar", () => { + it("gives native Claude HTTP tools the Runner lease budget without changing sibling providers", () => { + const source = readFileSync(new URL("./acpx-runtime-sidecar.ts", import.meta.url), "utf8"); + const start = source.indexOf(" environment:"); + const end = source.indexOf(" expectedIdentity:", start); + expect(start).toBeGreaterThan(0); expect(end).toBeGreaterThan(start); + const create = new Function("process", "params", "createAcpxSidecarHostEnvironment", `return ({ ${source.slice(start, end)} }).environment;`); + const original = { MCP_TOOL_TIMEOUT: "1000", ANTHROPIC_API_KEY: "selected-key", UNRELATED_SECRET: "private" }; + const claude = create({ env: original }, { agent: "claude", normalizedSessionId: "session" }, createAcpxSidecarHostEnvironment); + expect(createSanitizedAcpxSpawnInput(claude, "claude").env).toMatchObject({ MCP_TOOL_TIMEOUT: "3600000", ANTHROPIC_API_KEY: "selected-key" }); + expect(createSanitizedAcpxSpawnInput(claude, "claude").env).not.toHaveProperty("UNRELATED_SECRET"); + expect(original.MCP_TOOL_TIMEOUT).toBe("1000"); + const codex = create({ env: original }, { agent: "codex", normalizedSessionId: "session" }, createAcpxSidecarHostEnvironment); + expect(codex).toBe(original); + expect(createSanitizedAcpxSpawnInput(codex, "codex").env).not.toHaveProperty("MCP_TOOL_TIMEOUT"); + }); + it("projects the actual sidecar terminal diagnostic block without authoritative accounting", () => { const source = readFileSync(new URL("./acpx-runtime-sidecar.ts", import.meta.url), "utf8"); const start = source.indexOf(" try {", source.indexOf(" const usageAfter = await readSidecarHostStatusWithin(activeHost);")); @@ -279,6 +296,42 @@ describe("qualified ACPX runtime sidecar", () => { for (const agent of ["cursor", "codex", "pi"]) expect(create({ agent, tools: [] }, evidence, () => undefined).captureSemanticReceipt).toBeUndefined(); }); + it("keeps a native tool pending through controller downtime without a duplicate side effect", async () => { + const source = readFileSync(new URL("./acpx-runtime-sidecar.ts", import.meta.url), "utf8"); + const start = source.indexOf(" semanticTools: {"); + const end = source.indexOf(" onGoalUpdate:", start); + expect(start).toBeGreaterThan(0); expect(end).toBeGreaterThan(start); + const create = new Function("params", "activeCopilotEvidence", "waitForTool", `return ({ ${source.slice(start, end)} }).semanticTools;`); + let admitted!: () => void, complete!: (value: unknown) => void; + const started = new Promise(resolve => { admitted = resolve; }); + const result = new Promise(resolve => { complete = resolve; }); + let signal: AbortSignal | undefined; + const handler = vi.fn((call: RunnerToolCall) => { signal = call.signal; admitted(); return result; }); + const bridge = await startRunnerToolBridge(create({ agent: "claude", tools: [{ name: "register_deliverable", inputSchema: { type: "object" } }] }, undefined, handler)); + const call = () => fetch(bridge.url, { method: "POST", headers: { Authorization: `Bearer ${bridge.secret}`, "Content-Type": "application/json" }, + body: JSON.stringify({ jsonrpc: "2.0", id: "saved-call", method: "tools/call", params: { name: "register_deliverable", arguments: {} } }) }).then(response => response.json()); + let first: Promise | undefined; + try { + vi.useFakeTimers({ toFake: ["setTimeout", "clearTimeout"] }); + first = call(); + await started; + await vi.advanceTimersByTimeAsync(125_000); + expect(signal?.aborted).toBe(false); + // The same pending invocation can be read again after reconnect. It must + // share the original result rather than create another external write. + const duplicate = call(); + complete({ artifactId: "saved-artifact" }); + expect(await first).toMatchObject({ result: { content: [{ text: '{"artifactId":"saved-artifact"}' }] } }); + expect(await duplicate).toEqual(await first); + expect(handler).toHaveBeenCalledOnce(); + } finally { + complete({ artifactId: "saved-artifact" }); + await first?.catch(() => undefined); + await bridge.close(); + vi.useRealTimers(); + } + }); + it.each(["forwarded", "emit-failed"])("binds native v2 receipts to actual sidecar-normalized input: %s", async mode => { const raw = { reportedWorkDisposition: "done", summary: "Complete", evidence: [], verification: [], completionClaim: { contractRevision: "1", objectiveSatisfied: true, criteria: [], remainingWork: [] } }; diff --git a/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.ts b/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.ts index 2b728b2027e..bc1ffb523c3 100644 --- a/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.ts +++ b/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.ts @@ -305,12 +305,22 @@ async function dispatch( providerPolicy: params.providerPolicy, systemInstructions: params.systemInstructions, runtimeContext: params.runtimeContext, - environment: createAcpxSidecarHostEnvironment(process.env, params.agent, params.normalizedSessionId), + environment: params.agent === "claude" ? { + ...createAcpxSidecarHostEnvironment(process.env, params.agent, params.normalizedSessionId), + // Claude's HTTP MCP client otherwise aborts at 60s, before the + // native Runner connection lease. Cancellation and Runner shutdown + // still revoke the pending call; this is not a new authority lease. + MCP_TOOL_TIMEOUT: String(60 * 60 * 1_000), + } : createAcpxSidecarHostEnvironment(process.env, params.agent, params.normalizedSessionId), expectedIdentity: params.expectedIdentity, assignedMcpTransport: "prp", semanticTools: { tools: params.tools, handler: waitForTool, + // These calls are durable PRP requests. A controller outage must not + // turn a queued write into a provider-visible failure and retry. + // Runner lease expiry, turn cancellation and bridge close own them. + timeoutMs: null, ...(params.agent === "copilot" ? { captureSemanticReceipt: () => activeCopilotEvidence?.captureSemanticReceipt() } : {}), }, onGoalUpdate: (goal) => { diff --git a/packages/paperclip-runner/src/control-plane/runner-execution-driver.ts b/packages/paperclip-runner/src/control-plane/runner-execution-driver.ts index 06e94abc778..7a55b20b843 100644 --- a/packages/paperclip-runner/src/control-plane/runner-execution-driver.ts +++ b/packages/paperclip-runner/src/control-plane/runner-execution-driver.ts @@ -47,6 +47,10 @@ export interface RunnerExecutionDriver { }) => Promise>; /** Active-connection recovery budget. Omitted for the existing local mode. */ runnerReconnectGraceMs?: number; + /** Process lifetime while disconnected. Defaults to the active-connection + * recovery budget for compatibility. Null uses the authenticated connection + * lease as the lifetime bound, so controller downtime cannot kill its owner. */ + runnerProcessReconnectGraceMs?: number | null; /** * A verified Runner that outlived its controller. Adoption registers * the durable authority and waits for this exact process to reconnect; it diff --git a/packages/paperclip-runner/src/drivers/acpx/copilot-installation.test.ts b/packages/paperclip-runner/src/drivers/acpx/copilot-installation.test.ts index 6df0af8afb3..8bdf9b3950d 100644 --- a/packages/paperclip-runner/src/drivers/acpx/copilot-installation.test.ts +++ b/packages/paperclip-runner/src/drivers/acpx/copilot-installation.test.ts @@ -87,9 +87,19 @@ vi.mock("./installation-integrity.js", () => ({ verifyNativeAcpxInstallation: vi describe("Copilot build-owned installation", () => { it("admits the current profile with environment launch preparation", () => { - expect(QUALIFIED_ACPX_PROFILES.copilot.agentProfileVersion).toBe(58); + expect(QUALIFIED_ACPX_PROFILES.copilot.agentProfileVersion).toBe(59); expect(QUALIFIED_ACPX_PROFILES.copilot.qualificationStatus).toBeUndefined(); - expect(QUALIFIED_ACPX_PROFILES.copilot.commandDigest).toBe("sha256:d490dacdab02051d24c84aae49ac6fb12b8226cf8a47b014ec53f3d3a8410a94"); + expect(QUALIFIED_ACPX_PROFILES.copilot.commandDigest).toBe("sha256:2f39ffe54a6af832d044b5098c31df9e08d4d27aee0ac675ba4b0c9da7445c40"); + }); + + it("preserves v58 while binding the separate remote process reconnect policy in v59", () => { + const prior = JSON.parse(readFileSync(new URL("../../../test/fixtures/copilot-profile-v58-identity.json", import.meta.url), "utf8")); + const current = JSON.parse(readFileSync(new URL("../../../test/fixtures/copilot-profile-v59-identity.json", import.meta.url), "utf8")); + expect(Object.keys(current.declaration).filter(key => current.declaration[key] !== prior.declaration[key]).sort()).toEqual([ + "agentProfileVersion", "credentialBindingSourceSha256", "nativeInstructionsSourceSha256", "nativeSessionRuntimeSourceSha256", "profileCompatibilitySourceSha256", "runtimeRequestBridgeSourceSha256", "semanticBridgeSourceSha256", "semanticSidecarSourceSha256", "transportContractSourceSha256", + ]); + expect(isSupportedAcpxProfileVersion("copilot", 58)).toBe(true); + expect(current.commandDigest).not.toBe(prior.commandDigest); }); it("preserves v57 while binding the shared Pi setup and launch repairs in v58", () => { diff --git a/packages/paperclip-runner/src/drivers/acpx/environment.test.ts b/packages/paperclip-runner/src/drivers/acpx/environment.test.ts index 3bc038a101e..3c7cdee2fe9 100644 --- a/packages/paperclip-runner/src/drivers/acpx/environment.test.ts +++ b/packages/paperclip-runner/src/drivers/acpx/environment.test.ts @@ -6,6 +6,13 @@ afterEach(() => vi.unstubAllEnvs()); import { ACPX_CREDENTIAL_BINDING_ENV, createAcpxCredentialBinding, createAcpxSidecarHostEnvironment, createSanitizedAcpxSpawnInput } from "./environment.js"; describe("ACPX launch environment", () => { + it("retains an explicit Claude MCP timeout without inheriting an ambient or sibling setting", () => { + vi.stubEnv("MCP_TOOL_TIMEOUT", "999999"); + expect(createSanitizedAcpxSpawnInput(undefined, "claude").env).not.toHaveProperty("MCP_TOOL_TIMEOUT"); + expect(createSanitizedAcpxSpawnInput({ MCP_TOOL_TIMEOUT: "120000" }, "claude").env.MCP_TOOL_TIMEOUT).toBe("120000"); + expect(createSanitizedAcpxSpawnInput({ MCP_TOOL_TIMEOUT: "120000" }, "codex").env).not.toHaveProperty("MCP_TOOL_TIMEOUT"); + }); + it("excludes general IAM from the Pi sidecar and launch, including selected task values", () => { const iam = { AWS_ACCESS_KEY_ID: "general-key", AWS_SECRET_ACCESS_KEY: "general-secret", AWS_SESSION_TOKEN: "general-session" }; const source = { ...iam, ...configuredEnvironmentProjection({ ...iam, CUSTOM_SETTING: "allowed" }), AWS_BEARER_TOKEN_BEDROCK: "provider-key" }; diff --git a/packages/paperclip-runner/src/drivers/acpx/environment.ts b/packages/paperclip-runner/src/drivers/acpx/environment.ts index 1fb79afba46..f11aeacfa08 100644 --- a/packages/paperclip-runner/src/drivers/acpx/environment.ts +++ b/packages/paperclip-runner/src/drivers/acpx/environment.ts @@ -131,6 +131,7 @@ export function createSanitizedAcpxSpawnInput( ...(environment === undefined ? [] : ["PAPERCLIP_AGENT_KEY_ID", "PAPERCLIP_AGENT_PUBLIC_KEY", "PAPERCLIP_AGENT_PRIVATE_KEY"]), ...credentialNames, ...(agent === "claude" ? CLAUDE_ROUTING_ENV_KEYS : []), + ...(agent === "claude" && environment !== undefined ? ["MCP_TOOL_TIMEOUT"] : []), ...(usesPreconfiguredHome(source) ? [...PRECONFIGURED_HOME_KEYS, "PAPERCLIP_RUNNER_HOME_MODE", "ACPX_CLAUDE_INCLUDE_USER_SETTINGS"] : []), ]); let retainedBytes = 0; diff --git a/packages/paperclip-runner/src/drivers/acpx/generated-profiles.ts b/packages/paperclip-runner/src/drivers/acpx/generated-profiles.ts index ca6e53d79d7..36f2bdf29e1 100644 --- a/packages/paperclip-runner/src/drivers/acpx/generated-profiles.ts +++ b/packages/paperclip-runner/src/drivers/acpx/generated-profiles.ts @@ -22,12 +22,12 @@ export const QUALIFIED_ACPX_PROFILE_DATA = { "protocolVersion": 1, "acpxVersion": "0.13.1", "agent": "pi", - "agentProfileVersion": 28, + "agentProfileVersion": 29, "agentServerPackage": "pi-acp", "agentServerVersion": "0.0.33", "agentRuntimePackage": "@earendil-works/pi-coding-agent", "agentRuntimeVersion": "1.0.0", - "commandDigest": "sha256:4918a4d5472d66b22b1ec642b68b5c60a8092a39c84ec1ab1c9ad0289cb8c68c", + "commandDigest": "sha256:e80c6dc7bde12790371345ee5a12f0201d30941d26f1a780aec3d090545b2a06", "permissionPolicy": "interactive" }, "cursor": { @@ -48,12 +48,12 @@ export const QUALIFIED_ACPX_PROFILE_DATA = { "protocolVersion": 1, "acpxVersion": "0.13.1", "agent": "copilot", - "agentProfileVersion": 58, + "agentProfileVersion": 59, "agentServerPackage": "@github/copilot", "agentServerVersion": "1.0.88", "agentRuntimePackage": null, "agentRuntimeVersion": null, - "commandDigest": "sha256:d490dacdab02051d24c84aae49ac6fb12b8226cf8a47b014ec53f3d3a8410a94", + "commandDigest": "sha256:2f39ffe54a6af832d044b5098c31df9e08d4d27aee0ac675ba4b0c9da7445c40", "permissionPolicy": "interactive" }, "claude": { diff --git a/packages/paperclip-runner/src/drivers/acpx/pi-installation.test.ts b/packages/paperclip-runner/src/drivers/acpx/pi-installation.test.ts index 7695e64219c..8d81addb3cf 100644 --- a/packages/paperclip-runner/src/drivers/acpx/pi-installation.test.ts +++ b/packages/paperclip-runner/src/drivers/acpx/pi-installation.test.ts @@ -155,11 +155,20 @@ describe("Pi installation factory", () => { expect(current.commandDigest).not.toBe(prior.commandDigest); }); + it("preserves profile 28 while binding the Claude-only explicit MCP client budget source in profile 29", async () => { + const read = async (version: number) => JSON.parse(await readFile(new URL(`../../../test-fixtures/pi-acp/profile-v${version}-identity.json`, import.meta.url), "utf8")); + const prior = await read(28), current = await read(29); + expect(Object.keys(current.declaration).filter(key => JSON.stringify(current.declaration[key]) !== JSON.stringify(prior.declaration[key])).sort()) + .toEqual(["agentProfileVersion", "credentialEnvironmentSourceSha256"]); + expect(current.declaration.closure).toEqual(prior.declaration.closure); + expect(current.commandDigest).not.toBe(prior.commandDigest); + }); + it("binds the profile declaration to the reviewed patch and platform closure pins", async () => { const hash = (bytes: string | Uint8Array) => createHash("sha256").update(bytes).digest("hex"); const canonical = (value: any): string => Array.isArray(value) ? `[${value.map(canonical).join(",")}]` : value && typeof value === "object" ? `{${Object.keys(value).sort().map(key => `${JSON.stringify(key)}:${canonical(value[key])}`).join(",")}}` : JSON.stringify(value); - const identity = JSON.parse(await readFile(new URL("../../../test-fixtures/pi-acp/profile-v28-identity.json", import.meta.url), "utf8")); + const identity = JSON.parse(await readFile(new URL("../../../test-fixtures/pi-acp/profile-v29-identity.json", import.meta.url), "utf8")); const declaration = identity.declaration; expect(declaration.sharedRuntimeContract).toBe("paperclip.acpx-runtime-contract.v1"); expect(identity.commandDigest).toBe(`sha256:${hash(canonical(declaration))}`); @@ -190,21 +199,21 @@ describe("Pi installation factory", () => { it("rejects legacy profiles and tampered executable identities", () => { expect(() => assertPiInstallationProfile(candidate())).not.toThrow(); - expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 1 })).toThrow("version 28"); - expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 2 })).toThrow("version 28"); - expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 3 })).toThrow("version 28"); - expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 4 })).toThrow("version 28"); - expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 5 })).toThrow("version 28"); - expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 6 })).toThrow("version 28"); - expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 7 })).toThrow("version 28"); - expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 8 })).toThrow("version 28"); - expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 9 })).toThrow("version 28"); - expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 10 })).toThrow("version 28"); - expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 11 })).toThrow("version 28"); - expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 12 })).toThrow("version 28"); - expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 13 })).toThrow("version 28"); - expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 14 })).toThrow("version 28"); - for (const priorVersion of [15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27]) expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: priorVersion })).toThrow("version 28"); + expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 1 })).toThrow("version 29"); + expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 2 })).toThrow("version 29"); + expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 3 })).toThrow("version 29"); + expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 4 })).toThrow("version 29"); + expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 5 })).toThrow("version 29"); + expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 6 })).toThrow("version 29"); + expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 7 })).toThrow("version 29"); + expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 8 })).toThrow("version 29"); + expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 9 })).toThrow("version 29"); + expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 10 })).toThrow("version 29"); + expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 11 })).toThrow("version 29"); + expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 12 })).toThrow("version 29"); + expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 13 })).toThrow("version 29"); + expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 14 })).toThrow("version 29"); + for (const priorVersion of [15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28]) expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: priorVersion })).toThrow("version 29"); for (const changed of [{ agentServerVersion: "latest" }, { commandDigest: `sha256:${"0".repeat(64)}` }, { agentRuntimePackage: "ambient-pi" }]) { expect(() => assertPiInstallationProfile({ ...candidate(), ...changed })).toThrow("trusted declaration"); } diff --git a/packages/paperclip-runner/src/drivers/acpx/profile-compatibility.ts b/packages/paperclip-runner/src/drivers/acpx/profile-compatibility.ts index 1119594c0ed..56a5878aa39 100644 --- a/packages/paperclip-runner/src/drivers/acpx/profile-compatibility.ts +++ b/packages/paperclip-runner/src/drivers/acpx/profile-compatibility.ts @@ -2,13 +2,13 @@ import { QUALIFIED_ACPX_PROFILE_DATA } from "./generated-profiles.js"; /** Decodable historical profile revisions, not permission to launch them. * Exact current profile/package/digest admission is checked separately. */ -type HistoricalAcpxProfileVersion = 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57; +type HistoricalAcpxProfileVersion = 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58; type CurrentAcpxProfileVersion = typeof QUALIFIED_ACPX_PROFILE_DATA[keyof typeof QUALIFIED_ACPX_PROFILE_DATA]["agentProfileVersion"]; export type AcpxProfileVersion = HistoricalAcpxProfileVersion | CurrentAcpxProfileVersion; const historicalVersions: Readonly> = { - pi: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27], claude: [1, 2, 3, 4, 5], codex: [1, 2, 3, 4, 5], - grok: [1, 2, 3, 4, 5], copilot: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57], + pi: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28], claude: [1, 2, 3, 4, 5], codex: [1, 2, 3, 4, 5], + grok: [1, 2, 3, 4, 5], copilot: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58], cursor: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15], }; diff --git a/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.test.ts b/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.test.ts index a63abc777f8..5d568eddf86 100644 --- a/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.test.ts +++ b/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.test.ts @@ -1513,7 +1513,7 @@ describe("OpenCodeServerDriver", () => { ); const isolatedHome = runtimeEnvironment.home; expect(startupFailurePending).toBe(false); - expect(dirname(isolatedHome)).toBe(sessionRoot); + expect(dirname(isolatedHome)).toBe(join(sessionRoot, "homes")); expect(basename(isolatedHome)).toMatch(/^home-/); await expect( readFile( @@ -1553,6 +1553,59 @@ describe("OpenCodeServerDriver", () => { readFile(join(instructionRoot, "sibling.md"), "utf8"), ).resolves.toBe("instruction sibling\n"); await session.close({ reason: "test" }); + await expect(readdir(join(sessionRoot, "homes"))).resolves.toEqual([]); + }); + + it("removes isolated homes left by earlier launches when a session reopens", async () => { + await chmod(fixture, 0o755); + const root = await mkdtemp(join(tmpdir(), "paperclip-opencode-legacy-home-")); + const workspace = await mkdtemp( + join(tmpdir(), "paperclip-opencode-legacy-home-workspace-"), + ); + const skillRoot = join(root, "skill-source"); + const instructionRoot = join(root, "instruction-source"); + roots.push(root, workspace); + await Promise.all([ + mkdir(skillRoot, { recursive: true }), + mkdir(instructionRoot, { recursive: true }), + ]); + await writeFile(join(skillRoot, "SKILL.md"), "# Assigned\n"); + await writeFile(join(instructionRoot, "AGENTS.md"), "instructions\n"); + const sessionRoot = join(root, "legacy"); + const legacyHome = join(sessionRoot, "home-AbC123"); + const sealedSkill = join(legacyHome, ".claude", "skills", "assigned"); + await mkdir(sealedSkill, { recursive: true }); + await mkdir(join(legacyHome, ".npm", "_cacache"), { recursive: true }); + await writeFile(join(legacyHome, ".npm", "_cacache", "entry"), "cache"); + await writeFile(join(sealedSkill, "SKILL.md"), "# Assigned\n"); + await chmod(join(sealedSkill, "SKILL.md"), 0o444); + await chmod(sealedSkill, 0o555); + await mkdir(join(sessionRoot, "data", "opencode"), { recursive: true }); + await writeFile(join(sessionRoot, "data", "opencode", "opencode.db"), "history"); + const driver = new OpenCodeServerDriver({ + model: "openrouter/deepseek/deepseek-v4-flash-0731", + runtimeDirectory: root, + command: fixture, + runtimeContext: runtimeContext(skillRoot, instructionRoot), + environment: { + PATH: process.env.PATH, + OPENROUTER_API_KEY: "fixture-key", + }, + }); + const session = await driver.openSession({ + runId: "run-legacy-home", + normalizedSessionId: "legacy", + workingDirectory: workspace, + }); + await expect(lstat(legacyHome)).rejects.toMatchObject({ code: "ENOENT" }); + await session.close({ reason: "test" }); + expect( + (await readdir(sessionRoot)).filter((entry) => entry.startsWith("home-")), + ).toEqual([]); + await expect(readdir(join(sessionRoot, "homes"))).resolves.toEqual([]); + await expect( + readFile(join(sessionRoot, "data", "opencode", "opencode.db"), "utf8"), + ).resolves.toBe("history"); }); it("resends system instructions without repeating the task envelope on recovery", async () => { diff --git a/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.ts b/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.ts index a7f80aac85e..4ab25b38b83 100644 --- a/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.ts +++ b/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.ts @@ -6,6 +6,7 @@ import { accessSync, constants, readFileSync, realpathSync, statSync } from "nod import { createRequire } from "node:module"; import { chmod, + lstat, mkdir, mkdtemp, readFile, @@ -85,7 +86,10 @@ import { type OpenCodeMcpBridge, } from "./mcp-bridge.js"; import { nativeMcpLaunchBinding } from "../native-mcp.js"; -import { materializeNativeRuntimeSkills } from "../runtime-context-materializer.js"; +import { + materializeNativeRuntimeSkills, + releaseMaterializedNativeRuntimeSkills, +} from "../runtime-context-materializer.js"; export const OPENCODE_SERVER_DRIVER_KIND = "opencode_server" as const; export const QUALIFIED_OPENCODE_VERSION = "1.18.34" as const; @@ -369,6 +373,7 @@ export class OpenCodeServerDriver implements HarnessDriver { ); await mkdir(root, { recursive: true, mode: 0o700 }); await writeFile(join(root, "workspace"), `${cwd}\n`, { mode: 0o600 }); + await removeLegacyIsolatedHomes(root); const trace = await createProviderTraceFileSink({ path: this.#options.environment?.PAPERCLIP_PROVIDER_TRACE_PATH, provider: "opencode", @@ -2281,7 +2286,9 @@ async function startRuntime(input: { // HOME contains a read-only skill snapshot, whose destination must be fresh. // Keep OpenCode's XDG data stable for provider-session recovery while giving // every launch (including retries and recovery) a new isolated HOME. - const isolatedHome = await mkdtemp(join(input.root, "home-")); + const homesRoot = join(input.root, OPENCODE_ISOLATED_HOMES_DIRECTORY); + await mkdir(homesRoot, { recursive: true, mode: 0o700 }); + const isolatedHome = await mkdtemp(join(homesRoot, "home-")); await chmod(isolatedHome, 0o700); try { await materializeNativeRuntimeSkills( @@ -2289,18 +2296,14 @@ async function startRuntime(input: { join(isolatedHome, ".claude", "skills"), ); } catch (error) { - await rm(isolatedHome, { recursive: true, force: true }).catch( - () => undefined, - ); + await removeIsolatedHome(isolatedHome).catch(() => undefined); throw error; } const bridge = await startOpenCodeMcpBridge({ tools: input.options.dynamicTools, handler: input.dispatch, }).catch(async (error) => { - await rm(isolatedHome, { recursive: true, force: true }).catch( - () => undefined, - ); + await removeIsolatedHome(isolatedHome).catch(() => undefined); throw error; }); const assignedMcp = nativeMcpLaunchBinding( @@ -2335,7 +2338,7 @@ async function startRuntime(input: { const providerProxy = modelProvider === "paperclip" && input.options.environment?.PAPERCLIP_AI_PROVIDER_URL ? await startOpenCodeProviderProxy(input.options.environment.PAPERCLIP_AI_PROVIDER_URL, input.options.environment.PAPERCLIP_AI_PROVIDER_KEY ?? "", providerModelId, input.options.environment).catch(async error => { await bridge.close().catch(() => {}); - await rm(isolatedHome, { recursive: true, force: true }).catch(() => {}); + await removeIsolatedHome(isolatedHome).catch(() => {}); throw error; }) : null; @@ -2568,9 +2571,7 @@ async function startRuntime(input: { await rm(join(configHome, "opencode", "opencode.json"), { force: true, }).catch(() => undefined); - await rm(isolatedHome, { recursive: true, force: true }).catch( - () => undefined, - ); + await removeIsolatedHome(isolatedHome).catch(() => undefined); if (closeInput.finalizeTrace !== false) { await input.trace?.finish({ reason: closeInput.reason ?? null }); } @@ -2585,9 +2586,7 @@ async function startRuntime(input: { await rm(join(configHome, "opencode", "opencode.json"), { force: true, }).catch(() => undefined); - await rm(isolatedHome, { recursive: true, force: true }).catch( - () => undefined, - ); + await removeIsolatedHome(isolatedHome).catch(() => undefined); throw error; } } @@ -2962,8 +2961,9 @@ function sanitizedEnvironmentKeys(): string[] { ]; } -function sessionRoot( - runtimeDirectory: string, +export const OPENCODE_ISOLATED_HOMES_DIRECTORY = "homes"; + +export function opencodeRuntimeSessionDirectoryName( normalizedSessionId: string, ): string { const safe = normalizedSessionId @@ -2971,7 +2971,35 @@ function sessionRoot( .slice(0, 120); if (!safe || safe === "." || safe === "..") throw new Error("Invalid normalized OpenCode session id"); - return join(resolve(runtimeDirectory), safe); + return safe; +} + +function sessionRoot( + runtimeDirectory: string, + normalizedSessionId: string, +): string { + return join( + resolve(runtimeDirectory), + opencodeRuntimeSessionDirectoryName(normalizedSessionId), + ); +} + +async function removeIsolatedHome(home: string): Promise { + const skillsParent = await lstat(join(home, ".claude")).catch(() => null); + if (skillsParent?.isDirectory()) { + await releaseMaterializedNativeRuntimeSkills( + join(home, ".claude", "skills"), + ).catch(() => undefined); + } + await rm(home, { recursive: true, force: true }); +} + +async function removeLegacyIsolatedHomes(root: string): Promise { + const entries = await readdir(root, { withFileTypes: true }).catch(() => []); + for (const entry of entries) { + if (!entry.isDirectory() || !entry.name.startsWith("home-")) continue; + await removeIsolatedHome(join(root, entry.name)).catch(() => undefined); + } } function validateWorkspace(value: string): string { diff --git a/packages/paperclip-runner/src/drivers/runner-tool-bridge.test.ts b/packages/paperclip-runner/src/drivers/runner-tool-bridge.test.ts index 152b8a9b6ea..4b212476c49 100644 --- a/packages/paperclip-runner/src/drivers/runner-tool-bridge.test.ts +++ b/packages/paperclip-runner/src/drivers/runner-tool-bridge.test.ts @@ -246,7 +246,7 @@ describe("runner semantic MCP bridge", () => { }); }); - it("honors MCP cancellation after the handler starts", async () => { + it.each([60_000, null])("honors MCP cancellation after the handler starts (timeout=%s)", async timeoutMs => { let markStarted!: () => void; const started = new Promise((resolve) => { markStarted = resolve; @@ -254,7 +254,7 @@ describe("runner semantic MCP bridge", () => { const aborted = vi.fn(); const bridge = await startRunnerToolBridge({ tools: [tool("documents.read")], - timeoutMs: 60_000, + timeoutMs, handler: ({ signal }) => new Promise((_resolve, reject) => { signal.addEventListener("abort", () => { aborted(); @@ -291,6 +291,23 @@ describe("runner semantic MCP bridge", () => { expect(aborted).toHaveBeenCalledOnce(); }); + it("cancels a lease-owned native tool when its bridge closes", async () => { + let admitted!: () => void; + const started = new Promise(resolve => { admitted = resolve; }); + let signal: AbortSignal | undefined; + const bridge = await startRunnerToolBridge({ tools: [tool("documents.read")], timeoutMs: null, + handler: call => { signal = call.signal; admitted(); return new Promise(() => {}); }, + }); + bridges.push(bridge); + const pending = rpc(bridge, { id: "close-me", method: "tools/call", params: { name: "documents.read", arguments: {} } }); + await started; + await bridge.close(); + expect(signal?.aborted).toBe(true); + expect(await (await pending).json()).toMatchObject({ result: { isError: true, + content: [{ text: "Paperclip tool call cancelled" }], + } }); + }); + it("preserves successful mutation identity when the result is oversized", async () => { const result = { text: `snowman-${"☃".repeat(65 * 1024)}` }; const handler = vi.fn(async () => result); diff --git a/packages/paperclip-runner/src/drivers/runner-tool-bridge.ts b/packages/paperclip-runner/src/drivers/runner-tool-bridge.ts index 3a82d69d209..624569fa195 100644 --- a/packages/paperclip-runner/src/drivers/runner-tool-bridge.ts +++ b/packages/paperclip-runner/src/drivers/runner-tool-bridge.ts @@ -45,7 +45,9 @@ export interface RunnerToolBridgeOptions { handler(call: RunnerToolCall): Promise; /** Copilot-only opt-in; capture the active invocation scope before dispatch. */ captureSemanticReceipt?: () => ((receipt: SemanticToolReceipt) => void) | undefined; - timeoutMs?: number; + /** Null delegates the wait to an owning durable runtime's lease and close. + * Ordinary callers retain the bounded request timeout. */ + timeoutMs?: number | null; privateToolTimeoutMs?: number; maxBodyBytes?: number; /** Internal test seam; production bounds headers and body reads to 30 seconds. */ @@ -143,7 +145,7 @@ export async function startRunnerToolBridge( controllers, handler: options.handler, captureSemanticReceipt: options.captureSemanticReceipt, - timeoutMs: positiveBoundedInteger( + timeoutMs: options.timeoutMs === null ? null : positiveBoundedInteger( options.timeoutMs, DEFAULT_TIMEOUT_MS, 1, @@ -214,7 +216,7 @@ async function handleRequest( controllers: Map; handler: RunnerToolBridgeOptions["handler"]; captureSemanticReceipt: RunnerToolBridgeOptions["captureSemanticReceipt"]; - timeoutMs: number; + timeoutMs: number | null; privateToolTimeoutMs: number; privateToolNames: ReadonlySet; maxBodyBytes: number; @@ -599,24 +601,24 @@ function closeServer(server: Server): Promise { function withCancellationAndTimeout( promise: Promise, controller: AbortController, - timeoutMs: number, + timeoutMs: number | null, ): Promise { return new Promise((resolve, reject) => { let settled = false; const finish = (callback: () => void) => { if (settled) return; settled = true; - clearTimeout(timer); + if (timer !== undefined) clearTimeout(timer); controller.signal.removeEventListener("abort", onAbort); callback(); }; const onAbort = () => finish(() => reject(new Error("Paperclip tool call cancelled"))); - const timer = setTimeout(() => { + const timer = timeoutMs === null ? undefined : setTimeout(() => { finish(() => reject(new Error("Paperclip tool call timed out"))); controller.abort(); }, timeoutMs); - timer.unref(); + timer?.unref(); controller.signal.addEventListener("abort", onAbort, { once: true }); promise.then( (value) => finish(() => resolve(value)), diff --git a/packages/paperclip-runner/src/live/runnerd-codex-transport.ts b/packages/paperclip-runner/src/live/runnerd-codex-transport.ts index 21452870d1a..b70954e9aa8 100644 --- a/packages/paperclip-runner/src/live/runnerd-codex-transport.ts +++ b/packages/paperclip-runner/src/live/runnerd-codex-transport.ts @@ -857,6 +857,11 @@ function runnerCloseGraceMs(options: Pick; + #resolveHandlerReady!: () => void; #core: DurablePrpControlPlane | null = null; #handle: RunnerProcessHandle | null = null; #adoptedRunnerMonitor: NodeJS.Timeout | null = null; @@ -3622,6 +3630,9 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { this.#failureSignal = new Promise((_resolve, reject) => { this.#rejectFailureSignal = reject; }); + this.#handlerReady = new Promise(resolve => { + this.#resolveHandlerReady = resolve; + }); this.#closedSignal = new Promise(resolve => { this.#resolveClosedSignal = resolve; }); @@ -3997,6 +4008,8 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { setServerRequestHandler(handler: CodexServerRequestHandler): void { this.#handler = handler; + this.#handlerInstalled = true; + this.#resolveHandlerReady(); } async #awaitWarmRunAttachmentReady(): Promise { @@ -5074,7 +5087,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { maxOutboxBytes: RUNNERD_MAX_OUTBOX_BYTES, p0ReserveBytes: RUNNERD_P0_RESERVE_BYTES, maxRuntimeMs: 0, - reconnectGraceMs: this.options.runnerReconnectGraceMs, + reconnectGraceMs: runnerProcessReconnectGraceMs(this.options), lifecyclePolicy: this.options.lifecyclePolicy, runnerBinaryPath, runnerVersion: runnerArtifact.version, @@ -5759,7 +5772,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { maxOutboxBytes: RUNNERD_MAX_OUTBOX_BYTES, p0ReserveBytes: RUNNERD_P0_RESERVE_BYTES, maxRuntimeMs: 0, - reconnectGraceMs: this.options.runnerReconnectGraceMs, + reconnectGraceMs: runnerProcessReconnectGraceMs(this.options), lifecyclePolicy: this.options.lifecyclePolicy, runnerBinaryPath, runnerVersion: runnerArtifact.version, @@ -5915,6 +5928,18 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { : await Promise.race([admission.settled, this.#failureSignal]).catch(() => { throw new SemanticToolNotDispatchedError(); }); + // An adopted Runner can authenticate and deliver queued tools before the + // driver has finished reconstructing its session. Handler publication is + // the dispatch barrier; the default RPC diagnostic is not a tool outcome. + // The owning startup/transport lifetime bounds this wait, and the original + // authority/epoch checks below still apply after it becomes ready. + if (!this.#handlerInstalled) { + await Promise.race([ + this.#handlerReady, + this.#failureSignal, + this.#closedSignal.then(() => { throw new SemanticToolNotDispatchedError(); }), + ]).catch(() => { throw new SemanticToolNotDispatchedError(); }); + } try { this.#throwIfFailed(); } catch { diff --git a/packages/paperclip-runner/src/live/runnerd-controller-outage.test.ts b/packages/paperclip-runner/src/live/runnerd-controller-outage.test.ts new file mode 100644 index 00000000000..2988e4845a9 --- /dev/null +++ b/packages/paperclip-runner/src/live/runnerd-controller-outage.test.ts @@ -0,0 +1,110 @@ +import { createServer } from "node:http"; +import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; +import { expect, it, vi } from "vitest"; +import type { DurablePrpControlPlane } from "../control-plane/durable-prp-control-plane.js"; +import { codexSemanticToolSpecs } from "../drivers/codex/codex-app-server-driver.js"; +import { + createCapabilityRunnerdCodexTransport, + defaultCapabilityRunnerdBinary, +} from "./runnerd-codex-transport.js"; + +it.each([ + { policy: "connection lease", processGraceMs: null, survives: true }, + { policy: "explicit process deadline", processGraceMs: 500, survives: false }, +])("preserves the $policy policy across controller downtime beyond the connection recovery budget", async ({ processGraceMs, survives }) => { + const root = await mkdtemp(join(tmpdir(), "runner-controller-outage-")); + const server = createServer(); + let authority: DurablePrpControlPlane | null = null; + server.on("upgrade", (request, socket, head) => { + if (!authority) { socket.destroy(); return; } + authority.handleUpgrade(request, socket, "/runner", head); + }); + await new Promise(resolveListen => server.listen(0, "127.0.0.1", resolveListen)); + const address = server.address(); + if (!address || typeof address === "string") throw new Error("Missing outage test listener"); + const registration = async (next: DurablePrpControlPlane) => { + authority = next; + return { + connectUrl: `ws://127.0.0.1:${address.port}/runner`, + release: () => { if (authority === next) authority = null; }, + }; + }; + const shared = { + runnerBinary: defaultCapabilityRunnerdBinary(), + codexCommand: resolve(import.meta.dirname, "../../runner/target/debug/fake-codex-app-server"), + codexArgs: ["--state-file", join(root, "provider-state.json"), "--hold-first-durable-turn"], + stateDirectory: root, + prpIdentity: { + runnerInstanceId: "runner-outage", environmentLeaseId: "lease-outage", runId: "run-outage", + normalizedSessionId: "session-outage", turnId: "turn-outage", itemId: "item-outage", + }, + // Scale the production 120-second reconnect budget down. The live process + // must be bounded by its authenticated connection lease, independently of + // how long a controller spends retrying a connection or booting its server. + runnerReconnectGraceMs: 500, + runnerProcessReconnectGraceMs: processGraceMs, + controlPlaneRegistration: registration, + }; + const first = createCapabilityRunnerdCodexTransport(shared); + let adopted: ReturnType | undefined; + let pid: number | null = null; + try { + const opened = await first.transport.request("thread/start", { + cwd: root, dynamicTools: codexSemanticToolSpecs(), + }); + await first.transport.request("turn/start", { input: [{ type: "text", text: "Keep this task active." }] }); + for await (const notification of first.transport.notifications()) { + if (notification.method === "turn/started") break; + } + pid = first.evidence().runnerPid; + const providerPid = first.evidence().codexPid; + expect(pid).toEqual(expect.any(Number)); + expect(providerPid).toEqual(expect.any(Number)); + await writeFile(join(root, "saved-work.txt"), "completed work survives controller replacement"); + await first.detachControllerForRestart(); + // The deliberate fault is an offline controller, not a slow test assertion. + await new Promise(resolveWait => setTimeout(resolveWait, 1_250)); + const runnerState = JSON.parse(await readFile(join(root, "runner", "runner-state.json"), "utf8")); + if (!survives) { + expect(runnerState.recoverableFailure).toBe("transport_reconnect_grace_exceeded"); + await vi.waitFor(() => expect(() => process.kill(pid!, 0)).toThrow()); + return; + } + expect(runnerState.recoverableFailure).not.toBe("transport_reconnect_grace_exceeded"); + expect(() => process.kill(pid!, 0)).not.toThrow(); + expect(() => process.kill(providerPid!, 0)).not.toThrow(); + const duplicateLaunch = vi.fn(() => { throw new Error("Must adopt the existing runner"); }); + const thread = opened.thread as Record; + adopted = createCapabilityRunnerdCodexTransport({ + ...shared, + runnerReconnectGraceMs: 5_000, + resumeDynamicTools: codexSemanticToolSpecs(), + resumeProviderSession: { driverSessionId: String(thread.id), providerSessionId: String(thread.sessionId) }, + runnerProcessLauncher: duplicateLaunch, + adoptExistingRunner: { + pid: pid!, processGroupId: pid, startedAt: new Date().toISOString(), + isAlive: () => { try { process.kill(pid!, 0); return true; } catch { return false; } }, + signal: signal => { try { process.kill(-pid!, signal); return true; } catch { return false; } }, + }, + }); + adopted.transport.setServerRequestHandler(async () => ({ success: true, contentItems: [] })); + const restored = await adopted.transport.request("thread/read", {}); + expect(restored.thread).toMatchObject({ id: thread.id, sessionId: thread.sessionId }); + expect(adopted.evidence()).toMatchObject({ runnerPid: pid, codexPid: providerPid, runnerExited: false }); + expect(duplicateLaunch).not.toHaveBeenCalled(); + expect(await readFile(join(root, "saved-work.txt"), "utf8")).toBe("completed work survives controller replacement"); + // A real Stop still reaches the adopted owner; the lease-only disconnect + // policy cannot turn a detached process into an uncancellable executor. + await adopted.transport.close(); + expect(adopted.evidence().runnerExited).toBe(true); + } finally { + await first.transport.close().catch(() => undefined); + await adopted?.transport.close().catch(() => undefined); + if (pid) { try { process.kill(-pid, "SIGKILL"); } catch { /* already stopped */ } } + server.closeAllConnections(); + if (server.listening) await new Promise(resolveClose => server.close(() => resolveClose())); + await rm(root, { recursive: true, force: true }); + } +}, 30_000); diff --git a/packages/paperclip-runner/src/live/runnerd-semantic-handler-readiness.test.ts b/packages/paperclip-runner/src/live/runnerd-semantic-handler-readiness.test.ts new file mode 100644 index 00000000000..f669c4d4aca --- /dev/null +++ b/packages/paperclip-runner/src/live/runnerd-semantic-handler-readiness.test.ts @@ -0,0 +1,54 @@ +import { mkdtemp, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; +import { expect, it, vi } from "vitest"; +import type { DurablePrpControlPlane } from "../control-plane/durable-prp-control-plane.js"; +import type { CodexRpcServerRequest } from "../drivers/codex/app-server-transport.js"; +import { createCapabilityRunnerdCodexTransport, defaultCapabilityRunnerdBinary } from "./runnerd-codex-transport.js"; + +it.each(["install", "close", "detach"] as const)("holds an authenticated semantic call until handler readiness or %s", async mode => { + const root = await mkdtemp(join(tmpdir(), "runner-handler-readiness-")); + let core!: DurablePrpControlPlane; + const bundle = createCapabilityRunnerdCodexTransport({ + runnerBinary: defaultCapabilityRunnerdBinary(), + codexCommand: resolve(import.meta.dirname, "../../runner/target/debug/fake-codex-app-server"), + codexArgs: ["--state-file", join(root, "provider.json"), "--emit-tool-call"], + stateDirectory: root, + controlPlaneRegistration: async authority => { + core = authority; + await authority.start(); + return { release: () => undefined }; + }, + }); + const handler = vi.fn(async (_request: CodexRpcServerRequest) => ({ success: true, contentItems: [{ type: "inputText", text: "one retained result" }] })); + let pid: number | null = null; + try { + await bundle.transport.request("thread/start", { cwd: root, dynamicTools: [{ name: "get_task_context", description: "Read the task.", inputSchema: { type: "object", properties: {}, additionalProperties: false } }] }); + await bundle.transport.request("turn/start", { input: [{ type: "text", text: "Read the task." }] }); + pid = bundle.evidence().runnerPid; + expect(bundle.evidence().runnerProcessGroupId).toBe(pid); + await vi.waitFor(() => expect(core.store.state.committedEvents.some(event => event.eventType === "semantic_tool.input")).toBe(true)); + // The authenticated provider is already asking for a tool, but the driver + // has not published its recovered session/handler yet. No error result may + // be manufactured in this gap, and no external effect has been dispatched. + await new Promise(resolveWait => setTimeout(resolveWait, 50)); + expect(core.store.state.commands.filter(command => command.type === "semantic_tool.result")).toHaveLength(0); + expect(core.semanticToolResultsSettled()).toBe(false); + if (mode === "close") await bundle.transport.close(); + if (mode === "detach") await bundle.detachControllerForRestart(); + bundle.transport.setServerRequestHandler(handler); + if (mode === "install") { + await vi.waitFor(() => expect(handler).toHaveBeenCalledOnce()); + await vi.waitFor(() => expect(core.store.state.commands.filter(command => command.type === "semantic_tool.result")).toHaveLength(1)); + expect(core.store.state.commands.find(command => command.type === "semantic_tool.result")?.payload).toMatchObject({ isError: false }); + expect(handler.mock.calls[0]?.[0]).toMatchObject({ method: "item/tool/call", params: { tool: "get_task_context" } }); + } else { + await new Promise(resolveWait => setTimeout(resolveWait, 50)); + expect(handler).not.toHaveBeenCalled(); + } + } finally { + await bundle.transport.close().catch(() => undefined); + if (pid) { try { process.kill(-pid, "SIGKILL"); } catch { /* owned child already stopped */ } } + await rm(root, { recursive: true, force: true }); + } +}, 20_000); diff --git a/packages/paperclip-runner/test-fixtures/pi-acp/profile-v29-identity.json b/packages/paperclip-runner/test-fixtures/pi-acp/profile-v29-identity.json new file mode 100644 index 00000000000..664c0ab00c7 --- /dev/null +++ b/packages/paperclip-runner/test-fixtures/pi-acp/profile-v29-identity.json @@ -0,0 +1,51 @@ +{ + "commandDigest": "sha256:e80c6dc7bde12790371345ee5a12f0201d30941d26f1a780aec3d090545b2a06", + "declaration": { + "schema": "paperclip.acpx_profile_declaration.v1", + "agent": "pi", + "agentProfileVersion": 29, + "acpxVersion": "0.13.1", + "agentServerVersion": "0.0.33", + "agentRuntimeVersion": "1.0.0", + "nodeVersion": "24.21.0", + "closure": { + "darwin-arm64": "282022db10150c6632b3444df421342e7d534bdf5d5fb1097a2e79d0625a2bcf", + "darwin-x64": "64e251e19009f755c0b04f73ce2138246faab71a961b0f13d75ebfcc34bef12e", + "linux-x64": "713b1fdff42fb56a1518bdc084f181d70bee8ebadc3e4b1d76321ed9108c8410" + }, + "wrapperSha256": "9d129b3d38772e93e97080aa6c4e574ac5df4e47ce5bc331a484a9fbf8188b36", + "helperSha256": "41e0490b617da0d60e0c8ec58ef311236f945129d99f816cff6897f78da7f91d", + "extensionSha256": "f5818573a355702388072b28926897db546a5cf346b5171f646582ce5f9102a4", + "agentFilesBinding": "registered-runtime-context-v1", + "nativeQuestions": "select-confirm-input-editor-v4-root-field-types", + "pathPolicy": "sdk-normalization-v1", + "mcpToolNames": "provider-safe-exact-source-v1", + "acpxPatchSha256": "00d6af17216cb0cad902b48633eb19bf0ec6b05837a984cb24661fca593f09c1", + "nativeAssistantMessages": "pi1-rpc-deltas-v3-bound-message-and-tool-receipts", + "nativeNotices": "session-extension-v2-bounded-runtime-failure", + "messageProjectionSha256": "17d1e91b948dfa6a56e1905645add52b502041ad843c8d50a13b07956f22aadd", + "noticeProjectionSha256": "014485adc690998e395d334ea3ae67f72be6060a2fc78a1185df7b8d1da28fbf", + "sharedRuntimeContract": "paperclip.acpx-runtime-contract.v1", + "nativeInputDisposition": "queued-only-v1", + "cacheWarming": "disabled-owned-decision-v1", + "thinkingLevel": { + "supported": [ + "off", + "low", + "high", + "max" + ], + "newConfigurationDefault": "low", + "admission": "explicit-native-effective-before-prompt-v1", + "recovery": "mode-bound-v1" + }, + "modelSelection": "caller-selected-native-verified-v1", + "providerConfigurationSourceSha256": "75a9a13f13f224dc092bd8ebf49e8747162172943761e44476c8eb18d80c52d4", + "credentialEnvironmentSourceSha256": "7586d44b47c83395498dc8806287cd3538ef58177ba7c5949f5766ed105a0161", + "runtimeSandboxSourceSha256": "92dfa53f5a3ed0062913ae417eeccec243b430d444495efde2a8e9c4c02e78ea", + "recoveryIdentitySourceSha256": "d2c9fc161fcc135fb9760110bfa3d984ee2bb38d87fa5ce7cb7418a49dd89e66", + "nativeCancellation": "acknowledged-cancelled-terminal-preserves-usage-v1", + "dependencySecurityPatchSha256": "5273a195b952f233336122842faaf650193c04f36b9359d20d4d63079b0ce814", + "openRouterAttributionSourceSha256": "1211191529d6073b733b311c3d69cada4973eede0f3e540c46e261f78e0b90f9" + } +} diff --git a/packages/paperclip-runner/test/fixtures/copilot-profile-v59-identity.json b/packages/paperclip-runner/test/fixtures/copilot-profile-v59-identity.json new file mode 100644 index 00000000000..b087d433186 --- /dev/null +++ b/packages/paperclip-runner/test/fixtures/copilot-profile-v59-identity.json @@ -0,0 +1,85 @@ +{ + "commandDigest": "sha256:2f39ffe54a6af832d044b5098c31df9e08d4d27aee0ac675ba4b0c9da7445c40", + "declaration": { + "schema": "paperclip.acpx_profile_declaration.v1", + "agent": "copilot", + "agentProfileVersion": 59, + "acpxVersion": "0.13.1", + "agentServerVersion": "1.0.88", + "protocolPolicyRevision": "copilot-agent-manual-v1", + "acpxPatchSha256": "00d6af17216cb0cad902b48633eb19bf0ec6b05837a984cb24661fca593f09c1", + "policySha256": "a4be69be43db84052560420a3ca546a665c9165573c24d543eb33d144ba138a3", + "distributionSourceSha256": "c0a65e247fc47b263bd369a3d5ab505c4b323d4e6493be8a9ecbeb0e9dbb9107", + "agentFilesBinding": "registered-runtime-context-v1", + "systemInstructionDelivery": "COPILOT_HOME/copilot-instructions.md:replace-under-lifetime-lease-before-launch:v1", + "sharedRuntimeContract": "paperclip.acpx-runtime-contract.v1", + "innerDistributionSourceSha256": "9bb08347eed01f1ab44e065c8b06ead9be3a25973d6e6329dc4c0a099096279a", + "upstreamAppSha256": "7b48282a19b5b0814a0c96ad5e3a125173d792cc55f9525b2effea962f6063c0", + "patchedAppSha256": "61f8eebe0c30cc03b311c6818b2acb158a6d16ca58c8bfac5ebd42db095c2679", + "messageIdentityContract": "copilot-native-message-id-v1", + "ownedDistributionDelivery": "COPILOT_CLI_DIST_DIR:lease-owned-guarded-inner-distribution:v1", + "permissionContextContract": "copilot-edit-permission-context-v1", + "permissionContextSourceSha256": "de2d38d702b3c574161af0c0c04790fb6f6347916eb3916ce7a131a8c229121b", + "permissionAdapterSourceSha256": "ee5929ed2ee647ad0369465dee9d51a00b2dd28e1ff735b43e969ddc9c164668", + "permissionLocationsSourceSha256": "f39a81da395b6e23aa2a1a45572a28846e5e81bfd3d764d9a456f74a79c752e9", + "permissionRedactionSourceSha256": "ad6770c69269087951790e2e87e22ac088e3d432769adf17ebc865ed512e6c4f", + "permissionClassifierSourceSha256": "1161eabd82e26133e3b3db3ab567f65d52808b03f176e93497e02c8330349fb4", + "semanticToolReceiptContract": "paperclip.semantic_tool_receipt.v2", + "semanticReceiptSourceSha256": "b2795726b2234a663fec1a6ca11f0551ae69882fe9e82aa39686e952d036e791", + "semanticBridgeSourceSha256": "8764b8c9032fdc7d1415b01e75a9fc57c550e7a5e11da4bed7d9100082872ae5", + "toolEvidenceSourceSha256": "6eef5039afc75bb851eaf9cd82a6f099e4bd398ca2deb59dd66f5b2a81a7a706", + "semanticNormalizedInputContract": "validated-forwarded-input-commit-v1", + "semanticSidecarSourceSha256": "17361d1782c0dcebb2b7972635f45c890f2a4081f8bc0ef2d2276550e0b91629", + "semanticDirectDriverSourceSha256": "ff572638f2cb5d9f8249ff8e2f412311c1e3af209ef0bf7e5ea9f10a21d91103", + "semanticValidationSourceSha256": "02fbb92681c78c5de5d2f1aae25aabc25a0e5df6b0ff4baccac652192a8da975", + "semanticNormalizationSourceSha256": "8b628b82e8eacc91cdcc6fdca2210ba4cf282f01e9c6f74e95ff271655b5047e", + "semanticCompletionContractSourceSha256": "3343df6362252d44d12b079ff78c8f6539d1544f3ce5ec36ff36db70ab0fac31", + "semanticValidatorsSourceSha256": "f22d9caf2035b600abc1a6625ae23ca97f1409d7e29ed5b7264bb8f9784c7dc9", + "semanticSchemaBundleSourceSha256": "016391a087cc217aafed4ba1f4efafc6e4881a7525952dce824b7f4d24530603", + "semanticSidecarProtocolSourceSha256": "03c0c95b900f9dc18f29f6be27b8c79bc0bcab29780190d46a74e95328081236", + "semanticSidecarReceiptCommitContract": "correlated-tool-resolve-v1", + "displayEventSourceSha256": "d72f9a6035c247bc6a42c067c05753076c54322c4b58984e93922f3325dc6877", + "displayProjectionSourceSha256": "75952606b26dd93bbc582c1f0a4424487377272ea745fb3a27ceae031fd5dc7e", + "configuredEnvironmentSourceSha256": "5c9008d121979a3a8987e6b8a5752172eafdc4a2b5d725a73f3bbb8fb83be8ff", + "configuredEnvironmentRustSourceSha256": "85c87fae62b8a9377e4f0a4c00302c7519fae4397b1c48253f8f2b7ba7faf994", + "profileActivitySourceSha256": "4fdb6b0f9125fd8372ba6f27945fa5c8a3d3b2ec891df544092e3dc7a75d5282", + "profileExtensionsSourceSha256": "a80ddc95eb4505095bfdbbf85c758199949ab20be9c84b34c77f762eb7180571", + "profileInstallationSourceSha256": "3e61c4cdd4b78e38fcb16eaadae4f1fd65ce3c7127184ea7a80948c8e5d77f88", + "attachmentPolicySourceSha256": "4f7638b7cc64e18f6a4ba33fd46de2b3d49225e32aeab41f5c695b268fa4f8e2", + "singleReadEvidenceSourceSha256": "4069393405089bde613cecec04f5123c426594811da4c668bd96f0e7e283550f", + "nativeSessionRuntimeSourceSha256": "1fe6fa9e6b96d84d70432b85a57c473f833606d4a521563d5022b4d3c41576ce", + "nativeRuntimeContextSourceSha256": "71607f80dbb8dc5cc6c123d0b8f3995d6c5407277695a8c82cbedc90fb8f4842", + "nativeRuntimePromptsSourceSha256": "103b83066da72434602e9e461e6278dfb40bb993d2f3b49b2e93df81bbb08b72", + "nativeSessionContractSourceSha256": "22a8c96ce41ba4f03e976fc37f0cc7c5d05cee99fa789941da31ee1dab440b90", + "harnessSessionContractSourceSha256": "03f4cd7f76f34f6d82e009b66aa51f18f6e23dc235b660778060e35b6ea07412", + "harnessBackendSourceSha256": "dc827e281ae6731545b166ee4d709a4f67f594a556dcbe63c70fe8f1aca26249", + "harnessSessionSourceSha256": "81361733e86b56dd5a66415aefc6f72e69be24acc8d3417370154ebb7fef6e94", + "transportContractSourceSha256": "03fca93d36a89b0cc0917335e293ad097fdf04356d21b307827bb0ea4c13f523", + "nativeBackendSourceSha256": "972c08e6e320d1384221e180391fe817ea1ba632db687201e457d64e68ef4360", + "nativeBackendFactorySourceSha256": "7e431c926b76d53c4c08e3a4105353dcd34bd940f6588a1a68f79631f6feeb44", + "nativeInstructionsSourceSha256": "9173ebccf09cb56a781d2bf5c7eb27fdafb13d3450d99342c6aff86b42d6b43a", + "providerLaunchSourceSha256": "d4f4a69baeeb2f3b2f350840f81f711e40f08588c28ea28039748dde90302a51", + "launchPreparationStoreSourceSha256": "ff6025568bbc0aa6ce545772d73dfca2d3da226740bd3a7b386b069e76cff296", + "runtimeRequestBridgeSourceSha256": "7497083b6cf1fc0268187aff948352f3bce6e054ac6e845005846473efdaff15", + "runtimeRequestFacadeSourceSha256": "d0ed28e0c5ddad34db8f437e704c8a755eb0b47f27e08e44185f66d1d2205295", + "runtimeRequestNotificationSourceSha256": "6196c6175f11c69295dc00fc773201f4fdeca26c500653870253c43b81e65211", + "guardedInstallationSourceSha256": "56a177d3e702914818dca4ca50e061f0d25f988bfc4c0582c36ce23c1696d82f", + "nativeExecutionSourceSha256": "fc7c6568feb0d8d709e9baa51839420c9b90336b4865a9b8405aba06e4d1e9c4", + "profileCompatibilitySourceSha256": "6a26b102b78f282137ff93b0f75eb8788c603fc177802c58715c5ad7b2f3b9e3", + "installationSourceSha256": "a1c7c6f103b898216973e77ffbac0ef7572e250aa574d1a4065270e9dfa0dc2d", + "assetOwnershipSourceSha256": "1785e3f31e5dad5a13b53e25d8cf9482a9bd89a27b6c127830aed6b81acc605c", + "assetInstallerSourceSha256": "d3bd9514e5920d0cc3567aa50487fb534c57a6f66b8da0d3f22c5f6e499c841f", + "credentialBindingSourceSha256": "7586d44b47c83395498dc8806287cd3538ef58177ba7c5949f5766ed105a0161", + "sidecarErrorSourceSha256": "b09af4577f8bf42eee7036b2ad9842202cfc763e72b53c3287f82c6090cdad80", + "profileSourceSha256": "f5538b3dbefa7634a332b87d5ededcefb56aa7e5790cede1c7a0d948a548a456", + "metadataProbeSourceSha256": "be4c8416edd27061109bc1f33d0c26071b4facbb3f651beaae6b3c0275cce727", + "permissionToolIdentitySourceSha256": "770b47305e571a81f344883057d0e4a482a1cd59ac77d045836593a8e739deb1", + "semanticContinuationSourceSha256": "d248e47a0fe7f2536d1e64cea0b0b410f34104800cd643d995b4473104ecc8e0", + "runtimeHostSourceSha256": "94a73749267f5a80b5df19413a65b4b9263c00a9bf7e98fda00f00b8f9fd32ec", + "runtimeAdapterSourceSha256": "e4c764c2c3c7dd655a965b39472bd1ad8d577d1e7258b25594b171c9429752b8", + "runtimeSandboxSourceSha256": "92dfa53f5a3ed0062913ae417eeccec243b430d444495efde2a8e9c4c02e78ea", + "cancellationSourceSha256": "a07d1a9e982dec0ae130c1ad1efa5bc75c6a39e013b730ba3ee45256f09ceeca", + "agentFilesSourceSha256": "fb32f6badc13dfa22f231479509b631b2f0eba0c99b1ab9f3cd5c0c601aa03ba", + "nativeDistributionSourceSha256": "aed327c389e957483a503cb1bb8b247ee7cd7ff4b528fb4567c39bcb4577690b" + } +} diff --git a/packages/shared/src/index.ts b/packages/shared/src/index.ts index 99f0a885507..5ad42e5a7a1 100644 --- a/packages/shared/src/index.ts +++ b/packages/shared/src/index.ts @@ -889,6 +889,7 @@ export type { AgentDetail, ClearAgentErrorResponse, AgentPermissions, + AgentProjectAccess, AgentPublicIdentity, AgentInstructionsBundleMode, AgentInstructionsFileSummary, @@ -1130,6 +1131,7 @@ export type { IssueExecutionStagePrincipal, IssueExecutionDecision, IssueComment, + IssueRecentComments, IssueQueuedCommentEntry, IssueQueuedCommentProtocol, IssueQueuedCommentQueue, @@ -1882,6 +1884,7 @@ export { qualifyAgentRunnerSchema, agentPermissionsSchema, updateAgentPermissionsSchema, + addAgentProjectAccessSchema, type CreateAgent, type BuiltInAgentProvision, type BuiltInAgentReset, diff --git a/packages/shared/src/types/agent.ts b/packages/shared/src/types/agent.ts index ac1970305b7..ba4b594389b 100644 --- a/packages/shared/src/types/agent.ts +++ b/packages/shared/src/types/agent.ts @@ -25,6 +25,11 @@ export interface AgentPermissions extends Record { authorizationPolicy?: TrustAuthorizationPolicy; } +export interface AgentProjectAccess { + projectIds: string[]; + canManage: boolean; +} + export type AgentRuntimeConfig = Record & { aiConnection?: import("../ai-connection-router.js").AiRuntimeConnectionBinding; }; diff --git a/packages/shared/src/types/execution-projection.ts b/packages/shared/src/types/execution-projection.ts index 23c7d3d688d..ff28fd01dc1 100644 --- a/packages/shared/src/types/execution-projection.ts +++ b/packages/shared/src/types/execution-projection.ts @@ -104,7 +104,9 @@ export interface TaskInterruptionPresentation { state: "working" | "retry_scheduled" | "failed" | "waiting_for_user" | "resolved"; nextRetryAt: string | null; canRetry: boolean; - userAction: "approval" | "reconnect" | "assign_agent" | "choose_model" | "resume" | "budget" | null; + userAction: "approval" | "reconnect" | "assign_agent" | "choose_model" | "resume" | "budget" | "project_access" | null; + /** Only the task's own project id is exposed. Other source projects stay unnamed. */ + projectAccess?: { agentId: string; projectId: string | null; ready: boolean }; sourceRunIds: string[]; observedRunId: string | null; } diff --git a/packages/shared/src/types/index.ts b/packages/shared/src/types/index.ts index c89aa62b1e3..48f9fe6b6f4 100644 --- a/packages/shared/src/types/index.ts +++ b/packages/shared/src/types/index.ts @@ -280,6 +280,7 @@ export type { AgentDetail, ClearAgentErrorResponse, AgentPermissions, + AgentProjectAccess, AgentPublicIdentity, AgentRuntimeConfig, AgentInstructionsBundleMode, @@ -731,6 +732,7 @@ export type { IssueReviewRequest, IssueExecutionDecision, IssueComment, + IssueRecentComments, IssueQueuedCommentEntry, IssueQueuedCommentProtocol, IssueQueuedCommentQueue, diff --git a/packages/shared/src/types/issue.ts b/packages/shared/src/types/issue.ts index 0e7b0c2a133..cb6b7a5b622 100644 --- a/packages/shared/src/types/issue.ts +++ b/packages/shared/src/types/issue.ts @@ -1720,3 +1720,10 @@ export interface IssueAttachment { openPath?: string; downloadPath?: string; } + +/** Bounded cold-open window; the complete chronological feed stays paginated. */ +export interface IssueRecentComments { + latestCommentId: string | null; + comments: IssueComment[]; + hasMore: boolean; +} diff --git a/packages/shared/src/validators/agent.ts b/packages/shared/src/validators/agent.ts index 32d65555e6d..1e9e42e2876 100644 --- a/packages/shared/src/validators/agent.ts +++ b/packages/shared/src/validators/agent.ts @@ -322,6 +322,10 @@ export const updateAgentPermissionsSchema = z.object({ export type UpdateAgentPermissions = z.infer; +export const addAgentProjectAccessSchema = z.object({ + projectIds: z.array(z.string().uuid()).min(1).max(100), +}).strict(); + export const resolveAgentInstructionCandidateSchema = z.object({ baseRevisionId: z.string().uuid().nullable(), content: z.string().max(1024 * 1024), diff --git a/packages/shared/src/validators/index.ts b/packages/shared/src/validators/index.ts index 05519dd262f..8f1ba1bf907 100644 --- a/packages/shared/src/validators/index.ts +++ b/packages/shared/src/validators/index.ts @@ -370,6 +370,7 @@ export { qualifyAgentRunnerSchema, agentPermissionsSchema, updateAgentPermissionsSchema, + addAgentProjectAccessSchema, type CreateAgent, type BuiltInAgentProvision, type BuiltInAgentReset, diff --git a/scripts/pi-public-install-probe.mjs b/scripts/pi-public-install-probe.mjs index 5e6672cc68f..945d8c902de 100644 --- a/scripts/pi-public-install-probe.mjs +++ b/scripts/pi-public-install-probe.mjs @@ -52,8 +52,8 @@ assert.equal(JSON.parse(await readFile(join(server, 'package.json'), 'utf8')).na const { resolvePaperclipRunnerBinary } = await import(pathToFileURL(join(server, 'dist/services/native-runtime/native-codex-runner.js'))); const { createCapabilityRunnerdCodexTransport } = await import(pathToFileURL(join(server, 'dist/vendor/paperclip-runner/live/runnerd-codex-transport.js'))); const { QUALIFIED_ACPX_PROFILES } = await import(pathToFileURL(join(server, 'dist/vendor/paperclip-runner/drivers/acpx/qualified-profiles.js'))); -assert.equal(QUALIFIED_ACPX_PROFILES.pi.agentProfileVersion, 28); -assert.equal(QUALIFIED_ACPX_PROFILES.pi.commandDigest, 'sha256:4918a4d5472d66b22b1ec642b68b5c60a8092a39c84ec1ab1c9ad0289cb8c68c'); +assert.equal(QUALIFIED_ACPX_PROFILES.pi.agentProfileVersion, 29); +assert.equal(QUALIFIED_ACPX_PROFILES.pi.commandDigest, 'sha256:e80c6dc7bde12790371345ee5a12f0201d30941d26f1a780aec3d090545b2a06'); assert.equal(Object.hasOwn(QUALIFIED_ACPX_PROFILES.pi, 'qualificationModel'), false); assert.equal(Object.hasOwn(QUALIFIED_ACPX_PROFILES.pi, 'reportedModelId'), false); const daemon = resolvePaperclipRunnerBinary(); diff --git a/server/scripts/bundle-runtime.test.mjs b/server/scripts/bundle-runtime.test.mjs index 14b0d46c681..6e9375f61c4 100644 --- a/server/scripts/bundle-runtime.test.mjs +++ b/server/scripts/bundle-runtime.test.mjs @@ -2,7 +2,7 @@ import { execFile } from "node:child_process"; import { mkdirSync, mkdtempSync, realpathSync, rmSync, symlinkSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; import path from "node:path"; -import { pathToFileURL } from "node:url"; +import { fileURLToPath, pathToFileURL } from "node:url"; import { promisify } from "node:util"; import { afterEach, beforeEach, describe, expect, it } from "vitest"; @@ -85,6 +85,33 @@ describe("bundleRuntime", () => { expect(output.entryDir).toBe(path.join(root, "server", "dist")); }); + it("keeps database caller locations public in a source-mapped runtime bundle", async () => { + const captureModule = fileURLToPath(new URL("../../packages/db/src/response-capture.ts", import.meta.url)); + write(path.join(root, "server", "dist", "services", "response-caller.js"), [ + `import { publicDatabaseCallers } from ${JSON.stringify(captureModule)};`, + 'export function callers() { return publicDatabaseCallers(new Error("private-capture-message").stack); }', + ].join("\n")); + write(path.join(root, "server", "dist", "index.js"), [ + 'import { callers } from "./services/response-caller.js";', + 'console.log(JSON.stringify({ callers: callers() }));', + ].join("\n")); + const summary = await bundleRuntime({ entry: path.join(root, "server", "dist", "index.js"), outdir: path.join(root, "server", "dist-runtime") }); + const env = { ...process.env }; + delete env.NODE_OPTIONS; + delete env.PAPERCLIP_DB_RESPONSE_CAPTURE_WINDOW; + const options = { cwd: path.join(root, "server"), env, timeout: 30_000 }; + const plain = JSON.parse((await execFileAsync(process.execPath, [summary.outfile], options)).stdout); + const mapped = JSON.parse((await execFileAsync(process.execPath, ["--enable-source-maps", summary.outfile], options)).stdout); + expect(plain.callers).toEqual([]); + expect(mapped.callers).toEqual([expect.stringMatching(/^services\/response-caller\.js:\d+:\d+$/)]); + for (const result of [plain, mapped]) { + const output = JSON.stringify(result); + expect(output).not.toContain(root); + expect(output).not.toContain(captureModule); + expect(output).not.toContain("private-capture-message"); + } + }); + it("emits a server module that loads without starting the server", async () => { const outdir = path.join(root, "server", "dist-runtime"); const summary = await bundleRuntime({ entry: path.join(root, "server", "dist", "index.js"), outdir }); diff --git a/server/src/__tests__/activity-routes.test.ts b/server/src/__tests__/activity-routes.test.ts index 578c8a2738a..e8f7d67d258 100644 --- a/server/src/__tests__/activity-routes.test.ts +++ b/server/src/__tests__/activity-routes.test.ts @@ -5,6 +5,7 @@ import { beforeEach, describe, expect, it, vi } from "vitest"; const mockActivityService = vi.hoisted(() => ({ list: vi.fn(), forIssue: vi.fn(), + inputFactsForIssue: vi.fn(), runsForIssue: vi.fn(), issuesForRun: vi.fn(), create: vi.fn(), @@ -152,6 +153,22 @@ describe("activity routes", () => { expect(res.body).toEqual([{ action: "issue.recovery_action_resolved", actorType: "system" }]); }); + it("reads bounded input facts with the same issue access checks", async () => { + const commentId = "11111111-1111-4111-8111-111111111111"; + mockIssueService.getByIdentifier.mockResolvedValue({ id: "issue-1", companyId: "company-1" }); + mockActivityService.inputFactsForIssue.mockResolvedValue([{ action: "issue.queued_comment_steered" }]); + const app = await createApp({ type: "agent", companyId: "company-1", agentId: "agent-1" }); + const res = await requestApp(app, baseUrl => request(baseUrl).get(`/api/issues/TEST-1/activity?view=comment-inputs&commentIds=${commentId}`)); + expect(res.status).toBe(200); + expect(mockActivityService.inputFactsForIssue).toHaveBeenCalledWith("company-1", "issue-1", [commentId]); + expect(mockAccessService.decide).toHaveBeenCalledWith(expect.objectContaining({ action: "issue:read" })); + const invalid = await requestApp(app, baseUrl => request(baseUrl).get("/api/issues/TEST-1/activity?view=comment-inputs&commentIds=invalid")); + expect(invalid.status).toBe(400); + const oversized = await requestApp(app, baseUrl => request(baseUrl).get(`/api/issues/TEST-1/activity?view=comment-inputs&commentIds=${Array(101).fill(commentId).join(",")}`)); + expect(oversized.status).toBe(400); + expect(mockActivityService.inputFactsForIssue).toHaveBeenCalledTimes(1); + }); + it.each(["boundary", "company"])("denies agent activity reads outside the %s scope", async (kind) => { const issue = { id: "issue-1", companyId: kind === "company" ? "company-2" : "company-1" }; mockIssueService.getById.mockResolvedValue(issue); @@ -161,6 +178,9 @@ describe("activity routes", () => { const res = await requestApp(app, baseUrl => request(baseUrl).get("/api/issues/issue-1/activity")); expect(res.status).toBe(kind === "company" ? 404 : 403); expect(mockActivityService.forIssue).not.toHaveBeenCalled(); + const inputs = await requestApp(app, baseUrl => request(baseUrl).get("/api/issues/issue-1/activity?view=comment-inputs")); + expect(inputs.status).toBe(kind === "company" ? 404 : 403); + expect(mockActivityService.inputFactsForIssue).not.toHaveBeenCalled(); }); it("returns redacted all-actors rows to a basic company reader", async () => { diff --git a/server/src/__tests__/agent-auth-identity-transfer.test.ts b/server/src/__tests__/agent-auth-identity-transfer.test.ts new file mode 100644 index 00000000000..819b23dc64a --- /dev/null +++ b/server/src/__tests__/agent-auth-identity-transfer.test.ts @@ -0,0 +1,102 @@ +import { randomUUID } from "node:crypto"; +import express from "express"; +import request from "supertest"; +import { eq } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; +import { agents, companies, createDb, heartbeatRunEvents, heartbeatRuns, issueComments, issues } from "@paperclipai/db"; +import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { createLocalAgentJwt } from "../agent-auth-jwt.js"; +import { actorMiddleware } from "../middleware/auth.js"; +import { errorHandler } from "../middleware/error-handler.js"; +import { captureRunIdentity, initializeRunIdentity, reserveSteeredIdentity } from "../services/run-identity.js"; + +const support = await getEmbeddedPostgresTestSupport(); +(support.supported ? describe : describe.skip)("agent authentication identity transfer", () => { + let database: Awaited>; + let db: ReturnType; + beforeAll(async () => { + vi.stubEnv("PAPERCLIP_AGENT_JWT_SECRET", "identity-transfer-test-secret"); + vi.stubEnv("PAPERCLIP_INSTANCE_ID", "default"); + database = await startEmbeddedPostgresTestDatabase("paperclip-auth-identity-transfer-"); + db = createDb(database.connectionString); + }, 30_000); + afterAll(async () => { + vi.unstubAllEnvs(); + await db?.$client.end({ timeout: 5 }); + await database?.cleanup(); + }, 60_000); + + async function seed() { + const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), runId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Authentication transfer", issuePrefix: companyId.slice(0, 8) }); + await db.insert(agents).values({ id: agentId, companyId, name: "Reader", role: "engineer", adapterType: "codex_local", status: "idle" }); + await db.insert(issues).values({ id: issueId, companyId, title: "Authentication task", responsibleUserId: "owner" }); + const padding = "retained input ".repeat(32_768); + const payloads = { + contextSnapshot: { issueId, launchInput: padding }, + runnerProfileJson: { nativeExecutionInput: { savedInput: padding } }, + resultJson: { savedResult: padding }, stdoutExcerpt: padding, stderrExcerpt: padding, + }; + await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running", ...payloads }); + const identity = await initializeRunIdentity(db, { companyId, runId, issueId, messageIds: [], responsibleUserId: "owner", cause: "instruction" }); + const app = express(); + app.use(actorMiddleware(db, { deploymentMode: "authenticated", resolveSession: async () => null })); + app.get("/actor", (req, res) => res.json(req.actor)); + app.patch("/actor", (req, res) => res.json(req.actor)); + app.use(errorHandler); + const token = createLocalAgentJwt(agentId, companyId, "codex_local", runId, "stale-user"); + expect(token).not.toBeNull(); + return { companyId, agentId, issueId, runId, app, token: token!, identity, payloads }; + } + + it("bounds actual middleware responses and preserves the complete stored run", async () => { + const fixture = await seed(); + const events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(db, { heartbeatRuns }, capture); + try { + const response = await request(fixture.app).get("/actor").set("Authorization", `Bearer ${fixture.token}`); + expect(response.status).toBe(200); + expect(response.body).toMatchObject({ type: "agent", runId: fixture.runId, + identityContextId: fixture.identity.id, onBehalfOfUserId: "owner" }); + } finally { capture.flush(); } + const reads = events.filter(event => event.event === "paperclip.db_response_capture.entry.v1" + && event.table === "heartbeat_runs" && (event.operation === "select" || event.operation === "execute")); + expect(reads.reduce((sum, event) => sum + Number(event.calls), 0)).toBeGreaterThanOrEqual(3); + expect(reads.reduce((sum, event) => sum + Number(event.responseValueBytes), 0)).toBeLessThan(16_384); + expect((await captureRunIdentity(db, fixture)).run).toMatchObject(fixture.payloads); + }); + + it("blocks uncertain steering, accepts its durable acknowledgement, and rechecks cancellation", async () => { + const fixture = await seed(), messageId = randomUUID(); + await db.insert(issueComments).values({ id: messageId, companyId: fixture.companyId, issueId: fixture.issueId, + authorUserId: "next-user", body: "Next instruction" }); + const pending = await reserveSteeredIdentity(db, { ...fixture, messageId }); + expect(pending).not.toBeNull(); + const blocked = await request(fixture.app).get("/actor").set("Authorization", `Bearer ${fixture.token}`); + expect(blocked.status).toBe(409); + await db.insert(heartbeatRunEvents).values({ companyId: fixture.companyId, agentId: fixture.agentId, runId: fixture.runId, + seq: 1, eventType: "item.completed", sourceEventId: randomUUID(), + payload: { prpEvent: { turnId: "auth-turn", itemId: `auth-turn:steer:${messageId}`, + payload: { kind: "steering_acknowledgement" } } } }); + const accepted = await request(fixture.app).get("/actor").set("Authorization", `Bearer ${fixture.token}`); + expect(accepted.status).toBe(200); + expect(accepted.body).toMatchObject({ identityContextId: pending!.id, onBehalfOfUserId: "next-user" }); + await db.update(heartbeatRuns).set({ status: "cancelled" }).where(eq(heartbeatRuns.id, fixture.runId)); + const revoked = await request(fixture.app).patch("/actor").set("Authorization", `Bearer ${fixture.token}`); + expect(revoked.status).toBe(403); + expect(revoked.body.code).toBe("agent_run_cancelled"); + const [stored] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId)); + expect(stored).toMatchObject(fixture.payloads); + }); + + it.each(["company", "agent", "missing", "finished"] as const)("rejects %s run identity for the compact scope", async (fault) => { + const fixture = await seed(); + if (fault === "finished") await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, fixture.runId)); + const input = { companyId: fault === "company" ? randomUUID() : fixture.companyId, + agentId: fault === "agent" ? randomUUID() : fixture.agentId, + runId: fault === "missing" ? randomUUID() : fixture.runId }; + await expect(captureRunIdentity(db, input, { readScope: "request_authentication" })).rejects.toThrow(/active run/); + }); +}); diff --git a/server/src/__tests__/agent-live-run-routes.test.ts b/server/src/__tests__/agent-live-run-routes.test.ts index 94f6d525268..7d877cd9e54 100644 --- a/server/src/__tests__/agent-live-run-routes.test.ts +++ b/server/src/__tests__/agent-live-run-routes.test.ts @@ -16,6 +16,7 @@ const mockHeartbeatService = vi.hoisted(() => ({ getActiveRunIssueSummaryForAgent: vi.fn(), getRunReadAccess: vi.fn(), getRunLogAccess: vi.fn(), + getRunWorkspaceAccess: vi.fn(), readLog: vi.fn(), wakeup: vi.fn(), getRun: vi.fn(), @@ -378,6 +379,10 @@ describe("agent live run routes", () => { logStore: "local_file", logRef: "logs/aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa.ndjson", }); + mockHeartbeatService.getRunWorkspaceAccess.mockResolvedValue({ + id: "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa", companyId: "company-1", scopeKind: "company", issueId: null, + executionWorkspaceId: null, + }); mockHeartbeatService.readLog.mockResolvedValue({ runId: "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa", store: "local_file", @@ -609,10 +614,10 @@ describe("agent live run routes", () => { }); }); - it("uses narrow run log metadata lookups for log polling", async () => { + it.each([false, true])("uses narrow run log metadata lookups (aligned tail: %s)", async (aligned) => { const res = await requestApp(await createApp(), (baseUrl) => request(baseUrl).get( - "/api/heartbeat-runs/aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa/log?offset=12&limitBytes=64", + `/api/heartbeat-runs/aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa/log?offset=12&limitBytes=64${aligned ? "&alignStartToRecord=true" : ""}`, ), ); @@ -630,6 +635,8 @@ describe("agent live run routes", () => { { offset: 12, limitBytes: 64, + signal: expect.any(AbortSignal), + ...(aligned ? { alignStartToRecord: true } : {}), }, ); expect(res.body).toEqual({ @@ -687,6 +694,21 @@ describe("agent live run routes", () => { }, ); + it("lists workspace operations using narrow current run access", async () => { + mockHeartbeatService.getRunWorkspaceAccess.mockResolvedValue({ + id: "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa", companyId: "company-1", scopeKind: "company", issueId: null, + executionWorkspaceId: " workspace-one ", + }); + mockWorkspaceOperationService.listForRun.mockResolvedValue([]); + const response = await requestApp(await createApp(), baseUrl => request(baseUrl) + .get("/api/heartbeat-runs/aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa/workspace-operations")); + expect(response.status).toBe(200); + expect(response.body).toEqual([]); + expect(mockHeartbeatService.getRunWorkspaceAccess).toHaveBeenCalledWith("aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa"); + expect(mockHeartbeatService.getRun).not.toHaveBeenCalled(); + expect(mockWorkspaceOperationService.listForRun).toHaveBeenCalledWith("aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa", "workspace-one"); + }); + it("caps company live run polling by default", async () => { const rows = Array.from({ length: 75 }, (_, index) => ({ id: `run-${index}`, diff --git a/server/src/__tests__/agent-project-access.test.ts b/server/src/__tests__/agent-project-access.test.ts new file mode 100644 index 00000000000..5b7b26baa34 --- /dev/null +++ b/server/src/__tests__/agent-project-access.test.ts @@ -0,0 +1,197 @@ +import { randomUUID } from "node:crypto"; +import express from "express"; +import request from "supertest"; +import { and, eq } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { agents, agentWakeupRequests, activityLog, authUsers, companies, companyMemberships, createDb, + executionWorkspaces, heartbeatRuns, issues, projects, projectWorkspaces, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS } from "@paperclipai/db"; +import { registerAgentProjectAccessRoutes } from "../routes/agent-project-access.js"; +import { errorHandler } from "../middleware/error-handler.js"; +import { assertTaskWorkspaceExecutionAccess, assertTaskWorkspaceSourceProjectAccess } from "../services/task-workspace-access.js"; +import { taskInterruptionPresentation } from "../services/task-interruption-presentation.js"; +import { ProjectAccessDeniedError } from "../services/project-access-denial.js"; +import { issueService } from "../services/issues.js"; +import { executionWorkspaceService } from "../services/execution-workspaces.js"; +import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; + +describe("additive project access", () => { + let database: Awaited> | undefined; + let db: ReturnType; + beforeAll(async () => { + const external = process.env.PAPERCLIP_TEST_DATABASE_URL; + if (external) db = createDb(external); + else { database = await startEmbeddedPostgresTestDatabase("agent-project-access-"); db = createDb(database.connectionString); } + }, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS); + afterAll(async () => { await db?.$client.end(); await database?.cleanup(); }); + + async function seed() { + const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), runId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Project access", issuePrefix: `A${companyId.slice(0, 6)}` }); + const selected = await db.insert(projects).values(["Existing", "Needed", "Third"].map(name => ({ companyId, name }))).returning(); + const permissions = { canCreateAgents: false, canCreateSkills: false, trustPreset: "low_trust_review", + authorizationPolicy: { reviewPreset: { id: "low_trust_review", version: 1, rawOutputDisposition: "quarantine" }, + customPolicy: { preserve: true }, trustBoundary: { mode: "low_trust_review", companyId, + projectIds: [selected[0]!.id], issueIds: [issueId], allowedToolClasses: ["git.read"], allowedSecretBindingIds: [] } } }; + await db.insert(agents).values({ id: agentId, companyId, name: "Reviewer", adapterType: "process", status: "idle", permissions }); + await db.insert(companyMemberships).values({ companyId, principalType: "agent", principalId: agentId, membershipRole: "member", status: "active" }); + const [issue] = await db.insert(issues).values({ id: issueId, companyId, title: "Saved message", status: "blocked", assigneeAgentId: agentId, projectId: selected[1]!.id }).returning(); + await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, issueId, status: "failed", errorCode: "setup_failed", + error: "Project is outside this low-trust boundary.", contextSnapshot: { issueId } }); + return { companyId, agentId, issue: issue!, runId, selected, permissions }; + } + function app(actor: express.Request["actor"]) { + const app = express(); app.use(express.json()); app.use((req, _res, next) => { req.actor = actor; next(); }); + registerAgentProjectAccessRoutes(app, db); app.use(errorHandler); return app; + } + const board = { type: "board" as const, source: "local_implicit" as const }; + + it("keeps a historical denial blocked after the task moves to an allowed project but retains its source", async () => { + const f = await seed(); + const [source] = await db.insert(projectWorkspaces).values({ companyId: f.companyId, projectId: f.selected[1]!.id, + name: "Original repository", cwd: "/tmp/project-access-original" }).returning(); + await db.update(issues).set({ projectWorkspaceId: source!.id, + workspaceSelection: { version: 1, source: "project", selection: { + kind: "configured_source", projectWorkspaceId: source!.id, mode: "managed_isolated", + } } }).where(eq(issues.id, f.issue.id)); + const moved = await issueService(db).update(f.issue.id, { projectId: f.selected[0]!.id }); + expect(moved).toMatchObject({ projectId: f.selected[0]!.id, projectWorkspaceId: source!.id }); + const actor = { type: "agent" as const, source: "agent_key" as const, agentId: f.agentId, companyId: f.companyId }; + await expect(assertTaskWorkspaceExecutionAccess(db, actor, f.companyId, { issueId: f.issue.id })) + .rejects.toBeInstanceOf(ProjectAccessDeniedError); + expect(await taskInterruptionPresentation(db, moved!)).toMatchObject({ + canRetry: false, userAction: "project_access", projectAccess: { ready: false, projectId: null }, + }); + expect((await request(app(board)).post(`/agents/${f.agentId}/project-access`) + .send({ projectIds: [f.selected[1]!.id] })).status).toBe(200); + expect(await taskInterruptionPresentation(db, moved!)).toMatchObject({ canRetry: true, projectAccess: { ready: true } }); + await expect(assertTaskWorkspaceExecutionAccess(db, actor, f.companyId, { issueId: f.issue.id })).resolves.toBeDefined(); + }); + + it.each(["applied", "pending"] as const)("rechecks the selected source after an %s repair instead of the old denied project", async repair => { + const f = await seed(); + const [source] = await db.insert(projectWorkspaces).values({ companyId: f.companyId, projectId: f.selected[0]!.id, + name: "Allowed repository", cwd: "/tmp/project-access-allowed" }).returning(); + await db.update(heartbeatRuns).set({ errorCode: "project_access_denied", error: null, + resultJson: { projectAccessDenied: { projectId: f.selected[1]!.id } } }).where(eq(heartbeatRuns.id, f.runId)); + const moved = await issueService(db).update(f.issue.id, { projectId: f.selected[0]!.id }); + if (repair === "applied") { + await issueService(db).update(f.issue.id, { projectWorkspaceId: source!.id, workspaceSelectionActor: board }); + } else { + await executionWorkspaceService(db).selectTaskWorkspace({ companyId: f.companyId, issueId: f.issue.id, actor: board, + selection: { kind: "configured_source", projectWorkspaceId: source!.id, mode: "managed_isolated" }, + expectedBindingRevision: 0, requestKey: randomUUID() }); + } + expect(await taskInterruptionPresentation(db, moved!)).toMatchObject({ canRetry: true, projectAccess: { ready: true } }); + // The original denied source was not granted, and presentation creates no execution. + expect((await db.select().from(agents).where(eq(agents.id, f.agentId)))[0]!.permissions).toEqual(f.permissions); + expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId))).toHaveLength(0); + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId))).toHaveLength(1); + if (repair === "pending") { + await db.update(issues).set({ workspaceBindingRevision: 2 }).where(eq(issues.id, f.issue.id)); + const stale = await taskInterruptionPresentation(db, moved!); + expect(stale).toMatchObject({ state: "failed", canRetry: false, userAction: null }); + expect(stale).not.toHaveProperty("projectAccess"); + } + }); + + it("keeps a pending existing selection blocked if its workspace is archived before retry", async () => { + const f = await seed(); + const task = await issueService(db).update(f.issue.id, { projectId: f.selected[0]!.id }); + const [workspace] = await db.insert(executionWorkspaces).values({ companyId: f.companyId, + projectId: f.selected[0]!.id, name: "Selected files", cwd: "/tmp/project-access-selected", + mode: "isolated_workspace", strategyType: "git_worktree" }).returning(); + const service = executionWorkspaceService(db); + await service.selectTaskWorkspace({ companyId: f.companyId, issueId: f.issue.id, actor: board, + selection: { kind: "existing", workspaceId: workspace!.id }, expectedBindingRevision: 0, requestKey: randomUUID() }); + expect(await taskInterruptionPresentation(db, task!)).toMatchObject({ canRetry: true, projectAccess: { ready: true } }); + expect(await service.archiveWorkspaceUnderLifecycleLock({ id: workspace!.id, patch: {}, closedAt: new Date() })) + .toMatchObject({ outcome: "archived" }); + await expect(service.applyPendingTaskWorkspaceSelection({ companyId: f.companyId, issueId: f.issue.id, + runId: f.runId, actor: { type: "agent", source: "agent_key", agentId: f.agentId, companyId: f.companyId } })) + .rejects.toThrow("Workspace is unavailable or inaccessible"); + const archived = await taskInterruptionPresentation(db, task!); + expect(archived).toMatchObject({ state: "failed", canRetry: false, userAction: null }); + expect(archived).not.toHaveProperty("projectAccess"); + }); + + it.each(["denied_project", "foreign_company"] as const)("retains the %s gate on a bound workspace even when its source is allowed", async boundary => { + const f = await seed(), foreign = boundary === "foreign_company" ? await seed() : null; + const [source] = await db.insert(projectWorkspaces).values({ companyId: f.companyId, projectId: f.selected[0]!.id, + name: "Allowed source", cwd: "/tmp/project-access-source" }).returning(); + const [workspace] = await db.insert(executionWorkspaces).values({ companyId: foreign?.companyId ?? f.companyId, + projectId: foreign?.selected[0]!.id ?? f.selected[1]!.id, projectWorkspaceId: foreign ? null : source!.id, + name: "Retained files", cwd: "/tmp/project-access-bound", mode: "isolated_workspace", strategyType: "git_worktree" }).returning(); + const [task] = await db.update(issues).set({ projectId: f.selected[0]!.id, projectWorkspaceId: source!.id, + executionWorkspaceId: workspace!.id }).where(eq(issues.id, f.issue.id)).returning(); + const blocked = await taskInterruptionPresentation(db, task!); + if (boundary === "denied_project") { + expect(blocked).toMatchObject({ canRetry: false, + userAction: "project_access", projectAccess: { ready: false, projectId: null } }); + } else { + expect(blocked).toMatchObject({ state: "failed", canRetry: false, userAction: null }); + expect(blocked).not.toHaveProperty("projectAccess"); + } + }); + + it("preserves containment, audits concurrent additions, and changes the persisted denial into an explicit retry", async () => { + const f = await seed(); + const actor = { type: "agent" as const, source: "agent_key" as const, agentId: f.agentId, companyId: f.companyId }; + await expect(assertTaskWorkspaceSourceProjectAccess(db, actor, f.companyId, f.selected[1]!.id)).rejects.toBeInstanceOf(ProjectAccessDeniedError); + expect(await taskInterruptionPresentation(db, f.issue)).toMatchObject({ userAction: "project_access", canRetry: false }); + const api = app(board); + const results = await Promise.all(f.selected.slice(1).map(project => request(api).post(`/agents/${f.agentId}/project-access`).send({ projectIds: [project.id] }))); + expect(results.map(result => result.status)).toEqual([200, 200]); + const [saved] = await db.select().from(agents).where(eq(agents.id, f.agentId)); + expect(saved!.permissions).toEqual({ ...f.permissions, authorizationPolicy: { ...f.permissions.authorizationPolicy, + trustBoundary: { ...f.permissions.authorizationPolicy.trustBoundary, projectIds: expect.arrayContaining(f.selected.map(project => project.id)) } } }); + expect((saved!.permissions as typeof f.permissions).authorizationPolicy.trustBoundary.projectIds).toHaveLength(3); + await expect(assertTaskWorkspaceSourceProjectAccess(db, actor, f.companyId, f.selected[1]!.id)).resolves.toBeUndefined(); + expect(await taskInterruptionPresentation(db, f.issue)).toMatchObject({ canRetry: true, projectAccess: { ready: true } }); + const again = await request(api).post(`/agents/${f.agentId}/project-access`).send({ projectIds: [f.selected[1]!.id] }); + expect(again.status).toBe(200); + expect(await db.select().from(activityLog).where(and(eq(activityLog.companyId, f.companyId), eq(activityLog.action, "agent.project_access_added")))).toHaveLength(2); + expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId))).toHaveLength(0); + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId))).toHaveLength(1); + }); + + it("rejects foreign projects, agent self-grants, unknown policy fields, and pending-approval agents", async () => { + const f = await seed(), other = await seed(), api = app(board); + expect((await request(api).post(`/agents/${f.agentId}/project-access`).send({ projectIds: [other.selected[1]!.id] })).status).toBe(404); + expect((await request(api).post(`/agents/${f.agentId}/project-access`).send({ projectIds: [f.selected[1]!.id], trustPreset: "standard" })).status).toBe(400); + expect((await request(app({ type: "agent", agentId: f.agentId, companyId: f.companyId, source: "agent_key" })) + .post(`/agents/${f.agentId}/project-access`).send({ projectIds: [f.selected[1]!.id] })).status).toBe(403); + await db.update(agents).set({ status: "pending_approval" }).where(eq(agents.id, f.agentId)); + expect((await request(api).post(`/agents/${f.agentId}/project-access`).send({ projectIds: [f.selected[1]!.id] })).status).toBe(409); + const [saved] = await db.select().from(agents).where(eq(agents.id, f.agentId)); expect(saved!.permissions).toEqual(f.permissions); + }); + + it.each(["standard", "root_issue"])("rechecks actual project authority after a %s repair and acknowledges an accepted retry", async repair => { + const f = await seed(); + const permissions = repair === "standard" ? { canCreateAgents: false, trustPreset: "standard", authorizationPolicy: {} } + : { ...f.permissions, authorizationPolicy: { ...f.permissions.authorizationPolicy, + trustBoundary: { ...f.permissions.authorizationPolicy.trustBoundary, rootIssueId: f.issue.id } } }; + await db.update(agents).set({ permissions }).where(eq(agents.id, f.agentId)); + expect(await taskInterruptionPresentation(db, f.issue)).toMatchObject({ canRetry: true, projectAccess: { ready: true } }); + await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agentId, + source: "on_demand", status: "deferred_issue_execution", requestedByActorType: "user", payload: { issueId: f.issue.id } }); + const accepted = await taskInterruptionPresentation(db, f.issue); + expect(accepted).toMatchObject({ state: "working", canRetry: false }); + expect(accepted).not.toHaveProperty("projectAccess"); + // Current project policy remains authoritative even after agent access changes. + await db.update(projects).set({ executionWorkspacePolicy: { trustPreset: "low_trust_review", authorizationPolicy: { + trustBoundary: { mode: "low_trust_review", companyId: f.companyId, projectIds: [f.selected[0]!.id] }, + } } }).where(eq(projects.id, f.selected[1]!.id)); + if (repair === "standard") expect(await taskInterruptionPresentation(db, f.issue)) + .toMatchObject({ canRetry: false, userAction: "project_access" }); + }); + + it("shows read-only access to viewers and denies their saves and cross-company reads", async () => { + const f = await seed(), other = await seed(), userId = randomUUID(); + await db.insert(authUsers).values({ id: userId, name: "Viewer", email: `${userId}@example.test`, emailVerified: true, createdAt: new Date(), updatedAt: new Date() }); + await db.insert(companyMemberships).values({ companyId: f.companyId, principalType: "user", principalId: userId, membershipRole: "viewer", status: "active" }); + const api = app({ type: "board", source: "session", userId, companyIds: [f.companyId], memberships: [{ companyId: f.companyId, membershipRole: "viewer", status: "active" }] }); + expect((await request(api).get(`/agents/${f.agentId}/project-access`)).body).toMatchObject({ canManage: false }); + expect((await request(api).post(`/agents/${f.agentId}/project-access`).send({ projectIds: [f.selected[1]!.id] })).status).toBe(403); + expect((await request(api).get(`/agents/${other.agentId}/project-access`)).status).toBe(404); + }); +}); diff --git a/server/src/__tests__/agent-record-read-projections.test.ts b/server/src/__tests__/agent-record-read-projections.test.ts new file mode 100644 index 00000000000..bf0a5183202 --- /dev/null +++ b/server/src/__tests__/agent-record-read-projections.test.ts @@ -0,0 +1,132 @@ +import { randomUUID } from "node:crypto"; +import { agents, companies, costEvents, createDb } from "@paperclipai/db"; +import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture"; +import { eq } from "drizzle-orm"; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { agentRecordQueries } from "../lib/agent-records.js"; +import { agentService } from "../services/agents.js"; + +const support = await getEmbeddedPostgresTestSupport(); + +describe.skipIf(!support.supported)("agent roster read projections", () => { + let database: Awaited>; + let db: ReturnType; + + beforeAll(async () => { + database = await startEmbeddedPostgresTestDatabase("agent-roster-read-"); + db = createDb(database.connectionString); + }, 60_000); + + beforeEach(() => { + vi.useFakeTimers({ toFake: ["Date"] }); + vi.setSystemTime(new Date("2020-06-15T12:00:00Z")); + }); + + afterEach(() => { vi.useRealTimers(); }); + + afterAll(async () => { + await db?.$client.end(); + await database?.cleanup(); + }); + + async function fixture() { + const companyId = randomUUID(), foreignCompanyId = randomUUID(); + const managerId = randomUUID(), workerId = randomUUID(), unrelatedId = randomUUID(), archivedId = randomUUID(), foreignId = randomUUID(); + await db.insert(companies).values([ + { id: companyId, name: "Roster fixture", issuePrefix: `R${companyId.slice(0, 7).toUpperCase()}` }, + { id: foreignCompanyId, name: "Foreign roster", issuePrefix: `F${foreignCompanyId.slice(0, 7).toUpperCase()}` }, + ]); + const bulk = "retained-other-agent-configuration🙂".repeat(32_768); + await db.insert(agents).values([ + { id: managerId, companyId, name: "Manager", status: "terminated", adapterType: "process", adapterConfig: { instructions: bulk } }, + { id: unrelatedId, companyId, name: "Other worker", status: "idle", adapterType: "process", adapterConfig: { instructions: bulk } }, + { id: archivedId, companyId, name: "Archived worker", status: "terminated", adapterType: "process", runtimeConfig: { retained: bulk } }, + { id: foreignId, companyId: foreignCompanyId, name: "Foreign manager", status: "idle", adapterType: "process", adapterConfig: { instructions: bulk } }, + ]); + const [worker] = await db.insert(agents).values({ id: workerId, companyId, name: "Selected worker", status: "idle", + reportsTo: managerId, adapterType: "process", adapterConfig: { instructions: "selected instructions", model: "fixture-model" }, + runtimeConfig: { heartbeat: { enabled: false } }, metadata: { retained: "selected metadata" }, + permissions: { canCreateAgents: true }, spentMonthlyCents: 999, + }).returning(); + const now = new Date(); + await db.insert(costEvents).values([ + { companyId, agentId: workerId, provider: "fixture", model: "fixture-model", costCents: 19.5, occurredAt: now }, + { companyId, agentId: workerId, provider: "fixture", model: "fixture-model", costCents: 50, + occurredAt: new Date(Date.UTC(now.getUTCFullYear(), now.getUTCMonth() - 1, 15)) }, + { companyId: foreignCompanyId, agentId: foreignId, provider: "fixture", model: "fixture-model", costCents: 100, occurredAt: now }, + ]); + return { companyId, foreignCompanyId, managerId, workerId, unrelatedId, archivedId, foreignId, worker, bulk }; + } + + async function observe(run: (observed: ReturnType) => Promise) { + const observed = createDb(database.connectionString), events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(observed, { agents, costEvents }, capture); + try { + const result = await run(observed); + capture.flush(); + const reads = events.filter(event => event.table === "agents" && event.operation === "select"); + return { result, calls: reads.reduce((n, event) => n + Number(event.calls), 0), + bytes: reads.reduce((n, event) => n + Number(event.responseValueBytes), 0), events }; + } finally { + capture.flush(); + await observed.$client.end(); + } + } + + it.each([false, true])("reads one full roster for list, preserving terminated-manager evidence and configurations (includeTerminated=%s)", async includeTerminated => { + const f = await fixture(); + const captured = await observe(observed => agentService(observed).list(f.companyId, { includeTerminated })); + expect(captured.calls).toBe(1); + expect(captured.result.map(row => row.id).sort()).toEqual((includeTerminated + ? [f.workerId, f.managerId, f.archivedId, f.unrelatedId] : [f.workerId, f.unrelatedId]).sort()); + const selected = captured.result.find(row => row.id === f.workerId)!; + expect(selected).toMatchObject({ adapterConfig: f.worker.adapterConfig, runtimeConfig: f.worker.runtimeConfig, + metadata: f.worker.metadata, permissions: { canCreateAgents: true }, spentMonthlyCents: 19.5, + orgChainHealth: { status: "invalid_org_chain", reason: "terminated_ancestor", firstInvalidAncestor: { id: f.managerId } } }); + expect(captured.result.find(row => row.id === f.unrelatedId)?.adapterConfig.instructions).toBe(f.bulk); + if (includeTerminated) expect(captured.result.find(row => row.id === f.archivedId)?.runtimeConfig.retained).toBe(f.bulk); + expect((await db.select().from(agents).where(eq(agents.id, f.workerId)))[0]).toEqual(f.worker); + expect(JSON.stringify(captured.events)).not.toContain("retained-other-agent-configuration"); + }); + + it("keeps detail reads small while returning the selected agent's complete current configuration", async () => { + const f = await fixture(); + const captured = await observe(observed => agentRecordQueries(observed).getById(f.workerId)); + expect(captured.calls).toBe(2); + expect(captured.bytes).toBeLessThan(8192); + expect(captured.result).toMatchObject({ id: f.workerId, companyId: f.companyId, adapterConfig: f.worker.adapterConfig, + runtimeConfig: f.worker.runtimeConfig, metadata: f.worker.metadata, permissions: { canCreateAgents: true }, spentMonthlyCents: 19.5, + orgChainHealth: { reason: "terminated_ancestor", firstInvalidAncestor: { id: f.managerId, name: "Manager" } } }); + expect((await db.select().from(agents).where(eq(agents.id, f.workerId)))[0]).toEqual(f.worker); + }); + + it("projects only current company chain fields for plugin normalization", async () => { + const f = await fixture(); + const captured = await observe(observed => agentRecordQueries(observed).listCompanyAgentEligibilityRows(f.companyId)); + expect(captured.calls).toBe(1); + expect(captured.bytes).toBeLessThan(4096); + expect(captured.result.map(row => row.id).sort()).toEqual([f.workerId, f.managerId, f.unrelatedId, f.archivedId].sort()); + for (const row of captured.result) expect(Object.keys(row).sort()).toEqual(["companyId", "id", "name", "reportsTo", "status"]); + const normalized = agentRecordQueries(db).normalizeAgentRow(f.worker, captured.result); + expect(normalized).toMatchObject({ adapterConfig: f.worker.adapterConfig, metadata: f.worker.metadata, + orgChainHealth: { reason: "terminated_ancestor", firstInvalidAncestor: { id: f.managerId } } }); + expect(await agentRecordQueries(db).listCompanyAgentEligibilityRows(randomUUID())).toEqual([]); + expect(await agentRecordQueries(db).getById(randomUUID())).toBeNull(); + }); + + it("rechecks manager status, name, selected configuration, missing managers, and cycles on later reads", async () => { + const f = await fixture(), records = agentRecordQueries(db); + expect((await records.getById(f.workerId))?.orgChainHealth.reason).toBe("terminated_ancestor"); + await db.update(agents).set({ status: "paused", name: "Current manager" }).where(eq(agents.id, f.managerId)); + await db.update(agents).set({ adapterConfig: { instructions: "changed instructions" } }).where(eq(agents.id, f.workerId)); + expect(await records.getById(f.workerId)).toMatchObject({ adapterConfig: { instructions: "changed instructions" }, + orgChainHealth: { status: "healthy", pausedAncestors: [{ id: f.managerId, name: "Current manager", status: "paused" }] } }); + await db.update(agents).set({ reportsTo: f.foreignId }).where(eq(agents.id, f.workerId)); + expect((await records.getById(f.workerId))?.orgChainHealth.reason).toBe("missing_manager"); + await db.update(agents).set({ reportsTo: f.managerId }).where(eq(agents.id, f.workerId)); + await db.update(agents).set({ reportsTo: f.workerId }).where(eq(agents.id, f.managerId)); + expect((await records.getById(f.workerId))?.orgChainHealth.reason).toBe("cycle"); + }); +}); diff --git a/server/src/__tests__/attention-service.test.ts b/server/src/__tests__/attention-service.test.ts index 26934470fbd..bf4879957cd 100644 --- a/server/src/__tests__/attention-service.test.ts +++ b/server/src/__tests__/attention-service.test.ts @@ -2,7 +2,8 @@ import { createAgentLifecycle } from "../services/agent-lifecycle.js"; import { randomUUID } from "node:crypto"; import express from "express"; import request from "supertest"; -import { eq } from "drizzle-orm"; +import { eq, sql } from "drizzle-orm"; +import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture"; import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest"; import { activityLog, @@ -1034,6 +1035,177 @@ describeEmbeddedPostgres("attention service", () => { expect(feed.items.filter((item) => item.sourceKind === "failed_run")).toEqual([]); }); + it("hides a failed run only after a newer run of the same agent on the same issue or task", async () => { + const { companyId, workerId, reviewerId } = await seedCompany("ATK"); + const taskIssueId = await insertIssue({ companyId, identifier: "ATK-1", title: "Task-keyed", status: "in_progress" }); + const issueKeyedId = await insertIssue({ companyId, identifier: "ATK-2", title: "Issue-keyed", status: "in_progress" }); + const otherIssueId = await insertIssue({ companyId, identifier: "ATK-3", title: "Other work", status: "in_progress" }); + const failedAt = new Date("2026-07-09T12:00:00.000Z"); + const laterAt = new Date("2026-07-09T12:01:00.000Z"); + const taskKeyedFailureId = randomUUID(); + const issueKeyedFailureId = randomUUID(); + const unkeyedHiddenFailureId = randomUUID(); + const unkeyedKeptFailureId = randomUUID(); + const failureIds = [taskKeyedFailureId, issueKeyedFailureId, unkeyedHiddenFailureId, unkeyedKeptFailureId]; + const sameMillisecondRunId = randomUUID(); + const run = (id: string, agentId: string, status: string, contextSnapshot: Record, at: Date) => ({ + id, companyId, agentId, invocationSource: "automation", status, contextSnapshot, + createdAt: at, updatedAt: at, finishedAt: at, ...(status === "failed" ? { error: "adapter failed" } : {}), + }); + + await db.insert(heartbeatRuns).values([ + run(taskKeyedFailureId, workerId, "failed", { taskId: taskIssueId }, failedAt), + run(issueKeyedFailureId, workerId, "failed", { issueId: issueKeyedId }, failedAt), + run(unkeyedHiddenFailureId, workerId, "failed", {}, failedAt), + run(unkeyedKeptFailureId, reviewerId, "failed", {}, failedAt), + run(randomUUID(), workerId, "succeeded", { taskId: taskIssueId }, laterAt), + run(randomUUID(), workerId, "succeeded", { wakeReason: "timer" }, laterAt), + run(randomUUID(), reviewerId, "succeeded", { issueId: issueKeyedId }, laterAt), + run(randomUUID(), reviewerId, "succeeded", { issueId: otherIssueId }, laterAt), + run(randomUUID(), workerId, "succeeded", { issueId: otherIssueId }, laterAt), + run(randomUUID(), workerId, "succeeded", { issueId: otherIssueId, taskId: issueKeyedId }, laterAt), + run(randomUUID(), workerId, "succeeded", { issueId: "", taskId: issueKeyedId }, laterAt), + run(sameMillisecondRunId, workerId, "succeeded", { issueId: issueKeyedId }, failedAt), + run(randomUUID(), workerId, "succeeded", { issueId: issueKeyedId }, new Date("2026-07-09T11:59:00.000Z")), + ]); + await db.execute(sql`update heartbeat_runs set created_at = created_at + interval '400 microseconds' + where id in (${sql.join(failureIds.map((id) => sql`${id}::uuid`), sql`, `)})`); + await db.execute(sql`update heartbeat_runs set created_at = created_at + interval '600 microseconds' + where id = ${sameMillisecondRunId}::uuid`); + await db.insert(heartbeatRunEvents).values( + failureIds.map((runId) => ({ + companyId, + runId, + agentId: runId === unkeyedKeptFailureId ? reviewerId : workerId, + seq: 1, + eventType: "lifecycle", + message: "Bounded retry exhausted after 4 scheduled attempts; no further automatic retry will be queued", + createdAt: new Date("2026-07-09T12:00:01.000Z"), + })), + ); + + const feed = await attentionService(db).list(companyId, { userId: "board-user" }); + + expect(feed.items.filter((item) => item.sourceKind === "failed_run").map((item) => item.subject.id).sort()) + .toEqual([issueKeyedFailureId, unkeyedKeptFailureId].sort()); + }); + + it("keeps newer-run response size bounded despite unrelated retained history", async () => { + const { companyId, workerId } = await seedCompany("ATW"); + const issueId = await insertIssue({ companyId, identifier: "ATW-1", title: "Exhausted task", status: "in_progress" }); + const otherIssueId = await insertIssue({ companyId, identifier: "ATW-2", title: "Other task", status: "done" }); + const failedId = randomUUID(), failedAt = new Date("2026-07-09T12:00:00.000Z"); + await db.insert(heartbeatRuns).values({ + id: failedId, companyId, agentId: workerId, status: "failed", contextSnapshot: { issueId }, + createdAt: failedAt, updatedAt: failedAt, finishedAt: failedAt, + }); + await db.insert(heartbeatRunEvents).values({ + companyId, agentId: workerId, runId: failedId, seq: 1, eventType: "lifecycle", + message: "Bounded retry exhausted after scheduled attempts", + }); + for (let batch = 0; batch < 4; batch += 1) { + await db.insert(heartbeatRuns).values(Array.from({ length: 500 }, (_, index) => ({ + id: randomUUID(), companyId, agentId: workerId, status: "succeeded", + contextSnapshot: { issueId: otherIssueId, prompt: "x".repeat(8_000) }, + createdAt: new Date(failedAt.getTime() + batch * 500 + index + 1), + updatedAt: failedAt, finishedAt: failedAt, + }))); + } + const events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(db, { heartbeatRuns }, capture); + try { + const feed = await attentionService(db).list(companyId, { userId: "board-user" }); + expect(feed.items.filter(item => item.sourceKind === "failed_run").map(item => item.subject.id)).toEqual([failedId]); + } finally { capture.flush(); } + const reads = events.filter(event => event.event === "paperclip.db_response_capture.entry.v1" + && event.table === "heartbeat_runs" && (event.operation === "select" || event.operation === "execute")); + expect(reads.reduce((sum, event) => sum + Number(event.calls), 0)).toBeGreaterThan(0); + expect(reads.reduce((sum, event) => sum + Number(event.rows), 0)).toBeLessThan(64); + expect(reads.reduce((sum, event) => sum + Number(event.responseValueBytes), 0)).toBeLessThan(4096); + expect(reads.filter(event => event.operation === "execute") + .reduce((sum, event) => sum + Number(event.calls), 0)).toBeGreaterThan(0); + const [remaining] = await db.select({ count: sql`count(*)::int` }).from(heartbeatRuns) + .where(eq(heartbeatRuns.companyId, companyId)); + expect(remaining?.count).toBe(2001); + }); + + it("reads keyless history once per agent despite thousands of failures", async () => { + const { companyId, workerId, reviewerId } = await seedCompany("ATB"); + const failedAt = new Date("2026-07-09T12:00:00.000Z"); + const laterAt = new Date("2026-07-09T12:01:00.000Z"); + const hiddenIds = Array.from({ length: 16_390 }, () => randomUUID()); + const keptIds = Array.from({ length: 10 }, () => randomUUID()); + const failure = (id: string, agentId: string) => ({ + id, companyId, agentId, invocationSource: "automation", status: "failed", error: "adapter failed", + contextSnapshot: {}, createdAt: failedAt, updatedAt: failedAt, finishedAt: failedAt, + }); + const failures = hiddenIds.map((id) => failure(id, workerId)); + keptIds.forEach((id, index) => failures.splice(index * 1_700, 0, failure(id, reviewerId))); + for (let start = 0; start < failures.length; start += 1_000) { + const chunk = failures.slice(start, start + 1_000); + await db.insert(heartbeatRuns).values(chunk); + await db.insert(heartbeatRunEvents).values(chunk.map((run) => ({ + companyId, runId: run.id, agentId: run.agentId, seq: 1, eventType: "lifecycle", + message: "Bounded retry exhausted after 4 scheduled attempts; no further automatic retry will be queued", + createdAt: new Date("2026-07-09T12:00:01.000Z"), + }))); + } + await db.insert(heartbeatRuns).values({ + id: randomUUID(), companyId, agentId: workerId, invocationSource: "automation", status: "succeeded", + contextSnapshot: { wakeReason: "timer" }, createdAt: laterAt, updatedAt: laterAt, finishedAt: laterAt, + }); + + const events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(db, { heartbeatRuns }, capture); + try { + const feed = await attentionService(db).list(companyId, { + includeDismissed: true, all: true, allowUnscopedAll: true, + }); + expect(feed.items.filter((item) => item.sourceKind === "failed_run").map((item) => item.subject.id).sort()) + .toEqual([...keptIds].sort()); + } finally { capture.flush(); } + const reads = events.filter(event => event.event === "paperclip.db_response_capture.entry.v1" + && event.table === "heartbeat_runs" && (event.operation === "select" || event.operation === "execute")); + expect(reads.reduce((sum, event) => sum + Number(event.calls), 0)).toBeGreaterThan(0); + expect(reads.reduce((sum, event) => sum + Number(event.calls), 0)).toBeLessThan(4); + }, 180_000); + + it("checks more failed runs for a newer run than one statement can bind", async () => { + const { companyId, workerId, reviewerId } = await seedCompany("ATQ"); + const issueId = await insertIssue({ companyId, identifier: "ATQ-1", title: "Keyed retained failures", status: "in_progress" }); + const failedAt = new Date("2026-07-09T12:00:00.000Z"); + const laterAt = new Date("2026-07-09T12:01:00.000Z"); + const hiddenIds = Array.from({ length: 16_390 }, () => randomUUID()); + const keptIds = Array.from({ length: 10 }, () => randomUUID()); + const failure = (id: string, agentId: string) => ({ + id, companyId, agentId, invocationSource: "automation", status: "failed", error: "adapter failed", + contextSnapshot: { issueId }, createdAt: failedAt, updatedAt: failedAt, finishedAt: failedAt, + }); + const failures = hiddenIds.map((id) => failure(id, workerId)); + keptIds.forEach((id, index) => failures.splice(index * 1_700, 0, failure(id, reviewerId))); + for (let start = 0; start < failures.length; start += 1_000) { + const chunk = failures.slice(start, start + 1_000); + await db.insert(heartbeatRuns).values(chunk); + await db.insert(heartbeatRunEvents).values(chunk.map((run) => ({ + companyId, runId: run.id, agentId: run.agentId, seq: 1, eventType: "lifecycle", + message: "Bounded retry exhausted after 4 scheduled attempts; no further automatic retry will be queued", + createdAt: new Date("2026-07-09T12:00:01.000Z"), + }))); + } + await db.insert(heartbeatRuns).values({ + id: randomUUID(), companyId, agentId: workerId, invocationSource: "automation", status: "succeeded", + contextSnapshot: { issueId }, createdAt: laterAt, updatedAt: laterAt, finishedAt: laterAt, + }); + + const feed = await attentionService(db).list(companyId, { + includeDismissed: true, all: true, allowUnscopedAll: true, + }); + expect(feed.items.filter(item => item.sourceKind === "failed_run").map(item => item.subject.id).sort()) + .toEqual([...keptIds].sort()); + }, 180_000); + it("enriches interaction details with project, workspace, plan metadata, and images", async () => { const { companyId, workerId } = await seedCompany("ATE"); const projectId = randomUUID(); diff --git a/server/src/__tests__/chat-channels.integration.test.ts b/server/src/__tests__/chat-channels.integration.test.ts index 4fd186e6271..f844a18bce3 100644 --- a/server/src/__tests__/chat-channels.integration.test.ts +++ b/server/src/__tests__/chat-channels.integration.test.ts @@ -5,6 +5,8 @@ import { createSystemRecovery } from "../services/system-recovery-state.js"; import { remoteTerminationReceipt } from "../services/remote-execution-termination.js"; import { stageReceiptReactionRemovals } from "../services/chat-receipt-reactions.js"; import type { Db } from "@paperclipai/db"; +import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture"; +import { githubResponseAuthorityFields } from "../services/chat-channels/github/response-authority-read.js"; import * as activityLogModule from "../services/activity-log.js"; import * as connectionCredentialBindings from "../services/connection-credential-bindings.js"; import { chatSlackManagerGrants, chatSlackRegistrations, toolOauthStates } from "@paperclipai/db"; @@ -24,7 +26,7 @@ import { toolActionRequests, toolInvocations, toolPolicies } from "@paperclipai/ import { GitHubPublicationLeaseLost, withGitHubPublicationLease } from "../services/chat-github-publication-lease.js"; import { secretService } from "../services/secrets.js"; import { githubChatWizardService } from "../services/chat-github-wizard.js"; -import { dispatchGitHubBotCloudEvent } from "../services/chat-github-cloud-ingress.js"; +import { dispatchGitHubBotCloudEvent, registerGitHubBotCloudSetupPoller } from "../services/chat-github-cloud-ingress.js"; import { githubChatManagementService } from "../services/chat-github-management.js"; import { githubChatReviewService } from "../services/chat-github-reviews.js"; import { resolveGitHubReviewInstructions } from "../services/chat-channels/github/review-instructions.js"; @@ -2585,6 +2587,81 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { .rejects.toMatchObject({ status: 403 }); }); + it("bounds GitHub response authorization reads with large saved chat content", async () => { + const f = await toolReplyFixture(); + const [delivery] = await db.select().from(chatDeliveries).where(eq(chatDeliveries.endpointId, f.endpoint.id)); + const [resource] = await db.select().from(chatEndpointResources).where(eq(chatEndpointResources.id, f.conversation.resourceId!)); + const [configuration] = await db.select().from(chatGitHubConfigurations).where(eq(chatGitHubConfigurations.endpointId, f.endpoint.id)); + const bulk = "unused-content-for-response-authorization".repeat(8_192); + const setup = { ...f.endpoint.setup, unusedContent: bulk }; + await db.update(chatEndpoints).set({ communicationInstructions: bulk, setup }).where(eq(chatEndpoints.id, f.endpoint.id)); + await db.update(chatConversations).set({ communicationGuidance: bulk }).where(eq(chatConversations.id, f.conversation.id)); + await db.update(chatEndpointResources).set({ metadata: { ...resource.metadata, unusedContent: bulk } }) + .where(eq(chatEndpointResources.id, resource.id)); + await db.update(chatGitHubConfigurations).set({ configuration: { ...configuration.configuration, + defaults: { ...configuration.configuration.defaults, instructions: bulk } } }) + .where(eq(chatGitHubConfigurations.endpointId, f.endpoint.id)); + const normalizedEvent = { ...delivery.normalizedEvent, unusedContent: bulk }; + await db.update(chatDeliveries).set({ normalizedEvent }).where(eq(chatDeliveries.id, delivery.id)); + const events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(db, { chatMessageLinks, chatDeliveries, chatEndpoints, chatConversations, + chatEndpointResources, chatGitHubConfigurations, chatExternalPrincipals, chatIdentityLinks, + companyMemberships, chatActions, heartbeatRuns }, capture); + try { + const receipt = await githubResponseCommentService(db, f.providerFetch).tryAcknowledge(delivery.id); + expect(receipt?.id).toBeTruthy(); + } finally { capture.flush(); } + expect(Number(events.at(-1)?.responseValueBytes)).toBeLessThan(24_000); + expect(events.filter(event => event.table === "chat_message_links").length).toBeGreaterThan(0); + expect((await db.select({ normalizedEvent: chatDeliveries.normalizedEvent }).from(chatDeliveries) + .where(eq(chatDeliveries.id, delivery.id)))[0].normalizedEvent).toEqual(normalizedEvent); + expect((await db.select({ setup: chatEndpoints.setup }).from(chatEndpoints) + .where(eq(chatEndpoints.id, f.endpoint.id)))[0].setup).toEqual(setup); + }); + + it.each(["githubAutomatic", "githubIssue"])("preserves %s policy context while omitting body content", async key => { + const f = await toolReplyFixture(); + const [delivery] = await db.select().from(chatDeliveries).where(eq(chatDeliveries.endpointId, f.endpoint.id)); + const context = { event: key === "githubIssue" ? "issue_opened" : "opened", repositoryId: "418", repository: "acme/repo", + author: { id: "42", login: "octocat", isBot: false }, sender: { id: "42", login: "octocat" }, + labels: ["review"], draft: false, baseBranch: "master", headSha: "a".repeat(40), + body: "unused-body".repeat(32_768), title: "unused-title".repeat(8_192) }; + const normalizedEvent = { text: "unrelated message text", [key]: { context, revision: 42, + policy: { instructions: "unused-policy-instructions".repeat(32_768), + prompts: { opened: "unused-policy-prompt".repeat(32_768) } }, + } }; + await db.update(chatDeliveries).set({ normalizedEvent }).where(eq(chatDeliveries.id, delivery.id)); + const events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(db, { chatDeliveries }, capture); + let projected: { event: Record }; + try { + [projected] = await db.select({ event: githubResponseAuthorityFields.delivery.normalizedEvent }) + .from(chatDeliveries).where(eq(chatDeliveries.id, delivery.id)); + } finally { capture.flush(); } + const { body: _body, title: _title, ...policy } = context; + expect(Number(events.at(-1)?.responseValueBytes)).toBeLessThan(2_048); + expect(projected.event).toEqual({ [key]: { context: policy } }); + expect((await db.select({ event: chatDeliveries.normalizedEvent }).from(chatDeliveries) + .where(eq(chatDeliveries.id, delivery.id)))[0].event).toEqual(normalizedEvent); + }); + + it.each([null, false, 7, "invalid-event", []])("preserves malformed GitHub authority input %j", async value => { + const f = await toolReplyFixture(); + const [delivery] = await db.select().from(chatDeliveries).where(eq(chatDeliveries.endpointId, f.endpoint.id)); + await db.update(chatDeliveries).set({ normalizedEvent: sql`${JSON.stringify(value)}::jsonb` }) + .where(eq(chatDeliveries.id, delivery.id)); + await db.update(chatEndpoints).set({ setup: sql`${JSON.stringify(value)}::jsonb` }) + .where(eq(chatEndpoints.id, f.endpoint.id)); + const [projected] = await db.select({ event: githubResponseAuthorityFields.delivery.normalizedEvent }) + .from(chatDeliveries).where(eq(chatDeliveries.id, delivery.id)); + const [endpoint] = await db.select({ setup: githubResponseAuthorityFields.endpoint.setup }) + .from(chatEndpoints).where(eq(chatEndpoints.id, f.endpoint.id)); + expect(projected.event).toEqual(value); + expect(endpoint.setup).toEqual(value); + }); + async function publicationRetryFixture() { const f = await toolReplyFixture("github:paperclipai/paperclip:418", true); const head = "b".repeat(40); @@ -5135,13 +5212,22 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { (await f.service.get(f.endpoint.id)).setup.webhookVerifiedAt, ).toBeNull(); expect(f.wakeup).not.toHaveBeenCalled(); - event.payload.githubApp.headers["x-hub-signature-256"] = - `sha256=${createHmac("sha256", f.webhookSecret).update(body).digest("hex")}`; - expect(await dispatchGitHubBotCloudEvent(db, event)).toBe(true); - expect( - (await f.service.get(f.endpoint.id)).setup.webhookVerifiedAt, - ).toBeTruthy(); - expect(f.wakeup).not.toHaveBeenCalled(); + const poll = vi.fn(async () => { await dispatchGitHubBotCloudEvent(db, event); }); + const unregister = registerGitHubBotCloudSetupPoller(db, poll); + try { + const waiting = await f.service.githubWizard.advance(f.endpoint.id, "owner-user"); + expect(waiting.state).toBe("verify"); + expect(poll).toHaveBeenCalledOnce(); + expect((await f.service.get(f.endpoint.id)).setup.webhookVerifiedAt).toBeNull(); + event.payload.githubApp.headers["x-hub-signature-256"] = + `sha256=${createHmac("sha256", f.webhookSecret).update(body).digest("hex")}`; + const ready = await f.service.githubWizard.advance(f.endpoint.id, "owner-user"); + expect(ready.state).not.toBe("verify"); + expect((await f.service.get(f.endpoint.id)).setup.webhookVerifiedAt).toBeTruthy(); + expect(f.wakeup).not.toHaveBeenCalled(); + await f.service.githubWizard.advance(f.endpoint.id, "owner-user"); + expect(poll).toHaveBeenCalledTimes(2); + } finally { unregister(); } }); it("legacy name-only registration preserves an explicit empty repository selection", async () => { const fixture = await seedCompany(); diff --git a/server/src/__tests__/company-skills-service.test.ts b/server/src/__tests__/company-skills-service.test.ts index 7217eb4f2be..c1c9a448232 100644 --- a/server/src/__tests__/company-skills-service.test.ts +++ b/server/src/__tests__/company-skills-service.test.ts @@ -16,6 +16,7 @@ import { projectWorkspaces, } from "@paperclipai/db"; import { parseFrontmatterMarkdown } from "@paperclipai/shared"; +import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase, @@ -722,6 +723,73 @@ describeEmbeddedPostgres("companySkillService.list", () => { expect(refreshed.filter((skill) => skill.key === key)).toHaveLength(1); }); + it("validates warm bundled releases without returning their stored file bodies", async () => { + const companyId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Release read checks", issuePrefix: "RRC" }); + await svc.list(companyId); + const expected = await svc.list(companyId); + const before = await db.select().from(companySkillVersions).where(eq(companySkillVersions.companyId, companyId)) + .orderBy(companySkillVersions.id); + expect(before.length).toBeGreaterThan(0); + const events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(db, { companySkillVersions }, capture); + try { expect(await companySkillService(db).list(companyId)).toEqual(expected); } + finally { capture.flush(); } + const reads = events.filter(event => event.event === "paperclip.db_response_capture.entry.v1" + && event.table === "company_skill_versions" && event.operation === "select"); + expect(reads.reduce((sum, event) => sum + Number(event.calls), 0)).toBeGreaterThan(0); + expect(reads.reduce((sum, event) => sum + Number(event.responseValueBytes), 0)).toBeLessThan(4096); + expect(await db.select().from(companySkillVersions).where(eq(companySkillVersions.companyId, companyId)) + .orderBy(companySkillVersions.id)).toEqual(before); + }); + + it.each(["agent", "board"] as const)("checks %s company access before using a warm release validation", async type => { + const companyId = randomUUID(), otherCompanyId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Release access checks", issuePrefix: "RAC" }); + await svc.list(companyId); + await svc.list(companyId); + const actor = type === "agent" + ? { type, companyId: otherCompanyId, agentId: randomUUID() } + : { type, companyIds: [otherCompanyId] }; + const events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(db, { companySkillVersions }, capture); + try { expect(await companySkillService(db, { actor }).list(companyId)).toEqual([]); } + finally { capture.flush(); } + expect(events.find(event => event.event === "paperclip.db_response_capture.summary.v1")?.completedQueries).toBe(0); + }); + + it("restores a deleted release after warm validation", async () => { + const companyId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Release deletion checks", issuePrefix: "RDC" }); + await svc.list(companyId); + await svc.list(companyId); + const before = await db.select().from(companySkillVersions).where(eq(companySkillVersions.companyId, companyId)); + const original = before[0]!; + await db.delete(companySkillVersions).where(eq(companySkillVersions.id, original.id)); + await companySkillService(db).list(companyId); + const after = await db.select().from(companySkillVersions).where(eq(companySkillVersions.companyId, companyId)); + expect(after).toHaveLength(before.length); + expect(after.find(version => version.releaseId === original.releaseId)).toMatchObject({ + companyId, companySkillId: original.companySkillId, fileInventory: original.fileInventory, + }); + expect(after.some(version => version.id === original.id)).toBe(false); + }); + + it("rejects a changed bundled snapshot after warm validation without overwriting it", async () => { + const companyId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Release integrity checks", issuePrefix: "RIC" }); + await svc.list(companyId); + await svc.list(companyId); + const [version] = await db.select().from(companySkillVersions).where(eq(companySkillVersions.companyId, companyId)); + const changed = [{ path: "SKILL.md", kind: "skill" as const, content: "Changed stored snapshot" }]; + await db.update(companySkillVersions).set({ fileInventory: changed }).where(eq(companySkillVersions.id, version!.id)); + await expect(companySkillService(db).list(companyId)).rejects.toThrow("does not match its seeded snapshot"); + expect((await db.select().from(companySkillVersions).where(eq(companySkillVersions.id, version!.id)))[0]!.fileInventory) + .toEqual(changed); + }); + it("seeds bundled skill releases idempotently and materializes the frozen champion snapshot", async () => { const companyId = randomUUID(); await db.insert(companies).values({ diff --git a/server/src/__tests__/conversation-settlement-transfer.test.ts b/server/src/__tests__/conversation-settlement-transfer.test.ts new file mode 100644 index 00000000000..6a128b00e95 --- /dev/null +++ b/server/src/__tests__/conversation-settlement-transfer.test.ts @@ -0,0 +1,105 @@ +import { randomUUID } from "node:crypto"; +import { and, eq, sql } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { agents, companies, createDb, heartbeatRuns, issueComments, issueThreadInteractions, issues } from "@paperclipai/db"; +import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { settleConversationTurnForRun } from "../services/agent-conversations.js"; + +const support = await getEmbeddedPostgresTestSupport(); +(support.supported ? describe : describe.skip)("conversation settlement transfer", () => { + let database: Awaited>; + let db: ReturnType; + beforeAll(async () => { + database = await startEmbeddedPostgresTestDatabase("paperclip-conversation-settlement-"); + db = createDb(database.connectionString); + }, 30_000); + afterAll(async () => { await db?.$client.end({ timeout: 5 }); await database?.cleanup(); }, 60_000); + + async function seed(reply: "response" | "interaction" | "reset" | "none" = "response") { + const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), runId = randomUUID(), wakeId = randomUUID(); + const padding = "saved conversation ".repeat(32_768); + await db.insert(companies).values({ id: companyId, name: "Settlement transfer", issuePrefix: companyId.slice(0, 8) }); + await db.insert(agents).values({ id: agentId, companyId, name: "Conversation agent", role: "engineer" }); + await db.insert(issues).values({ id: issueId, companyId, title: "Conversation", description: padding, + status: "in_progress", conversationState: "active", conversationUserId: "conversation-owner", + conversationAgentId: agentId, assigneeAgentId: agentId, conversationSessionGeneration: 3 }); + await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "succeeded", + contextSnapshot: { issueId, conversationSessionGeneration: 3, wakeCommentId: wakeId, + conversationReset: reply === "reset", savedInput: padding }, + runnerProfileJson: { savedProfile: padding }, resultJson: { savedResult: padding }, + stdoutExcerpt: padding, stderrExcerpt: padding }); + await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId)); + const wakeTime = new Date(Date.now() - 10_000); + await db.insert(issueComments).values({ id: wakeId, companyId, issueId, authorUserId: "conversation-owner", + body: padding, createdAt: wakeTime }); + if (reply === "response") await db.insert(issueComments).values({ companyId, issueId, + createdByRunId: runId, authorAgentId: agentId, origin: "comment", body: padding }); + if (reply === "interaction") await db.insert(issueThreadInteractions).values({ companyId, issueId, + sourceRunId: runId, createdByAgentId: agentId, status: "pending", kind: "request_confirmation", payload: { version: 1 } }); + return { companyId, agentId, issueId, runId, wakeId, wakeTime }; + } + + async function sourceDigests(input: Awaited>) { + return db.execute(sql`select + (select encode(sha256(convert_to(jsonb_build_array(context_snapshot,runner_profile_json,result_json,stdout_excerpt,stderr_excerpt)::text,'UTF8')),'hex') + from heartbeat_runs where id=${input.runId}::uuid) as run_digest, + (select encode(sha256(convert_to(description,'UTF8')),'hex') from issues where id=${input.issueId}::uuid) as description_digest, + (select encode(sha256(convert_to(jsonb_agg(jsonb_build_array(id,body) order by id)::text,'UTF8')),'hex') + from issue_comments where issue_id=${input.issueId}::uuid) as comments_digest`); + } + + it.each(["response", "interaction", "reset"] as const)("bounds %s settlement reads and retains all saved text and run payloads", async reply => { + const input = await seed(reply), before = await sourceDigests(input); + const events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(db, { heartbeatRuns, issues, issueComments }, capture); + try { expect(await settleConversationTurnForRun(db, input.companyId, input.runId)).toBe(true); } + finally { capture.flush(); } + const reads = events.filter(event => event.event === "paperclip.db_response_capture.entry.v1" + && ["heartbeat_runs", "issues", "issue_comments"].includes(String(event.table)) + && (event.operation === "select" || event.operation === "execute")); + expect(reads.reduce((sum, event) => sum + Number(event.calls), 0)).toBeGreaterThanOrEqual(4); + expect(reads.reduce((sum, event) => sum + Number(event.responseValueBytes), 0)).toBeLessThan(16_384); + const [issue] = await db.select({ status: issues.status, state: issues.conversationState }).from(issues).where(eq(issues.id, input.issueId)); + expect(issue).toEqual({ status: "in_review", state: "waiting" }); + expect(await sourceDigests(input)).toEqual(before); + }); + + it.each(["conversationReplyBoundaryCommentId", "wakeCommentId", "commentId"] as const)("keeps later user messages active through %s", async key => { + const input = await seed(); + await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: input.issueId, + conversationSessionGeneration: 3, [key]: input.wakeId } }).where(eq(heartbeatRuns.id, input.runId)); + await db.insert(issueComments).values({ companyId: input.companyId, issueId: input.issueId, + authorUserId: "conversation-owner", body: "A new instruction", createdAt: new Date(input.wakeTime.getTime() + 1000) }); + expect(await settleConversationTurnForRun(db, input.companyId, input.runId)).toBe(true); + const [issue] = await db.select({ status: issues.status, state: issues.conversationState }).from(issues).where(eq(issues.id, input.issueId)); + expect(issue).toEqual({ status: "in_progress", state: "active" }); + }); + + it.each(["missing", "company", "issue-company", "generation", "owner", "failed", "no-reply"] as const)("refuses to settle a %s mismatch", async fault => { + const input = await seed(fault === "no-reply" ? "none" : "response"); + if (fault === "generation") await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: input.issueId, + conversationSessionGeneration: 2, wakeCommentId: input.wakeId } }).where(eq(heartbeatRuns.id, input.runId)); + if (fault === "owner") { + const otherRun = randomUUID(); + await db.insert(heartbeatRuns).values({ id: otherRun, companyId: input.companyId, agentId: input.agentId }); + await db.update(issues).set({ executionRunId: otherRun }).where(eq(issues.id, input.issueId)); + } + if (fault === "failed") await db.update(heartbeatRuns).set({ status: "failed" }).where(eq(heartbeatRuns.id, input.runId)); + if (fault === "issue-company") { + const otherCompany = randomUUID(), otherIssue = randomUUID(), otherAgent = randomUUID(); + await db.insert(companies).values({ id: otherCompany, name: "Other company", issuePrefix: otherCompany.slice(0, 8) }); + await db.insert(agents).values({ id: otherAgent, companyId: otherCompany, name: "Other agent" }); + await db.insert(issues).values({ id: otherIssue, companyId: otherCompany, title: "Other conversation", + conversationUserId: "other-owner", conversationAgentId: otherAgent, assigneeAgentId: otherAgent, + conversationState: "active", conversationSessionGeneration: 3 }); + await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: otherIssue, conversationSessionGeneration: 3, + conversationReset: true } }).where(eq(heartbeatRuns.id, input.runId)); + } + expect(await settleConversationTurnForRun(db, fault === "company" ? randomUUID() : input.companyId, + fault === "missing" ? randomUUID() : input.runId)).toBe(false); + const [issue] = await db.select({ status: issues.status }).from(issues).where(and(eq(issues.id, input.issueId), eq(issues.companyId, input.companyId))); + expect(issue.status).toBe("in_progress"); + }); +}); diff --git a/server/src/__tests__/environment-custom-image-routes.test.ts b/server/src/__tests__/environment-custom-image-routes.test.ts index 82b8eda9d37..43b3ae382e4 100644 --- a/server/src/__tests__/environment-custom-image-routes.test.ts +++ b/server/src/__tests__/environment-custom-image-routes.test.ts @@ -92,6 +92,7 @@ vi.mock("../services/plugin-environment-driver.js", () => ({ REUSABLE_LEASE_WORKER_METHODS: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"], listReadyPluginEnvironmentDrivers: vi.fn(async () => []), resolvePluginSandboxProviderDriverByKey: vi.fn(async () => null), + resolvePluginSandboxProviderConfigSchema: vi.fn(async () => null), validatePluginEnvironmentDriverConfig: vi.fn(async ({ config }) => config), validatePluginSandboxProviderConfig: vi.fn(async ({ provider, config }) => ({ normalizedConfig: config, diff --git a/server/src/__tests__/environment-instance-routes.test.ts b/server/src/__tests__/environment-instance-routes.test.ts index 51664e01d98..293e12b7876 100644 --- a/server/src/__tests__/environment-instance-routes.test.ts +++ b/server/src/__tests__/environment-instance-routes.test.ts @@ -79,6 +79,7 @@ vi.mock("../services/plugin-environment-driver.js", () => ({ REUSABLE_LEASE_WORKER_METHODS: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"], listReadyPluginEnvironmentDrivers: vi.fn(async () => []), resolvePluginSandboxProviderDriverByKey: vi.fn(async () => null), + resolvePluginSandboxProviderConfigSchema: vi.fn(async () => null), validatePluginEnvironmentDriverConfig: vi.fn(async ({ config }) => config), validatePluginSandboxProviderConfig: vi.fn(async ({ provider, config }) => ({ normalizedConfig: config, diff --git a/server/src/__tests__/environment-probe.test.ts b/server/src/__tests__/environment-probe.test.ts index e00c2d3eb6f..9474fd8b5a9 100644 --- a/server/src/__tests__/environment-probe.test.ts +++ b/server/src/__tests__/environment-probe.test.ts @@ -25,6 +25,8 @@ vi.mock("../services/plugin-environment-driver.js", () => ({ probePluginEnvironmentDriver: mockProbePluginEnvironmentDriver, probePluginSandboxProviderDriver: mockProbePluginSandboxProviderDriver, resolvePluginSandboxProviderDriverByKey: mockResolvePluginSandboxProviderDriverByKey, + resolvePluginSandboxProviderConfigSchema: async (input: { db: unknown; driverKey: string }) => + (await mockResolvePluginSandboxProviderDriverByKey(input))?.driver?.configSchema ?? null, })); vi.mock("../services/environment-runtime.js", () => ({ diff --git a/server/src/__tests__/environment-routes.test.ts b/server/src/__tests__/environment-routes.test.ts index 07c90052f37..9ad0bf972c4 100644 --- a/server/src/__tests__/environment-routes.test.ts +++ b/server/src/__tests__/environment-routes.test.ts @@ -129,6 +129,8 @@ vi.mock("../services/plugin-environment-driver.js", () => ({ REUSABLE_LEASE_WORKER_METHODS: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"], listReadyPluginEnvironmentDrivers: mockListReadyPluginEnvironmentDrivers, resolvePluginSandboxProviderDriverByKey: mockResolvePluginSandboxProviderDriverByKey, + resolvePluginSandboxProviderConfigSchema: async (input: { db: unknown; driverKey: string }) => + (await mockResolvePluginSandboxProviderDriverByKey(input))?.driver?.configSchema ?? null, startPluginEnvironmentInteractiveSetup: mockStartPluginEnvironmentInteractiveSetup, getPluginEnvironmentInteractiveSetup: mockGetPluginEnvironmentInteractiveSetup, capturePluginEnvironmentTemplate: mockCapturePluginEnvironmentTemplate, diff --git a/server/src/__tests__/git-credentials.test.ts b/server/src/__tests__/git-credentials.test.ts index c286f832b42..e5aee1eb96f 100644 --- a/server/src/__tests__/git-credentials.test.ts +++ b/server/src/__tests__/git-credentials.test.ts @@ -9,6 +9,7 @@ import { GIT_CREDENTIAL_TOKEN_ENV_KEY, buildGitAuthInvocation, createGitRemoteAuthProvider, + createIssueGitRemoteAuthProvider, describeGitAuthFailure, isGitHubHttpsRemoteUrl, scrubGitCredentialText, @@ -379,3 +380,39 @@ describe("DEFAULT_GITHUB_TOKEN_SECRET_NAMES", () => { ]); }); }); + +describe("createIssueGitRemoteAuthProvider", () => { + const githubUrl = "https://github.com/paperclipai/private-repo.git"; + function issueDb(rows: unknown[]) { + const where = vi.fn(async () => rows); + return { db: { select: vi.fn(() => ({ from: () => ({ where }) })) } as unknown as Db, where }; + } + + it("resolves credentials with the issue's responsible user and assigned agent", async () => { + const { db } = issueDb([{ responsibleUserId: "user-1", assigneeAgentId: "agent-1" }]); + const inner = vi.fn(async () => ({ env: {}, configArgs: [], source: "managed_connection" })); + const create = vi.fn(() => inner) as unknown as typeof createGitRemoteAuthProvider; + const provider = createIssueGitRemoteAuthProvider(db, "company-1", "issue-1", create); + await expect(provider(githubUrl)).resolves.toMatchObject({ source: "managed_connection" }); + expect(create).toHaveBeenCalledWith(db, "company-1", { issueId: "issue-1", responsibleUserId: "user-1", agentId: "agent-1" }); + expect(inner).toHaveBeenCalledWith(githubUrl); + }); + + it("looks the issue up once across calls", async () => { + const { db } = issueDb([{ responsibleUserId: "user-1", assigneeAgentId: null }]); + const inner = vi.fn(async () => null); + const create = vi.fn(() => inner) as unknown as typeof createGitRemoteAuthProvider; + const provider = createIssueGitRemoteAuthProvider(db, "company-1", "issue-1", create); + await provider(githubUrl); await provider(githubUrl); + expect(db.select).toHaveBeenCalledTimes(1); + expect(create).toHaveBeenCalledTimes(1); + expect(inner).toHaveBeenCalledTimes(2); + }); + + it("falls back to the issue-only context when the issue is missing", async () => { + const { db } = issueDb([]); + const create = vi.fn(() => vi.fn(async () => null)) as unknown as typeof createGitRemoteAuthProvider; + await createIssueGitRemoteAuthProvider(db, "company-1", "issue-1", create)(githubUrl); + expect(create).toHaveBeenCalledWith(db, "company-1", { issueId: "issue-1", responsibleUserId: null, agentId: null }); + }); +}); diff --git a/server/src/__tests__/github-connection-events.test.ts b/server/src/__tests__/github-connection-events.test.ts index 77d34f15e6a..6bcd2913396 100644 --- a/server/src/__tests__/github-connection-events.test.ts +++ b/server/src/__tests__/github-connection-events.test.ts @@ -7,6 +7,7 @@ import { connectionGrants, createDb, externalObjects, + EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, toolApplications, toolConnections, } from "@paperclipai/db"; @@ -30,7 +31,7 @@ describeEmbeddedPostgres("GitHub connection event delivery", () => { beforeAll(async () => { tempDb = await startEmbeddedPostgresTestDatabase("paperclip-github-events-"); db = createDb(tempDb.connectionString); - }, 20_000); + }, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS); afterEach(async () => { await db.delete(activityLog); @@ -46,7 +47,7 @@ describeEmbeddedPostgres("GitHub connection event delivery", () => { await tempDb?.cleanup(); }); - it("applies a normalized merged PR once while retaining undecryptable siblings for retry", async () => { + it.each([false, true])("applies a normalized merged PR once while retaining undecryptable siblings for retry (active setup=%s)", async activeSetup => { const companyId = randomUUID(); const userId = `github-owner-${randomUUID()}`; const applicationId = randomUUID(); @@ -151,8 +152,21 @@ describeEmbeddedPostgres("GitHub connection event delivery", () => { let currentTime = new Date("2026-09-04T12:00:05.000Z"); const service = githubConnectionEventService(db, { connector, now: () => currentTime }); + if (activeSetup) { + // An existing personal connection can leave a long empty-queue backoff + // before a new dedicated App starts its signed-ping verification. + for (const delay of [0, 10_000, 20_000, 40_000, 80_000]) { + currentTime = new Date(currentTime.getTime() + delay); + vi.mocked(connector.leaseEvents).mockResolvedValueOnce(null); + await service.pollOnce(); + } + currentTime = new Date(currentTime.getTime() + 5_000); + await expect(service.pollOnce()).resolves.toMatchObject({ leased: 0 }); + expect(connector.leaseEvents).toHaveBeenCalledTimes(5); + } + const warning = vi.spyOn(logger, "warn").mockImplementation(() => undefined); - await expect(service.pollOnce()).resolves.toMatchObject({ leased: 2, processed: 1, duplicate: 0, failed: 1 }); + await expect(service.pollOnce({ forSetup: activeSetup })).resolves.toMatchObject({ leased: 2, processed: 1, duplicate: 0, failed: 1 }); expect(warning.mock.calls).toEqual([[{ deliveryId: "undecryptable-delivery", reason: "inner_event_authentication_failed", }, "GitHub connector event retained for retry"]]); diff --git a/server/src/__tests__/github-operation-credentials.test.ts b/server/src/__tests__/github-operation-credentials.test.ts index 02d1c864bcd..699249b5b58 100644 --- a/server/src/__tests__/github-operation-credentials.test.ts +++ b/server/src/__tests__/github-operation-credentials.test.ts @@ -6,6 +6,7 @@ import { createRuntimeToolsToken } from "../runtime-tools-token.js"; import { errorHandler } from "../middleware/index.js"; import { randomUUID } from "node:crypto"; import { eq } from "drizzle-orm"; +import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture"; import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; import { agents, @@ -31,6 +32,7 @@ import { } from "./helpers/embedded-postgres.js"; import { initializeRunIdentity, + captureRunIdentity, reserveSteeredIdentity, acceptSteeredIdentity, } from "../services/run-identity.js"; @@ -192,6 +194,72 @@ const support = await getEmbeddedPostgresTestSupport(); }); return { id, connectionId, secretId, definitionId }; } + it.each(["config", "transportConfig"] as const)("reads only current GitHub identity connections via %s", async location => { + const input = await seed(), selected = await grant(input, "A"); + const [connection] = await db.select().from(toolConnections).where(eq(toolConnections.id, selected.connectionId)); + await db.update(toolConnections).set({ config: {}, transportConfig: {}, [location]: { sourceTemplateKey: "github" } }) + .where(eq(toolConnections.id, selected.connectionId)); + const payload = "unrelated-connection-config🙂".repeat(10_000); + const irrelevant = [null, true, 42, ["github"], { name: "github" }, "GitHub", " github ", "other"].map(sourceTemplateKey => { + const id = randomUUID(); + return { id, companyId: input.companyId, applicationId: connection.applicationId, name: id, uid: id, + transport: "mcp_remote" as const, config: { sourceTemplateKey, payload }, transportConfig: { sourceTemplateKey } }; + }); + await db.insert(toolConnections).values(irrelevant); + const observed = createDb(database.connectionString), events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(observed, { toolConnections }, capture); + try { + expect(await resolveManagedGitHubIdentitySelection(observed, input.companyId, { agentId: input.agentId, responsibleUserId: "A" })) + .toMatchObject({ configured: true, grant: { id: selected.id } }); + capture.flush(); + const reads = events.filter(event => event.table === "tool_connections" && event.operation === "select"); + expect(reads).toHaveLength(1); + expect(reads[0].rows).toBe(1); + expect(reads[0].responseValueBytes).toBeLessThan(8192); + await db.update(toolConnections).set({ enabled: false }).where(eq(toolConnections.id, selected.connectionId)); + expect((await resolveManagedGitHubIdentitySelection(observed, input.companyId, { agentId: input.agentId, responsibleUserId: "A" })).grant) + .toBeUndefined(); + await db.update(toolConnections).set({ [location]: { sourceTemplateKey: "other" } }).where(eq(toolConnections.id, selected.connectionId)); + expect(await resolveManagedGitHubIdentitySelection(observed, input.companyId, { agentId: input.agentId, responsibleUserId: "A" })) + .toEqual({ configured: false }); + expect(await resolveManagedGitHubIdentitySelection(observed, randomUUID(), { agentId: input.agentId, responsibleUserId: "A" })) + .toEqual({ configured: false }); + const saved = await db.select().from(toolConnections).where(eq(toolConnections.companyId, input.companyId)); + for (const row of irrelevant) expect(saved.find(candidate => candidate.id === row.id)?.config).toEqual(row.config); + } finally { capture.flush(); await observed.$client.end(); } + }); + + it("resolves current GitHub credentials without transferring execution snapshots", async () => { + const input = await seed(), selected = await grant(input, "A"); + const payload = "github-credential-saved-input🙂".repeat(25_000); + await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: input.issueId, prompt: payload }, + runnerProfileJson: { nativeExecutionInput: { savedInput: payload }, savedInput: payload }, + resultJson: { stdout: payload }, stdoutExcerpt: payload, stderrExcerpt: payload, + }).where(eq(heartbeatRuns.id, input.runId)); + const [before] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, input.runId)); + const observed = createDb(database.connectionString), events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(observed, { heartbeatRuns }, capture); + try { + expect(await resolveGitHubOperationCredentials(observed, input)).toMatchObject({ + status: "available", login: "A", grantId: selected.id, + }); + capture.flush(); + const reads = events.filter(event => event.table === "heartbeat_runs" && event.operation === "select"); + expect(reads.length).toBeGreaterThan(0); + expect(reads.reduce((bytes, event) => bytes + Number(event.responseValueBytes), 0)).toBeLessThan(8192); + expect(JSON.stringify(events)).not.toContain("github-credential-saved-input"); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, input.runId)))[0]).toEqual(before); + const full = await captureRunIdentity(observed, input); + expect(full.run.runnerProfileJson).toEqual(before.runnerProfileJson); + expect(full.run.resultJson).toEqual(before.resultJson); + expect(full.run.contextSnapshot).toEqual(before.contextSnapshot); + await db.update(connectionGrants).set({ status: "revoked" }).where(eq(connectionGrants.id, selected.id)); + expect(await resolveGitHubOperationCredentials(observed, input)).toMatchObject({ status: "unavailable" }); + } finally { capture.flush(); await observed.$client.end(); } + }); + async function switchTo( input: Awaited>, user: string, diff --git a/server/src/__tests__/heartbeat-run-event-sequencing.test.ts b/server/src/__tests__/heartbeat-run-event-sequencing.test.ts index 28c2dd00d24..d8e7bd0b9cb 100644 --- a/server/src/__tests__/heartbeat-run-event-sequencing.test.ts +++ b/server/src/__tests__/heartbeat-run-event-sequencing.test.ts @@ -1,5 +1,6 @@ import { describe, expect, it } from "vitest"; -import { eq } from "drizzle-orm"; +import { eq, sql } from "drizzle-orm"; +import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture"; import { agents, companies, @@ -15,6 +16,62 @@ import { import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; describe("P6-11..13 / P6-17 canonical event allocator", () => { + it("keeps native cursor responses bounded with long histories and a late gap fill", async () => { + const temporary = await startEmbeddedPostgresTestDatabase("paperclip-native-cursor-budget-"); + const db = createDb(temporary.connectionString); + const companyId = "20000000-0000-4000-8000-000000000001"; + const agentId = "20000000-0000-4000-8000-000000000002"; + const runId = "20000000-0000-4000-8000-000000000003"; + const otherRunId = "20000000-0000-4000-8000-000000000004"; + try { + await db.insert(companies).values({ id: companyId, name: "Cursor budget", issuePrefix: "CBG" }); + await db.insert(agents).values({ id: agentId, companyId, name: "Cursor agent" }); + await db.insert(heartbeatRuns).values([ + { id: runId, companyId, agentId, status: "running", nextEventSeq: 5005 }, + { id: otherRunId, companyId, agentId, status: "running" }, + ]); + for (let batch = 0; batch < 10; batch += 1) { + await db.insert(heartbeatRunEvents).values(Array.from({ length: 500 }, (_, index) => { + const seq = batch * 500 + index + 1; + return { companyId, agentId, runId, seq, eventType: "item.delta", sourceInstanceId: "runner-a", + sourceSeq: seq, sourceEventId: `seed:${seq}` }; + })); + } + await db.insert(heartbeatRunEvents).values([ + ...[-1, 0, null].map((sourceSeq, index) => ({ + companyId, agentId, runId, seq: 5001 + index, eventType: "item.delta", + sourceInstanceId: "runner-a", sourceSeq, sourceEventId: `legacy:${index}`, + })), + { companyId, agentId, runId, seq: 5004, eventType: "item.delta", sourceInstanceId: "runner-b", + sourceSeq: 5001, sourceEventId: "other-source:5001" }, + { companyId, agentId, runId: otherRunId, seq: 1, eventType: "item.delta", sourceInstanceId: "runner-a", + sourceSeq: 5001, sourceEventId: "other-run:5001" }, + ]); + const append = (sourceSeq: number) => appendHeartbeatRunEvent(db, { + companyId, agentId, runId, eventType: "item.delta", payload: { sourceSeq }, + nativeSource: { sourceInstanceId: "runner-a", sourceSeq, sourceEventId: `runner-a:${sourceSeq}`, + protocolSchemaVersion: 1, canonicalPayload: { sourceSeq } }, + }); + const events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(db, { heartbeatRunEvents }, capture); + try { expect((await append(5002)).highestContiguousSourceSeq).toBe(5000); } + finally { capture.flush(); } + const reads = events.filter(event => event.event === "paperclip.db_response_capture.entry.v1" + && event.table === "heartbeat_run_events" && (event.operation === "select" || event.operation === "execute")); + expect(reads.reduce((sum, event) => sum + Number(event.calls), 0)).toBeGreaterThan(0); + expect(reads.reduce((sum, event) => sum + Number(event.rows), 0)).toBeLessThan(16); + expect(reads.reduce((sum, event) => sum + Number(event.responseValueBytes), 0)).toBeLessThan(2048); + expect((await append(5001)).highestContiguousSourceSeq).toBe(5002); + const replay = await append(5002); + expect(replay.disposition).toBe("duplicate"); + expect(replay.highestContiguousSourceSeq).toBe(5002); + const [remaining] = await db.select({ count: sql`count(*)::int` }).from(heartbeatRunEvents) + .where(eq(heartbeatRunEvents.runId, runId)); + expect(remaining?.count).toBe(5006); + } finally { await temporary.cleanup(); } + }, 60_000); + it.each([false, true])("deduplicates concurrent retry exhaustion (historical receipt: %s)", async (historical) => { const temporary = await startEmbeddedPostgresTestDatabase("paperclip-exhaustion-events-"); const db = createDb(temporary.connectionString); @@ -165,4 +222,46 @@ describe("P6-11..13 / P6-17 canonical event allocator", () => { await temporary.cleanup(); } }, 60_000); + + it("reports the highest contiguous source seq per source instance across gaps and late arrivals", async () => { + const temporary = await startEmbeddedPostgresTestDatabase("paperclip-native-cursor-"); + const db = createDb(temporary.connectionString); + const companyId = "20000000-0000-4000-8000-000000000001"; + const agentId = "20000000-0000-4000-8000-000000000002"; + const runId = "20000000-0000-4000-8000-000000000003"; + try { + await db.insert(companies).values({ id: companyId, name: "Cursor fixture", issuePrefix: "CUR" }); + await db.insert(agents).values({ id: agentId, companyId, name: "Cursor agent" }); + await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running" }); + const append = async (sourceInstanceId: string, sourceSeq: number) => (await appendHeartbeatRunEvent(db, { + companyId, + runId, + agentId, + eventType: "item.delta", + payload: { sourceInstanceId, sourceSeq }, + nativeSource: { + sourceInstanceId, + sourceEventId: `${sourceInstanceId}:${sourceSeq}`, + sourceSeq, + protocolSchemaVersion: 1, + canonicalPayload: { sourceInstanceId, sourceSeq }, + }, + })).highestContiguousSourceSeq; + + expect(await append("runner-a", 2)).toBe(0); + expect(await append("runner-a", 1)).toBe(2); + expect(await append("runner-a", 3)).toBe(3); + expect(await append("runner-a", 5)).toBe(3); + expect(await append("runner-a", 6)).toBe(3); + expect(await append("runner-b", 1)).toBe(1); + expect(await append("runner-a", 4)).toBe(6); + for (let sourceSeq = 7; sourceSeq <= 200; sourceSeq += 1) { + expect(await append("runner-a", sourceSeq)).toBe(sourceSeq); + } + expect(await append("runner-b", 3)).toBe(1); + expect(await append("runner-b", 2)).toBe(3); + } finally { + await temporary.cleanup(); + } + }, 60_000); }); diff --git a/server/src/__tests__/heartbeat-run-privacy-routes.test.ts b/server/src/__tests__/heartbeat-run-privacy-routes.test.ts index 03914cadeda..ed10d2548f2 100644 --- a/server/src/__tests__/heartbeat-run-privacy-routes.test.ts +++ b/server/src/__tests__/heartbeat-run-privacy-routes.test.ts @@ -1,6 +1,7 @@ import { randomUUID } from "node:crypto"; import { eq } from "drizzle-orm"; import { drizzle } from "drizzle-orm/postgres-js"; +import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture"; import express from "express"; import request from "supertest"; import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; @@ -257,6 +258,29 @@ describeEmbeddedPostgres("heartbeat run privacy routes", { concurrent: false }, return app; } + it("lists workspace operations with small run reads and checks a revoked issue grant again", async () => { + const f = await seedFixture(), payload = "workspace-operation-saved-input🙂".repeat(25_000); + await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: f.issueId, prompt: payload }, + runnerProfileJson: { instructions: payload }, stdoutExcerpt: payload, stderrExcerpt: payload, + }).where(eq(heartbeatRuns.id, f.privateRunId)); + const [before] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.privateRunId)); + const observed = createDb(tempDb!.connectionString), events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(observed, { heartbeatRuns }, capture); + try { + const app = createApp(f.companyId, f.grantedAgentId, observed); + const response = await request(app).get(`/api/heartbeat-runs/${f.privateRunId}/workspace-operations`).expect(200); + expect(response.body).toEqual([expect.objectContaining({ id: f.operationId })]); + capture.flush(); + const reads = events.filter(event => event.table === "heartbeat_runs" && event.operation === "select"); + expect(reads.length).toBeGreaterThan(0); + expect(reads.reduce((bytes, event) => bytes + Number(event.responseValueBytes), 0)).toBeLessThan(2048); + await db.delete(issueAccessGrants).where(eq(issueAccessGrants.subjectId, f.grantedAgentId)); + await request(app).get(`/api/heartbeat-runs/${f.privateRunId}/workspace-operations`).expect(404); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.privateRunId)))[0]).toEqual(before); + } finally { capture.flush(); await observed.$client.end(); } + }); + it("returns 404 for direct private-run content and metadata-only list stubs to a non-member", async () => { const fixture = await seedFixture(); const app = createApp(fixture.companyId, fixture.otherAgentId); @@ -459,6 +483,105 @@ describeEmbeddedPostgres("heartbeat run privacy routes", { concurrent: false }, process.env.PAPERCLIP_ISSUE_PRIVACY_MODE = "enforce"; } }); + it("batches distinct issue bindings without changing current authorization decisions", async () => { + const fixture = await seedFixture(); + const ids = Array.from({ length: 64 }, () => randomUUID()); + await db.insert(issues).values(ids.map((id, index) => ({ + id, companyId: fixture.companyId, title: `History ${index}`, status: "in_progress", + visibility: index % 2 ? "private" : "open", privacyRootIssueId: index % 2 ? id : null, + assigneeAgentId: fixture.ownerAgentId, responsibleUserId: fixture.ownerUserId, + }))); + const queries: string[] = []; + const observedDb = drizzle(db.$client, { schema, logger: { logQuery(query) { queries.push(query); } } }) as typeof db; + const access = authorizationService(observedDb); + const actor = { type: "agent" as const, companyId: fixture.companyId, agentId: fixture.grantedAgentId }; + const bindings = [...ids, fixture.issueId].map(issueId => ({ companyId: fixture.companyId, scopeKind: "issue" as const, issueId })); + const runs = [...bindings, ...bindings, { ...bindings[0], companyId: randomUUID() }, + { ...bindings[0], issueId: randomUUID() }, { companyId: fixture.companyId, scopeKind: "company" as const }, + { companyId: fixture.companyId, scopeKind: "issue" as const }]; + const individual = await getHeartbeatRunReadDecisions(observedDb, { decide: access.decide }, actor, runs); + const individualCount = queries.length; + queries.length = 0; + const batched = await getHeartbeatRunReadDecisions(observedDb, access, actor, runs); + expect(batched).toEqual(individual); + expect(batched.slice(0, 64)).toEqual(ids.map((_, index) => index % 2 === 0)); + expect(batched.at(64)).toBe(true); + expect(batched.slice(-4)).toEqual([false, false, true, false]); + expect(queries.length).toBeLessThan(25); + expect(queries.length).toBeLessThan(individualCount / 4); + await db.update(issueAccessGrants).set({ revokedAt: new Date() }).where(eq(issueAccessGrants.subjectId, fixture.grantedAgentId)); + expect((await getHeartbeatRunReadDecisions(observedDb, access, actor, runs)).at(64)).toBe(false); + }); + + it.each(["enforce", "shadow", "off"])("keeps missing and foreign issue bindings closed in %s mode", async mode => { + const fixture = await seedFixture(); + process.env.PAPERCLIP_ISSUE_PRIVACY_MODE = mode; + try { + const access = authorizationService(db); + const actor = { type: "board" as const, userId: fixture.ownerUserId }; + const binding = { companyId: fixture.companyId, scopeKind: "issue" as const, issueId: fixture.issueId }; + const runs = [binding, { ...binding, issueId: randomUUID() }, { ...binding, companyId: randomUUID() }]; + const expected = await getHeartbeatRunReadDecisions(db, { decide: access.decide }, actor, runs); + expect(await getHeartbeatRunReadDecisions(db, access, actor, runs)).toEqual(expected); + expect(expected).toEqual([true, false, false]); + await db.delete(companyMemberships).where(eq(companyMemberships.principalId, fixture.ownerUserId)); + expect(await getHeartbeatRunReadDecisions(db, access, actor, [binding])).toEqual( + await getHeartbeatRunReadDecisions(db, { decide: access.decide }, actor, [binding]), + ); + } finally { process.env.PAPERCLIP_ISSUE_PRIVACY_MODE = "enforce"; } + }); + + it("checks more than 500 task bindings with bounded company-scoped queries", async () => { + const fixture = await seedFixture(); + const ids = Array.from({ length: 501 }, () => randomUUID()); + await db.insert(issues).values(ids.map((id, index) => ({ + id, companyId: fixture.companyId, title: `Batch ${index}`, visibility: "open", status: "in_progress", + }))); + const queries: { query: string; params: unknown[] }[] = []; + const observedDb = drizzle(db.$client, { schema, logger: { logQuery(query, params) { queries.push({ query, params }); } } }) as typeof db; + const access = authorizationService(observedDb); + const actor = { type: "agent" as const, companyId: fixture.companyId, agentId: fixture.ownerAgentId }; + const runs = ids.map(issueId => ({ companyId: fixture.companyId, scopeKind: "issue" as const, issueId })); + expect(await getHeartbeatRunReadDecisions(observedDb, access, actor, runs)).toEqual(ids.map(() => true)); + const existence = queries.filter(({ query }) => query.startsWith('select "id" from "issues"')); + expect(existence.map(({ params }) => params.length).sort((a, b) => a - b)).toEqual([2, 501]); + expect(existence.every(({ query, params }) => query.includes('"company_id"') && params[0] === fixture.companyId)).toBe(true); + }); + + it("captures batch inputs before awaiting reads and rejects a failed batch", async () => { + const fixture = await seedFixture(); + const access = authorizationService(db); + const actor = { type: "agent" as const, companyId: fixture.companyId, agentId: fixture.ownerAgentId }; + const binding = { companyId: fixture.companyId, scopeKind: "issue" as const, issueId: fixture.issueId }; + const decideReads = vi.spyOn(access, "decideReads"); + const pending = getHeartbeatRunReadDecisions(db, access, actor, [binding, binding]); + binding.issueId = randomUUID(); + expect(await pending).toEqual([true, true]); + expect(decideReads).toHaveBeenCalledExactlyOnceWith(actor, [expect.objectContaining({ issueId: fixture.issueId })]); + const error = new Error("authorization unavailable"); + decideReads.mockRejectedValueOnce(error); + const current = { ...binding, issueId: fixture.issueId }; + await expect(getHeartbeatRunReadDecisions(db, access, actor, [current])).rejects.toBe(error); + const transient = Object.assign(new Error("temporary disconnect"), { code: "CONNECTION_CLOSED" }); + decideReads.mockRejectedValueOnce(transient); + expect(await getHeartbeatRunReadDecisions(db, access, actor, [current])).toEqual([true]); + }); + + it("does no database or authorization work for company-only or empty batches", async () => { + const fixture = await seedFixture(); + const queries: string[] = []; + const observedDb = drizzle(db.$client, { schema, logger: { logQuery(query) { queries.push(query); } } }) as typeof db; + const access = authorizationService(observedDb); + const decideReads = vi.spyOn(access, "decideReads"); + const actor = { type: "agent" as const, companyId: fixture.companyId, agentId: fixture.ownerAgentId }; + expect(await getHeartbeatRunReadDecisions(observedDb, access, actor, [])).toEqual([]); + expect(await getHeartbeatRunReadDecisions(observedDb, access, actor, [ + { companyId: fixture.companyId, scopeKind: "company" }, { companyId: fixture.companyId, scopeKind: "issue" }, + ])).toEqual([true, false]); + expect(queries).toEqual([]); + expect(decideReads).not.toHaveBeenCalled(); + }); + it.each(["owner", "outsider"])("checks concurrent duplicate issue runs once for the %s", async (viewer) => { const fixture = await seedFixture(); const access = authorizationService(db); @@ -469,7 +592,7 @@ describeEmbeddedPostgres("heartbeat run privacy routes", { concurrent: false }, agentId: viewer === "owner" ? fixture.ownerAgentId : fixture.otherAgentId, }; const binding = { companyId: fixture.companyId, scopeKind: "issue" as const, issueId: fixture.issueId }; - const decisions = await getHeartbeatRunReadDecisions(db, access, actor, Array.from({ length: 200 }, () => ({ ...binding }))); + const decisions = await getHeartbeatRunReadDecisions(db, { decide: access.decide }, actor, Array.from({ length: 200 }, () => ({ ...binding }))); expect(decisions).toEqual(Array(200).fill(viewer === "owner")); expect(decide).toHaveBeenCalledTimes(1); decide.mockRestore(); @@ -493,7 +616,7 @@ describeEmbeddedPostgres("heartbeat run privacy routes", { concurrent: false }, const decide = vi.spyOn(access, "decide"); const actor = { type: "agent" as const, companyId: fixture.companyId, agentId: fixture.ownerAgentId }; const binding = { companyId: fixture.companyId, scopeKind: "issue" as const, issueId: fixture.issueId }; - const pending = getHeartbeatRunReadDecisions(db, access, actor, [binding, binding]); + const pending = getHeartbeatRunReadDecisions(db, { decide: access.decide }, actor, [binding, binding]); binding.issueId = randomUUID(); expect(await pending).toEqual([true, true]); expect(decide).toHaveBeenCalledExactlyOnceWith(expect.objectContaining({ @@ -567,7 +690,7 @@ describeEmbeddedPostgres("heartbeat run privacy routes", { concurrent: false }, const decide = vi.spyOn(access, "decide").mockRejectedValue(error); const actor = { type: "agent" as const, companyId: fixture.companyId, agentId: fixture.ownerAgentId }; const binding = { companyId: fixture.companyId, scopeKind: "issue" as const, issueId: fixture.issueId }; - await expect(getHeartbeatRunReadDecisions(db, access, actor, Array(20).fill(binding))).rejects.toBe(error); + await expect(getHeartbeatRunReadDecisions(db, { decide: access.decide }, actor, Array(20).fill(binding))).rejects.toBe(error); expect(decide).toHaveBeenCalledTimes(1); decide.mockRestore(); expect(await getHeartbeatRunReadDecisions(db, access, actor, [binding])).toEqual([true]); diff --git a/server/src/__tests__/heartbeat-run-summary.test.ts b/server/src/__tests__/heartbeat-run-summary.test.ts index 33b03986b83..c0ed3d6b113 100644 --- a/server/src/__tests__/heartbeat-run-summary.test.ts +++ b/server/src/__tests__/heartbeat-run-summary.test.ts @@ -25,10 +25,10 @@ describe("model-facing run errors", () => { }); }); -describe("scheduled continuation presentation", () => { +describe.each(["scheduled_monitor_waiting", "scheduled_monitor_dispatched", "scheduled_monitor_preserved"])("scheduled continuation presentation (%s)", finalizationReasonCode => { const resultJson = { finalizationPhase: "committed", - finalizationReasonCode: "scheduled_monitor_waiting", + finalizationReasonCode, nativeResult: { schema: "paperclip.run_result.v1", reportedWorkDisposition: "yielded", diff --git a/server/src/__tests__/issue-creation-source-read.test.ts b/server/src/__tests__/issue-creation-source-read.test.ts new file mode 100644 index 00000000000..2b47f3af580 --- /dev/null +++ b/server/src/__tests__/issue-creation-source-read.test.ts @@ -0,0 +1,109 @@ +import { randomUUID } from "node:crypto"; +import { activityLog, agents, companies, createDb, heartbeatRuns, issues } from "@paperclipai/db"; +import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture"; +import { eq } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { resolveIssueCreationSource, runBoundIssueReference } from "../services/issue-creation-source.js"; + +const support = await getEmbeddedPostgresTestSupport(); + +describe.skipIf(!support.supported)("creation source run projections", () => { + const companyId = randomUUID(), otherCompanyId = randomUUID(), agentId = randomUUID(); + const sourceId = randomUUID(), secondId = randomUUID(), foreignId = randomUUID(); + let database: Awaited>; + let db: ReturnType; + let childNumber = 100; + + beforeAll(async () => { + database = await startEmbeddedPostgresTestDatabase("creation-source-read-"); + db = createDb(database.connectionString); + await db.insert(companies).values([ + { id: companyId, name: "Creation source fixture", issuePrefix: "CSR" }, + { id: otherCompanyId, name: "Foreign source fixture", issuePrefix: "FSR" }, + ]); + await db.insert(agents).values({ id: agentId, companyId, name: "Source agent", adapterType: "process" }); + await db.insert(issues).values([ + { id: sourceId, companyId, title: "Source task", identifier: "CSR-1", issueNumber: 1 }, + { id: secondId, companyId, title: "Second source task", identifier: "CSR-2", issueNumber: 2 }, + { id: foreignId, companyId: otherCompanyId, title: "Foreign task", identifier: "FSR-1", issueNumber: 1 }, + ]); + }, 60_000); + + afterAll(async () => { + await db?.$client.end(); + await database?.cleanup(); + }); + + async function fixture(values: Partial) { + const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId, status: "succeeded", ...values }).returning(); + const number = childNumber++; + const [child] = await db.insert(issues).values({ companyId, title: "Created task", identifier: `CSR-${number}`, + issueNumber: number, originRunId: run.id }).returning(); + return { run, child }; + } + + it.each(["scoped", "legacy task key", "legacy nested identifier"])("omits saved run payloads for %s bindings while preserving provenance and storage", async (binding) => { + const bulk = "retained-source-context🙂".repeat(32_768); + const contextSnapshot = binding === "scoped" + ? { issueId: sourceId, paperclipIssue: { id: sourceId, retainedDescription: bulk }, retainedLaunchInput: bulk } + : binding === "legacy task key" + ? { taskKey: "CSR-1", paperclipIssue: { retainedDescription: bulk }, retainedLaunchInput: bulk } + : { paperclipIssue: { id: "CSR-1", retainedDescription: bulk }, retainedLaunchInput: bulk }; + const { run, child } = await fixture({ contextSnapshot, + runnerProfileJson: { retainedInput: bulk }, stdoutExcerpt: bulk, stderrExcerpt: bulk, resultJson: { retainedReceipt: bulk } }); + expect(run.issueId).toBe(binding === "scoped" ? sourceId : null); + expect(run.nativeIssueId).toBeNull(); + const observed = createDb(database.connectionString), events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(observed, { heartbeatRuns, issues }, capture); + try { + const result = await resolveIssueCreationSource(observed, child); + capture.flush(); + const reads = events.filter(event => event.table === "heartbeat_runs" && event.operation === "select"); + expect(reads.reduce((count, event) => count + Number(event.calls), 0)).toBe(1); + expect(reads.reduce((bytes, event) => bytes + Number(event.responseValueBytes), 0)).toBeLessThan(8192); + expect(result).toMatchObject({ run: { id: run.id, companyId, agentId }, sourceIssue: { id: sourceId } }); + expect(runBoundIssueReference(result!.run)).toBe(binding === "scoped" ? sourceId : "CSR-1"); + for (const key of ["runnerProfileJson", "stdoutExcerpt", "stderrExcerpt", "resultJson"]) expect(result!.run).not.toHaveProperty(key); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0]).toEqual(run); + expect(JSON.stringify(events)).not.toContain("retained-source-context"); + } finally { + capture.flush(); + await observed.$client.end(); + } + }); + + it.each([ + { label: "native identity", nativeIssueId: sourceId, issueId: secondId, contextSnapshot: { issueId: secondId }, expected: sourceId }, + { label: "issue scope", issueId: sourceId, contextSnapshot: { issueId: secondId }, expected: sourceId }, + { label: "legacy issue id", contextSnapshot: { issueId: sourceId, taskId: secondId }, expected: sourceId }, + { label: "typed task id fallback", contextSnapshot: { issueId: 42, taskId: sourceId }, expected: sourceId }, + { label: "nested issue identifier", contextSnapshot: { issueId: " ", paperclipIssue: { id: "CSR-1", ignored: "payload" }, taskKey: "CSR-2" }, expected: sourceId }, + { label: "typed nested id fallback", contextSnapshot: { paperclipIssue: { id: 42 }, taskKey: "CSR-1" }, expected: sourceId }, + { label: "foreign context", contextSnapshot: { taskId: foreignId }, expected: null }, + { label: "null context", contextSnapshot: null, expected: null }, + ])("preserves binding precedence for $label", async ({ label: _label, expected, ...values }) => { + const { child } = await fixture(values); + expect((await resolveIssueCreationSource(db, child))?.sourceIssue.id ?? null).toBe(expected); + }); + + it.each([42, "scalar", ["issueId", sourceId]])("does not invent a task binding from malformed context %j", async (context) => { + const { child } = await fixture({ contextSnapshot: context as unknown as Record }); + expect(await resolveIssueCreationSource(db, child)).toBeNull(); + }); + + it("preserves historical activity lookup, current bindings, and missing or foreign source denial", async () => { + const { run, child } = await fixture({ contextSnapshot: { taskKey: "CSR-1" } }); + await db.update(issues).set({ originRunId: null }).where(eq(issues.id, child.id)); + await db.insert(activityLog).values({ companyId, actorType: "system", actorId: "fixture", action: "issue.created", + entityType: "issue", entityId: child.id, runId: run.id }); + expect((await resolveIssueCreationSource(db, { ...child, originRunId: null }))?.sourceIssue.id).toBe(sourceId); + await db.update(heartbeatRuns).set({ contextSnapshot: { taskKey: "CSR-2" } }).where(eq(heartbeatRuns.id, run.id)); + expect((await resolveIssueCreationSource(db, child))?.sourceIssue.id).toBe(secondId); + expect(await resolveIssueCreationSource(db, { ...child, companyId: otherCompanyId })).toBeNull(); + expect(await resolveIssueCreationSource(db, { ...child, originRunId: randomUUID() })).toBeNull(); + await db.update(heartbeatRuns).set({ contextSnapshot: { taskId: child.id } }).where(eq(heartbeatRuns.id, run.id)); + expect(await resolveIssueCreationSource(db, child)).toBeNull(); + }); +}); diff --git a/server/src/__tests__/issue-list-assignee-filter-routes.test.ts b/server/src/__tests__/issue-list-assignee-filter-routes.test.ts index 269f5d20010..3c685dcfa3d 100644 --- a/server/src/__tests__/issue-list-assignee-filter-routes.test.ts +++ b/server/src/__tests__/issue-list-assignee-filter-routes.test.ts @@ -791,6 +791,62 @@ describeEmbeddedPostgres("issue list routes assigneeAgentId filter", () => { }); }); + it.each([ + "projectId", + "parentId", + "parentIssueId", + "workspaceId", + "executionWorkspaceId", + "descendantOf", + "labelId", + "participantAgentId", + ])("returns 422 instead of a DB error for malformed %s filters", async (param) => { + const companyId = randomUUID(); + await db.insert(companies).values({ + id: companyId, + name: "Paperclip", + issuePrefix: uniqueIssuePrefix(), + requireBoardApprovalForNewAgents: false, + }); + await seedCloudTenantMember(companyId); + + const app = createApp(companyId); + const res = await request(app) + .get(`/api/companies/${companyId}/issues`) + .query({ status: "todo", [param]: "d", limit: "20" }); + + expect(res.status, JSON.stringify(res.body)).toBe(422); + expect(res.body).toMatchObject({ error: `${param} must be a UUID` }); + }); + + it.each([ + "projectId", + "parentId", + "parentIssueId", + "workspaceId", + "executionWorkspaceId", + "descendantOf", + "labelId", + "participantAgentId", + ])("returns 422 from issues/count for malformed %s filters", async (param) => { + const companyId = randomUUID(); + await db.insert(companies).values({ + id: companyId, + name: "Paperclip", + issuePrefix: uniqueIssuePrefix(), + requireBoardApprovalForNewAgents: false, + }); + await seedCloudTenantMember(companyId); + + const app = createApp(companyId); + const res = await request(app) + .get(`/api/companies/${companyId}/issues/count`) + .query({ attention: "blocked", [param]: "d" }); + + expect(res.status, JSON.stringify(res.body)).toBe(422); + expect(res.body).toMatchObject({ error: `${param} must be a UUID` }); + }); + it("returns opt-in live descendant counts for offscreen live descendants only", async () => { const companyId = randomUUID(); const otherCompanyId = randomUUID(); diff --git a/server/src/__tests__/issue-queued-comments-routes.test.ts b/server/src/__tests__/issue-queued-comments-routes.test.ts index bb42982787b..e02d9708a38 100644 --- a/server/src/__tests__/issue-queued-comments-routes.test.ts +++ b/server/src/__tests__/issue-queued-comments-routes.test.ts @@ -4,6 +4,7 @@ import { randomUUID } from "node:crypto"; import express from "express"; import request from "supertest"; import { eq, sql } from "drizzle-orm"; +import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture"; import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; import { agentRuntimeState, @@ -79,7 +80,7 @@ describeEmbeddedPostgres("issue queued-comment routes", () => { await tempDb?.cleanup(); }); - function app(companyId: string, userId = "queue-owner", agentActor?: { agentId: string; runId: string }) { + function app(companyId: string, userId = "queue-owner", agentActor?: { agentId: string; runId: string }, database = db) { const testApp = express(); testApp.use(express.json()); testApp.use((req, _res, next) => { @@ -94,7 +95,7 @@ describeEmbeddedPostgres("issue queued-comment routes", () => { }; next(); }); - testApp.use("/api", issueRoutes(db, {} as any, {})); + testApp.use("/api", issueRoutes(database, {} as any, {})); testApp.use(errorHandler); return testApp; } @@ -192,6 +193,104 @@ describeEmbeddedPostgres("issue queued-comment routes", () => { return { companyId, agentId, issueId, runId, wakeId, commentIds }; } + it.each([false, true])("polls queued comments without transferring run snapshots (agent fallback: %s)", async fallback => { + const f = await seedQueue(), payload = "saved-execution-input🙂".repeat(25_000); + await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: f.issueId, prompt: payload }, + runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner", adapterConfig: { instructions: payload } } }, + stdoutExcerpt: payload, stderrExcerpt: payload, + }).where(eq(heartbeatRuns.id, f.runId)); + if (fallback) await db.update(issues).set({ executionRunId: null }).where(eq(issues.id, f.issueId)); + const [before] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)); + const observed = createDb(tempDb!.connectionString), events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(observed, { heartbeatRuns }, capture); + try { + const response = await request(app(f.companyId, "queue-owner", undefined, observed)) + .get(`/api/issues/${f.issueId}/queued-comments`).expect(200); + expect(response.body.queueId).toBe(f.wakeId); + expect(response.body.targetRunId).toBe(f.runId); + expect(response.body.entries.map((entry: { comment: { id: string } }) => entry.comment.id)).toEqual(f.commentIds); + capture.flush(); + const reads = events.filter(event => event.table === "heartbeat_runs" && event.operation === "select"); + expect(reads.length).toBeGreaterThan(0); + expect(reads.reduce((bytes, event) => bytes + Number(event.responseValueBytes), 0)).toBeLessThan(2048); + await request(app(randomUUID(), "queue-owner", undefined, observed)) + .get(`/api/issues/${f.issueId}/queued-comments`).expect(404); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0]).toEqual(before); + } finally { capture.flush(); await observed.$client.end(); } + }); + + it.each(["deferred", "queued"])("reads only queue display inputs from large %s wake payloads", async state => { + const f = await seedQueue(); + if (state === "queued") await promoteQueue(f); + const client = app(f.companyId); + const path = `/api/issues/${f.issueId}/queued-comments`; + const expected = await request(client).get(path).expect(200); + const [wake] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, f.wakeId)); + const payload = "saved-wake-input🙂".repeat(30_000); + const context = wake.payload?._paperclipWakeContext as Record; + await db.update(agentWakeupRequests).set({ payload: { + ...wake.payload, prompt: payload, + _paperclipWakeContext: { ...context, prompt: payload, nativeExecutionInput: { instructions: payload } }, + } }).where(eq(agentWakeupRequests.id, f.wakeId)); + await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: f.issueId, prompt: payload }, + runnerProfileJson: { instructions: payload }, resultJson: { output: payload }, + }).where(eq(heartbeatRuns.companyId, f.companyId)); + const [before] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, f.wakeId)); + const observed = createDb(tempDb!.connectionString), events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(observed, { agentWakeupRequests, heartbeatRuns }, capture); + try { + const response = await request(app(f.companyId, "queue-owner", undefined, observed)).get(path).expect(200); + expect(response.body).toEqual(expected.body); + capture.flush(); + const reads = events.filter(event => ["agent_wakeup_requests", "heartbeat_runs"].includes(String(event.table)) && event.operation === "select"); + expect(reads.some(event => event.table === "agent_wakeup_requests")).toBe(true); + expect(reads.reduce((bytes, event) => bytes + Number(event.responseValueBytes), 0)).toBeLessThan(4096); + expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, f.wakeId)))[0]).toEqual(before); + await request(app(randomUUID(), "queue-owner", undefined, observed)).get(path).expect(404); + } finally { capture.flush(); await observed.$client.end(); } + }); + + it.each(["deferred", "queued"])("filters unrelated task wakes before reading the %s comment queue", async state => { + const f = await seedQueue(); + if (state === "queued") await promoteQueue(f); + const path = `/api/issues/${f.issueId}/queued-comments`; + const expected = await request(app(f.companyId)).get(path).expect(200); + const unrelated = Array.from({ length: 80 }, (_, index) => ({ + companyId: f.companyId, agentId: f.agentId, source: "issue_comment", + status: index % 2 ? "queued" : "deferred_issue_execution", + requestedAt: new Date("2020-01-01T00:00:00Z"), + payload: { issueId: randomUUID(), _paperclipWakeContext: { + wakeCommentIds: Array.from({ length: 60 }, () => randomUUID()), + } }, + })); + const malformed = [null, [f.issueId], { id: f.issueId }, f.issueId.toUpperCase()]; + await db.insert(agentWakeupRequests).values([ + ...unrelated, + ...malformed.map(issueId => ({ companyId: f.companyId, agentId: f.agentId, + source: "issue_comment", status: "deferred_issue_execution", + requestedAt: new Date("2020-01-01T00:00:00Z"), + payload: { issueId, _paperclipWakeContext: { wakeCommentIds: f.commentIds } }, + })), + ]); + const observed = createDb(tempDb!.connectionString), events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(observed, { agentWakeupRequests }, capture); + try { + const response = await request(app(f.companyId, "queue-owner", undefined, observed)).get(path).expect(200); + expect(response.body).toEqual(expected.body); + capture.flush(); + const reads = events.filter(event => event.table === "agent_wakeup_requests" && event.operation === "select"); + expect(reads.length).toBeGreaterThan(0); + expect(reads.reduce((bytes, event) => bytes + Number(event.responseValueBytes), 0)).toBeLessThan(4096); + expect(reads.reduce((rows, event) => rows + Number(event.rows), 0)).toBe(1); + expect(await db.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests) + .where(eq(agentWakeupRequests.companyId, f.companyId))).toHaveLength(85); + await request(app(randomUUID(), "queue-owner", undefined, observed)).get(path).expect(404); + } finally { capture.flush(); await observed.$client.end(); } + }); + it.each(["cancelled", "running"])("rejects a late Done from an interrupted %s task run at the write boundary", async status => { const seeded = await seedQueue(); await db.update(heartbeatRuns).set({ status, resultJson: { executionCancellation: { state: "requested" } } }) diff --git a/server/src/__tests__/issue-tree-control-routes.test.ts b/server/src/__tests__/issue-tree-control-routes.test.ts index 7931bb768ec..682650b7c5d 100644 --- a/server/src/__tests__/issue-tree-control-routes.test.ts +++ b/server/src/__tests__/issue-tree-control-routes.test.ts @@ -14,6 +14,7 @@ const mockTreeControlService = vi.hoisted(() => ({ cancelIssueStatusesForHold: vi.fn(), restoreIssueStatusesForHold: vi.fn(), getHold: vi.fn(), + getActivePauseHoldGate: vi.fn(), releaseHold: vi.fn(), cancelUnclaimedWakeupsForTree: vi.fn(), })); @@ -72,6 +73,7 @@ describe("issue tree control routes", () => { mockReplayBlocks.mockResolvedValue([]); mockExecutionBlocker.mockResolvedValue(null); mockTreeControlService.getHold.mockResolvedValue(null); + mockTreeControlService.getActivePauseHoldGate.mockResolvedValue(null); mockIssueService.getById.mockResolvedValue({ id: "11111111-1111-4111-8111-111111111111", companyId: "company-2", @@ -213,6 +215,36 @@ describe("issue tree control routes", () => { }, ); + it.each([ + { skipReason: "terminal_status", stillPaused: false }, + { skipReason: "already_held", stillPaused: false }, + { skipReason: "already_held", stillPaused: true }, + ])("rechecks skipped pause members with $skipReason and remaining pause=$stillPaused", async ({ skipReason, stillPaused }) => { + const rootId = "11111111-1111-4111-8111-111111111111"; + const holdId = "33333333-3333-4333-8333-333333333333"; + const childId = "44444444-4444-4444-8444-444444444444"; + const root = { id: rootId, companyId: "company-2", status: "todo", assigneeAgentId: null }; + const child = { id: childId, companyId: "company-2", status: "todo", assigneeAgentId: "current-agent" }; + mockIssueService.getById.mockImplementation(async (id: string) => id === rootId ? root : child); + const hold = { id: holdId, rootIssueId: rootId, mode: "pause", members: [ + { issueId: childId, skipped: true, skipReason, issueStatus: "done" }, + ] }; + mockTreeControlService.getHold.mockResolvedValue(hold); + mockTreeControlService.releaseHold.mockResolvedValue({ ...hold, status: "released" }); + mockTreeControlService.getActivePauseHoldGate.mockResolvedValue(stillPaused ? { holdId: "remaining-hold" } : null); + mockReplayBlocks.mockResolvedValue([{ id: childId, identifier: "TEST-2" }]); + const app = await createApp({ type: "board", userId: "user-1", companyIds: ["company-2"], source: "session" }); + const response = await request(app) + .post(`/api/issues/${rootId}/tree-holds/${holdId}/release`) + .send({ metadata: { wakeAgents: true } }); + expect(response.status).toBe(200); + expect(mockExecutionBlocker).toHaveBeenCalledWith(expect.anything(), "company-2", childId); + expect(mockHeartbeatService.wakeup).toHaveBeenCalledTimes(stillPaused ? 0 : 1); + if (!stillPaused) expect(mockHeartbeatService.wakeup).toHaveBeenCalledWith("current-agent", expect.objectContaining({ + payload: expect.objectContaining({ issueId: childId }), + })); + }); + it("resumes sandbox-backed work through the shared plugin worker manager", async () => { const rootId = "11111111-1111-4111-8111-111111111111"; const holdId = "33333333-3333-4333-8333-333333333333"; @@ -284,14 +316,14 @@ describe("issue tree control routes", () => { ).toEqual([rootId, "child"]); }); - it("keeps the hold active when resume requests an unsafe execution replay", async () => { + it.each([false, true])("keeps the hold active when resume requests an unsafe execution replay (previously skipped=%s)", async (skipped) => { const rootId = "11111111-1111-4111-8111-111111111111"; const holdId = "33333333-3333-4333-8333-333333333333"; mockTreeControlService.getHold.mockResolvedValue({ id: holdId, rootIssueId: rootId, mode: "pause", - members: [{ issueId: rootId }], + members: [{ issueId: rootId, skipped, skipReason: skipped ? "terminal_status" : null }], }); mockReplayBlocks.mockResolvedValue([{ id: rootId, identifier: "TEST-1" }]); mockExecutionBlocker.mockResolvedValue({ nextAction: "Resume without waking agents until the previous execution stops.", diff --git a/server/src/__tests__/issue-tree-control-service.test.ts b/server/src/__tests__/issue-tree-control-service.test.ts index 43c620053a5..923f399d3bd 100644 --- a/server/src/__tests__/issue-tree-control-service.test.ts +++ b/server/src/__tests__/issue-tree-control-service.test.ts @@ -6,6 +6,8 @@ import { agents, agentWakeupRequests, companies, + applyPendingMigrations, + closeRegisteredClients, createDb, heartbeatRuns, issueComments, @@ -20,8 +22,10 @@ import { } from "./helpers/embedded-postgres.js"; import { issueTreeControlService } from "../services/issue-tree-control.js"; import { issueService } from "../services/issues.js"; +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; -const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport(); +const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_ISSUE_TREE_TEST_DATABASE_URL"); +const embeddedPostgresSupport = externalUrl ? { supported: true } : await getEmbeddedPostgresTestSupport(); const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip; if (!embeddedPostgresSupport.supported) { @@ -35,6 +39,11 @@ describeEmbeddedPostgres("issueTreeControlService", () => { let tempDb: Awaited> | null = null; beforeAll(async () => { + if (externalUrl) { + await applyPendingMigrations(externalUrl); + db = createDb(externalUrl); + return; + } tempDb = await startEmbeddedPostgresTestDatabase("paperclip-issue-tree-control-"); db = createDb(tempDb.connectionString); }, 20_000); @@ -53,6 +62,7 @@ describeEmbeddedPostgres("issueTreeControlService", () => { }); afterAll(async () => { + if (externalUrl) await closeRegisteredClients(externalUrl); await tempDb?.cleanup(); }); @@ -623,6 +633,58 @@ describeEmbeddedPostgres("issueTreeControlService", () => { expect(checkedOutLegacyFullPauseRoot.checkoutRunId).toBe(rootRunId); }); + it("previews only executable current assignees as resume wake targets", async () => { + const companyId = randomUUID(); + const rootId = randomUUID(); + const readyAgentId = randomUUID(); + const parkedAgentId = randomUUID(); + const previousAgentId = randomUUID(); + const runId = randomUUID(); + await db.insert(companies).values({ + id: companyId, name: "Resume preview", issuePrefix: "RESUME", + }); + await db.insert(agents).values([readyAgentId, parkedAgentId, previousAgentId].map((id) => ({ + id, companyId, name: id, role: "engineer", status: "idle", adapterType: "process", + }))); + const children = ["todo", "in_progress", "in_review", "blocked", "backlog", "done"].map((status) => ({ + id: randomUUID(), companyId, parentId: rootId, title: status, status, + assigneeAgentId: status === "blocked" || status === "backlog" ? parkedAgentId : readyAgentId, + })); + await db.insert(issues).values([ + { id: rootId, companyId, title: "Unassigned root", status: "todo" }, + ...children, + ]); + const svc = issueTreeControlService(db); + const pause = await svc.createHold(companyId, rootId, { + mode: "pause", + actor: { actorType: "user", actorId: "board-user", userId: "board-user" }, + }); + expect(pause.hold.members?.find((member) => member.issueId === children[5].id)).toMatchObject({ + skipped: true, skipReason: "terminal_status", + }); + await db.update(issues).set({ status: "todo" }).where(eq(issues.id, children[5].id)); + // A previous executor is observable while stopping, but is not the current assignee to wake. + await db.insert(heartbeatRuns).values({ + id: runId, companyId, agentId: previousAgentId, invocationSource: "assignment", + status: "running", contextSnapshot: { issueId: children[0].id }, + }); + await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, children[0].id)); + + const preview = await svc.preview(companyId, rootId, { mode: "resume" }); + expect(preview.totals.affectedIssues).toBe(7); + expect(preview.affectedAgents).toEqual([{ agentId: readyAgentId, issueCount: 4, activeRunCount: 0 }]); + expect(preview.totals.affectedAgents).toBe(1); + expect(preview.warnings).toContainEqual({ + code: "resume_tasks_remain_idle", + message: "Blocked, backlog, and unassigned tasks will be unpaused but will not start.", + issueIds: [rootId, children[3].id, children[4].id].sort(), + }); + const rows = await db.select().from(issues).where(eq(issues.companyId, companyId)); + expect(rows.find((row) => row.id === children[3].id)?.status).toBe("blocked"); + expect(rows.find((row) => row.id === children[4].id)?.status).toBe("backlog"); + expect((await svc.getHold(companyId, pause.hold.id))?.status).toBe("active"); + }); + it("resumes subtree pauses by releasing matching pause holds", async () => { const companyId = randomUUID(); const rootIssueId = randomUUID(); diff --git a/server/src/__tests__/legacy-workspace-restore-recovery.test.ts b/server/src/__tests__/legacy-workspace-restore-recovery.test.ts index 493304a72fe..65e0e0cc147 100644 --- a/server/src/__tests__/legacy-workspace-restore-recovery.test.ts +++ b/server/src/__tests__/legacy-workspace-restore-recovery.test.ts @@ -1,6 +1,7 @@ -import { randomUUID } from "node:crypto"; +import { randomBytes, randomUUID } from "node:crypto"; +import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture"; import { and, eq, sql } from "drizzle-orm"; -import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; import { agents, companies, createDb, environmentLeases, environments, executionWorkspaces, heartbeatRuns, issueRecoveryActions, issues, projects } from "@paperclipai/db"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; import { recordLegacyWorkspaceRestoreFailure, legacyExecutionNeedsReconciliationWithEvidence, terminalizeLegacyExecution } from "../services/legacy-execution-recovery.js"; @@ -8,6 +9,7 @@ import { hasConversationContinuationPolicy, conversationRecoveryActionPredicate import { preserveWorkspaceRestoreRecoveryMetadataSql } from "../services/legacy-workspace-restore-recovery.js"; import { hasConfirmedSandboxStopAndRetain, prepareSandboxStopAndRetain, settleStopOnlyCleanup } from "../services/sandbox-stop-and-retain.js"; import { environmentService } from "../services/environments.js"; +import { deliverExecutionStatuses } from "../services/execution-status-delivery.js"; import { hasRequiredWorkspaceRecovery } from "../services/workspace-restore-recovery-state.js"; const externalTestDatabaseUrl = process.env.PAPERCLIP_TEST_DATABASE_URL?.trim(); @@ -45,6 +47,127 @@ const support = externalTestDatabaseUrl ? { supported: true } : await getEmbedde } }; const readLease = async (id: string) => (await db.select().from(environmentLeases).where(eq(environmentLeases.id, id)))[0]; + it("locks only terminal control state while returning and retaining the complete run", async () => { + const f = await seed(); + const bulk = randomBytes(128 * 1024).toString("base64"); + const [saved] = await db.update(heartbeatRuns).set({ contextSnapshot: { ...f.run.contextSnapshot, savedInput: bulk }, + runnerProfileJson: { savedProfile: bulk }, stdoutExcerpt: bulk, stderrExcerpt: bulk, + }).where(eq(heartbeatRuns.id, f.runId)).returning(); + const events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(db, { heartbeatRuns }, capture); + let returned: Awaited>; + try { returned = await terminalizeLegacyExecution({ db, run: saved!, status: "failed", patch }); } + finally { capture.flush(); } + const groups = events.filter(event => event.event === "paperclip.db_response_capture.entry.v1"); + const reads = groups.filter(group => group.table === "heartbeat_runs" && group.operation === "select"); + expect(reads.reduce((sum, group) => sum + Number(group.calls), 0)).toBe(1); + expect(reads.reduce((sum, group) => sum + Number(group.responseValueBytes), 0)).toBeLessThan(4096); + expect(returned).toMatchObject({ id: f.runId, status: "failed", contextSnapshot: saved!.contextSnapshot, + runnerProfileJson: saved!.runnerProfileJson, stdoutExcerpt: bulk, stderrExcerpt: bulk }); + expect(hasRequiredWorkspaceRecovery(returned!.resultJson)).toBe(true); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0]).toEqual(returned); + expect(await readLease(f.lease.id)).toMatchObject({ status: "pending_cleanup", leasePolicy: "retain_on_failure" }); + expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId))) + .toEqual([expect.objectContaining({ evidence: expect.objectContaining({ runId: f.runId }), ownerType: "board" })]); + expect(JSON.stringify(events)).not.toContain(bulk.slice(0, 64)); + }); + + it.each(["legacy", "native"])("returns one complete run when retained workspace evidence is unchanged for %s", async (runtimeMode) => { + const f = await seed(); + const bulk = randomBytes(128 * 1024).toString("base64"); + const [saved] = await db.update(heartbeatRuns).set({ runtimeMode, status: "failed", finishedAt: new Date(), + contextSnapshot: { ...f.run.contextSnapshot, savedInput: bulk }, runnerProfileJson: { savedProfile: bulk }, + stdoutExcerpt: bulk, stderrExcerpt: bulk, + }).where(eq(heartbeatRuns.id, f.runId)).returning(); + const recordRestoreFailureOnly = runtimeMode === "native"; + const failed = (await terminalizeLegacyExecution({ db, run: saved!, status: "failed", recordRestoreFailureOnly, + workspaceRestoreSource: f.lease, + patch: { resultJson: { workspaceRestoreFailure: "restore_failed", futureResult: bulk } }, + }))!; + const events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(db, { heartbeatRuns }, capture); + let returned: Awaited>; + try { returned = await terminalizeLegacyExecution({ db, run: failed, status: "failed", recordRestoreFailureOnly, + workspaceRestoreSource: f.lease, + ...(recordRestoreFailureOnly ? { patch: { resultJson: failed.resultJson } } : {}), + }); } + finally { capture.flush(); } + const writes = events.filter(event => event.event === "paperclip.db_response_capture.entry.v1" + && event.table === "heartbeat_runs" && event.operation === "update"); + expect(writes.reduce((sum, group) => sum + Number(group.rows), 0)).toBe(1); + expect(returned).toMatchObject({ id: f.runId, status: "failed", finishedAt: failed.finishedAt, + contextSnapshot: failed.contextSnapshot, runnerProfileJson: failed.runnerProfileJson, resultJson: failed.resultJson, + stdoutExcerpt: bulk, stderrExcerpt: bulk }); + if (!recordRestoreFailureOnly) expect(returned!.executionStatusDeliveryId).toBe(failed.executionStatusDeliveryId); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0]).toEqual(returned); + expect(await readLease(f.lease.id)).toMatchObject({ status: "pending_cleanup", leasePolicy: "retain_on_failure" }); + expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId))) + .toEqual([expect.objectContaining({ evidence: expect.objectContaining({ runId: f.runId }), ownerType: "board" })]); + expect(JSON.stringify(events)).not.toContain(bulk.slice(0, 64)); + }); + + it("preserves pending and acknowledged status delivery on unpatched legacy recovery retries", async () => { + await deliverExecutionStatuses(db, { publish: vi.fn() }); + const f = await seed(); + const failed = (await terminalizeLegacyExecution({ db, run: f.run, status: "failed", patch }))!; + const holds = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId)); + expect(holds).toHaveLength(1); + expect(hasRequiredWorkspaceRecovery(failed.resultJson)).toBe(true); + const repeated = (await terminalizeLegacyExecution({ db, run: failed, status: "failed" }))!; + expect(repeated.executionStatusDeliveryId).toBe(failed.executionStatusDeliveryId); + const publish = vi.fn(); + await deliverExecutionStatuses(db, { publish }); + await terminalizeLegacyExecution({ db, run: failed, status: "failed" }); + await deliverExecutionStatuses(db, { publish }); + expect(publish.mock.calls.filter(([event]) => event.payload.runId === f.runId)).toHaveLength(1); + const after = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId)); + expect(after).toHaveLength(1); + expect(after[0]).toMatchObject({ id: holds[0].id, status: holds[0].status, outcome: holds[0].outcome, + ownerType: holds[0].ownerType, wakePolicy: null, evidence: { ...holds[0].evidence, + automaticRecovery: expect.objectContaining({ policy: "preserve_without_replay_v1", replay: "blocked", + runId: f.runId, actionOutcome: "unknown" }), + } }); + expect(await readLease(f.lease.id)).toMatchObject({ status: "pending_cleanup", leasePolicy: "retain_on_failure" }); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0].resultJson).toEqual(failed.resultJson); + }); + + it("delivers new retained workspace evidence even when legacy run status is unchanged", async () => { + await deliverExecutionStatuses(db, { publish: vi.fn() }); + const f = await seed(); + const failed = (await terminalizeLegacyExecution({ db, run: f.run, status: "failed", patch }))!; + const publish = vi.fn(); + await deliverExecutionStatuses(db, { publish }); + const [added] = await db.insert(environmentLeases).values({ ...f.lease, id: randomUUID(), providerLeaseId: randomUUID() }).returning(); + const updated = (await terminalizeLegacyExecution({ db, run: failed, status: "failed" }))!; + expect(updated.executionStatusDeliveryId).toBeTruthy(); + expect(updated.executionStatusDeliveryId).not.toBe(failed.executionStatusDeliveryId); + expect(updated.resultJson?.workspaceRestoreRecovery).toMatchObject({ leaseIds: expect.arrayContaining([f.lease.id, added.id]) }); + await deliverExecutionStatuses(db, { publish }); + expect(publish.mock.calls.filter(([event]) => event.payload.runId === f.runId)).toHaveLength(2); + await terminalizeLegacyExecution({ db, run: updated, status: "failed" }); + await deliverExecutionStatuses(db, { publish }); + expect(publish.mock.calls.filter(([event]) => event.payload.runId === f.runId)).toHaveLength(2); + }); + + it("retains write conditions and delivers an explicit patch on unchanged legacy status", async () => { + await deliverExecutionStatuses(db, { publish: vi.fn() }); + const f = await seed(); + const failed = (await terminalizeLegacyExecution({ db, run: f.run, status: "failed", patch }))!; + const publish = vi.fn(); + await deliverExecutionStatuses(db, { publish }); + const holds = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId)); + expect(await terminalizeLegacyExecution({ db, run: failed, status: "failed", writeConditions: [sql`false`] })).toBeNull(); + await deliverExecutionStatuses(db, { publish }); + expect(publish.mock.calls.filter(([event]) => event.payload.runId === f.runId)).toHaveLength(1); + expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId))).toEqual(holds); + const patched = (await terminalizeLegacyExecution({ db, run: failed, status: "failed", patch: { error: "Retained workspace still needs repair" } }))!; + expect(patched.error).toBe("Retained workspace still needs repair"); + await deliverExecutionStatuses(db, { publish }); + expect(publish.mock.calls.filter(([event]) => event.payload.runId === f.runId)).toHaveLength(2); + }); + async function stopAndRetain(f: Awaited>, retry = false) { let stopping = (await prepareSandboxStopAndRetain(db, f.lease))!; expect(stopping.status).toBe("pending_cleanup"); diff --git a/server/src/__tests__/live-events-run-read.test.ts b/server/src/__tests__/live-events-run-read.test.ts index 76704f2f63c..b02b0d4cf45 100644 --- a/server/src/__tests__/live-events-run-read.test.ts +++ b/server/src/__tests__/live-events-run-read.test.ts @@ -58,9 +58,19 @@ describe("live-event run read boundaries", () => { }).returning({ id: heartbeatRuns.id }); const [maintenance] = await db.insert(heartbeatRuns).values({ companyId, agentId: agent.id, scopeKind: "company" }).returning({ id: heartbeatRuns.id }); const [foreign] = await db.insert(heartbeatRuns).values({ companyId: foreignCompanyId, agentId: foreignAgent.id, scopeKind: "company" }).returning({ id: heartbeatRuns.id }); + const hiddenIssueId = randomUUID(); + await db.insert(issues).values({ id: hiddenIssueId, companyId, title: "Other private task", visibility: "private", privacyRootIssueId: hiddenIssueId, responsibleUserId: owner }); + const [hiddenRun] = await db.insert(heartbeatRuns).values({ companyId, agentId: agent.id, scopeKind: "issue", issueId: hiddenIssueId }).returning({ id: heartbeatRuns.id }); const reads: string[] = []; + const queries: string[] = []; let failNextRunRead = false; + let failNextIssueRead = false; const observedDb = drizzle(db.$client, { schema, logger: { logQuery(query) { + queries.push(query); + if (query.startsWith("select ") && query.includes('from "issues"') && failNextIssueRead) { + failNextIssueRead = false; + throw new Error("fixture issue lookup unavailable"); + } if (query.startsWith("select ") && query.includes('from "heartbeat_runs"')) { reads.push(query); if (failNextRunRead) { failNextRunRead = false; throw new Error("fixture run lookup unavailable"); } @@ -78,17 +88,32 @@ describe("live-event run read boundaries", () => { socket.on("message", bytes => messages.push(JSON.parse(bytes.toString()))); try { await once(socket, "open"); - async function deliver(runId: string) { + let deliveries = 0; + async function deliver(runId: string, payloadIssueId?: string) { + deliveries += 1; const start = messages.length; const barrier = randomUUID(); - publishLiveEvent({ companyId, type: "heartbeat.run.status", payload: { runId, status: "running" } }); + publishLiveEvent({ companyId, type: "heartbeat.run.status", payload: { + runId, status: "running", ...(payloadIssueId ? { issueId: payloadIssueId } : {}), + } }); publishLiveEvent({ companyId, type: "agent.status", payload: { agentId: barrier, status: "idle" } }); await expect.poll(() => messages.some(message => message.payload.agentId === barrier)).toBe(true); return messages.slice(start).filter(message => message.type === "heartbeat.run.status"); } expect(await deliver(run.id)).toMatchObject([{ payload: { runId: run.id, status: "running" } }]); + const batchStart = queries.length; + expect(await deliver(run.id, issueId)).toMatchObject([{ payload: { runId: run.id, issueId, status: "running" } }]); + expect(queries.length - batchStart).toBeLessThanOrEqual(6); + expect(queries.slice(batchStart).filter(query => query.startsWith("select ") && query.includes('from "issues"')).length).toBeLessThanOrEqual(2); + expect(await deliver(hiddenRun.id, issueId)).toEqual([]); + expect(await deliver(run.id, hiddenIssueId)).toEqual([]); + expect(await deliver(run.id, randomUUID())).toEqual([]); + failNextIssueRead = true; + expect(await deliver(run.id, issueId)).toEqual([]); + expect(await deliver(run.id, issueId)).toHaveLength(1); await db.update(issueAccessGrants).set({ revokedAt: new Date() }).where(eq(issueAccessGrants.id, grant.id)); + expect(await deliver(run.id, issueId)).toEqual([]); expect(await deliver(run.id)).toEqual([]); expect(await deliver(maintenance.id)).toHaveLength(1); expect(await deliver(foreign.id)).toEqual([]); @@ -101,7 +126,7 @@ describe("live-event run read boundaries", () => { await db.delete(issues).where(eq(issues.id, issueId)); expect(await deliver(run.id)).toEqual([]); - expect(reads).toHaveLength(9); + expect(reads).toHaveLength(deliveries); for (const query of reads) { expect(query).toMatch(/^select "company_id", "scope_kind", "issue_id" from "heartbeat_runs"/); expect(query).not.toMatch(/context_snapshot|runner_profile_json|stdout_excerpt|result_json/); diff --git a/server/src/__tests__/live-events-toast.test.ts b/server/src/__tests__/live-events-toast.test.ts index 5c5fbe7bf6c..e82af433d73 100644 --- a/server/src/__tests__/live-events-toast.test.ts +++ b/server/src/__tests__/live-events-toast.test.ts @@ -14,7 +14,10 @@ vi.mock("../realtime/task-toast-eligibility.js", () => ({ taskToastEligible: vi. vi.mock("../services/authorization.js", () => ({ authorizationService: () => ({ decide: async ({ resource }: { resource: { issueId: string } }) => ({ allowed: resource.issueId !== "hidden" }) }), })); -vi.mock("../services/heartbeat-run-privacy.js", () => ({ canActorReadHeartbeatRun: async () => false })); +vi.mock("../services/heartbeat-run-privacy.js", () => ({ + getHeartbeatRunReadDecisions: async (_db: unknown, _access: unknown, _actor: unknown, runs: { issueId?: string }[]) => + runs.map(run => run.issueId !== "hidden"), +})); vi.mock("../services/case-access.js", () => ({ canActorReadCase: async (_db: unknown, _actor: unknown, record: { id: string }) => record.id !== "hidden-case", })); diff --git a/server/src/__tests__/native-harness-backup-expiration-config.test.ts b/server/src/__tests__/native-harness-backup-expiration-config.test.ts index 81967a9bdad..0b79c92e542 100644 --- a/server/src/__tests__/native-harness-backup-expiration-config.test.ts +++ b/server/src/__tests__/native-harness-backup-expiration-config.test.ts @@ -37,4 +37,15 @@ describe("native harness backup expiration config parsing", () => { vi.stubEnv("PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_DAYS", "0"); expect(loadConfig()).toMatchObject({ nativeHarnessBackupExpirationDays: 1 }); }); + + it("keeps native runner evidence for seven days by default and at least one day", () => { + vi.stubEnv("PAPERCLIP_NATIVE_RUNNER_EVIDENCE_RETENTION_DAYS", undefined); + expect(loadConfig()).toMatchObject({ nativeRunnerEvidenceRetentionDays: 7 }); + vi.stubEnv("PAPERCLIP_NATIVE_RUNNER_EVIDENCE_RETENTION_DAYS", "14"); + expect(loadConfig()).toMatchObject({ nativeRunnerEvidenceRetentionDays: 14 }); + vi.stubEnv("PAPERCLIP_NATIVE_RUNNER_EVIDENCE_RETENTION_DAYS", "0"); + expect(loadConfig()).toMatchObject({ nativeRunnerEvidenceRetentionDays: 1 }); + vi.stubEnv("PAPERCLIP_NATIVE_RUNNER_EVIDENCE_RETENTION_DAYS", "many"); + expect(loadConfig()).toMatchObject({ nativeRunnerEvidenceRetentionDays: 7 }); + }); }); diff --git a/server/src/__tests__/native-harness-backup-expiration.test.ts b/server/src/__tests__/native-harness-backup-expiration.test.ts index b6330409e72..0123d816a86 100644 --- a/server/src/__tests__/native-harness-backup-expiration.test.ts +++ b/server/src/__tests__/native-harness-backup-expiration.test.ts @@ -301,4 +301,378 @@ describeEmbeddedPostgres("native harness backup expiration", () => { expect(await exists(leftover)).toBe(false); expect(await exists(session.current)).toBe(true); }); + async function seedArchive(root: string, input: { stampedDaysAgo?: number; mtimeDaysAgo?: number; name?: string } = {}) { + const stampedAt = now.getTime() - (input.stampedDaysAgo ?? 10) * DAY_MS; + const archive = path.join(root, "continuity-breaks", input.name ?? `${stampedAt}-${randomUUID()}`); + await fs.mkdir(path.join(archive, "codex-home"), { recursive: true }); + await fs.writeFile(path.join(archive, "codex-home", "history.jsonl"), "x".repeat(64 * 1024)); + const mtime = new Date(now.getTime() - (input.mtimeDaysAgo ?? input.stampedDaysAgo ?? 10) * DAY_MS); + await fs.utimes(archive, mtime, mtime); + return archive; + } + + async function seedQuarantine( + scope: string, + input: { stampedDaysAgo?: number; name?: string } = {}, + ) { + if (!stateBase) stateBase = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-durable-sessions-")); + const stampedAt = now.getTime() - (input.stampedDaysAgo ?? 10) * DAY_MS; + const entry = path.join( + stateBase, + "quarantine", + input.name ?? `${scope}.identity_indeterminate.${stampedAt}.4242.${randomUUID()}`, + ); + await fs.mkdir(path.join(entry, "control-plane"), { recursive: true }); + await fs.writeFile(path.join(entry, "history.jsonl"), "x".repeat(64 * 1024)); + return entry; + } + + function evidenceService(overrides: Parameters[1] = {}) { + return nativeHarnessBackupExpirationService(db, { + stateBase, + now, + evidenceRetentionDays: 7, + isSessionBusy: () => false, + ...overrides, + }); + } + + async function emptyStateBase() { + if (!stateBase) stateBase = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-durable-sessions-")); + return createHash("sha256").update(randomUUID()).digest("hex"); + } + + it("removes continuity archives older than the retention window and keeps recent ones", async () => { + const session = await seedSession({ issueStatus: "in_progress" }); + const old = await seedArchive(session.root, { stampedDaysAgo: 9 }); + const recent = await seedArchive(session.root, { stampedDaysAgo: 2 }); + + const result = await evidenceService().tick(); + + expect(result).toMatchObject({ evidenceCandidates: 1, evidenceRemoved: 1, failed: 0 }); + expect(result.evidenceReclaimedBytes).toBeGreaterThan(0); + expect(result.skipped.recent_evidence).toBe(1); + expect(await exists(old)).toBe(false); + expect(await exists(recent)).toBe(true); + expect(await exists(session.current)).toBe(true); + }); + + it("removes the continuity-breaks folder once its archives are gone", async () => { + const session = await seedSession({ issueStatus: "in_progress" }); + await seedArchive(session.root); + + expect(await evidenceService().tick()).toMatchObject({ evidenceRemoved: 1 }); + expect(await exists(path.join(session.root, "continuity-breaks"))).toBe(false); + }); + + it("ages a continuity archive by its name and falls back to mtime for a future stamp", async () => { + const session = await seedSession({ issueStatus: "in_progress" }); + const stampedRecently = await seedArchive(session.root, { stampedDaysAgo: 1, mtimeDaysAgo: 30 }); + const futureStamp = await seedArchive(session.root, { + name: `${now.getTime() + DAY_MS}-${randomUUID()}`, + mtimeDaysAgo: 30, + }); + const futureRecent = await seedArchive(session.root, { + name: `${now.getTime() + 2 * DAY_MS}-${randomUUID()}`, + mtimeDaysAgo: 1, + }); + + expect(await evidenceService().tick()).toMatchObject({ evidenceRemoved: 1 }); + expect(await exists(stampedRecently)).toBe(true); + expect(await exists(futureStamp)).toBe(false); + expect(await exists(futureRecent)).toBe(true); + }); + + it("ignores folders that are not continuity archives", async () => { + const session = await seedSession({ issueStatus: "in_progress" }); + const other = await seedArchive(session.root, { name: "notes", stampedDaysAgo: 30 }); + + expect(await evidenceService().tick()).toMatchObject({ evidenceCandidates: 0 }); + expect(await exists(other)).toBe(true); + }); + + it("keeps the evidence of a session that is executing in this process", async () => { + const session = await seedSession({ issueStatus: "in_progress" }); + const archive = await seedArchive(session.root); + const quarantine = await seedQuarantine(session.scope); + + const result = await evidenceService({ isSessionBusy: (scope) => scope === session.scope }).tick(); + + expect(result).toMatchObject({ evidenceCandidates: 0, evidenceRemoved: 0 }); + expect(result.skipped.session_active).toBe(3); + expect(await exists(archive)).toBe(true); + expect(await exists(quarantine)).toBe(true); + }); + + it("keeps evidence when its session starts while the removal is being prepared", async () => { + const scope = await emptyStateBase(); + const root = path.join(stateBase, scope); + const archive = await seedArchive(root); + let checks = 0; + + const result = await evidenceService({ isSessionBusy: () => ++checks > 1 }).tick(); + + expect(result).toMatchObject({ evidenceCandidates: 1, evidenceRemoved: 0 }); + expect(result.skipped.session_active).toBe(1); + expect(await exists(archive)).toBe(true); + }); + + it("reports evidence without deleting it in dry run", async () => { + const session = await seedSession({ issueStatus: "in_progress" }); + const archive = await seedArchive(session.root); + const quarantine = await seedQuarantine(session.scope); + const leftover = path.join(session.root, "continuity-breaks", `.expired-${randomUUID()}`); + await fs.mkdir(leftover); + + const result = await evidenceService({ dryRun: true }).tick(); + + expect(result).toMatchObject({ evidenceCandidates: 2, evidenceRemoved: 0, evidenceReclaimedBytes: 0 }); + expect(result.evidenceCandidateBytes).toBeGreaterThan(0); + expect(await exists(archive)).toBe(true); + expect(await exists(quarantine)).toBe(true); + expect(await exists(leftover)).toBe(true); + }); + + it("skips symlinked evidence without following it", async () => { + const session = await seedSession({ issueStatus: "in_progress" }); + const outside = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-evidence-outside-")); + try { + await fs.writeFile(path.join(outside, "keep.txt"), "keep"); + const stampedAt = now.getTime() - 10 * DAY_MS; + await fs.mkdir(path.join(session.root, "continuity-breaks"), { recursive: true }); + const archiveLink = path.join(session.root, "continuity-breaks", `${stampedAt}-${randomUUID()}`); + await fs.symlink(outside, archiveLink); + await fs.mkdir(path.join(stateBase, "quarantine"), { recursive: true }); + const quarantineLink = path.join( + stateBase, + "quarantine", + `${session.scope}.identity_indeterminate.${stampedAt}.4242.${randomUUID()}`, + ); + await fs.symlink(outside, quarantineLink); + + const result = await evidenceService().tick(); + + expect(result).toMatchObject({ evidenceCandidates: 0, evidenceRemoved: 0 }); + expect(result.skipped.unsafe_path).toBe(2); + expect(await exists(archiveLink)).toBe(true); + expect(await exists(quarantineLink)).toBe(true); + expect(await exists(path.join(outside, "keep.txt"))).toBe(true); + } finally { + await fs.rm(outside, { recursive: true, force: true }); + } + }); + + it("skips symlinked evidence folders without following them", async () => { + const session = await seedSession({ issueStatus: "in_progress" }); + const outside = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-evidence-outside-")); + try { + const stampedAt = now.getTime() - 10 * DAY_MS; + const inside = path.join(outside, `${stampedAt}-${randomUUID()}`); + await fs.mkdir(inside); + await fs.utimes(inside, new Date(stampedAt), new Date(stampedAt)); + await fs.symlink(outside, path.join(session.root, "continuity-breaks")); + await fs.symlink(outside, path.join(stateBase, "quarantine")); + + const result = await evidenceService().tick(); + + expect(result).toMatchObject({ evidenceCandidates: 0, evidenceRemoved: 0 }); + expect(result.skipped.unsafe_path).toBe(2); + expect(await exists(inside)).toBe(true); + } finally { + await fs.rm(outside, { recursive: true, force: true }); + } + }); + + it("removes quarantine entries older than the retention window and keeps recent ones", async () => { + const scope = await emptyStateBase(); + const old = await seedQuarantine(scope, { stampedDaysAgo: 9 }); + const recent = await seedQuarantine(scope, { stampedDaysAgo: 2 }); + + const result = await evidenceService().tick(); + + expect(result).toMatchObject({ evidenceCandidates: 1, evidenceRemoved: 1, failed: 0 }); + expect(result.skipped.recent_evidence).toBe(1); + expect(await exists(old)).toBe(false); + expect(await exists(recent)).toBe(true); + expect(await fs.readdir(path.join(stateBase, "quarantine"))).toEqual([path.basename(recent)]); + }); + + it("keeps cleanup source archives in quarantine", async () => { + const scope = await emptyStateBase(); + const entry = await seedQuarantine(scope, { name: `${scope}.identity_indeterminate.cleanup.${randomUUID()}` }); + const old = new Date(now.getTime() - 30 * DAY_MS); + await fs.utimes(entry, old, old); + + const result = await evidenceService().tick(); + + expect(result.evidenceCandidates).toBe(0); + expect(result.skipped.cleanup_archive).toBe(1); + expect(await exists(entry)).toBe(true); + }); + + it("finishes removing evidence left over from an interrupted removal", async () => { + const session = await seedSession({ issueStatus: "in_progress" }); + const archive = await seedArchive(session.root, { stampedDaysAgo: 1 }); + const leftover = path.join(session.root, "continuity-breaks", `.expired-${randomUUID()}`); + await fs.mkdir(path.join(leftover, "codex-home"), { recursive: true }); + const quarantineLeftover = path.join(stateBase, "quarantine", `.expired-${randomUUID()}`); + await fs.mkdir(quarantineLeftover, { recursive: true }); + + await evidenceService().tick(); + + expect(await exists(leftover)).toBe(false); + expect(await exists(quarantineLeftover)).toBe(false); + expect(await exists(archive)).toBe(true); + }); + + it.each([false, true])("pages evidence removals across ticks (dry run %s)", async (dryRun) => { + const session = await seedSession({ issueStatus: "in_progress" }); + const archives = [await seedArchive(session.root), await seedArchive(session.root), await seedArchive(session.root)]; + const quarantine = await seedQuarantine(session.scope); + const retention = evidenceService({ dryRun, maxRemovals: 2 }); + + const results = []; + for (let i = 0; i < 4; i++) { + const result = await retention.tick(); + results.push(result); + if (!result.truncated) break; + } + + expect(results.map((result) => result.evidenceCandidates)).toEqual([2, 2]); + expect(results.map((result) => result.truncated)).toEqual([true, false]); + for (const archive of archives) expect(await exists(archive)).toBe(dryRun); + expect(await exists(quarantine)).toBe(dryRun); + }); + + it("does not recount a session's backup when its archives resume on the next tick", async () => { + const session = await seedSession(); + await seedArchive(session.root); + await seedArchive(session.root); + const retention = evidenceService({ dryRun: true, maxRemovals: 2 }); + + const first = await retention.tick(); + const second = await retention.tick(); + + expect(first).toMatchObject({ candidates: 1, evidenceCandidates: 1, truncated: true }); + expect(second).toMatchObject({ candidates: 0, evidenceCandidates: 1, truncated: false }); + }); + it("removes old quarantine entries even when their session's issue is still open", async () => { + const session = await seedSession({ issueStatus: "in_progress", backupAgeDays: 1 }); + const entry = await seedQuarantine(session.scope); + await fs.cp(path.join(session.root, "control-plane"), path.join(entry, "control-plane"), { recursive: true }); + + expect(await evidenceService().tick()).toMatchObject({ evidenceRemoved: 1 }); + expect(await exists(entry)).toBe(false); + }); + + it("stays quiet until retained evidence is due and wakes when new evidence is written", async () => { + const session = await seedSession({ issueStatus: "in_progress", backupAgeDays: 1 }); + const recent = await seedArchive(session.root, { stampedDaysAgo: 2 }); + const retention = evidenceService(); + const at = (days: number) => ({ now: new Date(now.getTime() + days * DAY_MS) }); + + expect((await retention.tick()).skipped.recent_evidence).toBe(1); + expect((await retention.tick(at(4))).skipped.recent_evidence).toBeUndefined(); + expect(await retention.tick(at(6))).toMatchObject({ evidenceRemoved: 1 }); + expect(await exists(recent)).toBe(false); + + const written = await seedArchive(session.root, { stampedDaysAgo: 0 }); + expect(await retention.tick(at(8))).toMatchObject({ evidenceCandidates: 0 }); + retention.noteEvidence(now.getTime()); + expect(await retention.tick(at(8))).toMatchObject({ evidenceRemoved: 1 }); + expect(await exists(written)).toBe(false); + }); + + it("retries evidence of a busy session on the next tick", async () => { + const session = await seedSession({ issueStatus: "in_progress", backupAgeDays: 1 }); + const archive = await seedArchive(session.root); + let busy = true; + const retention = evidenceService({ isSessionBusy: () => busy }); + + expect(await retention.tick()).toMatchObject({ evidenceRemoved: 0 }); + busy = false; + expect(await retention.tick()).toMatchObject({ evidenceRemoved: 1 }); + expect(await exists(archive)).toBe(false); + }); + + it("reports dry-run evidence once until new evidence is written", async () => { + const session = await seedSession({ issueStatus: "in_progress", backupAgeDays: 1 }); + await seedArchive(session.root); + const retention = evidenceService({ dryRun: true }); + + expect(await retention.tick()).toMatchObject({ evidenceCandidates: 1 }); + expect(await retention.tick()).toMatchObject({ evidenceCandidates: 0 }); + retention.noteEvidence(now.getTime()); + expect(await retention.tick({ now: new Date(now.getTime() + 7 * DAY_MS) })).toMatchObject({ evidenceCandidates: 1 }); + }); + + it("counts inspected archives and leftovers against the entry limit", async () => { + const session = await seedSession({ issueStatus: "in_progress", backupAgeDays: 1 }); + const archives = [ + await seedArchive(session.root, { stampedDaysAgo: 1 }), + await seedArchive(session.root, { stampedDaysAgo: 2 }), + await seedArchive(session.root, { stampedDaysAgo: 3 }), + ]; + const leftovers = [randomUUID(), randomUUID()].map((id) => path.join(session.root, "continuity-breaks", `.expired-${id}`)); + for (const leftover of leftovers) await fs.mkdir(leftover); + const retention = evidenceService({ maxEntries: 2 }); + + const first = await retention.tick(); + expect(first).toMatchObject({ checked: 2, truncated: true }); + expect(first.skipped.recent_evidence).toBeUndefined(); + expect((await fs.readdir(path.join(session.root, "continuity-breaks"))).filter((name) => name.startsWith(".expired-"))).toHaveLength(1); + + const results = [first]; + for (let i = 0; i < 6 && results.at(-1)!.truncated; i++) results.push(await retention.tick()); + expect(results.at(-1)!.truncated).toBe(false); + expect(results.every((result) => result.checked <= 2)).toBe(true); + expect(results.every((result) => (result.skipped.recent_evidence ?? 0) <= 2)).toBe(true); + expect(results.reduce((sum, result) => sum + (result.skipped.recent_evidence ?? 0), 0)).toBe(3); + for (const leftover of leftovers) expect(await exists(leftover)).toBe(false); + for (const archive of archives) expect(await exists(archive)).toBe(true); + }); + + it("stops removing leftovers once the sweep must stop", async () => { + const session = await seedSession({ issueStatus: "in_progress", backupAgeDays: 1 }); + await fs.mkdir(path.join(session.root, "continuity-breaks")); + const leftover = path.join(session.root, "continuity-breaks", `.expired-${randomUUID()}`); + await fs.mkdir(leftover); + let calls = 0; + + const retention = evidenceService({ shouldContinue: () => ++calls !== 2 }); + + await retention.tick(); + expect(await exists(leftover)).toBe(true); + await retention.tick(); + expect(await exists(leftover)).toBe(false); + }); + + it("keeps a due evidence pass pending when drain stops it before any session", async () => { + const session = await seedSession({ issueStatus: "in_progress", backupAgeDays: 1 }); + const archive = await seedArchive(session.root); + let draining = true; + const retention = evidenceService({ shouldContinue: () => !draining }); + + expect(await retention.tick()).toMatchObject({ checked: 0, truncated: true }); + draining = false; + expect(await retention.tick()).toMatchObject({ evidenceRemoved: 1 }); + expect(await exists(archive)).toBe(false); + }); + + it("finishes one-entry batches without recounting a resumed session", async () => { + const session = await seedSession({ issueStatus: "in_progress", backupAgeDays: 1 }); + const archives = [await seedArchive(session.root), await seedArchive(session.root)]; + const quarantine = await seedQuarantine(session.scope); + const retention = evidenceService({ maxEntries: 1 }); + + const results = []; + for (let i = 0; i < 8; i++) { + results.push(await retention.tick()); + if (!results.at(-1)!.truncated) break; + } + + expect(results.at(-1)!.truncated).toBe(false); + for (const archive of archives) expect(await exists(archive)).toBe(false); + expect(await exists(quarantine)).toBe(false); + }); }); diff --git a/server/src/__tests__/native-settled-process-release.test.ts b/server/src/__tests__/native-settled-process-release.test.ts index be5c00a4516..d2a60b89ccb 100644 --- a/server/src/__tests__/native-settled-process-release.test.ts +++ b/server/src/__tests__/native-settled-process-release.test.ts @@ -1,5 +1,6 @@ import { EMBEDDED_POSTGRES_TEST_TIMEOUT_MS } from "@paperclipai/db"; -import { randomUUID } from "node:crypto"; +import { randomBytes, randomUUID } from "node:crypto"; +import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture"; import { spawn } from "node:child_process"; import os from "node:os"; import { and, eq } from "drizzle-orm"; @@ -207,6 +208,40 @@ suite("settled native run process identity", () => { expect((await readRun(runId)).processPid).toBeNull(); }); + it("releases the stopped identity without reading saved execution payloads", async () => { + const { runId } = await seed(); + const bulk = randomBytes(128 * 1024).toString("base64"); + const before = await readRun(runId); + await db.update(heartbeatRuns).set({ contextSnapshot: { ...before.contextSnapshot, savedTask: bulk }, + runnerProfileJson: { ...before.runnerProfileJson, sessionCheckpoint: { providerSessionId: "thread", savedState: bulk }, + nativeExecutionInput: { task: bulk } }, resultJson: { savedResult: bulk }, stdoutExcerpt: bulk, + }).where(eq(heartbeatRuns.id, runId)); + const saved = await readRun(runId); + const events: Array> = []; + const now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(db, { heartbeatRuns }, capture); + try { expect(await releaseSettledNativeRunProcess(db, runId)).toBe("released"); } + finally { capture.flush(); } + const summary = events.find(event => event.event === "paperclip.db_response_capture.summary.v1")!; + expect(summary.responseValueBytes).toBeLessThan(8192); + const after = await readRun(runId); + expect(after).toEqual({ ...saved, processPid: null, processGroupId: null, processStartedAt: null, + nextEventSeq: saved.nextEventSeq + 1, updatedAt: after.updatedAt }); + expect(await hasNativeLocalProcessStop(db, companyId, runId)).toBe(true); + expect(JSON.stringify(events)).not.toContain(bulk.slice(0, 64)); + }); + + it.each([null, false, 0, "", {}, []].map(checkpoint => ({ checkpoint })))( + "preserves checkpoint presence semantics when releasing an identity ($checkpoint)", async ({ checkpoint }) => { + const { runId, pid } = await seed({ checkpoint: false }); + await db.update(heartbeatRuns).set({ runnerProfileJson: { sessionCheckpoint: checkpoint } }) + .where(eq(heartbeatRuns.id, runId)); + expect(await releaseSettledNativeRunProcess(db, runId)).toBe(checkpoint === null ? "skipped" : "released"); + expect((await readRun(runId)).processPid).toBe(checkpoint === null ? pid : null); + }, + ); + it("keeps the identity while the process is still alive, then releases it after exit", async () => { const child = startDetached(); children.push(child); diff --git a/server/src/__tests__/nested-clone-migration.test.ts b/server/src/__tests__/nested-clone-migration.test.ts index e77222c4b9e..b417a4ce897 100644 --- a/server/src/__tests__/nested-clone-migration.test.ts +++ b/server/src/__tests__/nested-clone-migration.test.ts @@ -14,7 +14,7 @@ import { realizeExecutionWorkspace } from "../services/workspace-runtime.js"; import { executionWorkspaceService } from "../services/execution-workspaces.js"; import { prepareProjectRepositoryWorkspaces } from "../services/heartbeat/workspaces.js"; import { expireReconstructableGitWorkspace } from "../services/git-workspace-expiration.js"; -import { currentRestorePointSnapshot } from "../services/git-restore-point.js"; +import { currentRestorePointSnapshot, restorePointHistoryDigest } from "../services/git-restore-point.js"; import { nestedRepositoryBaseCandidates, nestedWorktreeBranchPrefix, readLinkedNestedCheckout, withNestedRepositoryBaseLock } from "../services/nested-repository-worktrees.js"; import { listNestedCloneMigrationNames, migrateNestedRepositoryClone, nextNestedCloneMigrationBatch, type NestedCloneMigrationOptions } from "../services/nested-clone-migration.js"; import { getStorageService } from "../storage/index.js"; @@ -225,6 +225,41 @@ describeEmbeddedPostgres("nested repository clone migration", () => { expect(git(f.checkout, "cat-file", "-t", feature)).toBe("commit"); }); + it("expires a converted checkout whose only extra history is remote-tracking and restores that history", async () => { + const f = await fixture({ issueStatus: "done" }); + git(f.nested.author, "checkout", "-q", "-b", "squashed-away"); + await fs.writeFile(path.join(f.nested.author, "feature.txt"), "feature\n"); + git(f.nested.author, "add", "."); git(f.nested.author, "commit", "-qm", "Feature"); + const feature = git(f.nested.author, "rev-parse", "HEAD"); + git(f.nested.author, "push", "-q", f.nested.remote, "squashed-away"); + git(f.checkout, "fetch", "-q", "origin"); + git(f.nested.remote, "branch", "-D", "squashed-away"); + git(f.nested.remote, "reflog", "expire", "--expire=now", "--all"); + git(f.nested.remote, "gc", "-q", "--prune=now"); + expect(await f.migrate()).toMatchObject({ outcome: "migrated" }); + + expect(await f.expire()).toMatchObject({ expired: true, restorePoint: true, skipReason: null }); + for (const line of ownedRefs(f.base, f.prefix).split("\n").filter(Boolean)) git(f.base, "update-ref", "-d", line.split(" ")[0]!); + git(f.base, "reflog", "expire", "--expire=now", "--all"); + git(f.base, "gc", "-q", "--prune=now"); + expect(() => git(f.base, "cat-file", "-e", `${feature}^{commit}`)).toThrow(); + expect(await f.reopen()).toMatchObject({ ok: true, reopened: true }); + expect(git(f.checkout, "cat-file", "-t", feature)).toBe("commit"); + }); + + it("fingerprints a converted checkout's own history rather than the shared clone's remote refs", async () => { + const f = await fixture(); + expect(await f.migrate()).toMatchObject({ outcome: "migrated" }); + const head = git(f.checkout, "rev-parse", "HEAD"); + const before = await restorePointHistoryDigest(f.checkout); + git(f.base, "update-ref", "refs/remotes/origin/another-task", head); + expect(await restorePointHistoryDigest(f.checkout)).toBe(before); + git(f.base, "update-ref", `refs/heads/${f.prefix}remotes/origin/extra`, head); + expect(await restorePointHistoryDigest(f.checkout)).not.toBe(before); + for (let index = 0; index < 64; index++) git(f.base, "update-ref", `refs/heads/${f.prefix}remotes/origin/bulk-${index}`, head); + await expect(restorePointHistoryDigest(f.checkout)).rejects.toThrow("managed_checkout_local_tip_budget"); + }); + it("finishes an interrupted conversion after the workspace is archived and forgets one whose workspace is gone", async () => { const f = await fixture(); const { unpushed } = await nestedWork(f.checkout); diff --git a/server/src/__tests__/openapi-routes.test.ts b/server/src/__tests__/openapi-routes.test.ts index d0abe4e4b4b..ff43e560394 100644 --- a/server/src/__tests__/openapi-routes.test.ts +++ b/server/src/__tests__/openapi-routes.test.ts @@ -21,6 +21,7 @@ const apiPrefixes: Record = { "activity.ts": "/api", "adapters.ts": "/api", "agents.ts": "/api", + "agent-project-access.ts": "/api", "agent-commentary.ts": "/api", "agent-avatars.ts": "/api", "agent-profile-avatar.ts": "/api", @@ -987,6 +988,21 @@ describe("openapi routes", () => { }); }); + it("documents additive project access as board-only with a strict request", () => { + const { spec } = loadSpecRoutes(); + const path = spec.paths["/api/agents/{id}/project-access"]; + for (const operation of [path.get, path.post]) { + expect(operation["x-paperclip-authorization"]).toEqual({ actor: "board" }); + expect(operation.security).toEqual([{ BoardSessionAuth: [] }, { BoardApiKeyAuth: [] }]); + expect(operation.responses["200"].content["application/json"].schema.required).toEqual(["projectIds", "canManage"]); + } + expect(path.post.requestBody.content["application/json"].schema).toMatchObject({ + additionalProperties: false, required: ["projectIds"], + properties: { projectIds: { type: "array", items: { type: "string", format: "uuid" } } }, + }); + expect(path.post.responses["409"]).toBeDefined(); + }); + it("documents the authenticated personal primary-agent contract", () => { const { spec } = loadSpecRoutes(); const path = spec.paths["/api/companies/{companyId}/primary-agent/me"]; diff --git a/server/src/__tests__/run-identity.test.ts b/server/src/__tests__/run-identity.test.ts index d6dbbbfc3ff..069139a8123 100644 --- a/server/src/__tests__/run-identity.test.ts +++ b/server/src/__tests__/run-identity.test.ts @@ -29,6 +29,23 @@ const support = await getEmbeddedPostgresTestSupport(); } return { companyId, agentId, issueId, runId, messageIds }; } + it.each(["github_credentials", "request_authentication"] as const)("retains typed trust bindings for %s and keeps other identity reads complete", async (readScope) => { + const input = await seed(); + await initializeRunIdentity(db, { ...input, responsibleUserId: "owner", cause: "queued" }); + for (const value of [null, true, 42, " padded ", [], {}, { future: { opaque: true } }]) { + const contextSnapshot = { issueId: input.issueId, projectId: value, executionPolicy: value, savedInput: { retained: true } }; + const runnerProfileJson = { nativeExecutionInput: { opaque: value }, savedInput: "retained" }; + await db.update(heartbeatRuns).set({ contextSnapshot, runnerProfileJson }).where(eq(heartbeatRuns.id, input.runId)); + const narrow = await captureRunIdentity(db, input, { readScope }); + expect(narrow.run.contextSnapshot).toEqual({ issueId: input.issueId, taskId: null, projectId: value, executionPolicy: value }); + expect(narrow.run.runnerProfileJson).toBeNull(); + const full = await captureRunIdentity(db, input); + expect(full.run.contextSnapshot).toEqual(contextSnapshot); + expect(full.run.runnerProfileJson).toEqual(runnerProfileJson); + expect(narrow.context).toEqual(full.context); + } + }); + it("retains mixed-author delivery order without rewriting task ownership", async () => { const input = await seed(); await initializeRunIdentity(db, { ...input, responsibleUserId: "owner", cause: "queued" }); @@ -106,24 +123,25 @@ const support = await getEmbeddedPostgresTestSupport(); }, ); - it("holds acquisition during uncertain steering, preserves snapshots, and never rewinds on replay", async () => { + it.each([undefined, "github_credentials", "request_authentication"] as const)("holds acquisition during uncertain steering and never rewinds on replay with %s", async (readScope) => { + const options = readScope ? { readScope } : undefined; const input = await seed(); await initializeRunIdentity(db, { ...input, messageIds: [input.messageIds[0]], responsibleUserId: "A", cause: "instruction" }); - const startedUnderA = await captureRunIdentity(db, input); + const startedUnderA = await captureRunIdentity(db, input, options); const b = await reserveSteeredIdentity(db, { ...input, messageId: input.messageIds[1] }); expect(b).not.toBeNull(); - await expect(captureRunIdentity(db, input)).rejects.toThrow(/reconciled/); + await expect(captureRunIdentity(db, input, options)).rejects.toThrow(/reconciled/); await expect(reserveSteeredIdentity(db, { ...input, messageId: input.messageIds[2] })).rejects.toThrow(/reconciled/); await db.transaction(async tx => { await tx.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, input.runId)).for("update"); await acceptSteeredIdentity(tx, b!); }); - expect((await captureRunIdentity(db, input)).context?.responsibleUserId).toBe("B"); + expect((await captureRunIdentity(db, input, options)).context?.responsibleUserId).toBe("B"); expect(startedUnderA.context?.responsibleUserId).toBe("A"); const a = await reserveSteeredIdentity(db, { ...input, messageId: input.messageIds[2] }); await acceptSteeredIdentity(db, a!); await acceptSteeredIdentity(db, b!); - expect((await captureRunIdentity(db, input)).context?.responsibleUserId).toBe("A"); + expect((await captureRunIdentity(db, input, options)).context?.responsibleUserId).toBe("A"); }); it("keeps rejected steering unchanged and preserves the originating identity on retry", async () => { const input = await seed(); @@ -158,16 +176,17 @@ const support = await getEmbeddedPostgresTestSupport(); expect(captured.run.responsibleUserId).toBeNull(); expect(captured.context?.parentContextId).toBe(origin.id); }); - it("recovers an uncertain acknowledgement from the authenticated native event journal", async () => { + it.each([undefined, "github_credentials", "request_authentication"] as const)("recovers an uncertain native acknowledgement with %s", async (readScope) => { + const options = readScope ? { readScope } : undefined; const input = await seed(); await initializeRunIdentity(db, { ...input, messageIds: [], responsibleUserId: "A", cause: "instruction" }); const pending = await reserveSteeredIdentity(db, { ...input, messageId: input.messageIds[1] }); - await expect(captureRunIdentity(db, input)).rejects.toThrow(/reconciled/); + await expect(captureRunIdentity(db, input, options)).rejects.toThrow(/reconciled/); await db.insert(heartbeatRunEvents).values({ companyId: input.companyId, runId: input.runId, agentId: input.agentId, seq: 1, eventType: "item.completed", sourceEventId: randomUUID(), payload: { prpEvent: { turnId: "turn-1", itemId: `turn-1:steer:${pending!.messageId}`, payload: {kind: "steering_acknowledgement"} } }, }); - expect((await captureRunIdentity(db, input)).context?.responsibleUserId).toBe("B"); + expect((await captureRunIdentity(db, input, options)).context?.responsibleUserId).toBe("B"); await db.update(heartbeatRuns).set({status: "succeeded"}).where(eq(heartbeatRuns.id, input.runId)); expect((await reserveSteeredIdentity(db, { ...input, messageId: input.messageIds[1] }))?.status).toBe("accepted"); }); diff --git a/server/src/__tests__/sandbox-provider-schema-reads.test.ts b/server/src/__tests__/sandbox-provider-schema-reads.test.ts new file mode 100644 index 00000000000..55f81f8bfbe --- /dev/null +++ b/server/src/__tests__/sandbox-provider-schema-reads.test.ts @@ -0,0 +1,173 @@ +import { randomBytes, randomUUID } from "node:crypto"; +import { eq, sql, type SQL } from "drizzle-orm"; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it } from "vitest"; +import { createDb, plugins, type Db } from "@paperclipai/db"; +import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture"; +import type { PaperclipPluginManifestV1, PluginEnvironmentDriverDeclaration } from "@paperclipai/shared"; +import { resolvePluginSandboxProviderConfigSchema, resolvePluginSandboxProviderDriverByKey } from "../services/plugin-environment-driver.js"; +import { resolveEnvironmentDriverConfigForRuntime, resolveSandboxProviderSecretRefPaths } from "../services/environment-config.js"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; + +const support = await getEmbeddedPostgresTestSupport(); + +describe.skipIf(!support.supported)("sandbox provider schema reads", () => { + let temporary: Awaited>; + let db: Db; + const clients: Db[] = []; + const key = "fixture-schema-provider"; + const driver = (driverKey = key, configSchema: unknown = { type: "object" }) => ({ + driverKey, kind: "sandbox_provider", displayName: "Schema fixture", configSchema, + }) as PluginEnvironmentDriverDeclaration; + const manifest = (id: string, declarations: PluginEnvironmentDriverDeclaration[]): PaperclipPluginManifestV1 => ({ + id: `test.schema.${id}`, apiVersion: 1, version: "1.0.0", displayName: "Schema fixture", + description: "Fixture", author: "Test", categories: ["automation"], + capabilities: ["environment.drivers.register"], entrypoints: { worker: "worker.js" }, environmentDrivers: declarations, + }); + async function seed(declarations = [driver()], installOrder = 0) { + const id = randomUUID(); + await db.insert(plugins).values({ + id, pluginKey: `test.schema.${id}`, packageName: "@test/schema", version: "1.0.0", + installOrder, status: "disabled", manifestJson: manifest(id, declarations), + }); + return id; + } + function observer() { + const observed = createDb(temporary.connectionString); + clients.push(observed); + return observed; + } + const lookup = (client: Db, driverKey = key) => resolvePluginSandboxProviderConfigSchema({ db: client, driverKey }); + async function measure(client: Db, operation: () => Promise) { + const events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event), () => now); + installDatabaseResponseCapture(client, { plugins }, capture); + try { + const value = await operation(); + capture.flush(); + const entries = events.filter(event => event.event === "paperclip.db_response_capture.entry.v1"); + return { value, bytes: entries.reduce((sum, entry) => sum + Number(entry.responseValueBytes), 0), + calls: entries.reduce((sum, entry) => sum + Number(entry.calls), 0) }; + } finally { capture.flush(); } + } + + beforeAll(async () => { + temporary = await startEmbeddedPostgresTestDatabase("sandbox-schema-reads-"); + db = createDb(temporary.connectionString); + }, 120_000); + beforeEach(async () => { await db.delete(plugins); }); + afterEach(async () => { await Promise.all(clients.splice(0).map(client => client.$client.end())); }); + afterAll(async () => { await temporary?.cleanup(); }); + + it("keeps runtime secret paths current without retransferring an unchanged schema", async () => { + const schema = { type: "object", description: randomBytes(16 * 1024).toString("hex"), properties: { + credentials: { type: "object", properties: { token: { type: "string", format: "secret-ref" } } }, + } }; + const id = await seed([driver(key, schema)]), observed = observer(); + const before = await db.select().from(plugins).where(eq(plugins.id, id)); + const cold = await measure(observed, () => resolveSandboxProviderSecretRefPaths(observed, key)); + const warm = await measure(observed, () => resolveSandboxProviderSecretRefPaths(observed, key)); + expect(cold.value).toEqual(new Set(["credentials.token"])); + expect(warm.value).toEqual(cold.value); + expect(cold.calls).toBe(1); + expect(warm.calls).toBe(1); + expect(cold.bytes).toBeGreaterThan(32 * 1024); + expect(warm.bytes).toBeLessThan(256); + const changed = { type: "object", allOf: [{ properties: { apiKey: { type: "string", format: "secret-ref" } } }] }; + await db.update(plugins).set({ manifestJson: manifest(id, [driver(key, changed)]) }).where(eq(plugins.id, id)); + expect(await resolveSandboxProviderSecretRefPaths(observed, key)).toEqual(new Set(["apiKey"])); + const environment = { id: randomUUID(), driver: "sandbox" as const, config: { provider: key, apiKey: randomUUID() } }; + await expect(resolveEnvironmentDriverConfigForRuntime(observed, null, environment)).rejects.toThrow("companyId"); + await db.update(plugins).set({ manifestJson: before[0].manifestJson }).where(eq(plugins.id, id)); + expect(await db.select().from(plugins).where(eq(plugins.id, id))).toEqual(before); + }); + + it("preserves install order, first matching declarations, and default readiness checks", async () => { + const a = await seed([driver(key, { title: "first" }), driver(key, { title: "second" })], 1); + const b = await seed([driver(key, { title: "other plugin" })], 2), observed = observer(); + expect((await lookup(observed))).toEqual({ title: "first" }); + const full = await resolvePluginSandboxProviderDriverByKey({ db: observed, driverKey: key }); + expect(full?.plugin.id).toBe(a); + expect(full?.plugin.status).toBe("disabled"); + expect(await resolvePluginSandboxProviderDriverByKey({ db: observed, driverKey: key, requireRunning: true })).toBeNull(); + await db.update(plugins).set({ installOrder: 0 }).where(eq(plugins.id, b)); + expect((await lookup(observed))).toEqual({ title: "other plugin" }); + await db.delete(plugins).where(eq(plugins.id, b)); + expect((await lookup(observed))).toEqual({ title: "first" }); + await db.update(plugins).set({ manifestJson: manifest(a, [driver("replacement")]) }).where(eq(plugins.id, a)); + expect(await lookup(observed)).toBeNull(); + }); + + it.each([null, true, "invalid", [], undefined])("preserves non-object schema behavior: %s", async value => { + const first = driver(key, value); + if (value === undefined) delete (first as Partial).configSchema; + await seed([first, driver(key, { properties: { secret: { format: "secret-ref" } } })]); + const observed = observer(); + expect(await resolveSandboxProviderSecretRefPaths(observed, key)).toEqual(new Set()); + expect(await resolveSandboxProviderSecretRefPaths(observed, key)).toEqual(new Set()); + }); + + it("returns independent schema objects and keeps separate database clients cold", async () => { + await seed([driver(key, { description: "x".repeat(4096), properties: { token: { format: "secret-ref" } } })]); + const a = observer(), b = observer(); + const first = await lookup(a); + first!.properties = {}; + expect(await resolveSandboxProviderSecretRefPaths(a, key)).toEqual(new Set(["token"])); + const warm = await measure(a, () => lookup(a)), independent = await measure(b, () => lookup(b)); + expect(warm.bytes).toBeLessThan(256); + expect(independent.bytes).toBeGreaterThan(4096); + warm.value!.properties = {}; + expect(await resolveSandboxProviderSecretRefPaths(a, key)).toEqual(new Set(["token"])); + }); + + it("bounds retained schemas and does not retain oversized definitions", async () => { + const declarations = Array.from({ length: 33 }, (_, index) => driver(`provider-${index}`, { description: "x".repeat(2048) })); + declarations.push(driver("oversized", { description: "x".repeat(128 * 1024) })); + await seed(declarations); + const observed = observer(); + for (let index = 0; index < 33; index++) await lookup(observed, `provider-${index}`); + expect((await measure(observed, () => lookup(observed, "provider-32"))).bytes).toBeLessThan(256); + expect((await measure(observed, () => lookup(observed, "provider-0"))).bytes).toBeGreaterThan(2048); + await lookup(observed, "oversized"); + expect((await measure(observed, () => lookup(observed, "oversized"))).bytes).toBeGreaterThan(128 * 1024); + }); + + it("binds unusual keys and never serves retained schemas after a lookup error", async () => { + const unusual = "provider-\"quoted' OR true --"; + await seed([driver(unusual, { title: "exact provider" })]); + const observed = observer(); + let fail = false; + const guarded = { execute: async (query: SQL) => { + if (fail) throw new Error("fixture database unavailable"); + return observed.execute(query); + } } as unknown as Db; + expect((await lookup(guarded, unusual))).toEqual({ title: "exact provider" }); + fail = true; + await expect(lookup(guarded, unusual)).rejects.toThrow("fixture database unavailable"); + fail = false; + expect((await lookup(guarded, unusual))).toEqual({ title: "exact provider" }); + expect(await lookup(guarded, "unknown-provider")).toBeNull(); + }); + + it("keeps each concurrent lookup tied to the database revision it observed", async () => { + const id = await seed([driver(key, { title: "old" })]), observed = observer(); + let pause = false, release!: () => void, reached!: () => void; + const gate = new Promise(resolve => { release = resolve; }); + const paused = new Promise(resolve => { reached = resolve; }); + const raced = { execute: async (query: SQL) => { + const rows = await observed.execute(query); + if (pause) { pause = false; reached(); await gate; } + return rows; + } } as unknown as Db; + expect((await lookup(raced))).toEqual({ title: "old" }); + pause = true; + const oldRead = lookup(raced); + await paused; + try { + await db.update(plugins).set({ manifestJson: manifest(id, [driver(key, { title: "new" })]) }).where(eq(plugins.id, id)); + expect((await lookup(raced))).toEqual({ title: "new" }); + } finally { release(); } + expect((await oldRead)).toEqual({ title: "old" }); + expect((await lookup(raced))).toEqual({ title: "new" }); + expect((await db.execute<{ title: string }>(sql`select ${plugins.manifestJson}->'environmentDrivers'->0->'configSchema'->>'title' as title from ${plugins} where ${plugins.id}=${id}`))[0].title).toBe("new"); + }); +}); diff --git a/server/src/__tests__/task-cold-open.test.ts b/server/src/__tests__/task-cold-open.test.ts new file mode 100644 index 00000000000..609aae1ce42 --- /dev/null +++ b/server/src/__tests__/task-cold-open.test.ts @@ -0,0 +1,125 @@ +import { randomUUID } from "node:crypto"; +import express from "express"; +import request from "supertest"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { activityLog, agents, companies, createDb, heartbeatRunEvents, heartbeatRuns, issueComments, issues } from "@paperclipai/db"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { issueRoutes } from "../routes/issues.js"; +import { errorHandler } from "../middleware/index.js"; +import { createHeartbeatRunState } from "../services/heartbeat/run-state.js"; +import { activityService } from "../services/activity.js"; + +const support = await getEmbeddedPostgresTestSupport(); +(support.supported ? describe : describe.skip)("task cold-open reads", () => { + let fixture: Awaited>; + let db: ReturnType; + const companyId = randomUUID(); + const issueId = randomUUID(); + const agentId = randomUUID(); + const runId = randomUUID(); + beforeAll(async () => { + fixture = await startEmbeddedPostgresTestDatabase("task-cold-open-"); + db = createDb(fixture.connectionString); + await db.insert(companies).values({ id: companyId, name: "Cold open", issuePrefix: `T${companyId.slice(0, 6)}` }); + await db.insert(agents).values({ id: agentId, companyId, name: "Writer", role: "engineer", adapterType: "codex_local" }); + await db.insert(issues).values({ id: issueId, companyId, title: "Original instruction", description: "Read this first", identifier: `T${companyId.slice(0, 6)}-1`, issueNumber: 1 }); + await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, issueId, runtimeMode: "native", status: "succeeded" }); + }, 30_000); + afterAll(async () => { await fixture?.cleanup(); }); + function app(allowedCompany = companyId) { + const app = express(); + app.use((req, _res, next) => { + (req as any).actor = { type: "board", userId: "local-board", source: allowedCompany === companyId ? "local_implicit" : "cloud_tenant", companyIds: [allowedCompany], isInstanceAdmin: allowedCompany === companyId }; + next(); + }); + app.use("/api", issueRoutes(db, {} as any)); + app.use(errorHandler); + return app; + } + it("returns the core instruction without related detail and keeps company checks", async () => { + const response = await request(app()).get(`/api/issues/${issueId}?view=core`); + expect(response.status, JSON.stringify(response.body)).toBe(200); + expect(response.body).toMatchObject({ id: issueId, description: "Read this first" }); + expect(response.body).not.toHaveProperty("ancestors"); + expect(response.body).not.toHaveProperty("workProducts"); + const denied = await request(app(randomUUID())).get(`/api/issues/${issueId}?view=core`); + expect([403, 404]).toContain(denied.status); + }); + it("returns a bounded recent exchange plus the latest human instruction", async () => { + const humanId = randomUUID(); + await db.insert(issueComments).values([ + { id: humanId, companyId, issueId, body: "Human direction", authorType: "user", authorUserId: "local-board", createdAt: new Date(0) }, + ...Array.from({ length: 20 }, (_, i) => ({ companyId, issueId, authorType: "agent" as const, authorAgentId: agentId, body: `Reply ${i}`, createdAt: new Date(1000 + i) })), + ]); + const response = await request(app()).get(`/api/issues/${issueId}/recent-comments`); + expect(response.status, JSON.stringify(response.body)).toBe(200); + expect(response.body.hasMore).toBe(true); + expect(response.body.comments.map((c: { body: string }) => c.body)).toEqual(["Human direction", "Reply 19", "Reply 18", "Reply 17", "Reply 16"]); + expect(response.body.latestCommentId).toBe(response.body.comments[1].id); + expect([403, 404]).toContain((await request(app(randomUUID())).get(`/api/issues/${issueId}/recent-comments`)).status); + }); + it.each(["user", null] as const)("preserves initial and latest human instructions without a description (authorType=%s)", async (authorType) => { + const bareIssueId = randomUUID(); + await db.insert(issues).values({ id: bareIssueId, companyId, title: "No description" }); + await db.insert(issueComments).values([ + { companyId, issueId: bareIssueId, body: "Human direction", authorType, authorUserId: "local-board", createdAt: new Date(0) }, + { companyId, issueId: bareIssueId, body: "Newest direction", authorType, authorUserId: "local-board", createdAt: new Date(50_000) }, + ...Array.from({ length: 6 }, (_, i) => ({ companyId, issueId: bareIssueId, body: `New reply ${i}`, authorType: "agent" as const, authorAgentId: agentId, createdAt: new Date(60_000 + i) })), + ]); + const response = await request(app()).get(`/api/issues/${bareIssueId}/recent-comments`); + expect(response.status).toBe(200); + expect(response.body.comments.map((c: { body: string }) => c.body)).toEqual([ + "Human direction", "Newest direction", "New reply 5", "New reply 4", "New reply 3", "New reply 2", + ]); + }); + it("loads the original input receipts outside the recent activity window and ignores replay receipts", async () => { + const commentId = randomUUID(); + const receiptIssueId = randomUUID(); + await db.insert(issues).values({ id: receiptIssueId, companyId, title: "Steering receipts" }); + await db.insert(activityLog).values([ + { companyId, actorId: "test", entityType: "issue", entityId: receiptIssueId, action: "issue.queued_comment_steered", createdAt: new Date(1), details: { commentId, targetRunId: runId } }, + { companyId, actorId: "test", entityType: "issue", entityId: receiptIssueId, action: "issue.queued_comment_steered", createdAt: new Date(2), details: { commentId, targetRunId: runId, duplicate: true } }, + ...Array.from({ length: 51 }, () => ({ companyId, actorId: "test", entityType: "issue", entityId: receiptIssueId, action: "issue.updated" })), + ]); + const activity = activityService(db); + expect((await activity.forIssue(receiptIssueId, 50)).some(row => row.action === "issue.queued_comment_steered")).toBe(false); + const facts = await activity.inputFactsForIssue(companyId, receiptIssueId, [commentId]); + expect(facts).toHaveLength(1); + expect(facts[0].createdAt).toEqual(new Date(1)); + expect(await activity.inputFactsForIssue(companyId, receiptIssueId, [randomUUID()])).toEqual([]); + expect(await activity.inputFactsForIssue(randomUUID(), receiptIssueId, [commentId])).toEqual([]); + expect(await activity.inputFactsForIssue(companyId, randomUUID(), [commentId])).toEqual([]); + }); + + it("pages conversation events without tools, including identity-less deltas and requests", async () => { + const event = (seq: number, eventType: string, payload: Record, itemId?: string) => ({ + companyId, runId, agentId, seq, eventType, payload: { prpEvent: { schema: "paperclip.prp.event.v1", eventType, itemId, payload } }, + }); + await db.insert(heartbeatRunEvents).values([ + event(1, "item.started", { kind: "toolCall", arguments: "large tool arguments" }, "tool"), + event(2, "item.completed", { kind: "toolCall", output: "x".repeat(100_000) }, "tool"), + event(3, "item.started", { kind: "agentMessage" }, "message"), + event(4, "item.delta", { text: "Current progress" }, "message"), + event(5, "runtime_request.opened", { requestId: "request", prompt: "Which option?" }), + event(6, "runtime_request.resolved", { requestId: "request" }), + event(7, "item.completed", { kind: "assistant_message", text: "Final answer" }), + event(8, "turn.completed", {}), + ]); + const service = createHeartbeatRunState(db); + const first = await service.listEvents(runId, 0, 2, true); + expect(first.map(e => e.seq)).toEqual([3, 4]); + const rest = await service.listEvents(runId, 4, 1000, true); + expect(rest.map(e => e.seq)).toEqual([5, 6, 7, 8]); + expect(JSON.stringify([...first, ...rest])).not.toContain("large tool arguments"); + expect(await service.listEvents(runId)).toHaveLength(8); + await db.insert(heartbeatRuns).values({ companyId, agentId, issueId, status: "succeeded", createdAt: new Date(Date.now() + 1000) }); + const activity = activityService(db); + await db.insert(activityLog).values(Array.from({ length: 55 }, () => ({ + companyId, actorId: "test", action: "issue.updated", entityType: "issue", entityId: issueId, + }))); + expect(await activity.forIssue(issueId, 50)).toHaveLength(50); + expect(await activity.forIssue(issueId)).toHaveLength(55); + expect(await activity.runsForIssue(companyId, issueId, null, 1)).toHaveLength(1); + expect(await activity.runsForIssue(companyId, issueId)).toHaveLength(2); + }); +}); diff --git a/server/src/__tests__/tool-effective-assignments.test.ts b/server/src/__tests__/tool-effective-assignments.test.ts index 2f69d9e62d0..4b99a23eec2 100644 --- a/server/src/__tests__/tool-effective-assignments.test.ts +++ b/server/src/__tests__/tool-effective-assignments.test.ts @@ -1,10 +1,11 @@ -import { randomUUID } from "node:crypto"; +import { randomBytes, randomUUID } from "node:crypto"; import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest"; import { eq, sql } from "drizzle-orm"; import { agents, companies, createDb, toolApplications, toolCatalogEntries, toolConnections, toolProfileBindings, toolProfileEntries, toolProfiles, type Db, } from "@paperclipai/db"; +import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture"; import { toolAccessService } from "../services/tool-access.js"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; @@ -49,6 +50,11 @@ describeDatabase("effective tool assignments", () => { expect(assignments).toEqual({ ...full, allowedTools: full.allowedTools.map( ({ id, applicationId, connectionId, toolName }) => ({ id, applicationId, connectionId, toolName }), ) }); + const connections = await service.getConnectionAssignmentsForAgent(f.company.id, f.agent.id, [f.connection.id]); + expect(connections).toEqual({ + installedConnections: full.installedConnections, + allowedConnectionIds: [...new Set(full.allowedTools.filter(tool => tool.connectionId === f.connection.id).map(tool => tool.connectionId))], + }); return { service, full, assignments }; } @@ -130,4 +136,161 @@ describeDatabase("effective tool assignments", () => { expect(observed[1].bytes).toBeLessThan(observed[0].bytes / 100); expect(full.allowedTools[0].inputSchema).toEqual(f.read.inputSchema); }); + it("scopes catalog reads, omits rule payloads, and keeps current full definitions", async () => { + const f = await fixture(); + const bulk = randomBytes(128 * 1024).toString("base64"); + const [rule] = await db.insert(toolProfileEntries).values({ companyId: f.company.id, profileId: f.profile.id, + selectorType: "application", effect: "include", applicationId: f.application.id, conditions: { savedRuleMetadata: bulk } }).returning(); + const [unrelated] = await db.insert(toolConnections).values({ companyId: f.company.id, applicationId: f.application.id, + name: "Other connection", uid: randomUUID(), transport: "mcp_remote", status: "active", enabled: true }).returning(); + await db.insert(toolCatalogEntries).values(Array.from({ length: 100 }, (_, index) => ({ companyId: f.company.id, + applicationId: f.application.id, connectionId: unrelated.id, name: `other_${index}`, toolName: `other_${index}`, versionHash: `v${index}` }))); + const service = toolAccessService(db); + const before = await service.getEffectiveToolAssignmentsForAgent(f.company.id, f.agent.id); + expect(before.allowedTools).toHaveLength(102); + const events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(db, { toolCatalogEntries, toolProfileEntries }, capture); + let scoped: Awaited>; + try { scoped = await service.getConnectionAssignmentsForAgent(f.company.id, f.agent.id, [f.connection.id]); } + finally { capture.flush(); } + expect(Number(events.at(-1)?.completedQueries)).toBeGreaterThan(0); + expect(Number(events.at(-1)?.responseValueBytes)).toBeLessThan(16 * 1024); + const catalogQueries = events.filter(event => event.event === "paperclip.db_response_capture.entry.v1" + && event.table === "tool_catalog_entries" && event.operation === "select"); + expect(catalogQueries).toHaveLength(1); + expect(catalogQueries[0]).toMatchObject({ calls: 1, rows: 2 }); + expect(scoped!).toEqual({ allowedConnectionIds: [f.connection.id], installedConnections: before.installedConnections }); + expect(JSON.stringify(events)).not.toContain(bulk.slice(0, 64)); + expect((await db.select().from(toolProfileEntries).where(eq(toolProfileEntries.id, rule.id)))[0]).toEqual(rule); + expect(await service.getEffectiveToolAssignmentsForAgent(f.company.id, f.agent.id)).toEqual(before); + expect((await service.getConnectionAssignmentsForAgent(f.company.id, f.agent.id, [])).allowedConnectionIds).toEqual([]); + await db.delete(toolProfileBindings).where(eq(toolProfileBindings.profileId, f.profile.id)); + expect((await service.getConnectionAssignmentsForAgent(f.company.id, f.agent.id, [f.connection.id])).allowedConnectionIds).toEqual([]); + }); + + it("omits unrelated rules and bindings while retaining current connection decisions", async () => { + const f = await fixture(); + const [otherApplication] = await db.insert(toolApplications).values({ companyId: f.company.id, name: "Other app", type: "mcp_http" }).returning(); + const [otherConnection] = await db.insert(toolConnections).values({ companyId: f.company.id, applicationId: otherApplication.id, + name: "Other connection", uid: randomUUID(), transport: "mcp_remote" }).returning(); + const [otherTool] = await db.insert(toolCatalogEntries).values({ companyId: f.company.id, applicationId: otherApplication.id, + connectionId: otherConnection.id, name: "other", toolName: "other", versionHash: "other", riskLevel: "destructive" }).returning(); + await db.insert(toolProfileEntries).values(Array.from({ length: 200 }, (_, index) => [ + { selectorType: "application" as const, applicationId: otherApplication.id }, + { selectorType: "connection" as const, connectionId: otherConnection.id }, + { selectorType: "catalog_entry" as const, catalogEntryId: otherTool.id }, + { selectorType: "tool_name" as const, toolName: `unrelated_${index}` }, + { selectorType: "risk_level" as const, riskLevel: "destructive" as const }, + ].map(entry => ({ companyId: f.company.id, profileId: f.profile.id, effect: index % 2 ? "include" as const : "exclude" as const, ...entry }))).flat()); + await db.insert(toolProfileBindings).values(Array.from({ length: 200 }, () => ({ companyId: f.company.id, + profileId: f.profile.id, targetType: "agent" as const, targetId: randomUUID() }))); + const [include, exclude] = await db.insert(toolProfileEntries).values([ + { companyId: f.company.id, profileId: f.profile.id, selectorType: "connection", effect: "include", connectionId: f.connection.id }, + { companyId: f.company.id, profileId: f.profile.id, selectorType: "catalog_entry", effect: "exclude", catalogEntryId: f.write.id }, + ]).returning(); + const service = toolAccessService(db), before = await compare(f); + const savedRules = await db.select().from(toolProfileEntries); + const events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(db, { toolProfileEntries, toolProfileBindings }, capture); + try { + const scoped = await service.getConnectionAssignmentsForAgent(f.company.id, f.agent.id, [f.connection.id]); + capture.flush(); + const rules = events.filter(event => event.event === "paperclip.db_response_capture.entry.v1" && event.table === "tool_profile_entries"); + const bindings = events.filter(event => event.event === "paperclip.db_response_capture.entry.v1" && event.table === "tool_profile_bindings"); + expect(rules).toHaveLength(1); + expect(Number(rules[0].responseValueBytes)).toBeLessThan(4096); + expect(rules[0]).toMatchObject({ calls: 1, rows: 2 }); + expect(bindings).toHaveLength(1); + expect(bindings[0]).toMatchObject({ calls: 1, rows: 1 }); + expect(scoped).toEqual({ allowedConnectionIds: [f.connection.id], installedConnections: before.full.installedConnections }); + expect(await db.select().from(toolProfileEntries)).toEqual(savedRules); + await db.update(toolProfileEntries).set({ effect: "exclude" }).where(eq(toolProfileEntries.id, include.id)); + expect((await service.getConnectionAssignmentsForAgent(f.company.id, f.agent.id, [f.connection.id])).allowedConnectionIds).toEqual([]); + await db.delete(toolProfileEntries).where(eq(toolProfileEntries.id, include.id)); + expect((await service.getConnectionAssignmentsForAgent(f.company.id, f.agent.id, [f.connection.id])).allowedConnectionIds).toEqual([]); + expect((await db.select().from(toolProfileEntries).where(eq(toolProfileEntries.id, exclude.id)))[0]).toEqual(exclude); + } finally { capture.flush(); } + }); + + it("preserves null application matches and exact special-character tool names", async () => { + const f = await fixture(), toolName = "quoted' path\\tool_🙂"; + await db.update(toolCatalogEntries).set({ status: "removed" }).where(eq(toolCatalogEntries.id, f.write.id)); + await db.update(toolCatalogEntries).set({ applicationId: null, toolName }).where(eq(toolCatalogEntries.id, f.read.id)); + await db.update(toolProfiles).set({ defaultAction: "allow" }).where(eq(toolProfiles.id, f.profile.id)); + const [rule] = await db.insert(toolProfileEntries).values({ companyId: f.company.id, profileId: f.profile.id, + selectorType: "application", applicationId: null, effect: "exclude" }).returning(); + const service = toolAccessService(db); + expect((await compare(f)).assignments.allowedTools).toEqual([]); + expect((await service.getConnectionAssignmentsForAgent(f.company.id, f.agent.id, [f.connection.id])).allowedConnectionIds).toEqual([]); + await db.update(toolProfileEntries).set({ selectorType: "tool_name", toolName }).where(eq(toolProfileEntries.id, rule.id)); + expect((await compare(f)).assignments.allowedTools).toEqual([]); + expect((await service.getConnectionAssignmentsForAgent(f.company.id, f.agent.id, [f.connection.id])).allowedConnectionIds).toEqual([]); + await db.update(toolProfileEntries).set({ toolName: `${toolName}different` }).where(eq(toolProfileEntries.id, rule.id)); + expect((await compare(f)).assignments.allowedTools.map(tool => tool.id)).toEqual([f.read.id]); + expect((await service.getConnectionAssignmentsForAgent(f.company.id, f.agent.id, [f.connection.id])).allowedConnectionIds).toEqual([f.connection.id]); + }); + + it.each(["application", "connection", "catalog_entry", "tool_name", "risk_level"] as const)("denies the connection when a %s exclusion blocks its only active tool", async selectorType => { + const f = await fixture(); + await db.update(toolCatalogEntries).set({ status: "removed" }).where(eq(toolCatalogEntries.id, f.write.id)); + await db.update(toolProfiles).set({ defaultAction: "allow" }).where(eq(toolProfiles.id, f.profile.id)); + const [rule] = await db.insert(toolProfileEntries).values({ companyId: f.company.id, profileId: f.profile.id, selectorType, effect: "exclude", + ...(selectorType === "application" ? { applicationId: f.application.id } : {}), + ...(selectorType === "connection" ? { connectionId: f.connection.id } : {}), + ...(selectorType === "catalog_entry" ? { catalogEntryId: f.read.id } : {}), + ...(selectorType === "tool_name" ? { toolName: f.read.toolName } : {}), + ...(selectorType === "risk_level" ? { riskLevel: "read" as const } : {}), + }).returning(); + const service = toolAccessService(db); + expect((await service.getConnectionAssignmentsForAgent(f.company.id, f.agent.id, [f.connection.id])).allowedConnectionIds).toEqual([]); + expect((await compare(f)).assignments.allowedTools).toEqual([]); + await db.delete(toolProfileEntries).where(eq(toolProfileEntries.id, rule.id)); + expect((await service.getConnectionAssignmentsForAgent(f.company.id, f.agent.id, [f.connection.id])).allowedConnectionIds).toEqual([f.connection.id]); + }); + + it("filters rules against the catalog rows being evaluated when the catalog changes during the read", async () => { + const f = await fixture(); + await db.update(toolProfiles).set({ defaultAction: "allow" }).where(eq(toolProfiles.id, f.profile.id)); + await db.update(toolCatalogEntries).set({ status: "removed" }).where(eq(toolCatalogEntries.id, f.write.id)); + await db.insert(toolProfileEntries).values({ companyId: f.company.id, profileId: f.profile.id, + selectorType: "tool_name", toolName: f.read.toolName, effect: "exclude" }); + let changed = false; + const changingDb = new Proxy(db, { get(target, key) { + if (key !== "select") return Reflect.get(target, key); + return (...fields: Parameters) => { + const builder = target.select(...fields), from = builder.from.bind(builder); + builder.from = ((...args: Parameters) => { + const selection = from(...args); + if (args[0] === toolCatalogEntries) { + const execute = selection.execute.bind(selection); + selection.execute = async () => { + const rows = await execute(); + if (!changed) { + changed = true; + await db.update(toolCatalogEntries).set({ toolName: "changed-after-read" }).where(eq(toolCatalogEntries.id, f.read.id)); + } + return rows; + }; + } + return selection; + }) as typeof builder.from; + return builder; + }; + } }); + const service = toolAccessService(changingDb); + expect((await service.getConnectionAssignmentsForAgent(f.company.id, f.agent.id, [f.connection.id])).allowedConnectionIds).toEqual([]); + expect(changed).toBe(true); + expect((await service.getConnectionAssignmentsForAgent(f.company.id, f.agent.id, [f.connection.id])).allowedConnectionIds).toEqual([f.connection.id]); + }); + + it("keeps other companies' catalog rows out of scoped assignments", async () => { + const f = await fixture(); + const other = await fixture(); + await db.update(toolProfiles).set({ defaultAction: "allow" }).where(eq(toolProfiles.id, f.profile.id)); + expect((await toolAccessService(db).getConnectionAssignmentsForAgent(f.company.id, f.agent.id, [other.connection.id])).allowedConnectionIds).toEqual([]); + await expect(toolAccessService(db).getConnectionAssignmentsForAgent(f.company.id, other.agent.id, [f.connection.id])).rejects.toThrow("same company"); + }); + }); diff --git a/server/src/config.ts b/server/src/config.ts index 74e80c77ee1..dac9dba2142 100644 --- a/server/src/config.ts +++ b/server/src/config.ts @@ -95,6 +95,7 @@ export interface Config { nativeHarnessBackupExpirationEnabled: boolean; nativeHarnessBackupExpirationDryRun: boolean; nativeHarnessBackupExpirationDays: number; + nativeRunnerEvidenceRetentionDays: number; nativeWorkspaceSyncRetentionEnabled: boolean; nativeWorkspaceSyncRetentionDryRun: boolean; nativeWorkspaceSyncRetentionDays: number; @@ -351,6 +352,7 @@ export function loadConfig(): Config { const skillVersionRetentionKeepNewest = nonNegativeIntegerEnv("PAPERCLIP_SKILL_VERSION_RETENTION_KEEP", 0); const skillVersionRetentionGraceMinutes = nonNegativeIntegerEnv("PAPERCLIP_SKILL_VERSION_RETENTION_GRACE_MINUTES", 60); const nativeHarnessBackupExpirationDays = Math.max(1, nonNegativeIntegerEnv("PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_DAYS", 7)); + const nativeRunnerEvidenceRetentionDays = Math.max(1, nonNegativeIntegerEnv("PAPERCLIP_NATIVE_RUNNER_EVIDENCE_RETENTION_DAYS", 7)); const nativeWorkspaceSyncRetentionDays = Math.max(1, nonNegativeIntegerEnv("PAPERCLIP_NATIVE_WORKSPACE_SYNC_RETENTION_DAYS", 7)); const gitRestorePointRetentionDays = Math.max(1, nonNegativeIntegerEnv("PAPERCLIP_GIT_RESTORE_POINT_RETENTION_DAYS", 90)); const gitRestorePointTerminalDelayDays = nonNegativeIntegerEnv("PAPERCLIP_GIT_RESTORE_POINT_TERMINAL_DELAY_DAYS", 3); @@ -417,6 +419,7 @@ export function loadConfig(): Config { nativeHarnessBackupExpirationEnabled: process.env.PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_ENABLED !== "false", nativeHarnessBackupExpirationDryRun: process.env.PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_DRY_RUN === "true", nativeHarnessBackupExpirationDays, + nativeRunnerEvidenceRetentionDays, nativeWorkspaceSyncRetentionEnabled: process.env.PAPERCLIP_NATIVE_WORKSPACE_SYNC_RETENTION_ENABLED !== "false", nativeWorkspaceSyncRetentionDryRun: process.env.PAPERCLIP_NATIVE_WORKSPACE_SYNC_RETENTION_DRY_RUN === "true", nativeWorkspaceSyncRetentionDays, diff --git a/server/src/index.ts b/server/src/index.ts index 584e9e67697..4212387dd04 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -13,6 +13,7 @@ import { NATIVE_HARNESS_BACKUP_EXPIRATION_STARTUP_DELAY_MS, nativeHarnessBackupExpirationService, } from "./services/native-runtime/native-harness-backup-expiration.js"; +import { onNativeRunnerEvidence } from "./services/native-runtime/native-runner-evidence-signal.js"; import { gitRestorePointRetentionService, registerGitRestorePointWork } from "./services/git-restore-point-ledger.js"; import { nativeWorkspaceSyncRetentionService, @@ -38,6 +39,7 @@ import { chatCompletionDeliveryService } from "./services/chat-completion-delive // instrumentationReady before opening DB connections or constructing the // HTTP server, so trace coverage does not depend on incidental timing. import { instrumentationReady, shutdownInstrumentation } from "./instrumentation.js"; +import { createTrackedSingleFlight } from "./lib/single-flight.js"; import { sentryReady, shutdownSentry, captureException } from "./sentry.js"; import { setSentryRunLinkResolver } from "./sentry-links.js"; import { createSentryRunLinkResolver } from "./services/sentry-run-links.js"; @@ -135,6 +137,7 @@ import { } from "./services/device-login-reaper.js"; import { createProductionSetupTokenReaper } from "./services/setup-token-reaper.js"; import { localAiLoginService } from "./services/local-ai-login.js"; +import { registerGitHubBotCloudSetupPoller } from "./services/chat-github-cloud-ingress.js"; import { resolveWorktreeRunExecutionActivationState } from "./services/instance-settings.js"; import { parseAdapterRegistryEnv, @@ -1209,6 +1212,16 @@ async function startServerWithDatabaseTeardown( }); heartbeatSchedulerInFlight.add(tracked); }; + const runPeriodicRecovery = createTrackedSingleFlight({ + trackWork: trackHeartbeatSchedulerWork, + onSkipped: (inFlightForMs) => { + if (inFlightForMs >= 3 * config.heartbeatSchedulerIntervalMs) { + logger.error({ inFlightForMs }, "periodic heartbeat recovery remains in flight; duplicate pass suppressed"); + } else { + logger.warn({ inFlightForMs }, "skipped periodic heartbeat recovery tick; previous chain still in flight"); + } + }, + }); const waitForHeartbeatSchedulerIdle = async () => { while (heartbeatSchedulerInFlight.size > 0) { await Promise.allSettled([...heartbeatSchedulerInFlight]); @@ -1257,8 +1270,10 @@ async function startServerWithDatabaseTeardown( const nativeHarnessBackupExpiration = nativeHarnessBackupExpirationService(db as any, { dryRun: config.nativeHarnessBackupExpirationDryRun, closedForDays: config.nativeHarnessBackupExpirationDays, + evidenceRetentionDays: config.nativeRunnerEvidenceRetentionDays, shouldContinue: () => !heartbeatSchedulerStopped && !isIdleTaskDrainActive() && !isWarmStandby(), }); + const unsubscribeNativeRunnerEvidence = onNativeRunnerEvidence(nativeHarnessBackupExpiration.noteEvidence); let nativeHarnessBackupExpirationInFlight = false; let nativeHarnessBackupExpirationContinuation: ReturnType | undefined; const sweepNativeHarnessBackups = () => { @@ -1266,7 +1281,7 @@ async function startServerWithDatabaseTeardown( nativeHarnessBackupExpirationInFlight = true; trackHeartbeatSchedulerWork(nativeHarnessBackupExpiration.tick() .then((result) => { - if (result.candidates > 0 || result.failed > 0) { + if (result.candidates > 0 || result.evidenceCandidates > 0 || result.failed > 0) { logger.info({ ...result, dryRun: config.nativeHarnessBackupExpirationDryRun }, "native harness backup expiration"); } if (result.truncated && !heartbeatSchedulerStopped) { @@ -1408,9 +1423,9 @@ async function startServerWithDatabaseTeardown( ?? process.env.PAPERCLIP_TOOL_RUNTIME_TRUSTED_HOST ?? null, }); - const scheduleGitHubConnectionEventPoll = () => { + const scheduleGitHubConnectionEventPoll = (forSetup = false) => { if (isIdleTaskDrainActive() || heartbeatSchedulerStopped) return; - trackHeartbeatSchedulerWork(githubConnectionEvents.pollOnce() + const work = githubConnectionEvents.pollOnce({ forSetup }) .then((result) => { if (result.leased > 0 || result.failed > 0) { logger.info(result, "GitHub connection event poll completed"); @@ -1418,8 +1433,13 @@ async function startServerWithDatabaseTeardown( }) .catch((err) => { logger.error({ err }, "GitHub connection event poll failed"); - })); + }); + trackHeartbeatSchedulerWork(work); + return work; }; + server.on("close", registerGitHubBotCloudSetupPoller(db, async () => { + await scheduleGitHubConnectionEventPoll(true); + })); const scheduleGitHubConnectionContinuitySweep = () => { if (isIdleTaskDrainActive() || heartbeatSchedulerStopped) return; trackHeartbeatSchedulerWork(tools.sweepGitHubConnectionContinuity() @@ -1887,7 +1907,7 @@ async function startServerWithDatabaseTeardown( if (!(await heartbeat.resolveSchedulingSuppression()).suppressed) { // Periodically reap orphaned runs (5-min staleness threshold) and make sure // persisted queued work is still being driven forward. - trackHeartbeatSchedulerWork(heartbeat + runPeriodicRecovery(() => heartbeat .reapOrphanedRuns({ staleThresholdMs: 5 * 60 * 1000 }) .then(() => heartbeat.promoteDueScheduledRetries()) .then(async (promotion) => { @@ -2078,6 +2098,7 @@ async function startServerWithDatabaseTeardown( clearInterval(nativeHarnessBackupExpirationInterval); clearTimeout(nativeHarnessBackupExpirationStartup); clearTimeout(nativeHarnessBackupExpirationContinuation); + unsubscribeNativeRunnerEvidence(); await skillVersionRetention.dispose(); await skillSourceSync.stop(); unsubscribeChatCompletions(); diff --git a/server/src/lib/agent-records.ts b/server/src/lib/agent-records.ts index 786f61af550..25c0ed7c9f5 100644 --- a/server/src/lib/agent-records.ts +++ b/server/src/lib/agent-records.ts @@ -98,6 +98,8 @@ export interface CreateAgentOptions { claudeLogin?: ClaudeLoginContext; } +type AgentEligibilityRow = Pick; + interface AgentShortnameRow { id: string; name: string; @@ -340,7 +342,7 @@ export function agentRecordQueries(db: Db) { }; } - function normalizeAgentRows(rows: (typeof agents.$inferSelect)[], allCompanyRows = rows) { + function normalizeAgentRows(rows: (typeof agents.$inferSelect)[], allCompanyRows: AgentEligibilityRow[] = rows) { const eligibilityAgents = allCompanyRows.map(toEligibilityAgent); return rows.map((row) => { const base = normalizeAgentBaseRow(row); @@ -357,7 +359,7 @@ export function agentRecordQueries(db: Db) { }); } - function normalizeAgentRow(row: typeof agents.$inferSelect, allCompanyRows?: (typeof agents.$inferSelect)[]) { + function normalizeAgentRow(row: typeof agents.$inferSelect, allCompanyRows?: AgentEligibilityRow[]) { return normalizeAgentRows([row], allCompanyRows)[0]!; } @@ -365,6 +367,11 @@ export function agentRecordQueries(db: Db) { return db.select().from(agents).where(eq(agents.companyId, companyId)); } + async function listCompanyAgentEligibilityRows(companyId: string): Promise { + return db.select({ id: agents.id, companyId: agents.companyId, name: agents.name, + status: agents.status, reportsTo: agents.reportsTo }).from(agents).where(eq(agents.companyId, companyId)); + } + async function getMonthlySpendByAgentIds(companyId: string, agentIds: string[]) { if (agentIds.length === 0) return new Map(); const { start, end } = currentUtcMonthWindow(); @@ -405,7 +412,7 @@ export function agentRecordQueries(db: Db) { .then((rows) => rows[0] ?? null); if (!row) return null; const [companyRows, hydrated] = await Promise.all([ - listCompanyAgentRows(row.companyId), + listCompanyAgentEligibilityRows(row.companyId), hydrateAgentSpend([row]).then((rows) => rows[0]!), ]); return normalizeAgentRow(hydrated, companyRows); @@ -472,6 +479,6 @@ export function agentRecordQueries(db: Db) { }); } - return { normalizeAgentRows, normalizeAgentRow, listCompanyAgentRows, hydrateAgentSpend, + return { normalizeAgentRows, normalizeAgentRow, listCompanyAgentRows, listCompanyAgentEligibilityRows, hydrateAgentSpend, getById, ensureManager, assertNoCycle, assertCompanyShortnameAvailable, assertBuiltInAgentMetadataMutationAllowed }; } diff --git a/server/src/lib/single-flight.test.ts b/server/src/lib/single-flight.test.ts new file mode 100644 index 00000000000..06dabc92485 --- /dev/null +++ b/server/src/lib/single-flight.test.ts @@ -0,0 +1,163 @@ +import { describe, expect, it, vi } from "vitest"; +import { createSingleFlight, createTrackedSingleFlight } from "./single-flight.js"; +import { beginIdleTrackedWork, idleWorkSnapshot } from "../services/task-admission.js"; + +function deferred() { + let resolve!: () => void; + let reject!: (err: unknown) => void; + const promise = new Promise((res, rej) => { + resolve = res; + reject = rej; + }); + return { promise, resolve, reject }; +} + +describe("createSingleFlight", () => { + it("runs work when idle and reports busy until it settles", async () => { + const gate = createSingleFlight(); + const gap = deferred(); + + const first = gate.run(() => gap.promise); + expect(first.started).toBe(true); + expect(gate.busy).toBe(true); + + gap.resolve(); + await (first as { settled: Promise }).settled; + expect(gate.busy).toBe(false); + }); + + it("skips a tick while the previous invocation is still in flight", async () => { + const clock = { nowMs: 1_000 }; + const gate = createSingleFlight(() => clock.nowMs); + const gap = deferred(); + const work = vi.fn(() => gap.promise); + + const first = gate.run(work); + clock.nowMs = 31_000; + const second = gate.run(work); + + expect(second).toEqual({ started: false, inFlightForMs: 30_000 }); + expect(work).toHaveBeenCalledTimes(1); + + gap.resolve(); + await (first as { settled: Promise }).settled; + }); + + it("admits new work after the previous invocation resolves", async () => { + const gate = createSingleFlight(); + const gap = deferred(); + + const first = gate.run(() => gap.promise); + gap.resolve(); + await (first as { settled: Promise }).settled; + + const second = gate.run(async () => {}); + expect(second.started).toBe(true); + await (second as { settled: Promise }).settled; + }); + + it("releases the gate when work rejects and never rejects the settled promise", async () => { + const gate = createSingleFlight(); + const gap = deferred(); + + const first = gate.run(() => gap.promise); + gap.reject(new Error("chain failed")); + await expect((first as { settled: Promise }).settled).resolves.toBeUndefined(); + expect(gate.busy).toBe(false); + + const second = gate.run(async () => {}); + expect(second.started).toBe(true); + await (second as { settled: Promise }).settled; + }); + + it("propagates a synchronous throw and leaves the gate free", async () => { + const gate = createSingleFlight(); + + expect(() => + gate.run(() => { + throw new Error("sync failure"); + }), + ).toThrow("sync failure"); + expect(gate.busy).toBe(false); + + const second = gate.run(async () => {}); + expect(second.started).toBe(true); + await (second as { settled: Promise }).settled; + }); + + it("keeps the owned pass active after a caller stops waiting", async () => { + const gate = createSingleFlight(), work = deferred(), observer = deferred(); + const first = gate.run(() => work.promise); + if (!first.started) throw new Error("initial pass was not admitted"); + const stopped = Promise.race([first.settled, observer.promise]); + observer.reject(new Error("observer stopped")); + await expect(stopped).rejects.toThrow("observer stopped"); + const duplicate = vi.fn(async () => undefined); + expect(gate.run(duplicate).started).toBe(false); + expect(duplicate).not.toHaveBeenCalled(); + expect(gate.busy).toBe(true); + work.resolve(); + await first.settled; + expect(gate.busy).toBe(false); + }); + + it("keeps cleanup joined after the main operation completes", async () => { + const gate = createSingleFlight(), operation = deferred(), cleanup = deferred(); + const first = gate.run(() => operation.promise.finally(() => cleanup.promise)); + if (!first.started) throw new Error("initial pass was not admitted"); + operation.resolve(); + await Promise.resolve(); + expect(gate.busy).toBe(true); + const duplicate = vi.fn(async () => undefined); + expect(gate.run(duplicate).started).toBe(false); + expect(duplicate).not.toHaveBeenCalled(); + cleanup.resolve(); + await first.settled; + expect(gate.busy).toBe(false); + const next = gate.run(duplicate); + expect(next.started).toBe(true); + if (next.started) await next.settled; + expect(duplicate).toHaveBeenCalledTimes(1); + }); + + it("tracks one periodic pass through scheduler shutdown and idle sleep", async () => { + vi.useFakeTimers(); + const before = idleWorkSnapshot().active; + const operation = deferred(), cleanup = deferred(); + const tracked: Promise[] = []; + const trackWork = vi.fn((work: Promise) => { + const finish = beginIdleTrackedWork(); + tracked.push(work.finally(finish)); + }); + const onSkipped = vi.fn(); + const tick = createTrackedSingleFlight({ trackWork, onSkipped }); + const heartbeat = vi.fn(() => operation.promise.finally(() => cleanup.promise)); + const timer = setInterval(() => tick(heartbeat), 30_000); + try { + await vi.advanceTimersByTimeAsync(60_000); + expect(heartbeat).toHaveBeenCalledOnce(); + expect(trackWork).toHaveBeenCalledOnce(); + expect(onSkipped).toHaveBeenCalledExactlyOnceWith(30_000); + clearInterval(timer); + let shutdownComplete = false; + const shutdown = Promise.all(tracked).then(() => { shutdownComplete = true; }); + await Promise.resolve(); + expect(shutdownComplete).toBe(false); + expect(idleWorkSnapshot().active).toBe(before + 1); + operation.resolve(); + await Promise.resolve(); + expect(shutdownComplete).toBe(false); + expect(idleWorkSnapshot().active).toBe(before + 1); + cleanup.resolve(); + await shutdown; + expect(shutdownComplete).toBe(true); + expect(idleWorkSnapshot().active).toBe(before); + } finally { + clearInterval(timer); + operation.resolve(); + cleanup.resolve(); + await Promise.all(tracked); + vi.useRealTimers(); + } + }); +}); diff --git a/server/src/lib/single-flight.ts b/server/src/lib/single-flight.ts new file mode 100644 index 00000000000..68dc1b035f4 --- /dev/null +++ b/server/src/lib/single-flight.ts @@ -0,0 +1,51 @@ +export interface SingleFlightSkip { + started: false; + inFlightForMs: number; +} + +export interface SingleFlightStart { + started: true; + settled: Promise; +} + +export type SingleFlightResult = SingleFlightSkip | SingleFlightStart; + +export function createSingleFlight(now: () => number = Date.now) { + let inFlight: Promise | null = null; + let startedAtMs = 0; + + return { + get busy(): boolean { + return inFlight !== null; + }, + run(work: () => Promise): SingleFlightResult { + if (inFlight) { + return { started: false, inFlightForMs: now() - startedAtMs }; + } + startedAtMs = now(); + const settled = Promise.resolve(work()) + .then( + () => undefined, + () => undefined, + ) + .finally(() => { + inFlight = null; + }); + inFlight = settled; + return { started: true, settled }; + }, + }; +} + +export function createTrackedSingleFlight(options: { + trackWork: (work: Promise) => void; + onSkipped: (inFlightForMs: number) => void; + now?: () => number; +}) { + const gate = createSingleFlight(options.now); + return (work: () => Promise) => { + const tick = gate.run(work); + if (tick.started) options.trackWork(tick.settled); + else options.onSkipped(tick.inFlightForMs); + }; +} diff --git a/server/src/middleware/auth.ts b/server/src/middleware/auth.ts index 8504fc3c853..86347bb40d5 100644 --- a/server/src/middleware/auth.ts +++ b/server/src/middleware/auth.ts @@ -429,7 +429,8 @@ export function actorMiddleware(db: Db, opts: ActorMiddlewareOptions): RequestHa return; } if (identityRun?.activeIdentityContextId && identityRun.status === "running") { - const captured = await captureRunIdentity(db, { companyId: claims.company_id, agentId: claims.sub, runId: claims.run_id }); + const captured = await captureRunIdentity(db, { companyId: claims.company_id, agentId: claims.sub, runId: claims.run_id }, + { readScope: "request_authentication" }); identityRun.activeIdentityContextId = captured.context?.id ?? null; identityRun.responsibleUserId = captured.context?.responsibleUserId ?? null; } diff --git a/server/src/modules/wake-queue/adapters/postgres.test.ts b/server/src/modules/wake-queue/adapters/postgres.test.ts index 16479253a2f..83d5bb6ee44 100644 --- a/server/src/modules/wake-queue/adapters/postgres.test.ts +++ b/server/src/modules/wake-queue/adapters/postgres.test.ts @@ -206,7 +206,7 @@ describeEmbeddedPostgres("wake-queue postgres adapter", () => { return { recorded, rows }; } - it("keeps in-flight work and every recovery receipt while omitting output from release reads", async () => { + it.each(["legacy", "native"])("keeps %s in-flight work and every recovery receipt while omitting output from release reads", async runtimeMode => { const companyId = await seedCompany(); const agentId = await seedAgent({ companyId }); const issueId = await seedIssue({ companyId, assigneeAgentId: agentId }); @@ -218,7 +218,10 @@ describeEmbeddedPostgres("wake-queue postgres adapter", () => { nativeCancellation: { schema: "paperclip.native-cancellation.v1", runId, companyId, issueId }, executionCancellation: { state: "acknowledged" }, executionRecovery: { kind: "interrupted", providerStopped: true }, configurationIncomplete: { reason: "workspace_unavailable" }, unknownFutureReceipt: { retained: true } }; - await db.update(heartbeatRuns).set({ resultJson: { ...evidence, stdout: payload.repeat(3), stderr: payload } }).where(eq(heartbeatRuns.id, runId)); + await db.update(heartbeatRuns).set({ runtimeMode, + ...(runtimeMode === "native" ? { runnerProfileJson: { savedInput: payload }, stdoutExcerpt: payload, stderrExcerpt: payload, + contextSnapshot: { issueId, paperclipWake: { savedTask: payload }, paperclipTaskMarkdown: payload } } : {}), + resultJson: { ...evidence, stdout: payload.repeat(3), stderr: payload } }).where(eq(heartbeatRuns.id, runId)); await db.update(issues).set({ executionRunId: runId, checkoutRunId: runId }).where(eq(issues.id, issueId)); const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); const [issue] = await db.select().from(issues).where(eq(issues.id, issueId)); @@ -230,7 +233,10 @@ describeEmbeddedPostgres("wake-queue postgres adapter", () => { ); expect(result.outcome.kind).toBe("released"); expect(rows).toHaveLength(1); - expect(rows[0]).toEqual([{ ...run, resultJson: evidence }]); + expect(rows[0]).toEqual([{ ...run, resultJson: evidence, + ...(runtimeMode === "native" ? { runnerProfileJson: null, stdoutExcerpt: null, stderrExcerpt: null, + contextSnapshot: { issueId } } : {}) }]); + expect(result.run.contextSnapshot).toEqual({ issueId }); expect(Buffer.byteLength(JSON.stringify(rows))).toBeLessThan(8 * 1024); expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)))[0]).toEqual(run); expect((await db.select().from(issues).where(eq(issues.id, issueId)))[0]).toEqual(issue); diff --git a/server/src/modules/wake-queue/adapters/postgres.ts b/server/src/modules/wake-queue/adapters/postgres.ts index 6db54a82725..cd2f1aa7a48 100644 --- a/server/src/modules/wake-queue/adapters/postgres.ts +++ b/server/src/modules/wake-queue/adapters/postgres.ts @@ -5,7 +5,8 @@ import { isCompletedOnboardingHandoffWake } from "../../../services/chat-complet import { instanceSettingsService } from "../../../services/instance-settings.js"; import { currentConversationCommentCondition } from "../../../services/agent-conversations.js"; import { getExecutionBlocker, hasPendingRunCleanup } from "../../../services/execution-blocker.js"; -import { and, asc, eq, getTableColumns, inArray, isNull, notInArray, or, sql } from "drizzle-orm"; +import { and, asc, eq, inArray, isNull, notInArray, or, sql } from "drizzle-orm"; +import { readReleaseRun, type ReleaseRun } from "./release-run-read.js"; import type { Db } from "@paperclipai/db"; import { extractIssueReferenceIdentifiers, isUuidLike } from "@paperclipai/shared"; import { @@ -74,14 +75,6 @@ const EXECUTION_PATH_HEARTBEAT_RUN_STATUSES = ["queued", "running", "scheduled_r type HeartbeatRunRow = typeof heartbeatRuns.$inferSelect; type IssueRow = typeof issues.$inferSelect; -const releaseRunFields = { - ...getTableColumns(heartbeatRuns), - resultJson: sql | null>`case - when jsonb_typeof(${heartbeatRuns.resultJson}) = 'object' - then ${heartbeatRuns.resultJson} - 'stdout' - 'stderr' - else ${heartbeatRuns.resultJson} end`, -}; - function normalizeAgentNameKey(value: string | null | undefined): string | null { if (typeof value !== "string") return null; const normalized = value.trim().toLowerCase(); @@ -94,7 +87,7 @@ function toRequestedByActorType(value: string | null): "user" | "agent" | "syste return value === "user" || value === "agent" || value === "system" ? value : null; } -function toRunSnapshot(row: HeartbeatRunRow): RunSnapshot { +function toRunSnapshot(row: ReleaseRun): RunSnapshot { const configurationIncompletePayload = parseObject(parseObject(row.resultJson).configurationIncomplete); return { id: row.id, @@ -206,7 +199,7 @@ function buildHost(_tx: Db, deps: WakeQueuePostgresAdapterDeps): WakeQueueHost { }; } -function buildTransaction(tx: Db, deps: WakeQueuePostgresAdapterDeps, db: Db, run: HeartbeatRunRow): WakeQueueTransaction { +function buildTransaction(tx: Db, deps: WakeQueuePostgresAdapterDeps, db: Db, run: ReleaseRun): WakeQueueTransaction { const treeControlSvc = issueTreeControlService(tx); const issuesSvc = issueService(tx); const interruptQueueId = run.runtimeMode !== "native" && run.status === "cancelled" @@ -750,7 +743,7 @@ function buildTransaction(tx: Db, deps: WakeQueuePostgresAdapterDeps, db: Db, ru }; } -async function recordNativeTerminalRecoveryIfNeeded(tx: Db, run: HeartbeatRunRow, issue: IssueRow, now: Date): Promise { +async function recordNativeTerminalRecoveryIfNeeded(tx: Db, run: ReleaseRun, issue: IssueRow, now: Date): Promise { const applies = run.runtimeMode === "native" && ["failed", "timed_out", "interrupted", "cancelled"].includes(run.status) && @@ -1058,11 +1051,8 @@ export function createPostgresWakeQueueAdapter(db: Db, deps: WakeQueuePostgresAd async withIssueExecutionLock(input, fn): Promise { return db.transaction(async (rawTx) => { const tx = rawTx as unknown as Db; - const run = await tx - .select(releaseRunFields) - .from(heartbeatRuns) - .where(and(eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId))) - .then((rows) => rows[0] ?? null); + const evidence = await readReleaseRun(tx, input); + const run = evidence.run; if (!run) { throw new Error(`wake-queue: run ${input.runId} was not found while releasing issue execution`); } @@ -1164,7 +1154,8 @@ export function createPostgresWakeQueueAdapter(db: Db, deps: WakeQueuePostgresAd issueStatus: issueRow?.status ?? "", hasAssigneeUser: Boolean(issueRow?.assigneeUserId), assigneeAgentMatchesRunAgent: issueRow?.assigneeAgentId === run.agentId, - legacyExecutionNeedsReconciliation: await legacyExecutionNeedsReconciliationWithEvidence(tx as unknown as Db, run), + legacyExecutionNeedsReconciliation: evidence.kind === "complete" && + await legacyExecutionNeedsReconciliationWithEvidence(tx as unknown as Db, evidence.run), // An operator stop never promotes old queued work by itself. The // next explicit wake adopts those messages atomically when it // queues a run. diff --git a/server/src/modules/wake-queue/adapters/release-run-read.test.ts b/server/src/modules/wake-queue/adapters/release-run-read.test.ts new file mode 100644 index 00000000000..6a669edf617 --- /dev/null +++ b/server/src/modules/wake-queue/adapters/release-run-read.test.ts @@ -0,0 +1,123 @@ +import { randomBytes, randomUUID } from "node:crypto"; +import { agents, companies, createDb, heartbeatRuns } from "@paperclipai/db"; +import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture"; +import { eq } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../../__tests__/helpers/embedded-postgres.js"; +import { readReleaseRun } from "./release-run-read.js"; + +const support = await getEmbeddedPostgresTestSupport(); + +describe.skipIf(!support.supported)("wake release run reads", () => { + let database: Awaited>; + let db: ReturnType; + let companyId: string; + let agentId: string; + + beforeAll(async () => { + database = await startEmbeddedPostgresTestDatabase("wake-release-read-"); + db = createDb(database.connectionString); + const [company] = await db.insert(companies).values({ name: "Wake release fixture", issuePrefix: "WRR" }).returning(); + const [agent] = await db.insert(agents).values({ companyId: company.id, name: "Release agent", adapterType: "process" }).returning(); + companyId = company.id; + agentId = agent.id; + }, 60_000); + + afterAll(async () => { + await db?.$client.end(); + await database?.cleanup(); + }); + + it("omits native bulk at the driver boundary and preserves the complete legacy fallback", async () => { + const bulk = randomBytes(128 * 1024).toString("base64"); + const context = { issueId: randomUUID(), opaqueFutureContext: { savedInput: "keep this input" }, explicitUserContinuation: true }; + const evidence = { nativeCancellation: { schema: "paperclip.native-cancellation.v1", acknowledged: true }, unknownFutureReceipt: [true, null, 42] }; + const [before] = await db.insert(heartbeatRuns).values({ + companyId, agentId, runtimeMode: "native", status: "running", contextSnapshot: context, + runnerProfileJson: { nativeExecutionInput: { savedInput: bulk } }, stdoutExcerpt: bulk, stderrExcerpt: bulk, + resultJson: { ...evidence, stdout: bulk, stderr: bulk }, + }).returning(); + const scope = { companyId, runId: before.id }; + const events: Array> = []; + const now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(db, { heartbeatRuns }, capture); + try { + const native = await db.transaction(tx => readReleaseRun(tx as unknown as typeof db, scope)); + expect(native.kind).toBe("native"); + expect(native.run?.contextSnapshot).toEqual(context); + expect(native.run?.resultJson).toEqual(evidence); + expect(native.run).not.toHaveProperty("runnerProfileJson"); + expect(native.run).not.toHaveProperty("stdoutExcerpt"); + expect(native.run).not.toHaveProperty("stderrExcerpt"); + } finally { capture.flush(); } + const summary = events.find(event => event.event === "paperclip.db_response_capture.summary.v1")!; + expect(summary.completedQueries).toBe(1); + expect(summary.responseValueBytes).toBeLessThan(8192); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, before.id)))[0]).toEqual(before); + expect(JSON.stringify(events)).not.toContain(bulk.slice(0, 64)); + await db.update(heartbeatRuns).set({ runtimeMode: "legacy" }).where(eq(heartbeatRuns.id, before.id)); + const legacy = await readReleaseRun(db, scope); + expect(legacy.kind).toBe("complete"); + expect(legacy.run).toEqual({ ...before, runtimeMode: "legacy", resultJson: evidence }); + expect((await readReleaseRun(db, { ...scope, companyId: randomUUID() })).run).toBeUndefined(); + expect((await readReleaseRun(db, { companyId, runId: randomUUID() })).run).toBeUndefined(); + }); + + it.each([null, ["stdout", { stderr: "evidence" }], "scalar", 42].map(result => ({ result })))( + "preserves non-object native result evidence $result", async ({ result }) => { + const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId, runtimeMode: "native", + resultJson: result as Record | null }).returning(); + expect((await readReleaseRun(db, { companyId, runId: run.id })).run?.resultJson).toEqual(result); + }, + ); + + it("keeps launch text in storage while releasing native runs with their control context", async () => { + const bulk = randomBytes(128 * 1024).toString("base64"); + const context = { + issueId: randomUUID(), taskId: randomUUID(), taskKey: "saved-task", + retryReason: "execution_review_participant_recovery", wakeReason: "issue_commented", + explicitUserContinuation: true, + interactionId: randomUUID(), interactionKind: "review", interactionStatus: "resolved", + wakeCommentIds: [randomUUID()], queuedCommentSourceWakeIds: [randomUUID()], + chatConversationId: randomUUID(), executionPolicy: { requireApproval: true }, + opaqueFutureContext: { retained: [null, false, "input"] }, + }; + const launch = { + executionContinuation: { instruction: bulk, state: "saved" }, + paperclipWake: { savedTask: bulk }, paperclipWakeComment: bulk, paperclipTaskMarkdown: bulk, + paperclipTaskMarkdownCompact: bulk, paperclipTaskMarkdownAssignment: bulk, + paperclipTaskMarkdownAssignmentCompact: bulk, paperclipWorkspace: { files: bulk }, + paperclipEnvironment: { input: bulk }, paperclipContinuationSummary: bulk, + }; + const evidence = { nativeCancellation: { schema: "paperclip.native-cancellation.v1", dispatched: true }, + executionCancellation: { state: "acknowledged" }, workspaceRestoreRecovery: { leaseIds: [randomUUID()] }, + configurationIncomplete: { reason: "secret_binding_missing" }, unknownFutureReceipt: { retained: true } }; + const [before] = await db.insert(heartbeatRuns).values({ companyId, agentId, runtimeMode: "native", + contextSnapshot: { ...context, ...launch }, resultJson: evidence }).returning(); + const events: Array> = []; + const now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(db, { heartbeatRuns }, capture); + let selected: Awaited>; + try { selected = await db.transaction(tx => readReleaseRun(tx as unknown as typeof db, { companyId, runId: before.id })); } + finally { capture.flush(); } + const summary = events.find(event => event.event === "paperclip.db_response_capture.summary.v1")!; + expect(summary.completedQueries).toBe(1); + expect(summary.responseValueBytes).toBeLessThan(8192); + expect(selected!.run?.contextSnapshot).toEqual(context); + expect(selected!.run?.resultJson).toEqual(evidence); + expect(JSON.stringify(events)).not.toContain(bulk.slice(0, 64)); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, before.id)))[0]).toEqual(before); + await db.update(heartbeatRuns).set({ runtimeMode: "legacy" }).where(eq(heartbeatRuns.id, before.id)); + expect((await readReleaseRun(db, { companyId, runId: before.id })).run?.contextSnapshot).toEqual(before.contextSnapshot); + }); + + it.each([null, ["paperclipWake", { savedInput: "keep" }], "scalar", 42].map(context => ({ context })))( + "preserves non-object native context $context", async ({ context }) => { + const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId, runtimeMode: "native", + contextSnapshot: context as Record | null }).returning(); + expect((await readReleaseRun(db, { companyId, runId: run.id })).run?.contextSnapshot).toEqual(context); + }, + ); +}); diff --git a/server/src/modules/wake-queue/adapters/release-run-read.ts b/server/src/modules/wake-queue/adapters/release-run-read.ts new file mode 100644 index 00000000000..3fa7371bc1a --- /dev/null +++ b/server/src/modules/wake-queue/adapters/release-run-read.ts @@ -0,0 +1,29 @@ +import { heartbeatRuns, type Db } from "@paperclipai/db"; +import { and, eq, getTableColumns, sql } from "drizzle-orm"; + +type Run = typeof heartbeatRuns.$inferSelect; +export type ReleaseRun = Omit; + +export async function readReleaseRun(db: Db, scope: { companyId: string; runId: string }) { + const [selected] = await db.select({ + ...getTableColumns(heartbeatRuns), + contextSnapshot: sql`case when ${heartbeatRuns.runtimeMode} = 'native' + and jsonb_typeof(${heartbeatRuns.contextSnapshot}) = 'object' + then ${heartbeatRuns.contextSnapshot} - array['executionContinuation', 'paperclipWake', 'paperclipWakeComment', + 'paperclipTaskMarkdown', 'paperclipTaskMarkdownCompact', 'paperclipTaskMarkdownAssignment', + 'paperclipTaskMarkdownAssignmentCompact', 'paperclipWorkspace', 'paperclipEnvironment', + 'paperclipContinuationSummary'] + else ${heartbeatRuns.contextSnapshot} end`, + runnerProfileJson: sql`case when ${heartbeatRuns.runtimeMode} = 'native' + then null else ${heartbeatRuns.runnerProfileJson} end`, + stdoutExcerpt: sql`case when ${heartbeatRuns.runtimeMode} = 'native' + then null else ${heartbeatRuns.stdoutExcerpt} end`, + stderrExcerpt: sql`case when ${heartbeatRuns.runtimeMode} = 'native' + then null else ${heartbeatRuns.stderrExcerpt} end`, + resultJson: sql`case when jsonb_typeof(${heartbeatRuns.resultJson}) = 'object' + then ${heartbeatRuns.resultJson} - 'stdout' - 'stderr' else ${heartbeatRuns.resultJson} end`, + }).from(heartbeatRuns).where(and(eq(heartbeatRuns.id, scope.runId), eq(heartbeatRuns.companyId, scope.companyId))); + if (!selected || selected.runtimeMode !== "native") return { kind: "complete" as const, run: selected }; + const { runnerProfileJson: _profile, stdoutExcerpt: _stdout, stderrExcerpt: _stderr, ...run } = selected; + return { kind: "native" as const, run }; +} diff --git a/server/src/realtime/live-events-ws.ts b/server/src/realtime/live-events-ws.ts index 6f299bfe672..95159016122 100644 --- a/server/src/realtime/live-events-ws.ts +++ b/server/src/realtime/live-events-ws.ts @@ -5,7 +5,7 @@ import { createRequire } from "node:module"; import type { Duplex } from "node:stream"; import { and, eq, isNull } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; -import { agentApiKeys, companyMemberships, instanceUserRoles, heartbeatRuns, issues, projects } from "@paperclipai/db"; +import { agentApiKeys, companyMemberships, instanceUserRoles, heartbeatRuns, projects } from "@paperclipai/db"; import type { DeploymentMode, LiveEvent } from "@paperclipai/shared"; import type { BetterAuthSessionResult } from "../auth/better-auth.js"; import { logger } from "../middleware/logger.js"; @@ -45,7 +45,7 @@ const { WebSocket, WebSocketServer } = require("ws") as { }; import { authorizationService, type AuthorizationActor } from "../services/authorization.js"; -import { canActorReadHeartbeatRun } from "../services/heartbeat-run-privacy.js"; +import { getHeartbeatRunReadDecisions } from "../services/heartbeat-run-privacy.js"; import { canActorReadCase } from "../services/case-access.js"; interface UpgradeContext { @@ -301,12 +301,22 @@ export function setupLiveEventsWebSocketServer( const payload = event.payload; const issueId = typeof payload.issueId === "string" ? payload.issueId : payload.entityType === "issue" && typeof payload.entityId === "string" ? payload.entityId : null; + const runId = typeof payload.runId === "string" ? payload.runId + : payload.entityType === "heartbeat_run" && typeof payload.entityId === "string" ? payload.entityId : null; + const readBindings: Parameters[3][number][] = []; if (issueId) { - const [issue] = await db.select({ id: issues.id }).from(issues) - .where(and(eq(issues.id, issueId), eq(issues.companyId, event.companyId))).limit(1); - if (!issue || !(await access.decide({ actor: context!.actor, action: "issue:read", - resource: { type: "issue", companyId: event.companyId, issueId } })).allowed) return null; + readBindings.push({ companyId: event.companyId, scopeKind: "issue", issueId }); } + if (runId) { + const [run] = await db.select({ + companyId: heartbeatRuns.companyId, + scopeKind: heartbeatRuns.scopeKind, + issueId: heartbeatRuns.issueId, + }).from(heartbeatRuns).where(and(eq(heartbeatRuns.id, runId), eq(heartbeatRuns.companyId, event.companyId))); + if (!run) return null; + readBindings.push(run); + } + if ((await getHeartbeatRunReadDecisions(db, access, context!.actor, readBindings)).some(allowed => !allowed)) return null; if (["company_skill", "company_skill_source"].includes(String(payload.entityType)) && typeof payload.entityId === "string") { try { const skills = skillAccessService(db, context!.actor); @@ -320,16 +330,6 @@ export function setupLiveEventsWebSocketServer( if (!project || !(await access.decide({ actor: context!.actor, action: "project:read", resource: { type: "project", companyId: event.companyId, projectId: project.id } })).allowed) return null; } - const runId = typeof payload.runId === "string" ? payload.runId - : payload.entityType === "heartbeat_run" && typeof payload.entityId === "string" ? payload.entityId : null; - if (runId) { - const [run] = await db.select({ - companyId: heartbeatRuns.companyId, - scopeKind: heartbeatRuns.scopeKind, - issueId: heartbeatRuns.issueId, - }).from(heartbeatRuns).where(and(eq(heartbeatRuns.id, runId), eq(heartbeatRuns.companyId, event.companyId))); - if (!run || !(await canActorReadHeartbeatRun(db, access, context!.actor, run))) return null; - } if (event.type.startsWith("heartbeat.run.")) return runId ? event : null; if (event.type === "agent.session.goal.changed") return issueId ? event : null; // Company-wide invalidations carry no task-derived content. Authorized diff --git a/server/src/routes/activity.ts b/server/src/routes/activity.ts index 5239562f7a6..fd46f3d1208 100644 --- a/server/src/routes/activity.ts +++ b/server/src/routes/activity.ts @@ -355,7 +355,15 @@ export function activityRoutes(db: Db) { const issue = await getAccessibleResource(req, res, resolveIssueByRef(rawId), "Issue not found"); if (!issue) return; if (!(await assertIssueReadAllowed(req, res, issue))) return; - const result = await svc.forIssue(issue.id); + if (req.query.view === "comment-inputs") { + const parsed = z.array(z.string().uuid()).max(100).safeParse( + typeof req.query.commentIds === "string" && req.query.commentIds ? req.query.commentIds.split(",") : [], + ); + if (!parsed.success) throw badRequest("commentIds must contain at most 100 UUIDs"); + res.json(await svc.inputFactsForIssue(issue.companyId, issue.id, parsed.data)); + return; + } + const result = await (req.query.view === "recent" ? svc.forIssue(issue.id, 50) : svc.forIssue(issue.id)); res.json(result); }); @@ -364,8 +372,10 @@ export function activityRoutes(db: Db) { const issue = await getAccessibleResource(req, res, resolveIssueByRef(rawId), "Issue not found"); if (!issue) return; if (!(await assertIssueReadAllowed(req, res, issue))) return; - const result = await svc.runsForIssue(issue.companyId, issue.id, - req.actor.type === "board" ? getActorInfo(req).actorId : null); + const viewerUserId = req.actor.type === "board" ? getActorInfo(req).actorId : null; + const result = await (req.query.view === "recent" + ? svc.runsForIssue(issue.companyId, issue.id, viewerUserId, 2) + : svc.runsForIssue(issue.companyId, issue.id, viewerUserId)); res.json(result); }); diff --git a/server/src/routes/agent-project-access.ts b/server/src/routes/agent-project-access.ts new file mode 100644 index 00000000000..08346180aab --- /dev/null +++ b/server/src/routes/agent-project-access.ts @@ -0,0 +1,25 @@ +import type { Router } from "express"; +import { agents, type Db } from "@paperclipai/db"; +import { eq } from "drizzle-orm"; +import { addAgentProjectAccessSchema } from "@paperclipai/shared"; +import { notFound } from "../errors.js"; +import { validate } from "../middleware/validate.js"; +import { addAgentProjectAccess, readAgentProjectAccess } from "../services/agent-project-access.js"; +import { assertBoard, assertCompanyAccess, hasCompanyAccess } from "./authz.js"; + +export function registerAgentProjectAccessRoutes(router: Router, db: Db) { + router.get("/agents/:id/project-access", async (req, res) => { + assertBoard(req); + const [agent] = await db.select({ id: agents.id, companyId: agents.companyId }).from(agents).where(eq(agents.id, req.params.id as string)); + if (!agent || !hasCompanyAccess(req, agent.companyId)) throw notFound("Agent not found"); + assertCompanyAccess(req, agent.companyId); + res.json(await readAgentProjectAccess(db, req.actor, agent.companyId, agent.id)); + }); + router.post("/agents/:id/project-access", validate(addAgentProjectAccessSchema), async (req, res) => { + assertBoard(req); + const [agent] = await db.select({ id: agents.id, companyId: agents.companyId }).from(agents).where(eq(agents.id, req.params.id as string)); + if (!agent || !hasCompanyAccess(req, agent.companyId)) throw notFound("Agent not found"); + assertCompanyAccess(req, agent.companyId); + res.json(await addAgentProjectAccess(db, req.actor, agent.companyId, agent.id, req.body.projectIds)); + }); +} diff --git a/server/src/routes/agents.ts b/server/src/routes/agents.ts index 540e1f7d8cc..7351ab8fb20 100644 --- a/server/src/routes/agents.ts +++ b/server/src/routes/agents.ts @@ -98,6 +98,7 @@ import { } from "@paperclipai/adapter-utils/server-utils"; import { trackAgentCreated } from "@paperclipai/shared/telemetry"; import { validate } from "../middleware/validate.js"; +import { registerAgentProjectAccessRoutes } from "./agent-project-access.js"; import { inheritNativeRunnerAdapterConfig } from "../services/native-runtime/native-agent-runtime-inheritance.js"; import { agentInstructionRevisionService } from "../services/agent-instruction-revisions.js"; import { agentInstructionWorkingCopyService } from "../services/agent-instruction-working-copies.js"; @@ -4918,6 +4919,8 @@ export function agentRoutes( res.status(201).json({ ...redactAgentRowForResponse(agent), ...(configurationWarnings.length ? { configurationWarnings } : {}) }); }); + registerAgentProjectAccessRoutes(router, db); + router.patch("/agents/:id/permissions", validate(updateAgentPermissionsSchema), async (req, res) => { const id = req.params.id as string; const existing = await getAccessibleResource(req, res, svc.getById(id), "Agent not found"); @@ -7555,7 +7558,11 @@ export function agentRoutes( const afterSeq = Number(req.query.afterSeq ?? 0); const limit = Number(req.query.limit ?? 200); - const events = await heartbeat.listEvents(runId, Number.isFinite(afterSeq) ? afterSeq : 0, Number.isFinite(limit) ? limit : 200); + const cursor = Number.isFinite(afterSeq) ? afterSeq : 0; + const pageLimit = Number.isFinite(limit) ? limit : 200; + const events = await (req.query.view === "conversation" + ? heartbeat.listEvents(runId, cursor, pageLimit, true) + : heartbeat.listEvents(runId, cursor, pageLimit)); const currentUserRedactionOptions = await getCurrentUserRedactionOptions(); const redactedEvents = events.map((event) => redactCurrentUserValue({ @@ -7575,24 +7582,32 @@ export function agentRoutes( const offset = Number(req.query.offset ?? 0); const limitBytes = readRunLogLimitBytes(req.query.limitBytes); - const result = await heartbeat.readLog(run, { - offset: Number.isFinite(offset) ? offset : 0, - limitBytes, - }); + const controller = new AbortController(); + const cancelRead = () => controller.abort(); + res.once("close", cancelRead); + try { + const result = await heartbeat.readLog(run, { + offset: Number.isFinite(offset) ? offset : 0, + limitBytes, + ...(req.query.alignStartToRecord === "true" ? { alignStartToRecord: true } : {}), + signal: controller.signal, + }); - res.set("Cache-Control", "no-cache, no-store"); - res.json(await runRedactions.redactForRun(run.companyId, run.id, result)); + res.set("Cache-Control", "no-cache, no-store"); + res.json(await runRedactions.redactForRun(run.companyId, run.id, result)); + } finally { + res.off("close", cancelRead); + } }); router.get("/heartbeat-runs/:runId/workspace-operations", async (req, res) => { const runId = readHeartbeatRunId(req); - const run = await getAccessibleResource(req, res, heartbeat.getRun(runId), "Heartbeat run not found"); + const run = await getAccessibleResource(req, res, heartbeat.getRunWorkspaceAccess(runId), "Heartbeat run not found"); if (!run) return; if (!(await assertRunTelemetryReadAllowed(req, res, run.companyId))) return; if (!(await assertRunReadAllowed(req, res, run))) return; - const context = asRecord(run.contextSnapshot); - const executionWorkspaceId = asNonEmptyString(context?.executionWorkspaceId); + const executionWorkspaceId = asNonEmptyString(run.executionWorkspaceId); const operations = await workspaceOperations.listForRun(runId, executionWorkspaceId); const visibleOperations = []; for (const operation of operations) { diff --git a/server/src/routes/issue-tree-control.ts b/server/src/routes/issue-tree-control.ts index c9a00491ba3..05c208756e9 100644 --- a/server/src/routes/issue-tree-control.ts +++ b/server/src/routes/issue-tree-control.ts @@ -6,6 +6,7 @@ import { } from "@paperclipai/db"; import { and, eq, inArray, isNotNull } from "drizzle-orm"; import { getExecutionBlocker } from "../services/execution-blocker.js"; +import { ISSUE_TREE_RESUME_EXECUTABLE_STATUSES } from "../services/issue-tree-control.js"; import { conflict } from "../errors.js"; import { createIssueTreeHoldSchema, @@ -26,7 +27,6 @@ import { assertBoard, getAccessibleResource, getActorInfo } from "./authz.js"; import type { PluginWorkerManager } from "../services/plugin-worker-manager.js"; const TREE_RUN_CANCELLATION_RESPONSE_WAIT_MS = 1_000; -const RESUME_EXECUTABLE_STATUSES = ["todo", "in_progress", "in_review"]; function errorToMessage(error: unknown) { return error instanceof Error ? error.message : String(error); @@ -436,15 +436,13 @@ export function issueTreeControlRoutes( const activeHold = await treeControlSvc.getHold(root.companyId, holdId); const issueIds = activeHold?.mode === "pause" && activeHold.rootIssueId === root.id - ? (activeHold.members ?? []) - .filter((member) => !member.skipped) - .map((member) => member.issueId) + ? (activeHold.members ?? []).map((member) => member.issueId) : []; if (issueIds.length > 0) { const candidates = await db.select({ id: issueRows.id, identifier: issueRows.identifier }) .from(issueRows).where(and( eq(issueRows.companyId, root.companyId), inArray(issueRows.id, issueIds), - inArray(issueRows.status, RESUME_EXECUTABLE_STATUSES), isNotNull(issueRows.assigneeAgentId), + inArray(issueRows.status, ISSUE_TREE_RESUME_EXECUTABLE_STATUSES), isNotNull(issueRows.assigneeAgentId), )); for (const task of candidates) { const blocked = await getExecutionBlocker(db, root.companyId, task.id); @@ -489,16 +487,18 @@ export function issueTreeControlRoutes( const wakeFailures: Array<{ issueId: string; message: string }> = []; if (hold.mode === "pause" && req.body.metadata?.wakeAgents === true) { for (const member of hold.members ?? []) { - if (member.skipped) continue; try { const issue = await issuesSvc.getById(member.issueId); if ( !issue || issue.companyId !== root.companyId || !issue.assigneeAgentId || - !RESUME_EXECUTABLE_STATUSES.includes(issue.status) + !ISSUE_TREE_RESUME_EXECUTABLE_STATUSES.includes(issue.status) ) continue; + // The snapshot may predate reopening a task or releasing another + // pause. Current task state and remaining holds govern execution. + if (await treeControlSvc.getActivePauseHoldGate(root.companyId, issue.id)) continue; await heartbeat.wakeup(issue.assigneeAgentId, { source: "assignment", triggerDetail: "system", diff --git a/server/src/routes/issues.ts b/server/src/routes/issues.ts index 69398e41a4c..842df00a10d 100644 --- a/server/src/routes/issues.ts +++ b/server/src/routes/issues.ts @@ -3190,6 +3190,42 @@ function decrementIssueListActorClientInflight(actorClientKey: string) { else issueListActorClientInflight.set(actorClientKey, next); } +const ISSUE_LIST_UUID_QUERY_PARAMS = [ + "participantAgentId", + "projectId", + "workspaceId", + "executionWorkspaceId", + "parentId", + "parentIssueId", + "descendantOf", + "createdFromIssueId", + "labelId", +] as const; + +type IssueListUuidQueryParam = (typeof ISSUE_LIST_UUID_QUERY_PARAMS)[number]; + +function parseIssueListUuidQueryParams( + query: Record, +): + | { ok: true; values: Partial> } + | { ok: false; error: string } { + const values: Partial> = {}; + for (const name of ISSUE_LIST_UUID_QUERY_PARAMS) { + const raw = query[name]; + if (raw === undefined) continue; + if (typeof raw !== "string") { + return { ok: false, error: `${name} must be a UUID` }; + } + const trimmed = raw.trim(); + if (trimmed.length === 0) continue; + if (!isUuidLike(trimmed)) { + return { ok: false, error: `${name} must be a UUID` }; + } + values[name] = trimmed; + } + return { ok: true, values }; +} + async function coordinateIssueListGet(input: { req: Request; companyId: string; @@ -7211,18 +7247,20 @@ export function issueRoutes( id: string; assigneeAgentId: string | null; executionRunId?: string | null; - }) { + }, options?: { queueStateOnly: true }) { let runToInterrupt = issue.executionRunId - ? await heartbeat.getRun(issue.executionRunId) + ? options?.queueStateOnly + ? await heartbeat.getRunIssueControl(issue.executionRunId) + : await heartbeat.getRun(issue.executionRunId) : null; if ( (!runToInterrupt || runToInterrupt.status !== "running") && issue.assigneeAgentId ) { - const activeRun = await heartbeat.getActiveRunForAgent( - issue.assigneeAgentId, - ); + const activeRun = options?.queueStateOnly + ? await heartbeat.getActiveRunIssueControlForAgent(issue.assigneeAgentId) + : await heartbeat.getActiveRunForAgent(issue.assigneeAgentId); const activeIssueId = activeRun && activeRun.contextSnapshot && @@ -7249,9 +7287,9 @@ export function issueRoutes( type IssueQueueWake = typeof agentWakeupRequests.$inferSelect; type IssueQueueRun = typeof heartbeatRuns.$inferSelect; type IssueQueueState = { - wake: IssueQueueWake; + wake: Pick; state: "deferred" | "queued"; - queueRun: IssueQueueRun | null; + queueRun: Pick | null; }; async function findQueuedCommentWake( @@ -7260,12 +7298,29 @@ export function issueRoutes( ): Promise { if (!issue.assigneeAgentId) return null; const rows = await executor - .select() + .select({ + id: agentWakeupRequests.id, companyId: agentWakeupRequests.companyId, + status: agentWakeupRequests.status, runId: agentWakeupRequests.runId, + payload: sql>`jsonb_build_object( + 'issueId', ${agentWakeupRequests.payload}->'issueId', + 'mutation', ${agentWakeupRequests.payload}->'mutation', + 'interactionId', ${agentWakeupRequests.payload}->'interactionId', + 'interactionStatus', ${agentWakeupRequests.payload}->'interactionStatus', + 'externalChatContinuation', ${agentWakeupRequests.payload}->'externalChatContinuation', + 'executionWait', jsonb_build_object( + 'reason', ${agentWakeupRequests.payload}->'executionWait'->'reason', + 'message', ${agentWakeupRequests.payload}->'executionWait'->'message'), + '_paperclipWakeContext', jsonb_build_object( + 'wakeCommentIds', ${agentWakeupRequests.payload}->'_paperclipWakeContext'->'wakeCommentIds', + 'forceFreshSession', ${agentWakeupRequests.payload}->'_paperclipWakeContext'->'forceFreshSession') + )`, + }) .from(agentWakeupRequests) .where( and( eq(agentWakeupRequests.companyId, issue.companyId), eq(agentWakeupRequests.agentId, issue.assigneeAgentId), + sql`${agentWakeupRequests.payload}->'issueId' = to_jsonb(${issue.id}::text)`, inArray(agentWakeupRequests.status, [ "deferred_issue_execution", "queued", @@ -7285,7 +7340,7 @@ export function issueRoutes( } if (!wake.runId) continue; const queueRun = await executor - .select() + .select({ runtimeMode: heartbeatRuns.runtimeMode }) .from(heartbeatRuns) .where( and( @@ -7306,7 +7361,7 @@ export function issueRoutes( async function queueCommentsForWake( executor: IssueQueueDb, issueId: string, - wake: IssueQueueWake | null, + wake: Pick | null, ) { if (wake && queuedInteractionId(wake.payload)) { const response = await readQueuedInteractionResponse(executor as Db, wake.companyId, issueId, wake.payload); @@ -8445,6 +8500,12 @@ export function issueRoutes( }); return; } + const uuidQuery = parseIssueListUuidQueryParams(req.query); + if (!uuidQuery.ok) { + res.status(422).json({ error: uuidQuery.error }); + return; + } + const uuidFilters = uuidQuery.values; const offset = parsedOffset ?? 0; const listFilters: IssueFilters = { @@ -8452,20 +8513,18 @@ export function issueRoutes( attention: attention === "blocked" ? "blocked" : undefined, status: req.query.status as string | string[] | undefined, assigneeAgentId, - participantAgentId: req.query.participantAgentId as string | undefined, + participantAgentId: uuidFilters.participantAgentId, assigneeUserId, touchedByUserId, inboxArchivedByUserId, unreadForUserId, - projectId: req.query.projectId as string | undefined, - workspaceId: req.query.workspaceId as string | undefined, - executionWorkspaceId: req.query.executionWorkspaceId as - string | undefined, - parentId: (req.query.parentId ?? req.query.parentIssueId) as - string | undefined, - descendantOf: req.query.descendantOf as string | undefined, - createdFromIssueId: req.query.createdFromIssueId as string | undefined, - labelId: req.query.labelId as string | undefined, + projectId: uuidFilters.projectId, + workspaceId: uuidFilters.workspaceId, + executionWorkspaceId: uuidFilters.executionWorkspaceId, + parentId: uuidFilters.parentId ?? uuidFilters.parentIssueId, + descendantOf: uuidFilters.descendantOf, + createdFromIssueId: uuidFilters.createdFromIssueId, + labelId: uuidFilters.labelId, originKind: req.query.originKind as string | undefined, originKindPrefix: req.query.originKindPrefix as string | undefined, originId: req.query.originId as string | undefined, @@ -8686,22 +8745,27 @@ export function issueRoutes( return; } + const countUuidQuery = parseIssueListUuidQueryParams(req.query); + if (!countUuidQuery.ok) { + res.status(422).json({ error: countUuidQuery.error }); + return; + } + const countUuidFilters = countUuidQuery.values; + const blockedCountFilters = { readCondition: await issueReadSqlCondition(db, req.actor), attention: "blocked", status: req.query.status as string | string[] | undefined, assigneeAgentId: req.query.assigneeAgentId as string | undefined, - participantAgentId: req.query.participantAgentId as string | undefined, + participantAgentId: countUuidFilters.participantAgentId, assigneeUserId: req.query.assigneeUserId as string | undefined, - projectId: req.query.projectId as string | undefined, - workspaceId: req.query.workspaceId as string | undefined, - executionWorkspaceId: req.query.executionWorkspaceId as - string | undefined, - parentId: (req.query.parentId ?? req.query.parentIssueId) as - string | undefined, - descendantOf: req.query.descendantOf as string | undefined, - createdFromIssueId: req.query.createdFromIssueId as string | undefined, - labelId: req.query.labelId as string | undefined, + projectId: countUuidFilters.projectId, + workspaceId: countUuidFilters.workspaceId, + executionWorkspaceId: countUuidFilters.executionWorkspaceId, + parentId: countUuidFilters.parentId ?? countUuidFilters.parentIssueId, + descendantOf: countUuidFilters.descendantOf, + createdFromIssueId: countUuidFilters.createdFromIssueId, + labelId: countUuidFilters.labelId, originKind: req.query.originKind as string | undefined, originKindPrefix: req.query.originKindPrefix as string | undefined, originId: req.query.originId as string | undefined, @@ -9253,6 +9317,13 @@ export function issueRoutes( ); if (!issue) return; if (!(await timing.time("authorization", () => assertIssueReadAllowed(req, res, issue, { allowBreakGlass: true })))) return; + // The cold-open read owns only the task row. Related objects and recovery + // projections retain their existing authorized detail path. + if (req.query.view === "core") { + res.setHeader("Server-Timing", timing.header()); + res.json(issue); + return; + } const inboxArchiveFieldsPromise = req.actor.type === "board" && req.actor.userId ? timing.time("inbox", () => svc.getActiveInboxArchiveFields(issue, req.actor.userId!)) @@ -16210,6 +16281,24 @@ export function issueRoutes( res.json(result); }); + router.get("/issues/:id/recent-comments", async (req, res) => { + const issue = await getAccessibleResource(req, res, getIssueById(req, req.params.id as string), "Issue not found"); + if (!issue || !(await assertIssueReadAllowed(req, res, issue))) return; + const [recent, human, firstInstruction] = await Promise.all([ + svc.listComments(issue.id, { order: "desc", limit: 5 }), + svc.listComments(issue.id, { order: "desc", limit: 1, humanOnly: true }), + issue.description?.trim() + ? Promise.resolve([]) + : svc.listComments(issue.id, { order: "asc", limit: 1, humanOnly: true }), + ]); + const comments = [...new Map([...firstInstruction, ...human, ...recent.slice(0, 4)].map((comment) => [comment.id, comment])).values()]; + res.json(await runRedactions.redactForIssue(issue.companyId, issue.id, { + comments, + latestCommentId: recent[0]?.id ?? null, + hasMore: recent.length > 4, + })); + }); + router.get("/issues/:id/comments", async (req, res) => { const id = req.params.id as string; const issue = await getAccessibleResource( @@ -16298,7 +16387,7 @@ export function issueRoutes( const queue = await buildQueuedCommentQueue({ executor: db, issue, - activeRun: await resolveActiveIssueRun(issue), + activeRun: await resolveActiveIssueRun(issue, { queueStateOnly: true }), actor: getActorInfo(req), }); res.json( diff --git a/server/src/routes/openapi.ts b/server/src/routes/openapi.ts index 650eb0670eb..3946cac4d74 100644 --- a/server/src/routes/openapi.ts +++ b/server/src/routes/openapi.ts @@ -48,6 +48,7 @@ import { updateAgentSchema, updatePrimaryAgentSchema, updateAgentPermissionsSchema, + addAgentProjectAccessSchema, updateAgentInstructionsPathSchema, updateAgentInstructionsBundleSchema, upsertAgentInstructionsFileSchema, @@ -1390,6 +1391,8 @@ const browserUseOperations = [ ] as const; const BOARD_ONLY_OPERATIONS = new Set([ + "GET /api/agents/{id}/project-access", + "POST /api/agents/{id}/project-access", "POST /api/issues/{id}/interactions/{interactionId}/dismiss", "POST /api/issues/{id}/retry", "GET /api/companies/{companyId}/prompt-injection-prevention", @@ -3790,6 +3793,26 @@ registry.registerPath({ responses: { 200: r.ok(), 400: r.badRequest, 401: r.unauthorized }, }); +const agentProjectAccessResponse = z.object({ projectIds: z.array(z.string().uuid()), canManage: z.boolean() }); +registry.registerPath({ + method: "get", + path: "/api/agents/{id}/project-access", + tags: ["agents"], + summary: "Read an agent's allowed projects and the current operator's management capability", + request: { params: z.object({ id: z.string() }) }, + responses: { 200: r.ok(agentProjectAccessResponse), 401: r.unauthorized, 403: r.forbidden, 404: r.notFound }, +}); +registry.registerPath({ + method: "post", + path: "/api/agents/{id}/project-access", + tags: ["agents"], + summary: "Add allowed projects without replacing an agent's other permissions", + description: "Requires board access and agents:create in the agent's company. Accepts 1 to 100 project IDs. Selected projects must be readable and in that company. Locks the low-trust agent, preserves existing containment, audits new additions, and rejects pending-approval agents or changed trust settings. Repeated additions are idempotent. Saving does not start or retry a task.", + request: { params: z.object({ id: z.string() }), body: jsonBody(addAgentProjectAccessSchema) }, + responses: { 200: r.ok(agentProjectAccessResponse), 400: r.badRequest, 401: r.unauthorized, + 403: r.forbidden, 404: r.notFound, 409: r.conflict }, +}); + registry.registerPath({ method: "patch", path: "/api/agents/{id}/instructions-path", @@ -4294,7 +4317,7 @@ registry.registerPath({ path: "/api/issues/{id}", tags: ["issues"], summary: "Get an issue", - request: { params: z.object({ id: z.string() }) }, + request: { params: z.object({ id: z.string() }), query: z.object({ view: z.literal("core").optional().describe("Core task fields without related enrichment.") }) }, responses: { 200: r.ok(), 401: r.unauthorized, 404: r.notFound }, }); @@ -4601,6 +4624,16 @@ registry.registerPath({ responses: { 200: r.ok(), 401: r.unauthorized }, }); +registry.registerPath({ + method: "get", + path: "/api/issues/{id}/recent-comments", + tags: ["issues"], + summary: "Read a bounded recent task exchange", + description: "Returns four recent comments, the latest human comment, and the first human instruction when the task has no description. Includes latestCommentId and hasMore. Uses the same authorization and redaction as full comment reads.", + request: { params: z.object({ id: z.string() }) }, + responses: { 200: r.ok(), 401: r.unauthorized, 403: r.forbidden, 404: r.notFound }, +}); + registry.registerPath({ method: "get", path: "/api/issues/{id}/comments", @@ -6057,8 +6090,11 @@ registry.registerPath({ path: "/api/issues/{id}/activity", tags: ["activity"], summary: "List activity for an issue", - request: { params: z.object({ id: z.string() }) }, - responses: { 200: r.ok(), 401: r.unauthorized }, + request: { params: z.object({ id: z.string() }), query: z.object({ + view: z.enum(["recent", "comment-inputs"]).optional().describe("Recent activity (at most 50 rows), or original comment/steering facts for selected comments."), + commentIds: z.string().optional().describe("For comment-inputs: up to 100 comma-separated comment UUIDs. Returns at most two original facts per comment."), + }) }, + responses: { 200: r.ok(), 400: r.badRequest, 401: r.unauthorized }, }); registry.registerPath({ @@ -6066,7 +6102,7 @@ registry.registerPath({ path: "/api/issues/{id}/runs", tags: ["activity"], summary: "List runs for an issue", - request: { params: z.object({ id: z.string() }) }, + request: { params: z.object({ id: z.string() }), query: z.object({ view: z.literal("recent").optional().describe("At most two recent run summaries.") }) }, responses: { 200: r.ok(), 401: r.unauthorized }, }); @@ -7798,7 +7834,11 @@ registry.registerPath({ path: "/api/heartbeat-runs/{runId}/events", tags: ["runs"], summary: "Get events for a heartbeat run", - request: { params: z.object({ runId: heartbeatRunIdParamSchema }) }, + request: { params: z.object({ runId: heartbeatRunIdParamSchema }), query: z.object({ + view: z.literal("conversation").optional().describe("Assistant text and request lifecycle events, excluding historical tool activity."), + afterSeq: z.coerce.number().int().min(0).optional(), + limit: z.coerce.number().int().min(1).max(1000).optional(), + }) }, responses: { 200: r.ok(), 400: r.badRequest, 401: r.unauthorized }, }); @@ -7807,7 +7847,14 @@ registry.registerPath({ path: "/api/heartbeat-runs/{runId}/log", tags: ["runs"], summary: "Get log for a heartbeat run", - request: { params: z.object({ runId: heartbeatRunIdParamSchema }) }, + request: { + params: z.object({ runId: heartbeatRunIdParamSchema }), + query: z.object({ + offset: z.coerce.number().int().min(0).optional(), + limitBytes: z.coerce.number().int().min(1).max(1024 * 1024).optional(), + alignStartToRecord: z.enum(["true", "false"]).optional().describe("Extend the tail backward to include its first complete log record. The response can exceed limitBytes by that record's prefix."), + }), + }, responses: { 200: r.ok(), 400: r.badRequest, 401: r.unauthorized }, }); diff --git a/server/src/services/access.ts b/server/src/services/access.ts index fced6f2c698..d7ae99b09a7 100644 --- a/server/src/services/access.ts +++ b/server/src/services/access.ts @@ -1162,6 +1162,7 @@ export function accessService(db: Db) { return { isInstanceAdmin, decide, + decideReads: authorization.decideReads, projectDiscoveryCandidateIds: authorization.projectDiscoveryCandidateIds, canUser, hasPermission, diff --git a/server/src/services/acknowledged-native-stop.ts b/server/src/services/acknowledged-native-stop.ts index bb6b7fa74c3..9d07f7b61a7 100644 --- a/server/src/services/acknowledged-native-stop.ts +++ b/server/src/services/acknowledged-native-stop.ts @@ -37,7 +37,8 @@ export function isAcknowledgedNativeStop(run: Parameters) { if (!isAcknowledgedNativeStop(run)) return false; const leases = await db.select().from(environmentLeases).where(and( eq(environmentLeases.companyId, run.companyId), eq(environmentLeases.heartbeatRunId, run.id), diff --git a/server/src/services/activity.ts b/server/src/services/activity.ts index d4f6b03983e..60ef56c4e84 100644 --- a/server/src/services/activity.ts +++ b/server/src/services/activity.ts @@ -396,8 +396,24 @@ export function activityService(db: Db) { .then((rows) => rows.map((r) => r.activityLog)); }, - forIssue: (issueId: string) => - db + inputFactsForIssue: async (companyId: string, issueId: string, commentIds: string[]) => { + if (!commentIds.length) return []; + const commentId = sql`${activityLog.details}->>'commentId'`; + return db.selectDistinctOn([commentId, activityLog.action]) + .from(activityLog) + .where(and( + eq(activityLog.companyId, companyId), + eq(activityLog.entityType, "issue"), + eq(activityLog.entityId, issueId), + inArray(commentId, commentIds), + inArray(activityLog.action, ["issue.queued_comment_steered", "issue.comment_added"]), + sql`${activityLog.details}->>'duplicate' is distinct from 'true'`, + )) + .orderBy(commentId, activityLog.action, asc(activityLog.createdAt), asc(activityLog.id)); + }, + + forIssue: (issueId: string, limit?: number) => { + const query = db .select() .from(activityLog) .where( @@ -407,11 +423,13 @@ export function activityService(db: Db) { sql`${activityLog.companyId} = (select company_id from issues where id = ${issueId})`), ), ) - .orderBy(desc(activityLog.createdAt)), + .orderBy(desc(activityLog.createdAt), desc(activityLog.id)); + return limit === undefined ? query : query.limit(limit); + }, - runsForIssue: async (companyId: string, issueId: string, viewerUserId: string | null = null) => { + runsForIssue: async (companyId: string, issueId: string, viewerUserId: string | null = null, limit?: number) => { scheduleRunLivenessBackfill(companyId, issueId); - const runs = await db + const runsQuery = db .select({ runId: heartbeatRuns.id, runtimeMode: heartbeatRuns.runtimeMode, @@ -473,7 +491,8 @@ export function activityService(db: Db) { ), ), ) - .orderBy(desc(heartbeatRuns.createdAt)); + .orderBy(desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id)); + const runs = await (limit === undefined ? runsQuery : runsQuery.limit(limit)); if (runs.length === 0) return runs; const runIds = runs.map((run) => run.runId); diff --git a/server/src/services/agent-conversations.ts b/server/src/services/agent-conversations.ts index a2ec8f74b01..b6cfd1a7337 100644 --- a/server/src/services/agent-conversations.ts +++ b/server/src/services/agent-conversations.ts @@ -282,7 +282,7 @@ export async function prepareConversationTurn( /** Finalizers only park a turn with a durable response, interaction, or processed /new. */ export async function settleConversationTurn( db: Db, - run: typeof heartbeatRuns.$inferSelect, + run: Pick, ) { if (run.status !== "succeeded") return false; const context = run.contextSnapshot ?? {}; @@ -291,7 +291,12 @@ export async function settleConversationTurn( let publication: ActivityPublication | null = null; const settled = await db.transaction(async (tx) => { const [issue] = await tx - .select() + .select({ + id: issues.id, companyId: issues.companyId, status: issues.status, + conversationAgentId: issues.conversationAgentId, conversationUserId: issues.conversationUserId, + conversationState: issues.conversationState, conversationSessionGeneration: issues.conversationSessionGeneration, + executionRunId: issues.executionRunId, + }) .from(issues) .where(and(eq(issues.id, issueId), eq(issues.companyId, run.companyId))) .for("update"); @@ -344,7 +349,7 @@ export async function settleConversationTurn( const [wake] = typeof wakeId === "string" ? await tx - .select() + .select({ id: issueComments.id }) .from(issueComments) .where(eq(issueComments.id, wakeId)) : []; @@ -407,6 +412,26 @@ export async function settleConversationTurn( return settled; } +export async function settleConversationTurnForRun( + db: Db, + companyId: string, + runId: string, +) { + const [run] = await db.select({ + id: heartbeatRuns.id, companyId: heartbeatRuns.companyId, status: heartbeatRuns.status, + contextSnapshot: sql | null>`case when jsonb_typeof(${heartbeatRuns.contextSnapshot}) = 'object' + then jsonb_build_object( + 'issueId', ${heartbeatRuns.contextSnapshot}->'issueId', + 'conversationReset', ${heartbeatRuns.contextSnapshot}->'conversationReset', + 'conversationSessionGeneration', ${heartbeatRuns.contextSnapshot}->'conversationSessionGeneration', + 'conversationReplyBoundaryCommentId', ${heartbeatRuns.contextSnapshot}->'conversationReplyBoundaryCommentId', + 'wakeCommentId', ${heartbeatRuns.contextSnapshot}->'wakeCommentId', + 'commentId', ${heartbeatRuns.contextSnapshot}->'commentId' + ) else null end`, + }).from(heartbeatRuns).where(and(eq(heartbeatRuns.id, runId), eq(heartbeatRuns.companyId, companyId))); + return run ? settleConversationTurn(db, run) : false; +} + /** Fresh provider context uses only messages in this session, up to this turn. */ export async function conversationReplay( db: Db, diff --git a/server/src/services/agent-project-access.ts b/server/src/services/agent-project-access.ts new file mode 100644 index 00000000000..4c6ee81659a --- /dev/null +++ b/server/src/services/agent-project-access.ts @@ -0,0 +1,69 @@ +import { and, eq, inArray } from "drizzle-orm"; +import { agents, projects, type Db } from "@paperclipai/db"; +import { trustAuthorizationPolicySchema, type AgentProjectAccess } from "@paperclipai/shared"; +import { conflict, forbidden, notFound } from "../errors.js"; +import { normalizeAgentPermissions } from "../lib/agent-permissions.js"; +import { accessService } from "./access.js"; +import type { AuthorizationActor } from "./authorization.js"; +import { logActivity, publishActivity, type ActivityPublication } from "./activity-log.js"; + +export async function readAgentProjectAccess(db: Db, actor: AuthorizationActor, companyId: string, agentId: string): Promise { + const [agent] = await db.select({ permissions: agents.permissions, status: agents.status }).from(agents) + .where(and(eq(agents.companyId, companyId), eq(agents.id, agentId))); + if (!agent) throw notFound("Agent not found"); + const decision = await accessService(db).decide({ actor, action: "agents:create", resource: { type: "company", companyId } }); + const permissions = normalizeAgentPermissions(agent.permissions); + const policy = trustAuthorizationPolicySchema.safeParse(permissions.authorizationPolicy ?? {}); + const boundary = policy.success ? policy.data.trustBoundary : undefined; + return { projectIds: policy.success ? policy.data.trustBoundary?.projectIds ?? [] : [], + canManage: actor.type === "board" && decision.allowed && agent.status !== "pending_approval" + && policy.success && permissions.trustPreset === "low_trust_review" + && (!boundary || (boundary.mode === "low_trust_review" && (!boundary.companyId || boundary.companyId === companyId))) }; +} + +/** An additive operation under the agent row lock, never a stale policy replacement. */ +export async function addAgentProjectAccess(db: Db, actor: AuthorizationActor, companyId: string, agentId: string, projectIds: string[]) { + if (actor.type !== "board") throw forbidden("Board access required"); + const publications: ActivityPublication[] = []; + const result = await db.transaction(async tx => { + const [agent] = await tx.select().from(agents) + .where(and(eq(agents.companyId, companyId), eq(agents.id, agentId))).for("update"); + if (!agent) throw notFound("Agent not found"); + const transactionDb = tx as unknown as Db; + const access = accessService(transactionDb); + const decision = await access.decide({ actor, action: "agents:create", resource: { type: "company", companyId } }); + if (!decision.allowed) throw forbidden(decision.explanation); + if (agent.status === "pending_approval") throw conflict("Pending approval agent permissions cannot be changed before board approval"); + const permissions = normalizeAgentPermissions(agent.permissions); + const parsedPolicy = trustAuthorizationPolicySchema.safeParse(permissions.authorizationPolicy ?? {}); + if (!parsedPolicy.success) throw conflict("The agent’s trust settings need review before adding project access."); + const policy = parsedPolicy.data; + const boundary = policy?.trustBoundary; + if (permissions.trustPreset !== "low_trust_review" || (boundary && (boundary.mode !== "low_trust_review" + || (boundary.companyId && boundary.companyId !== companyId)))) { + throw conflict("The agent’s trust settings changed. Refresh before adding project access."); + } + const requested = [...new Set(projectIds)]; + const rows = await tx.select({ id: projects.id }).from(projects) + .where(and(eq(projects.companyId, companyId), inArray(projects.id, requested))); + if (rows.length !== requested.length) throw notFound("A selected project is unavailable"); + for (const project of rows) { + const projectDecision = await access.decide({ actor, action: "project:read", resource: { type: "project", companyId, projectId: project.id } }); + if (!projectDecision.allowed) throw forbidden("A selected project is unavailable"); + } + const previous = boundary?.projectIds ?? []; + const added = requested.filter(id => !previous.includes(id)); + const next = [...previous, ...added]; + if (added.length) { + await tx.update(agents).set({ permissions: { ...agent.permissions, authorizationPolicy: { + ...policy, trustBoundary: { ...boundary, mode: "low_trust_review", companyId, projectIds: next }, + } }, updatedAt: new Date() }).where(and(eq(agents.id, agentId), eq(agents.companyId, companyId))); + await logActivity(transactionDb, { companyId, actorType: "user", actorId: actor.userId ?? "local-board", + action: "agent.project_access_added", entityType: "agent", entityId: agentId, + details: { addedProjectIds: added, trustPreset: "low_trust_review" } }, publications); + } + return { projectIds: next, canManage: true } satisfies AgentProjectAccess; + }); + for (const publication of publications) publishActivity(publication); + return result; +} diff --git a/server/src/services/agents.ts b/server/src/services/agents.ts index 4c4cbd7248f..c9dd824f495 100644 --- a/server/src/services/agents.ts +++ b/server/src/services/agents.ts @@ -1,6 +1,6 @@ import { type BudgetServiceHooks } from "./budgets.js"; -import { and, desc, eq, inArray, ne } from "drizzle-orm"; +import { and, desc, eq, inArray } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; import { agents, agentConfigRevisions, agentApiKeys, heartbeatRuns } from "@paperclipai/db"; import { isUuidLike, normalizeAgentApiKeyScope, normalizeAgentUrlKey, type AgentApiKeyScope } from "@paperclipai/shared"; @@ -31,14 +31,10 @@ export function agentService(db: Db, budgetHooks: BudgetServiceHooks = {}) { createAgentLifecycle(db, budgetHooks).updateConfiguration(...args); return { list: async (companyId: string, options?: { includeTerminated?: boolean }) => { - const conditions = [eq(agents.companyId, companyId)]; - if (!options?.includeTerminated) { - conditions.push(ne(agents.status, "terminated")); - } - const [rows, allCompanyRows] = await Promise.all([ - db.select().from(agents).where(and(...conditions)), - listCompanyAgentRows(companyId), - ]); + const allCompanyRows = await listCompanyAgentRows(companyId); + const rows = options?.includeTerminated + ? allCompanyRows + : allCompanyRows.filter((row) => row.status !== "terminated"); const hydrated = await hydrateAgentSpend(rows); return normalizeAgentRows(hydrated, allCompanyRows); }, diff --git a/server/src/services/attention.ts b/server/src/services/attention.ts index dd46de7d91e..90ed24f135e 100644 --- a/server/src/services/attention.ts +++ b/server/src/services/attention.ts @@ -1099,6 +1099,65 @@ function readRunIssueId(contextSnapshot: Record | null) { return typeof issueId === "string" && issueId.length > 0 ? issueId : null; } +type FailedRunProbe = { id: string; agentId: string; issueId: string | null; createdAt: Date }; + +const FAILED_RUN_PROBE_BATCH_SIZE = 1000; + +async function listFailedRunsWithNewerRun(db: Db, companyId: string, failedRuns: FailedRunProbe[]) { + const withNewerRun = new Set(); + const keyed = failedRuns.filter(run => run.issueId !== null); + for (let start = 0; start < keyed.length; start += FAILED_RUN_PROBE_BATCH_SIZE) { + const batch = keyed.slice(start, start + FAILED_RUN_PROBE_BATCH_SIZE); + for (const id of await listFailedRunBatchWithNewerRun(db, companyId, batch)) withNewerRun.add(id); + } + const unkeyed = failedRuns.filter(run => run.issueId === null); + if (unkeyed.length === 0) return withNewerRun; + const agentIds = [...new Set(unkeyed.map(run => run.agentId))]; + const oldestCreatedAt = unkeyed.reduce((oldest, run) => run.createdAt < oldest ? run.createdAt : oldest, + unkeyed[0]!.createdAt); + const latestByAgent = new Map(); + for (let start = 0; start < agentIds.length; start += FAILED_RUN_PROBE_BATCH_SIZE) { + const rows = await db.select({ + agentId: heartbeatRuns.agentId, + createdAt: sql`max(${heartbeatRuns.createdAt})`.mapWith(heartbeatRuns.createdAt), + }).from(heartbeatRuns).where(and( + eq(heartbeatRuns.companyId, companyId), + inArray(heartbeatRuns.agentId, agentIds.slice(start, start + FAILED_RUN_PROBE_BATCH_SIZE)), + gt(heartbeatRuns.createdAt, oldestCreatedAt), + sql`coalesce(${heartbeatRuns.contextSnapshot} ->> 'issueId', ${heartbeatRuns.contextSnapshot} ->> 'taskId', '') = ''`, + )).groupBy(heartbeatRuns.agentId); + for (const row of rows) latestByAgent.set(row.agentId, row.createdAt); + } + for (const run of unkeyed) { + if ((latestByAgent.get(run.agentId)?.getTime() ?? 0) > run.createdAt.getTime()) withNewerRun.add(run.id); + } + return withNewerRun; +} + +async function listFailedRunBatchWithNewerRun(db: Db, companyId: string, failedRuns: FailedRunProbe[]) { + if (failedRuns.length === 0) return []; + const failed = sql.join( + failedRuns.map((run) => sql`(${run.id}::uuid, ${run.agentId}::uuid, ${run.issueId}::text, ${new Date(run.createdAt.getTime() + 1).toISOString()}::timestamptz)`), + sql`, `, + ); + const runIssueId = sql`(${heartbeatRuns.contextSnapshot} ->> 'issueId')`; + const runTaskId = sql`(${heartbeatRuns.contextSnapshot} ->> 'taskId')`; + const newerRunExists = (match: SQL) => sql`exists ( + select 1 from ${heartbeatRuns} + where ${heartbeatRuns.companyId} = ${companyId} + and ${heartbeatRuns.agentId} = failed.agent_id + and ${heartbeatRuns.createdAt} >= failed.newer_from + and ${match} + )`; + const rows = await db.execute(sql` + select failed.id + from (values ${failed}) as failed (id, agent_id, issue_id, newer_from) + where ${newerRunExists(sql`${runIssueId} = failed.issue_id`)} + or ${newerRunExists(sql`${runTaskId} = failed.issue_id and ${runIssueId} is null`)} + `) as unknown as Array<{ id: string }>; + return rows.map((row) => row.id); +} + export function attentionService(db: Db, serviceOptions: AttentionServiceOptions = {}) { const openDecisionLimit = Math.min( Math.max(Math.trunc(serviceOptions.openDecisionLimit ?? OPEN_DECISION_DEFAULT_LIMIT), 1), @@ -1717,12 +1776,7 @@ export function attentionService(db: Db, serviceOptions: AttentionServiceOptions const failedRows = await listAttentionExhaustedRuns(db, companyId); const failedIssueIds = failedRows.map((row) => readRunIssueId(row.contextSnapshot)); - const failedAgentIds = [...new Set(failedRows.map((row) => row.agentId))]; - const oldestFailedRunCreatedAt = failedRows.reduce((oldest, row) => { - if (!oldest || row.createdAt < oldest) return row.createdAt; - return oldest; - }, null); - const [failedIssueMap, failedImageMap, newerRuns] = await Promise.all([ + const [failedIssueMap, failedImageMap, failedRunsWithNewerRun] = await Promise.all([ issueSummaryMap( db, companyId, @@ -1730,38 +1784,16 @@ export function attentionService(db: Db, serviceOptions: AttentionServiceOptions readCondition, ), issueImageMap(db, companyId, failedIssueIds, readCondition), - oldestFailedRunCreatedAt && failedAgentIds.length > 0 - ? db - .select({ - agentId: heartbeatRuns.agentId, - createdAt: heartbeatRuns.createdAt, - // Project just the ids readRunIssueId needs; pulling the whole - // context_snapshot detoasts megabytes per feed build. - runIssueId: sql`${heartbeatRuns.contextSnapshot} ->> 'issueId'`, - runTaskId: sql`${heartbeatRuns.contextSnapshot} ->> 'taskId'`, - }) - .from(heartbeatRuns) - .where(and( - eq(heartbeatRuns.companyId, companyId), - inArray(heartbeatRuns.agentId, failedAgentIds), - gt(heartbeatRuns.createdAt, oldestFailedRunCreatedAt), - )) - : Promise.resolve([]), + listFailedRunsWithNewerRun(db, companyId, failedRows.map((row, index) => ({ + id: row.id, + agentId: row.agentId, + issueId: failedIssueIds[index], + createdAt: row.createdAt, + }))), ]); - const latestRunCreatedAtByKey = new Map(); - for (const newerRun of newerRuns) { - const newerRunIssueId = readRunIssueId({ issueId: newerRun.runIssueId, taskId: newerRun.runTaskId }); - const newerRunKey = `${newerRun.agentId}:${newerRunIssueId ?? ""}`; - const latestCreatedAt = latestRunCreatedAtByKey.get(newerRunKey); - if (!latestCreatedAt || newerRun.createdAt > latestCreatedAt) { - latestRunCreatedAtByKey.set(newerRunKey, newerRun.createdAt); - } - } for (const run of failedRows) { const issueId = readRunIssueId(run.contextSnapshot); - const runKey = `${run.agentId}:${issueId ?? ""}`; - const hasNewerRun = (latestRunCreatedAtByKey.get(runKey)?.getTime() ?? 0) > run.createdAt.getTime(); - if (hasNewerRun) continue; + if (failedRunsWithNewerRun.has(run.id)) continue; const issue = issueId ? failedIssueMap.get(issueId) ?? null : null; if (issueId && !issue) continue; diff --git a/server/src/services/bundled-skill-release-validation.test.ts b/server/src/services/bundled-skill-release-validation.test.ts new file mode 100644 index 00000000000..4c4c1c42857 --- /dev/null +++ b/server/src/services/bundled-skill-release-validation.test.ts @@ -0,0 +1,103 @@ +import { createHash, randomUUID } from "node:crypto"; +import { and, eq } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; +import { companies, companySkills, companySkillVersions, createDb } from "@paperclipai/db"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js"; +import { verifyExistingBundledSkillRelease } from "./bundled-skill-release-validation.js"; + +const support = await getEmbeddedPostgresTestSupport(); +const hashInventory = (files: Array<{ path: string; content: string }>) => createHash("sha256") + .update(JSON.stringify(files.map(file => [file.path, file.content]))).digest("hex"); + +(support.supported ? describe : describe.skip)("bundled release validation with current database fingerprints", () => { + let db: ReturnType; + let temporary: Awaited>; + beforeAll(async () => { + temporary = await startEmbeddedPostgresTestDatabase("bundled-release-validation-"); + db = createDb(temporary.connectionString); + }, 30_000); + afterAll(async () => { await db?.$client.end(); await temporary?.cleanup(); }); + + async function seed(content = "saved release content") { + const companyId = randomUUID(), skillId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Release validation", issuePrefix: companyId.slice(0, 8) }); + await db.insert(companySkills).values({ id: skillId, companyId, key: "fixture/release", slug: "release", + name: "Release", markdown: "# Release", sourceType: "local_path" }); + const files = [{ path: "SKILL.md", kind: "skill" as const, content }]; + const [version] = await db.insert(companySkillVersions).values({ companyId, companySkillId: skillId, + revisionNumber: 1, releaseId: "release-1", fileInventory: files }).returning(); + const expectedInventoryHash = hashInventory(files); + const validateInventory = vi.fn((row: typeof companySkillVersions.$inferSelect) => { + if (hashInventory(row.fileInventory) !== expectedInventoryHash) throw new Error("snapshot mismatch"); + }); + return { version: version!, input: { db, companyId, skillId, releaseId: "release-1", expectedInventoryHash, validateInventory } }; + } + + it("reuses proof only within current company, skill and release bindings", async () => { + const f = await seed(); + expect(await verifyExistingBundledSkillRelease(f.input)).toBe(true); + expect(await verifyExistingBundledSkillRelease(f.input)).toBe(true); + expect(f.input.validateInventory).toHaveBeenCalledTimes(1); + expect(await verifyExistingBundledSkillRelease({ ...f.input, companyId: randomUUID() })).toBe(false); + expect(await verifyExistingBundledSkillRelease({ ...f.input, skillId: randomUUID() })).toBe(false); + expect(await verifyExistingBundledSkillRelease({ ...f.input, releaseId: "other-release" })).toBe(false); + expect(f.input.validateInventory).toHaveBeenCalledTimes(1); + expect(await verifyExistingBundledSkillRelease(f.input)).toBe(true); + }); + + it("revalidates when the expected on-disk inventory changes without poisoning valid proof", async () => { + const f = await seed(); + await verifyExistingBundledSkillRelease(f.input); + const changedHash = hashInventory([{ path: "SKILL.md", content: "new on-disk content" }]); + const validateChanged = vi.fn((row: typeof companySkillVersions.$inferSelect) => { + if (hashInventory(row.fileInventory) !== changedHash) throw new Error("snapshot mismatch"); + }); + await expect(verifyExistingBundledSkillRelease({ ...f.input, expectedInventoryHash: changedHash, + validateInventory: validateChanged })).rejects.toThrow("snapshot mismatch"); + expect(validateChanged).toHaveBeenCalledOnce(); + expect(await verifyExistingBundledSkillRelease(f.input)).toBe(true); + expect(f.input.validateInventory).toHaveBeenCalledOnce(); + }); + + it("rechecks current content after a change overlaps an earlier validation", async () => { + const f = await seed(); + let entered!: () => void, release!: () => void; + const validating = new Promise(resolve => { entered = resolve; }); + const held = new Promise(resolve => { release = resolve; }); + const first = verifyExistingBundledSkillRelease({ ...f.input, validateInventory: async row => { + f.input.validateInventory(row); + entered(); + await held; + } }); + await validating; + try { + await db.update(companySkillVersions).set({ fileInventory: [{ path: "SKILL.md", kind: "skill", content: "changed" }] }) + .where(and(eq(companySkillVersions.companyId, f.input.companyId), eq(companySkillVersions.id, f.version.id))); + } finally { release(); } + expect(await first).toBe(true); + await expect(verifyExistingBundledSkillRelease(f.input)).rejects.toThrow("snapshot mismatch"); + expect(f.input.validateInventory).toHaveBeenCalledTimes(2); + }); + + it("does not accept a deleted row from earlier proof", async () => { + const f = await seed(); + await verifyExistingBundledSkillRelease(f.input); + await db.delete(companySkillVersions).where(eq(companySkillVersions.id, f.version.id)); + expect(await verifyExistingBundledSkillRelease(f.input)).toBe(false); + expect(f.input.validateInventory).toHaveBeenCalledOnce(); + }); + + it("bounds retained proof and safely revalidates an evicted release", async () => { + const f = await seed(); + await verifyExistingBundledSkillRelease(f.input); + for (let revisionNumber = 2; revisionNumber <= 66; revisionNumber++) { + const releaseId = `release-${revisionNumber}`; + await db.insert(companySkillVersions).values({ companyId: f.input.companyId, companySkillId: f.input.skillId, + revisionNumber, releaseId, fileInventory: f.version.fileInventory }); + expect(await verifyExistingBundledSkillRelease({ ...f.input, releaseId })).toBe(true); + } + const before = f.input.validateInventory.mock.calls.length; + expect(await verifyExistingBundledSkillRelease(f.input)).toBe(true); + expect(f.input.validateInventory).toHaveBeenCalledTimes(before + 1); + }); +}); diff --git a/server/src/services/bundled-skill-release-validation.ts b/server/src/services/bundled-skill-release-validation.ts new file mode 100644 index 00000000000..b29f8ee28a6 --- /dev/null +++ b/server/src/services/bundled-skill-release-validation.ts @@ -0,0 +1,45 @@ +import { companySkillVersions, type Db } from "@paperclipai/db"; +import { and, eq, getTableColumns, sql } from "drizzle-orm"; + +type Version = typeof companySkillVersions.$inferSelect; +type Verification = { inventorySha256: string; expectedInventoryHash: string }; +const verifiedInventories = new WeakMap>(); +const inventorySha256 = sql`encode(sha256(convert_to(${companySkillVersions.fileInventory}::text, 'UTF8')), 'hex')`; + +export async function verifyExistingBundledSkillRelease(input: { + db: Db; + companyId: string; + skillId: string; + releaseId: string; + expectedInventoryHash: string; + validateInventory: (version: Version) => void | Promise; +}): Promise { + const scope = and(eq(companySkillVersions.companyId, input.companyId), + eq(companySkillVersions.companySkillId, input.skillId), eq(companySkillVersions.releaseId, input.releaseId)); + const [current] = await input.db.select({ id: companySkillVersions.id, inventorySha256 }) + .from(companySkillVersions).where(scope).limit(1); + if (!current) return false; + let verified = verifiedInventories.get(input.db); + if (!verified) { + verified = new Map(); + verifiedInventories.set(input.db, verified); + } + const previous = verified.get(current.id); + if (previous?.inventorySha256 === current.inventorySha256 + && previous.expectedInventoryHash === input.expectedInventoryHash) { + verified.delete(current.id); + verified.set(current.id, previous); + return true; + } + const [version] = await input.db.select({ ...getTableColumns(companySkillVersions), inventorySha256 }) + .from(companySkillVersions).where(scope).limit(1); + if (!version) return false; + await input.validateInventory(version); + verified.delete(version.id); + verified.set(version.id, { inventorySha256: version.inventorySha256, expectedInventoryHash: input.expectedInventoryHash }); + while (verified.size > 64) { + const oldest = verified.keys().next().value; + if (oldest !== undefined) verified.delete(oldest); + } + return true; +} diff --git a/server/src/services/chat-channels/authorization-run-read.test.ts b/server/src/services/chat-channels/authorization-run-read.test.ts new file mode 100644 index 00000000000..3d56b695222 --- /dev/null +++ b/server/src/services/chat-channels/authorization-run-read.test.ts @@ -0,0 +1,116 @@ +import { randomBytes, randomUUID } from "node:crypto"; +import { agents, companies, createDb, heartbeatRuns } from "@paperclipai/db"; +import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture"; +import { eq } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; +import { readChatAuthorizationRun } from "./authorization-run-read.js"; + +const support = await getEmbeddedPostgresTestSupport(); + +describe.skipIf(!support.supported)("chat authorization run reads", () => { + let database: Awaited>; + let db: ReturnType; + let companyId: string; + let agentId: string; + + beforeAll(async () => { + database = await startEmbeddedPostgresTestDatabase("chat-authorization-read-"); + db = createDb(database.connectionString); + const [company] = await db.insert(companies).values({ name: "Chat authorization fixture", issuePrefix: "CAR" }).returning(); + const [agent] = await db.insert(agents).values({ companyId: company.id, name: "Chat agent", adapterType: "process" }).returning(); + companyId = company.id; + agentId = agent.id; + }, 60_000); + + afterAll(async () => { + await db?.$client.end(); + await database?.cleanup(); + }); + + async function seed(contextSnapshot: unknown = {}) { + const [run] = await db.insert(heartbeatRuns).values({ + companyId, agentId, contextSnapshot: contextSnapshot as Record | null, + }).returning(); + return { companyId, agentId, failedRunId: run.id }; + } + + it("reads committed authority below the byte budget and keeps complete retry evidence", async () => { + const bulk = randomBytes(128 * 1024).toString("base64"); + const context = { + source: "chat:slack", issueId: randomUUID(), interactionId: null, taskKey: "CAR-1", + wakeCommentIds: [randomUUID()], wakeCommentId: null, + chatFailedRunRetry: { version: 1, actionId: randomUUID(), failedRunId: randomUUID() }, + retryOfRunId: randomUUID(), + }; + const scope = await seed({ ...context, savedInput: bulk }); + await db.update(heartbeatRuns).set({ runnerProfileJson: { savedInput: bulk }, stdoutExcerpt: bulk, stderrExcerpt: bulk }) + .where(eq(heartbeatRuns.id, scope.failedRunId)); + const [before] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, scope.failedRunId)); + const events: Array> = []; + const now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(db, { heartbeatRuns }, capture); + try { + const selected = await db.transaction(tx => readChatAuthorizationRun(tx, scope, true)); + expect(selected.kind).toBe("committed"); + expect(selected.run?.contextSnapshot).toEqual(context); + expect(selected.run).not.toHaveProperty("runnerProfileJson"); + expect(selected.run).not.toHaveProperty("stdoutExcerpt"); + } finally { capture.flush(); } + const summary = events.find(event => event.event === "paperclip.db_response_capture.summary.v1")!; + expect(summary.completedQueries).toBeGreaterThan(0); + expect(summary.responseValueBytes).toBeLessThan(4096); + const retry = await db.transaction(tx => readChatAuthorizationRun(tx, scope, false)); + expect(retry.kind).toBe("retry"); + expect(retry.run).toEqual(before); + const [after] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, scope.failedRunId)); + expect(after).toEqual(before); + expect(JSON.stringify(events)).not.toContain(bulk.slice(0, 64)); + }); + + it.each([{}, { wakeCommentId: null }, { wakeCommentId: false, wakeCommentIds: "invalid", interactionId: [] }])( + "preserves key presence and JSON types in %j", async context => { + const scope = await seed(context); + expect((await readChatAuthorizationRun(db, scope, true)).run?.contextSnapshot).toEqual(context); + }, + ); + + it.each([null, [], "invalid", 42])("handles non-object context %j", async context => { + const scope = await seed(context); + expect((await readChatAuthorizationRun(db, scope, true)).run?.contextSnapshot).toEqual(context === null ? null : {}); + expect((await readChatAuthorizationRun(db, scope, false)).run?.contextSnapshot).toEqual(context); + }); + + it("reads current authority and fails closed for another company or agent", async () => { + const scope = await seed({ source: "chat:slack", taskKey: "CAR-1" }); + expect((await readChatAuthorizationRun(db, scope, true)).run?.contextSnapshot).toEqual({ source: "chat:slack", taskKey: "CAR-1" }); + await db.update(heartbeatRuns).set({ contextSnapshot: { source: "chat:discord", taskKey: "CAR-2" } }) + .where(eq(heartbeatRuns.id, scope.failedRunId)); + expect((await readChatAuthorizationRun(db, scope, true)).run?.contextSnapshot).toEqual({ source: "chat:discord", taskKey: "CAR-2" }); + for (const committed of [true, false]) { + expect((await readChatAuthorizationRun(db, { ...scope, companyId: randomUUID() }, committed)).run).toBeUndefined(); + expect((await readChatAuthorizationRun(db, { ...scope, agentId: randomUUID() }, committed)).run).toBeUndefined(); + expect((await readChatAuthorizationRun(db, { ...scope, failedRunId: randomUUID() }, committed)).run).toBeUndefined(); + } + }); + + it("preserves nonblocking authorization row locks", async () => { + const scope = await seed(); + let acquired!: () => void, release!: () => void; + const ready = new Promise(resolve => { acquired = resolve; }); + const released = new Promise(resolve => { release = resolve; }); + const holder = db.transaction(async tx => { + await tx.update(heartbeatRuns).set({ status: "running" }).where(eq(heartbeatRuns.id, scope.failedRunId)); + acquired(); + await released; + }); + try { + await ready; + for (const committed of [true, false]) { + await expect(db.transaction(tx => readChatAuthorizationRun(tx, scope, committed))).rejects.toMatchObject({ cause: { code: "55P03" } }); + } + } finally { release(); await holder; } + expect((await readChatAuthorizationRun(db, scope, true)).run?.status).toBe("running"); + }); +}); diff --git a/server/src/services/chat-channels/authorization-run-read.ts b/server/src/services/chat-channels/authorization-run-read.ts new file mode 100644 index 00000000000..3390ba103b7 --- /dev/null +++ b/server/src/services/chat-channels/authorization-run-read.ts @@ -0,0 +1,46 @@ +import { heartbeatRuns, type Db } from "@paperclipai/db"; +import { and, eq, sql } from "drizzle-orm"; + +type Database = Db | Parameters[0]>[0]; +type Scope = { companyId: string; agentId: string; failedRunId: string }; + +const contextKeys = [ + "source", "issueId", "interactionId", "taskKey", "wakeCommentIds", + "wakeCommentId", "chatFailedRunRetry", "retryOfRunId", +] as const; + +const committedContext = sql | null>` + case when ${heartbeatRuns.contextSnapshot} is null then null else ( + select coalesce(jsonb_object_agg(field.key, field.value), '{}'::jsonb) + from jsonb_each(case when jsonb_typeof(${heartbeatRuns.contextSnapshot}) = 'object' + then ${heartbeatRuns.contextSnapshot} else '{}'::jsonb end) as field(key, value) + where field.key in (${sql.join(contextKeys.map(key => sql`${key}`), sql`, `)}) + ) end +`; + +export async function readChatAuthorizationRun(db: Database, scope: Scope, committed: boolean) { + const where = and( + eq(heartbeatRuns.id, scope.failedRunId), + eq(heartbeatRuns.companyId, scope.companyId), + eq(heartbeatRuns.agentId, scope.agentId), + ); + if (!committed) { + const [run] = await db.select().from(heartbeatRuns).where(where).for("share", { noWait: true }); + return { kind: "retry" as const, run }; + } + const [run] = await db.select({ + id: heartbeatRuns.id, + companyId: heartbeatRuns.companyId, + agentId: heartbeatRuns.agentId, + wakeupRequestId: heartbeatRuns.wakeupRequestId, + status: heartbeatRuns.status, + finishedAt: heartbeatRuns.finishedAt, + runtimeMode: heartbeatRuns.runtimeMode, + errorCode: heartbeatRuns.errorCode, + nativeIssueId: heartbeatRuns.nativeIssueId, + completionContractId: heartbeatRuns.completionContractId, + retryOfRunId: heartbeatRuns.retryOfRunId, + contextSnapshot: committedContext, + }).from(heartbeatRuns).where(where).for("share", { noWait: true }); + return { kind: "committed" as const, run }; +} diff --git a/server/src/services/chat-channels/failed-run-retry.ts b/server/src/services/chat-channels/failed-run-retry.ts index 2b5d2c9b026..6310d448a66 100644 --- a/server/src/services/chat-channels/failed-run-retry.ts +++ b/server/src/services/chat-channels/failed-run-retry.ts @@ -2,6 +2,7 @@ import { continuingTeamsFilePublication, type ContinuingTeamsFileTransfer } from import { readTaskChatRetryIntent, taskChatRetryCanAwaitRecovery, taskChatRetryExecutionStopped } from "./task-retry-authority.js"; import { notifyChatActionWork, notifyChatPublicationWork } from "../chat-work-notifications.js"; import { FAILED_RUN_RETRY_STALE_MS, dueChatAction } from "../chat-action-work.js"; +import { readChatAuthorizationRun } from "./authorization-run-read.js"; import { createHash, @@ -688,17 +689,8 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti ), ) .for("share", { noWait: true }); - const [failedRun] = await tx - .select() - .from(heartbeatRuns) - .where( - and( - eq(heartbeatRuns.id, input.failedRunId), - eq(heartbeatRuns.companyId, input.companyId), - eq(heartbeatRuns.agentId, input.agentId), - ), - ) - .for("share", { noWait: true }); + const evidence = await readChatAuthorizationRun(tx, input, Boolean(input.committedResponse)); + const failedRun = evidence.run; if ( !issue || issue.assigneeAgentId !== input.agentId || @@ -719,7 +711,7 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti "process_exit", "timeout", ].includes(failedRun.errorCode ?? "") || - (await isPreProviderReviewedChatFailure(tx, failedRun)) + (evidence.kind === "retry" && await isPreProviderReviewedChatFailure(tx, evidence.run)) )) || (!input.publication && ["backlog", "done", "cancelled"].includes(issue.status)) @@ -729,8 +721,8 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti companyId: input.companyId, issueId: issue.id, agentId: input.agentId, sourceRunId: failedRun.id, intentId: input.taskRetryIntentId, })) throw failedChatRetryDenied(); - if (failedRun.runtimeMode === "native" && !input.committedResponse && !input.taskRetryIntentId) - await failedNativeRetryCoordinator(tx, failedRun, issue.id); + if (failedRun.runtimeMode === "native" && evidence.kind === "retry" && !input.taskRetryIntentId) + await failedNativeRetryCoordinator(tx, evidence.run, issue.id); const context = failedRun.contextSnapshot ?? {}; const provider = CHAT_PROVIDERS.find( (value) => context.source === `chat:${value}`, diff --git a/server/src/services/chat-channels/github/response-authority-read.ts b/server/src/services/chat-channels/github/response-authority-read.ts new file mode 100644 index 00000000000..e5637bfbba5 --- /dev/null +++ b/server/src/services/chat-channels/github/response-authority-read.ts @@ -0,0 +1,50 @@ +import { chatConversations, chatDeliveries, chatEndpointResources, chatEndpoints, chatGitHubConfigurations } from "@paperclipai/db"; +import { sql } from "drizzle-orm"; + +const authorizationEvent = sql>`case + when jsonb_typeof(${chatDeliveries.normalizedEvent}) = 'object' then ( + select coalesce(jsonb_object_agg(field.key, case + when jsonb_typeof(field.value->'context') = 'object' then + jsonb_build_object('context', (field.value->'context') - 'body' - 'title') + else field.value end), '{}'::jsonb) + from jsonb_each(${chatDeliveries.normalizedEvent}) as field(key, value) + where field.key in ('githubAutomatic', 'githubIssue') + ) else ${chatDeliveries.normalizedEvent} end`; + +export const githubResponseAuthorityFields = { + endpoint: { + id: chatEndpoints.id, + companyId: chatEndpoints.companyId, + assignedAgentId: chatEndpoints.assignedAgentId, + botExternalId: chatEndpoints.botExternalId, + botUsername: chatEndpoints.botUsername, + provider: chatEndpoints.provider, + status: chatEndpoints.status, + allowUnlinkedPeople: chatEndpoints.allowUnlinkedPeople, + setup: sql`case when jsonb_typeof(${chatEndpoints.setup}) = 'object' + then jsonb_build_object('runtimeGeneration', ${chatEndpoints.setup}->'runtimeGeneration') + else ${chatEndpoints.setup} end`, + }, + conversation: { + id: chatConversations.id, + issueId: chatConversations.issueId, + externalThreadId: chatConversations.externalThreadId, + }, + delivery: { + id: chatDeliveries.id, + principalId: chatDeliveries.principalId, + state: chatDeliveries.state, + createdAt: chatDeliveries.createdAt, + normalizedEvent: authorizationEvent, + }, + resource: { + enabled: chatEndpointResources.enabled, + availability: chatEndpointResources.availability, + providerResourceId: chatEndpointResources.providerResourceId, + metadata: sql`case + when jsonb_typeof(${chatEndpointResources.metadata}) = 'object' + then jsonb_build_object('providerRepositoryId', ${chatEndpointResources.metadata}->'providerRepositoryId') + else ${chatEndpointResources.metadata} end`, + }, + toolsEnabled: sql`${chatGitHubConfigurations.configuration}->'toolsEnabled'`, +}; diff --git a/server/src/services/chat-channels/github/response-target.ts b/server/src/services/chat-channels/github/response-target.ts index 1e955b14482..b45ef3c8d81 100644 --- a/server/src/services/chat-channels/github/response-target.ts +++ b/server/src/services/chat-channels/github/response-target.ts @@ -4,9 +4,11 @@ import { chatActions, chatConversations, chatDeliveries, chatEndpoints, type Db import { forbidden, HttpError } from "../../../errors.js"; export type GitHubResponseSource = { - endpoint: typeof chatEndpoints.$inferSelect; - conversation: typeof chatConversations.$inferSelect; - delivery: typeof chatDeliveries.$inferSelect; + endpoint: Pick & { setup: unknown }; + conversation: Pick; + delivery: Pick; number: number; replyId: string | null; repositoryId?: string; diff --git a/server/src/services/chat-github-access.ts b/server/src/services/chat-github-access.ts index c906d75a479..22ba2c44c5b 100644 --- a/server/src/services/chat-github-access.ts +++ b/server/src/services/chat-github-access.ts @@ -1,4 +1,4 @@ -import { and, eq } from "drizzle-orm"; +import { and, eq, sql } from "drizzle-orm"; import { chatEndpoints, chatExternalPrincipals, @@ -13,12 +13,15 @@ type Database = Db | Parameters[0]>[0]; /** Current authority shared by task admission and every later publication. */ export async function githubChatPrincipalAccess( db: Database, - endpoint: typeof chatEndpoints.$inferSelect, + endpoint: Pick, principalId: string, ) { if (endpoint.provider !== "github") return null; const [saved] = await db - .select() + .select({ configuration: sql>`jsonb_build_object( + 'people', ${chatGitHubConfigurations.configuration}->'people', + 'memberAccess', ${chatGitHubConfigurations.configuration}->'memberAccess' + )` }) .from(chatGitHubConfigurations) .where( and( @@ -29,7 +32,7 @@ export async function githubChatPrincipalAccess( .for("share"); if (!saved) return null; // Preserve the existing connector until explicitly upgraded. const [principal] = await db - .select() + .select({ externalId: chatExternalPrincipals.externalId }) .from(chatExternalPrincipals) .where( and( @@ -50,7 +53,7 @@ export async function githubChatPrincipalAccess( (p) => p.githubUserId === principal.externalId, ); const [link] = await db - .select() + .select({ status: chatIdentityLinks.status, paperclipUserId: chatIdentityLinks.paperclipUserId }) .from(chatIdentityLinks) .where( and( @@ -62,7 +65,7 @@ export async function githubChatPrincipalAccess( .for("share"); const activeMember = async (userId: string) => { const [membership] = await db - .select() + .select({ membershipRole: companyMemberships.membershipRole }) .from(companyMemberships) .where( and( diff --git a/server/src/services/chat-github-checks.ts b/server/src/services/chat-github-checks.ts index dceb5ae3df5..a47cf3b1021 100644 --- a/server/src/services/chat-github-checks.ts +++ b/server/src/services/chat-github-checks.ts @@ -427,19 +427,24 @@ export function githubReviewCheckService(db: Db, fetchImpl = fetch) { and( inArray(chatGitHubReviews.state, ["queued", "running"]), isNull(chatGitHubReviews.assessment), - sql`exists ( - select 1 from ${chatActions} action - join ${chatDeliveries} delivery - on delivery.id::text = ${chatGitHubReviews.deliveryId} + sql`${chatGitHubReviews.deliveryId} ~ '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$'`, + sql`( + select action.id from ${chatActions} action where action.company_id = ${chatGitHubReviews.companyId} and action.endpoint_id = ${chatGitHubReviews.endpointId} - and delivery.endpoint_id = action.endpoint_id and action.kind = 'github_review_check' and action.status = 'cancelled' and action.result->>'code' = 'stale_head' - and action.payload->'event'->>'deliveryId' = - delivery.normalized_event->'githubAutomatic'->'context'->>'deliveryId' - )`, + and action.payload->'event'->>'deliveryId' = ( + select delivery.normalized_event->'githubAutomatic'->'context'->>'deliveryId' + from ${chatDeliveries} delivery + where delivery.id = case + when ${chatGitHubReviews.deliveryId} ~ '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$' + then ${chatGitHubReviews.deliveryId}::uuid end + and delivery.endpoint_id = ${chatGitHubReviews.endpointId} + ) + limit 1 + ) is not null`, ), ); const rows = await db diff --git a/server/src/services/chat-github-cloud-ingress.ts b/server/src/services/chat-github-cloud-ingress.ts index b1493d64779..2f8e75eb8f9 100644 --- a/server/src/services/chat-github-cloud-ingress.ts +++ b/server/src/services/chat-github-cloud-ingress.ts @@ -7,6 +7,18 @@ type Event = SealedConnectorEvents["events"][number]; type Endpoint = typeof chatEndpoints.$inferSelect; type Handler = (endpoint: Endpoint, request: Request) => Promise; const handlers = new WeakMap(); +const setupPollers = new WeakMap Promise>(); + +/** The active wizard reuses the server-owned, serialized event drain. */ +export function registerGitHubBotCloudSetupPoller(db: Db, poll: () => Promise) { + setupPollers.set(db, poll); + return () => { if (setupPollers.get(db) === poll) setupPollers.delete(db); }; +} + +export async function pollGitHubBotCloudSetupEvents(db: Db): Promise { + await setupPollers.get(db)?.(); +} + export function registerGitHubBotCloudIngress(db: Db, handler: Handler) { handlers.set(db, handler); return () => { diff --git a/server/src/services/chat-github-events.ts b/server/src/services/chat-github-events.ts index 4766843513a..c2656f04b73 100644 --- a/server/src/services/chat-github-events.ts +++ b/server/src/services/chat-github-events.ts @@ -20,6 +20,7 @@ import { import { effectiveGitHubReviewPolicy, githubReviewSchedulingDecision, + type GitHubAutomaticPolicyContext, } from "./chat-github-review-policy.js"; const id = z.union([ @@ -226,8 +227,8 @@ export function githubAutomaticReviewEvent( export async function githubAutomaticAdmission( db: Db | Parameters[0]>[0], - endpoint: typeof chatEndpoints.$inferSelect, - context: GitHubAutomaticEventContext, + endpoint: Pick, + context: GitHubAutomaticPolicyContext, ) { const [saved] = await db .select() diff --git a/server/src/services/chat-github-response-comments.ts b/server/src/services/chat-github-response-comments.ts index 8caff8894b9..c032ade25ba 100644 --- a/server/src/services/chat-github-response-comments.ts +++ b/server/src/services/chat-github-response-comments.ts @@ -11,7 +11,8 @@ import { githubAutomaticAdmission } from "./chat-github-events.js"; import { githubBotRepositoryToken, githubBotRequest } from "./chat-github-client.js"; import { withGitHubPublicationLease } from "./chat-github-publication-lease.js"; import { projectSafeChatPublicationText } from "./chat-publication-projection.js"; -import type { GitHubAutomaticEventContext } from "@paperclipai/shared"; +import type { GitHubAutomaticPolicyContext } from "./chat-github-review-policy.js"; +import { githubResponseAuthorityFields } from "./chat-channels/github/response-authority-read.js"; import { githubRunReplyState } from "./chat-run-publications.js"; import { logActivity } from "./activity-log.js"; import { boundRunIssueId } from "./run-task-binding.js"; @@ -159,8 +160,7 @@ export async function writeGitHubResponseComment( export function githubResponseCommentService(db: Db, fetchImpl = fetch) { async function context(deliveryId: string) { - const [row] = await db.select({ endpoint: chatEndpoints, conversation: chatConversations, - delivery: chatDeliveries, resource: chatEndpointResources, config: chatGitHubConfigurations, + const [row] = await db.select({ ...githubResponseAuthorityFields, assignee: issues.assigneeAgentId, issueStatus: issues.status, sourceMessageId: chatMessageLinks.commentId }).from(chatMessageLinks) .innerJoin(chatDeliveries, and(eq(chatDeliveries.id, chatMessageLinks.deliveryId), eq(chatDeliveries.companyId, chatMessageLinks.companyId))) .innerJoin(chatConversations, and(eq(chatConversations.id, chatMessageLinks.conversationId), eq(chatConversations.companyId, chatMessageLinks.companyId))) @@ -170,13 +170,13 @@ export function githubResponseCommentService(db: Db, fetchImpl = fetch) { .innerJoin(issues, and(eq(issues.id, chatConversations.issueId), eq(issues.companyId, chatEndpoints.companyId))) .where(and(eq(chatMessageLinks.deliveryId, deliveryId), eq(chatMessageLinks.direction, "inbound"), eq(chatEndpoints.provider, "github"))); if (!row || !row.delivery.principalId || !["processing", "processed"].includes(row.delivery.state) || - !["active", "verifying"].includes(row.endpoint.status) || !row.config.configuration.toolsEnabled || + !["active", "verifying"].includes(row.endpoint.status) || !row.toolsEnabled || !row.resource.enabled || row.resource.availability !== "available" || row.assignee !== row.endpoint.assignedAgentId) throw forbidden("GitHub request is no longer authorized"); const access = await githubChatPrincipalAccess(db, row.endpoint, row.delivery.principalId); if (!access?.allowed) throw forbidden("The initiating GitHub person is no longer authorized"); const automatic = (row.delivery.normalizedEvent.githubAutomatic ?? row.delivery.normalizedEvent.githubIssue) as - { context: GitHubAutomaticEventContext } | undefined; + { context: GitHubAutomaticPolicyContext } | undefined; if (automatic && !(await githubAutomaticAdmission(db, row.endpoint, automatic.context))?.allowed) throw forbidden("Automatic GitHub task authority changed"); const match = /^github:([^:]+):(?:issue:)?([1-9][0-9]*)(?::rc:([1-9][0-9]*))?$/.exec(row.conversation.externalThreadId); diff --git a/server/src/services/chat-github-review-policy.ts b/server/src/services/chat-github-review-policy.ts index cfb805e1be2..f014e4ae066 100644 --- a/server/src/services/chat-github-review-policy.ts +++ b/server/src/services/chat-github-review-policy.ts @@ -8,9 +8,14 @@ import { type GitHubReviewConclusion, type GitHubReviewEventContext, type GitHubAutomaticEventContext, + type GitHubIssueEventContext, type GitHubReviewPolicy, } from "@paperclipai/shared"; +export type GitHubAutomaticPolicyContext = + | Pick + | Pick; + export function effectiveGitHubReviewPolicy( configuration: GitHubChatConfiguration, repositoryId: string, @@ -85,7 +90,7 @@ export function githubReviewLineIsInPatch( export function githubReviewSchedulingDecision(input: { configuration: GitHubChatConfiguration; - context: GitHubAutomaticEventContext; + context: GitHubAutomaticPolicyContext; repositoryEnabled: boolean; /** Resolved against current company membership and the user's confirmed link. */ linkedMemberUserId: string | null; diff --git a/server/src/services/chat-github-review-repair.test.ts b/server/src/services/chat-github-review-repair.test.ts new file mode 100644 index 00000000000..01d3649aa2c --- /dev/null +++ b/server/src/services/chat-github-review-repair.test.ts @@ -0,0 +1,183 @@ +import { randomUUID } from "node:crypto"; +import { sql } from "drizzle-orm"; +import { createDb, getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase, type Db } from "@paperclipai/db"; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; +import { githubReviewCheckService } from "./chat-github-checks.js"; + +type QueryParameters = NonNullable[1]>; +type CapturedQuery = { text: string; parameters: QueryParameters }; +type Plan = { "Node Type"?: string; "Relation Name"?: string; "Index Name"?: string; "Actual Loops"?: number; Plans?: Plan[] }; +const support = await getEmbeddedPostgresTestSupport(); + +describe.skipIf(!support.supported)("GitHub stale review repair PostgreSQL behavior", () => { + let temporary: Awaited>; + let db: Db; + let queries: CapturedQuery[]; + const noProviderCall = vi.fn(async () => { throw new Error("Repair must not contact a provider"); }); + + beforeAll(async () => { + temporary = await startEmbeddedPostgresTestDatabase("github-review-repair-"); + const setup = createDb(temporary.connectionString, { maxConnections: 1 }); + try { + await setup.execute(sql`create schema github_review_repair_fixture`); + await setup.execute(sql`set search_path=github_review_repair_fixture`); + await setup.execute(sql`create table chat_github_reviews (id uuid primary key, company_id uuid not null, + endpoint_id uuid not null, delivery_id text not null, state text not null, assessment jsonb, + updated_at timestamptz not null default now())`); + await setup.execute(sql`create table chat_deliveries (id uuid primary key, endpoint_id uuid not null, normalized_event jsonb)`); + await setup.execute(sql`create table chat_actions (id uuid primary key, company_id uuid not null, + endpoint_id uuid not null, provider_action_id text, kind text not null, status text not null, + payload jsonb not null, result jsonb, updated_at timestamptz not null default now())`); + await setup.execute(sql`create unique index chat_actions_provider_action_uq + on chat_actions(company_id, endpoint_id, provider_action_id)`); + } finally { + await setup.$client.end(); + } + }, 90_000); + + beforeEach(async () => { + queries = []; + db = createDb(temporary.connectionString, { maxConnections: 1 }); + db.$client.options.debug = (_connection: number, text: string, parameters: QueryParameters) => { + if (/^update "chat_github_reviews"/i.test(text)) queries.push({ text, parameters: [...parameters] }); + }; + await db.execute(sql`set search_path=github_review_repair_fixture`); + await db.execute(sql`truncate chat_github_reviews, chat_deliveries, chat_actions`); + noProviderCall.mockClear(); + }); + afterEach(async () => { await db?.$client.end(); }); + afterAll(async () => { await temporary?.cleanup(); }); + + async function seed(overrides: Record = {}) { + const companyId = randomUUID(), endpointId = randomUUID(); + const deliveryId = "a" + randomUUID().slice(1), reviewId = randomUUID(), actionId = randomUUID(); + const providerDeliveryId = "provider-" + randomUUID(); + const values = { companyId, endpointId, deliveryId, reviewId, actionId, providerDeliveryId, + actionCompanyId: companyId, actionEndpointId: endpointId, deliveryEndpointId: endpointId, + actionDeliveryId: providerDeliveryId, reviewDeliveryId: deliveryId, state: "queued", assessment: null, + kind: "github_review_check", status: "cancelled", code: "stale_head", ...overrides }; + await db.execute(sql`insert into chat_deliveries(id, endpoint_id, normalized_event) + values (${deliveryId}::uuid, ${values.deliveryEndpointId}::uuid, + ${JSON.stringify({ githubAutomatic: { context: { deliveryId: providerDeliveryId } } })}::jsonb)`); + await db.execute(sql`insert into chat_actions(id, company_id, endpoint_id, kind, status, payload, result) + values (${actionId}::uuid, ${values.actionCompanyId}::uuid, ${values.actionEndpointId}::uuid, + ${values.kind}, ${values.status}, ${JSON.stringify({ event: { deliveryId: values.actionDeliveryId } })}::jsonb, + ${JSON.stringify({ code: values.code })}::jsonb)`); + await db.execute(sql`insert into chat_github_reviews(id, company_id, endpoint_id, delivery_id, state, assessment, updated_at) + values (${reviewId}::uuid, ${companyId}::uuid, ${endpointId}::uuid, ${values.reviewDeliveryId}, + ${values.state}, ${values.assessment === null ? null : JSON.stringify(values.assessment)}::jsonb, '2000-01-01T00:00:00Z')`); + return values; + } + + async function retainedRows() { + return { + reviews: await db.$client.unsafe("select * from chat_github_reviews order by id"), + deliveries: await db.$client.unsafe("select * from chat_deliveries order by id"), + actions: await db.$client.unsafe("select * from chat_actions order by id"), + }; + } + + it.each(["queued", "running"])("supersedes only the exact %s unassessed review", async state => { + const fixture = await seed({ state }); + const before = await retainedRows(); + const service = githubReviewCheckService(db, noProviderCall); + await service.processPending(); + const after = await retainedRows(); + expect(after.reviews).toHaveLength(1); + expect(after.reviews[0]).toMatchObject({ id: fixture.reviewId, state: "superseded", assessment: null }); + expect(after.deliveries).toEqual(before.deliveries); + expect(after.actions).toEqual(before.actions); + await service.processPending(); + expect(await retainedRows()).toEqual(after); + expect(noProviderCall).not.toHaveBeenCalled(); + }); + + it.each([ + ["another action company", { actionCompanyId: randomUUID() }], + ["another action endpoint", { actionEndpointId: randomUUID() }], + ["another delivery endpoint", { deliveryEndpointId: randomUUID() }], + ["another provider delivery", { actionDeliveryId: "another-delivery" }], + ["another cancellation reason", { code: "another_reason" }], + ["another action kind", { kind: "another_kind" }], + ["a processed action", { status: "processed" }], + ["an assessed review", { assessment: { retained: true } }], + ["a completed review", { state: "completed" }], + ])("preserves rows for %s", async (_name, overrides) => { + await seed(overrides as Record); + const before = await retainedRows(); + await githubReviewCheckService(db, noProviderCall).processPending(); + expect(await retainedRows()).toEqual(before); + expect(noProviderCall).not.toHaveBeenCalled(); + }); + + it.each(["malformed", "uppercase", "whitespace", "braces", "unhyphenated"])("preserves %s delivery references without cast errors", async kind => { + const fixture = await seed(); + const reference = kind === "malformed" ? "not-a-uuid" : kind === "uppercase" ? fixture.deliveryId.toUpperCase() + : kind === "whitespace" ? ` ${fixture.deliveryId} ` : kind === "braces" ? `{${fixture.deliveryId}}` + : fixture.deliveryId.replaceAll("-", ""); + await db.execute(sql`update chat_github_reviews set delivery_id=${reference} where id=${fixture.reviewId}::uuid`); + const before = await retainedRows(); + await githubReviewCheckService(db, noProviderCall).processPending(); + expect(await retainedRows()).toEqual(before); + expect(noProviderCall).not.toHaveBeenCalled(); + }); + + it.each(["missing delivery", "missing provider identity"])("preserves a review with %s", async kind => { + const fixture = await seed(); + if (kind === "missing delivery") { + await db.execute(sql`delete from chat_deliveries where id=${fixture.deliveryId}::uuid`); + } else { + await db.execute(sql`update chat_deliveries set normalized_event='{}'::jsonb where id=${fixture.deliveryId}::uuid`); + } + const before = await retainedRows(); + await githubReviewCheckService(db, noProviderCall).processPending(); + expect(await retainedRows()).toEqual(before); + expect(noProviderCall).not.toHaveBeenCalled(); + }); + + it("repairs a late review from the retained cancellation without replaying provider work", async () => { + const fixture = await seed(); + await db.execute(sql`delete from chat_github_reviews where id=${fixture.reviewId}::uuid`); + const service = githubReviewCheckService(db, noProviderCall); + await service.processPending(); + await db.execute(sql`insert into chat_github_reviews(id, company_id, endpoint_id, delivery_id, state) + values (${fixture.reviewId}::uuid, ${fixture.companyId}::uuid, ${fixture.endpointId}::uuid, ${fixture.deliveryId}, 'queued')`); + await service.processPending(); + const rows = await db.$client.unsafe("select state from chat_github_reviews"); + expect(rows).toEqual([{ state: "superseded" }]); + expect(noProviderCall).not.toHaveBeenCalled(); + }); + + it("resolves each candidate delivery once through its primary key with retained history", async () => { + const companyId = randomUUID(), endpointId = randomUUID(); + const candidates = 200; + await db.execute(sql`insert into chat_deliveries(id, endpoint_id, normalized_event) + select gen_random_uuid(), ${endpointId}::uuid, + jsonb_build_object('githubAutomatic', jsonb_build_object('context', jsonb_build_object('deliveryId', 'delivery-' || n))) + from generate_series(1,8000) n`); + await db.execute(sql`insert into chat_github_reviews(id, company_id, endpoint_id, delivery_id, state) + select gen_random_uuid(), ${companyId}::uuid, ${endpointId}::uuid, id::text, 'queued' + from chat_deliveries order by id limit ${candidates}`); + await db.execute(sql`insert into chat_actions(id, company_id, endpoint_id, provider_action_id, kind, status, payload, result) + select gen_random_uuid(), ${companyId}::uuid, ${endpointId}::uuid, 'action-' || n, + case when n<=100 then 'github_review_check' else 'another_kind' end, 'cancelled', + jsonb_build_object('event', jsonb_build_object('deliveryId', 'unmatched-' || n)), + jsonb_build_object('code', 'stale_head') from generate_series(1,1100) n`); + await db.execute(sql`analyze chat_deliveries`); + await db.execute(sql`analyze chat_actions`); + await db.execute(sql`analyze chat_github_reviews`); + await db.execute(sql`set statement_timeout='5s'`); + await githubReviewCheckService(db, noProviderCall).processPending(); + expect(queries).toHaveLength(1); + const query = queries[0]; + const explained = await db.$client.unsafe(`explain (analyze,buffers,format json) ${query.text}`, query.parameters); + const root = (explained[0]["QUERY PLAN"] as Array<{ Plan: Plan }>)[0].Plan; + function flatten(plan: Plan): Plan[] { return [plan, ...(plan.Plans ?? []).flatMap(flatten)]; } + const deliveries = flatten(root).filter(node => node["Relation Name"] === "chat_deliveries"); + expect(deliveries).toHaveLength(1); + expect(deliveries[0]["Index Name"]).toBe("chat_deliveries_pkey"); + expect(deliveries[0]["Actual Loops"]).toBeLessThanOrEqual(candidates); + expect((await db.$client.unsafe("select count(*)::int as count from chat_github_reviews where state='queued'"))[0].count).toBe(candidates); + expect(noProviderCall).not.toHaveBeenCalled(); + }, 30_000); +}); diff --git a/server/src/services/chat-github-wizard.ts b/server/src/services/chat-github-wizard.ts index 5f63f68db8c..701d77cba51 100644 --- a/server/src/services/chat-github-wizard.ts +++ b/server/src/services/chat-github-wizard.ts @@ -33,6 +33,7 @@ import { type PaperclipCloudConnector, } from "./paperclip-cloud-connector.js"; import { paperclipCloudConnectorEnrollmentStatus } from "./paperclip-cloud-connector-enrollment.js"; +import { pollGitHubBotCloudSetupEvents } from "./chat-github-cloud-ingress.js"; /** Loopback aliases are equivalent locally, but Cloud requires the approved exact origin. */ export function githubWizardBrowserOrigin( @@ -920,6 +921,10 @@ export function githubChatWizardService( // Keep the same vaulted App in the wizard instead of throwing JSON from // the browser callback or relaxing the configure action's signature gate. bot = await endpoint(id, userId); + if (!bot.setup.webhookVerifiedAt && bot.setup.github?.cloudRegistrationId) { + await pollGitHubBotCloudSetupEvents(db); + bot = await endpoint(id, userId); + } if (!bot.setup.webhookVerifiedAt) return { endpointId: id, diff --git a/server/src/services/chat-monitor-continuation.test.ts b/server/src/services/chat-monitor-continuation.test.ts index 01b094800e4..f8e277d587b 100644 --- a/server/src/services/chat-monitor-continuation.test.ts +++ b/server/src/services/chat-monitor-continuation.test.ts @@ -1,4 +1,4 @@ -import { randomUUID } from "node:crypto"; +import { randomBytes, randomUUID } from "node:crypto"; import { and, eq } from "drizzle-orm"; import { afterAll, beforeAll, describe, expect, it } from "vitest"; import { @@ -237,6 +237,12 @@ if (!support.supported) console.warn(`Monitor publication database unavailable: it("recovers the accepted continuation explanation after interrupted finalization without another provider result", async () => { const f = await fixture(); + const executionPayload = randomBytes(128 * 1024).toString("base64"); + await db.update(heartbeatRuns).set({ + contextSnapshot: { ...f.source.contextSnapshot, largePrompt: executionPayload, + paperclipWake: { prompt: executionPayload } }, + runnerProfileJson: { instructions: executionPayload }, stdoutExcerpt: executionPayload, + }).where(eq(heartbeatRuns.id, f.source.id)); const [contract] = await db.insert(completionContracts).values({ companyId: f.company.id, issueId: f.issue.id, revision: 1, schemaVersion: "paperclip.completion-contract.v1", policyVersion: "test", risk: "standard", completionAuthority: "server_arbiter", incompleteCriteriaPolicy: "preserve_non_terminal", @@ -270,6 +276,9 @@ if (!support.supported) console.warn(`Monitor publication database unavailable: expect(coordinator).toMatchObject({ phase: "committed" }); const [settledRun] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.source.id)); expect(settledRun).toMatchObject({ status: "succeeded", resultJson: { finalizationPhase: "committed" } }); + expect(settledRun.contextSnapshot?.largePrompt).toBe(executionPayload); + expect(settledRun.runnerProfileJson?.instructions).toBe(executionPayload); + expect(settledRun.stdoutExcerpt).toBe(executionPayload); const priorComments = await db.select({ body: issueComments.body }).from(issueComments).where(eq(issueComments.createdByRunId, f.source.id)); expect(priorComments).toEqual([{ body: summary }]); expect(await repair()).toBe(false); diff --git a/server/src/services/company-skills.ts b/server/src/services/company-skills.ts index ac114ee692f..f29f4ece222 100644 --- a/server/src/services/company-skills.ts +++ b/server/src/services/company-skills.ts @@ -1,4 +1,5 @@ import { ensureSkillCacheDirectory, touchSkillVersionCache, withSkillVersionCacheLock } from "./skill-version-cache.js"; +import { verifyExistingBundledSkillRelease } from "./bundled-skill-release-validation.js"; import { retainProjectSkillAccess, skillAccessService } from "./skill-access.js"; import type { AuthorizationActor } from "./authorization.js"; import { updateAgentConfigurationInTransaction } from "./agent-configuration-transaction.js"; @@ -3094,30 +3095,22 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo const fileInventory = serializeVersionFileInventory( await collectVersionFileInventoryFromDirectory(release.releaseDir), ); - const existing = await db - .select() - .from(companySkillVersions) - .where(and( - eq(companySkillVersions.companyId, companyId), - eq(companySkillVersions.companySkillId, paperclipSkill.id), - eq(companySkillVersions.releaseId, release.id), - )) - .then((rows) => rows[0] ?? null); - if (existing) { - const existingInventory = toCompanySkillVersion(existing).fileInventory; - const existingHash = buildInventoryContentHash(existingInventory.map((entry) => ({ - path: entry.path, - sha256: sha256Buffer(entry.content), - }))); - const registryHash = buildInventoryContentHash(fileInventory.map((entry) => ({ - path: entry.path, - sha256: sha256Buffer(entry.content), - }))); - if (existingHash !== registryHash) { - throw new Error(`Bundled skill release ${release.id} does not match its seeded snapshot.`); - } - continue; - } + const registryHash = buildInventoryContentHash(fileInventory.map((entry) => ({ + path: entry.path, + sha256: sha256Buffer(entry.content), + }))); + if (await verifyExistingBundledSkillRelease({ db, companyId, skillId: paperclipSkill.id, releaseId: release.id, + expectedInventoryHash: registryHash, validateInventory: existing => { + const existingInventory = toCompanySkillVersion(existing).fileInventory; + const existingHash = buildInventoryContentHash(existingInventory.map((entry) => ({ + path: entry.path, + sha256: sha256Buffer(entry.content), + }))); + if (existingHash !== registryHash) { + throw new Error(`Bundled skill release ${release.id} does not match its seeded snapshot.`); + } + }, + })) continue; const releasedAt = new Date(release.releasedAt); if (Number.isNaN(releasedAt.getTime())) { throw new Error(`Invalid bundled skill release date: ${release.releasedAt}`); diff --git a/server/src/services/connection-intents.ts b/server/src/services/connection-intents.ts index a6d07331c0c..77dabac142b 100644 --- a/server/src/services/connection-intents.ts +++ b/server/src/services/connection-intents.ts @@ -291,9 +291,9 @@ export function connectionIntentService(db: Db) { && connection.status !== "archived" && connection.connectionPurpose !== "ai" ); if (matching.length === 0) return null; - const effective = await access.getEffectiveToolAssignmentsForAgent(input.companyId, input.agentId); + const effective = await access.getConnectionAssignmentsForAgent(input.companyId, input.agentId, matching.map(connection => connection.id)); const installedIds = new Set(effective.installedConnections.map((connection) => connection.id)); - const permittedIds = new Set(effective.allowedTools.map((tool) => tool.connectionId)); + const permittedIds = new Set(effective.allowedConnectionIds); const usable = (connection: ToolConnection | undefined) => connection && installedIds.has(connection.id) && permittedIds.has(connection.id) && connection.status === "active" && connection.enabled diff --git a/server/src/services/conversation-continuation.ts b/server/src/services/conversation-continuation.ts index 36e8b7a4804..cd0c54a2dcd 100644 --- a/server/src/services/conversation-continuation.ts +++ b/server/src/services/conversation-continuation.ts @@ -111,7 +111,14 @@ export async function getConversationOwnershipBlocker(db: Db, companyId: string, and (${environmentLeases.releasedAt} is null or ${environmentLeases.status} = 'pending_cleanup' or ${environmentLeases.cleanupStatus} = 'failed'))`; - const candidates = await db.select({ run: heartbeatRuns, activeLease }).from(heartbeatRuns) + const candidates = await db.select({ run: { + id: heartbeatRuns.id, + agentId: heartbeatRuns.agentId, + processPid: heartbeatRuns.processPid, + processGroupId: heartbeatRuns.processGroupId, + processStartedAt: heartbeatRuns.processStartedAt, + resultJson: heartbeatRuns.resultJson, + }, activeLease }).from(heartbeatRuns) .where(and( eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.runtimeMode, "legacy"), conversationRunPredicate(), diff --git a/server/src/services/environment-config.ts b/server/src/services/environment-config.ts index b9c041a35d1..6431c4802ff 100644 --- a/server/src/services/environment-config.ts +++ b/server/src/services/environment-config.ts @@ -17,7 +17,7 @@ import { unprocessable } from "../errors.js"; import { parseObject } from "../adapters/utils.js"; import { secretService } from "./secrets.js"; import { - resolvePluginSandboxProviderDriverByKey, + resolvePluginSandboxProviderConfigSchema, validatePluginEnvironmentDriverConfig, validatePluginSandboxProviderConfig, } from "./plugin-environment-driver.js"; @@ -179,14 +179,10 @@ async function getSandboxProviderConfigSchema( db: Db, provider: string, ): Promise | null> { - const resolved = await resolvePluginSandboxProviderDriverByKey({ + return resolvePluginSandboxProviderConfigSchema({ db, driverKey: provider, }); - const schema = resolved?.driver.configSchema; - return schema && typeof schema === "object" && !Array.isArray(schema) - ? schema as Record - : null; } export async function resolveSandboxProviderSecretRefPaths( diff --git a/server/src/services/execution-blocker-read.test.ts b/server/src/services/execution-blocker-read.test.ts index 309385a25cb..5b16c521af1 100644 --- a/server/src/services/execution-blocker-read.test.ts +++ b/server/src/services/execution-blocker-read.test.ts @@ -100,7 +100,9 @@ describe.skipIf(!support.supported)("execution blocker PostgreSQL read projectio }); const input = { schema: "paperclip.native-execution-input.v6", binding: { runId: f.run.id, companyId: f.companyId, agentId: f.agentId, issueId: f.issueId }, - provider: { kind: "grok_bot" }, workspace: { access: "none", ...(hasCwd ? { cwd: null } : {}) } }; + provider: { kind: "grok_bot" }, workspace: { access: "none", ...(hasCwd ? { cwd: null } : {}) }, + task: { description: randomBytes(128 * 1024).toString("base64") }, + runtimeContext: { instructions: randomBytes(128 * 1024).toString("base64") } }; await db.update(heartbeatRuns).set({ nativeIssueId: f.issueId, runnerProfileJson: { ...f.run.runnerProfileJson, nativeExecutionInput: input } }).where(eq(heartbeatRuns.id, f.run.id)); expect(await compare(f)).toMatchObject({ workspaceRepairRequired: !hasCwd, canRetry: false }); @@ -108,6 +110,39 @@ describe.skipIf(!support.supported)("execution blocker PostgreSQL read projectio expect(stored.runnerProfileJson?.nativeExecutionInput).toEqual(input); }); + it.each(["run", "company", "agent", "issue", "provider", "schema", "access", "workspace", "binding", "missing_binding", + "null_input", "array_input", "scalar_input", "number_input"])( + "does not relax native workspace recovery for invalid input (%s)", async kind => { + const f = await fixture({ runtimeMode: "native", driverKind: "grok_bot_mcp" }, { + workspaceRestoreFailure: "restore_unsafe_archive", + workspaceRestoreRecovery: { schema: "paperclip.workspace-restore-recovery.v1", leaseIds: [] }, + }); + const input: Record = { + schema: "paperclip.native-execution-input.v6", + binding: { runId: f.run.id, companyId: f.companyId, agentId: f.agentId, issueId: f.issueId }, + provider: { kind: "grok_bot" }, workspace: { access: "none", cwd: null }, + task: { description: randomBytes(128 * 1024).toString("base64") }, + }; + const binding = input.binding as Record; + if (kind === "run") binding.runId = randomUUID(); + if (kind === "company") binding.companyId = randomUUID(); + if (kind === "agent") binding.agentId = randomUUID(); + if (kind === "issue") binding.issueId = randomUUID(); + if (kind === "provider") input.provider = { kind: "codex" }; + if (kind === "schema") input.schema = "unsupported"; + if (kind === "access") input.workspace = { cwd: null }; + if (kind === "workspace") input.workspace = "none"; + if (kind === "binding") input.binding = []; + if (kind === "missing_binding") delete input.binding; + const selectedInput = kind === "null_input" ? null : kind === "array_input" ? [] + : kind === "scalar_input" ? "input" : kind === "number_input" ? 42 : input; + await db.update(heartbeatRuns).set({ nativeIssueId: f.issueId, + runnerProfileJson: { ...f.run.runnerProfileJson, nativeExecutionInput: selectedInput } }).where(eq(heartbeatRuns.id, f.run.id)); + expect(await compare(f)).toMatchObject({ workspaceRepairRequired: true, canRetry: false }); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.run.id)))[0].runnerProfileJson?.nativeExecutionInput).toEqual(selectedInput); + }, + ); + it.each([null, [], "scalar", 42, { issueId: 42, taskId: false }].map(context => ({ context })))("preserves decisions for malformed context $context", async ({ context }) => { const f = await fixture({ contextSnapshot: context as Record | null }); expect(await compare(f)).toMatchObject({ canRetry: true }); diff --git a/server/src/services/execution-blocker.ts b/server/src/services/execution-blocker.ts index 4777d5947b3..3ee94aedb72 100644 --- a/server/src/services/execution-blocker.ts +++ b/server/src/services/execution-blocker.ts @@ -28,6 +28,11 @@ const blockerProfileColumn = sql | null>`case when ${heartbeatRuns.runnerProfileJson} is null then null else ( select coalesce(jsonb_object_agg(field.key, case field.key when 'adapterDispatch' then jsonb_build_object('adapterType', field.value->'adapterType') + when 'nativeExecutionInput' then case when jsonb_typeof(field.value) = 'object' then ( + select coalesce(jsonb_object_agg(input_field.key, input_field.value), '{}'::jsonb) + from jsonb_each(field.value) as input_field(key, value) + where input_field.key in ('schema', 'binding', 'provider', 'workspace') + ) else field.value end else field.value end), '{}'::jsonb) from jsonb_each(case when jsonb_typeof(${heartbeatRuns.runnerProfileJson}) = 'object' then ${heartbeatRuns.runnerProfileJson} else '{}'::jsonb end) as field(key, value) diff --git a/server/src/services/execution-workspaces.ts b/server/src/services/execution-workspaces.ts index 63800f66b9e..4d83e2c8a9c 100644 --- a/server/src/services/execution-workspaces.ts +++ b/server/src/services/execution-workspaces.ts @@ -98,7 +98,7 @@ import { type PullRequestMergeDetailsResolver, } from "./github-pull-request-merge.js"; import { visibleIssueCondition } from "./issue-visibility.js"; -import { createGitRemoteAuthProvider } from "./git-credentials.js"; +import { createIssueGitRemoteAuthProvider } from "./git-credentials.js"; import { readProjectWorkspaceRuntimeConfig } from "./project-workspace-runtime-config.js"; import { workspaceGitOperationScheduler } from "./workspace-git-operation-scheduler.js"; import { isRuntimeOwnedGitBranch } from "./execution-workspace-branch-ownership.js"; @@ -4113,7 +4113,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic try { recipe = readGitCacheRestoration(snapshot); } catch { return { ok: false, code: "rebuild_failed", message: new GitCacheRestorationFailure().message }; } const freshStart = recipe !== null && isClosedExecutionWorkspaceStatus(snapshot.status) && gitRestorePointExpired(recipe); - const gitAuth = createGitRemoteAuthProvider(db, snapshot.companyId, { issueId: issue.id }); + const gitAuth = createIssueGitRemoteAuthProvider(db, snapshot.companyId, issue.id); const [ { ensurePersistedExecutionWorkspaceAvailable }, { workspaceOperationService }, @@ -4168,9 +4168,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic }); }; }, - resolveGitAuth: createGitRemoteAuthProvider(db, snapshot.companyId, { - issueId: issue.id, - }), + resolveGitAuth: createIssueGitRemoteAuthProvider(db, snapshot.companyId, issue.id), }); resolvedBaseCwd = managedWorkspace.cwd; } catch { diff --git a/server/src/services/git-credentials.ts b/server/src/services/git-credentials.ts index 21899e5cef1..69bac2c029e 100644 --- a/server/src/services/git-credentials.ts +++ b/server/src/services/git-credentials.ts @@ -1,6 +1,7 @@ import { companySecrets, heartbeatRuns, + issues, companyMemberships, connectionGrantDelegations, connectionGrants, @@ -9,7 +10,7 @@ import { userSecretDefinitions, type Db, } from "@paperclipai/db"; -import { and, eq, inArray, or } from "drizzle-orm"; +import { and, eq, inArray, or, sql } from "drizzle-orm"; import { isGitHubDotCom } from "./github-fetch.js"; import { secretService } from "./secrets.js"; import { toolAccessService } from "./tool-access.js"; @@ -207,6 +208,19 @@ type GitCredentialSecretsDeps = { * provider instance so one run performs at most one secret resolution (and writes at most * one audit event) no matter how many git operations it authenticates. */ +export function createIssueGitRemoteAuthProvider(db: Db, companyId: string, issueId: string, + create: typeof createGitRemoteAuthProvider = createGitRemoteAuthProvider): GitRemoteAuthProvider { + let provider: Promise | null = null; + return async (remoteUrl: string) => { + provider ??= db.select({ responsibleUserId: issues.responsibleUserId, assigneeAgentId: issues.assigneeAgentId }).from(issues) + .where(and(eq(issues.id, issueId), eq(issues.companyId, companyId))) + .then(([issue]) => create(db, companyId, { + issueId, responsibleUserId: issue?.responsibleUserId ?? null, agentId: issue?.assigneeAgentId ?? null, + })); + return (await provider)(remoteUrl); + }; +} + export function createGitRemoteAuthProvider( db: Db, companyId: string, @@ -310,6 +324,10 @@ export async function resolveManagedGitHubIdentitySelection( }> { const connections = await db.select().from(toolConnections).where(and( eq(toolConnections.companyId, companyId), + or( + sql`${toolConnections.config}->'sourceTemplateKey' = '"github"'::jsonb`, + sql`${toolConnections.transportConfig}->'sourceTemplateKey' = '"github"'::jsonb`, + ), )); const githubConnections = connections.filter((connection) => { const config = connection.config && typeof connection.config === "object" ? connection.config as Record : {}; diff --git a/server/src/services/git-restore-point.ts b/server/src/services/git-restore-point.ts index 698b1395ade..3af9997b693 100644 --- a/server/src/services/git-restore-point.ts +++ b/server/src/services/git-restore-point.ts @@ -595,9 +595,21 @@ export async function restorePointHistoryDigest(input: string, signal?: AbortSig const cwd = await fs.realpath(input); const reflog = await gitText(cwd, ["reflog", "show", "--format=%H", "HEAD"], { signal }).catch(() => ""); const original = await gitText(cwd, ["rev-parse", "--verify", "-q", "ORIG_HEAD^{commit}"], { signal }).catch(() => ""); - const remotes = await gitText(cwd, ["for-each-ref", "--format=%(objectname) %(refname)", "refs/remotes"], { signal }).catch(() => ""); - const remoteReflogs = (await gitText(cwd, ["reflog", "show", "--all", "--format=%H %gD"], { signal }).catch(() => "")) - .split("\n").filter((line) => / refs\/remotes\//.test(line)); + const linked = (await fs.lstat(path.join(cwd, ".git")).catch(() => null))?.isFile() ?? false; + const nested = linked ? await import("./nested-repository-worktrees.js") : null; + const scope = nested ? `refs/heads/${await nested.nestedWorktreeBranchPrefix(cwd)}` : "refs/remotes"; + const remotes = await gitText(cwd, ["for-each-ref", ...nested ? [`--count=${nested.OWNED_NESTED_REF_LIMIT + 1}`] : [], + "--format=%(objectname) %(refname)", scope], { signal }).catch(() => ""); + const remoteReflogs: string[] = []; + if (nested) { + if (remotes.split("\n").filter(Boolean).length > nested.OWNED_NESTED_REF_LIMIT) throw new Error("managed_checkout_local_tip_budget"); + for (const ref of remotes.split("\n").map((line) => line.split(" ")[1]).filter(Boolean)) { + remoteReflogs.push(...(await gitText(cwd, ["reflog", "show", "--format=%H", ref!], { signal }).catch(() => "")).split("\n").filter((line) => SHA.test(line)).map((sha) => `${sha} ${ref}`)); + } + } else { + remoteReflogs.push(...(await gitText(cwd, ["reflog", "show", "--all", "--format=%H %gD"], { signal }).catch(() => "")) + .split("\n").filter((line) => / refs\/remotes\//.test(line))); + } return createHash("sha256").update(JSON.stringify([reflog.split("\n").filter((line) => SHA.test(line)), SHA.test(original) ? original : null, remotes.split("\n").filter(Boolean), remoteReflogs])).digest("hex"); } diff --git a/server/src/services/git-workspace-expiration.ts b/server/src/services/git-workspace-expiration.ts index 12cb8e52dda..5fdd85c7f96 100644 --- a/server/src/services/git-workspace-expiration.ts +++ b/server/src/services/git-workspace-expiration.ts @@ -6,7 +6,7 @@ import { acquireGitExpirationLocks } from "./git-workspace-expiration-locks.js"; import { and, eq, inArray, ne, sql } from "drizzle-orm"; import { executionWorkspaces, issues, heartbeatRuns, nativeRunFinalizations, environmentLeases, workspaceRuntimeServices, issueRecoveryActions, projectWorkspaces, withDedicatedDbConnection, type Db } from "@paperclipai/db"; -import { createGitRemoteAuthProvider } from "./git-credentials.js"; +import { createIssueGitRemoteAuthProvider } from "./git-credentials.js"; import type { StorageService } from "../storage/types.js"; import { TASK_DIRECTORY_RETENTION_KEY } from "./isolated-task-directory-snapshot.js"; import { currentRestorePointSnapshot, RestorePointSkip, type RestorePointLimits } from "./git-restore-point.js"; @@ -37,7 +37,7 @@ export function withAnonymousGitFallback(provider: GitRemoteAuthProvider): GitRe } function expirationGitAuth(db: Db, workspace: Workspace) { - return withAnonymousGitFallback(createGitRemoteAuthProvider(db, workspace.companyId, { issueId: workspace.sourceIssueId! })); + return withAnonymousGitFallback(createIssueGitRemoteAuthProvider(db, workspace.companyId, workspace.sourceIssueId!)); } const EXPIRATION_KEY = "gitCacheExpiration"; diff --git a/server/src/services/git-workspace-restoration.ts b/server/src/services/git-workspace-restoration.ts index f71c1851e7c..5aacb475a80 100644 --- a/server/src/services/git-workspace-restoration.ts +++ b/server/src/services/git-workspace-restoration.ts @@ -18,7 +18,7 @@ import { ensureGitRestorePointCommit, validateGitRestorePointRef, type GitRestor import { getStorageService } from "../storage/index.js"; import type { StorageService } from "../storage/types.js"; import { PROJECT_REPOSITORIES_DIR } from "@paperclipai/adapter-utils/git-workspace-sync"; -import { forgetExpiredNestedRepositoryWorktree, nestedRepositoryBaseCandidates, readLinkedNestedCheckout, recordedNestedRepositoryBase, type LinkedNestedCheckout } from "./nested-repository-worktrees.js"; +import { OWNED_NESTED_REF_LIMIT, forgetExpiredNestedRepositoryWorktree, nestedRepositoryBaseCandidates, readLinkedNestedCheckout, recordedNestedRepositoryBase, type LinkedNestedCheckout } from "./nested-repository-worktrees.js"; const execFile = promisify(execFileCallback); const shaPattern = /^[a-f0-9]{40}$/; @@ -249,7 +249,6 @@ async function assertCommitMessageSaved(cwd: string, gitDir: string, signal?: Ab if (!saved.includes(message.trim())) throw new Error("uncommitted_git_message"); } -const OWNED_NESTED_REF_LIMIT = 64; async function ownedNestedRefs(linked: LinkedNestedCheckout, signal?: AbortSignal) { const refs = (await gitCacheCommand(linked.checkout, ["for-each-ref", `--count=${OWNED_NESTED_REF_LIMIT + 1}`, @@ -380,9 +379,13 @@ async function provenNestedWorktreeHistory(linked: LinkedNestedCheckout, repoUrl const covered = new Set([head, ...shas(await gitCacheCommand(checkout, ["reflog", "show", "--format=%H", "HEAD"], signal).catch(() => "")), ...shas(await gitCacheCommand(checkout, ["rev-parse", "--verify", "-q", "ORIG_HEAD^{commit}"], signal).catch(() => ""))]); - const owned = await ownedNestedRefs(linked, signal); + const allOwned = await ownedNestedRefs(linked, signal); + const remotePrefix = `refs/heads/${linked.branchPrefix}remotes/`; + const owned = allOwned.filter(([, ref]) => !ref.startsWith(remotePrefix)); + const remoteTracked = [...new Set([...allOwned.filter(([, ref]) => ref.startsWith(remotePrefix)).map(([sha]) => sha), + ...await ownedNestedReflogs(linked, allOwned.filter(([, ref]) => ref.startsWith(remotePrefix)), signal)])]; const recorded = new Set([...owned.map(([sha]) => sha), ...await ownedNestedReflogs(linked, owned, signal)]); - if (recorded.size + covered.size > MANAGED_CHECKOUT_LOCAL_TIP_LIMIT) throw new Error("managed_checkout_local_tip_budget"); + if (recorded.size + covered.size + remoteTracked.length > MANAGED_CHECKOUT_LOCAL_TIP_LIMIT) throw new Error("managed_checkout_local_tip_budget"); const current = (await gitCacheCommand(checkout, ["symbolic-ref", "--quiet", "HEAD"], signal).catch(() => "")).trim(); const otherTips = owned.filter(([, ref]) => ref !== current).map(([sha]) => sha); if (otherTips.length && await hasCommitsMissingUpstream(checkout, otherTips, repoUrl, resolveGitAuth, signal, [head])) { @@ -393,7 +396,9 @@ async function provenNestedWorktreeHistory(linked: LinkedNestedCheckout, repoUrl throw new Error("unpushed_managed_checkout_branches"); } const headOnRemote = !await hasCommitsMissingUpstream(checkout, [head], repoUrl, resolveGitAuth, signal); - return { headOnRemote, preserve: [] }; + const preserve = remoteTracked.length + ? await tipsMissingUpstream(checkout, remoteTracked, repoUrl, resolveGitAuth, signal, [...covered, ...recorded]) : []; + return { headOnRemote, preserve }; } async function tipsMissingUpstream(checkout: string, tips: string[], repoUrl: string, diff --git a/server/src/services/github-connection-events.test.ts b/server/src/services/github-connection-events.test.ts new file mode 100644 index 00000000000..93031ecae62 --- /dev/null +++ b/server/src/services/github-connection-events.test.ts @@ -0,0 +1,74 @@ +import { describe, expect, it, vi } from "vitest"; +import { chatEndpoints, type Db } from "@paperclipai/db"; +import { githubConnectionEventService } from "./github-connection-events.js"; +import type { PaperclipCloudConnector } from "./paperclip-cloud-connector.js"; + +function fixture() { + let time = Date.parse("2026-09-04T12:00:00Z"); + const bindings = [{ + grant: { id: "grant", companyId: "company", kind: "user", subjectUserId: "owner", status: "active", + providerTenant: { github: { installationIds: ["101"] } } }, + connection: { id: "connection", config: { sourceTemplateKey: "github", oauth: { connectorProfile: "github.code" } } }, + }]; + const bots: Array<{ companyId: string }> = []; + // Only SQL is stubbed. Backoff, lease admission and concurrent execution use + // the real service; relational persistence has its own Postgres suite. + const select = vi.fn(() => ({ from: (table: unknown) => table === chatEndpoints + ? { where: async () => bots } + : { innerJoin: () => ({ where: async () => bindings }) } })); + const leaseEvents = vi.fn().mockResolvedValue(null); + const connector = { leaseEvents } as unknown as PaperclipCloudConnector; + const service = githubConnectionEventService({ select } as unknown as Db, { connector, now: () => new Date(time) }); + return { service, leaseEvents, select, bots, advance: (ms: number) => { time += ms; } }; +} + +describe("GitHub event polling during active App setup", () => { + it("reaches Cloud during an old empty-queue backoff without expanding idle polling", async () => { + const f = fixture(); + for (const delay of [0, 10_000, 20_000, 40_000, 80_000]) { + f.advance(delay); + await f.service.pollOnce(); + } + expect(f.leaseEvents).toHaveBeenCalledTimes(5); + const queries = f.select.mock.calls.length; + f.bots.push({ companyId: "company" }); + f.advance(5_000); + for (let index = 0; index < 3; index++) await f.service.pollOnce(); + expect(f.select).toHaveBeenCalledTimes(queries); + expect(f.leaseEvents).toHaveBeenCalledTimes(5); + await f.service.pollOnce({ forSetup: true }); + expect(f.leaseEvents).toHaveBeenCalledTimes(6); + const setupQueries = f.select.mock.calls.length; + await f.service.pollOnce({ forSetup: true }); + expect(f.leaseEvents).toHaveBeenCalledTimes(6); + expect(f.select).toHaveBeenCalledTimes(setupQueries); + f.advance(5_000); + await f.service.pollOnce({ forSetup: true }); + expect(f.leaseEvents).toHaveBeenCalledTimes(7); + }); + + it("shares one lease between setup and scheduled callers while Cloud is slow", async () => { + const f = fixture(); + let complete!: (value: null) => void; + f.leaseEvents.mockImplementationOnce(() => new Promise(resolve => { complete = resolve; })); + const scheduled = f.service.pollOnce(); + const setup = f.service.pollOnce({ forSetup: true }); + await vi.waitFor(() => expect(f.leaseEvents).toHaveBeenCalledTimes(1)); + expect(complete).toBeTypeOf("function"); + complete(null); + await Promise.all([scheduled, setup]); + expect(f.leaseEvents).toHaveBeenCalledTimes(1); + }); + + it("releases the shared attempt after an error and retries only after the setup throttle", async () => { + const f = fixture(); + f.leaseEvents.mockRejectedValueOnce(new Error("Cloud unavailable")); + await expect(f.service.pollOnce({ forSetup: true })).rejects.toThrow("Cloud unavailable"); + await f.service.pollOnce({ forSetup: true }); + await f.service.pollOnce(); + expect(f.leaseEvents).toHaveBeenCalledTimes(1); + f.advance(5_000); + await f.service.pollOnce({ forSetup: true }); + expect(f.leaseEvents).toHaveBeenCalledTimes(2); + }); +}); diff --git a/server/src/services/github-connection-events.ts b/server/src/services/github-connection-events.ts index 04022c647a1..6848875815f 100644 --- a/server/src/services/github-connection-events.ts +++ b/server/src/services/github-connection-events.ts @@ -221,6 +221,8 @@ export function githubConnectionEventService( const now = options.now ?? (() => new Date()); let nextPollAt = 0; let emptyPolls = 0; + let nextSetupPollAt = 0; + let inFlight: Promise | null = null; async function activeBindings() { const rows = await db.select({ grant: connectionGrants, connection: toolConnections }) @@ -458,88 +460,101 @@ export function githubConnectionEventService( return "processed" as const; } - return { - async pollOnce(): Promise { - if (now().getTime() < nextPollAt) { - return { leased: 0, processed: 0, duplicate: 0, ignored: 0, failed: 0 }; - } - const bindings = await activeBindings(); - const bots = await db.select().from(chatEndpoints).where(and(eq(chatEndpoints.provider, "github"), sql`${chatEndpoints.setup}->'github'->>'cloudRegistrationId' is not null`, sql`${chatEndpoints.status} <> 'archived'`)); - if (bindings.length === 0 && bots.length === 0) { - nextPollAt = now().getTime() + 5_000; - return { leased: 0, processed: 0, duplicate: 0, ignored: 0, failed: 0 }; - } - const config = options.connector ? null : paperclipCloudConnectorConfigFromEnv(options.env); - const connector = options.connector ?? (config ? createPaperclipCloudConnector({ config }) : null); - if (!connector) { - nextPollAt = now().getTime() + 5 * 60_000; - return { leased: 0, processed: 0, duplicate: 0, ignored: 0, failed: 0 }; - } - const first = bindings[0] ?? { subject: "github-bot-events", companyId: bots[0]!.companyId }; - const lease = await connector.leaseEvents({ subject: first.subject, companyId: first.companyId }); - if (!lease) { - emptyPolls += 1; - nextPollAt = now().getTime() + Math.min(bots.length ? 15_000 : 5 * 60_000, 5_000 * (2 ** Math.min(emptyPolls, 6))); - return { leased: 0, processed: 0, duplicate: 0, ignored: 0, failed: 0 }; - } - emptyPolls = 0; + async function runPoll(): Promise { + const bindings = await activeBindings(); + const bots = await db.select().from(chatEndpoints).where(and(eq(chatEndpoints.provider, "github"), sql`${chatEndpoints.setup}->'github'->>'cloudRegistrationId' is not null`, sql`${chatEndpoints.status} <> 'archived'`)); + if (bindings.length === 0 && bots.length === 0) { nextPollAt = now().getTime() + 5_000; - const result: GitHubConnectionEventPollResult = { - leased: lease.events.length + (lease.failedEventIds?.length ?? 0), - processed: 0, - duplicate: 0, - ignored: 0, - failed: lease.failedEventIds?.length ?? 0, - }; - const acknowledge: string[] = []; - for (const event of lease.events) { - if (event.payload.githubApp !== undefined) { - try { await dispatchGitHubBotCloudEvent(db, event); acknowledge.push(event.id); result.processed += 1; } - catch { result.failed += 1; } - continue; - } - const matched = bindings.filter((binding) => event.bindingIds.includes(binding.id)); - if (matched.length === 0) { - result.ignored += 1; - acknowledge.push(event.id); - continue; - } - try { - const companies = new Map(); - for (const binding of matched) companies.set(binding.companyId, [...(companies.get(binding.companyId) ?? []), binding]); - for (const [companyId, companyBindings] of companies) { - const status = await processForCompany(companyId, companyBindings, event); - result[status] += 1; - } - acknowledge.push(event.id); - if (event.event === "installation" && (event.action === "deleted" || event.action === "suspend")) { - await Promise.all(matched.map((binding) => connector.setWebhookBinding({ - subject: binding.subject, - companyId: binding.companyId, - id: binding.id, - installationId: binding.installationId, - connectionId: binding.connectionId, - grantId: binding.grantId, - active: false, - }))); - } - } catch { - result.failed += 1; - } + return { leased: 0, processed: 0, duplicate: 0, ignored: 0, failed: 0 }; + } + const config = options.connector ? null : paperclipCloudConnectorConfigFromEnv(options.env); + const connector = options.connector ?? (config ? createPaperclipCloudConnector({ config }) : null); + if (!connector) { + nextPollAt = now().getTime() + 5 * 60_000; + return { leased: 0, processed: 0, duplicate: 0, ignored: 0, failed: 0 }; + } + const first = bindings[0] ?? { subject: "github-bot-events", companyId: bots[0]!.companyId }; + const lease = await connector.leaseEvents({ subject: first.subject, companyId: first.companyId }); + if (!lease) { + emptyPolls += 1; + nextPollAt = now().getTime() + Math.min(bots.length ? 15_000 : 5 * 60_000, 5_000 * (2 ** Math.min(emptyPolls, 6))); + return { leased: 0, processed: 0, duplicate: 0, ignored: 0, failed: 0 }; + } + emptyPolls = 0; + nextPollAt = now().getTime() + 5_000; + const result: GitHubConnectionEventPollResult = { + leased: lease.events.length + (lease.failedEventIds?.length ?? 0), + processed: 0, + duplicate: 0, + ignored: 0, + failed: lease.failedEventIds?.length ?? 0, + }; + const acknowledge: string[] = []; + for (const event of lease.events) { + if (event.payload.githubApp !== undefined) { + try { await dispatchGitHubBotCloudEvent(db, event); acknowledge.push(event.id); result.processed += 1; } + catch { result.failed += 1; } + continue; } - if (acknowledge.length > 0) { - await connector.acknowledgeEvents({ - subject: first.subject, - companyId: first.companyId, - leaseId: lease.leaseId, - deliveryIds: acknowledge, - }); + const matched = bindings.filter((binding) => event.bindingIds.includes(binding.id)); + if (matched.length === 0) { + result.ignored += 1; + acknowledge.push(event.id); + continue; } - for (const deliveryId of lease.failedEventIds ?? []) { - logger.warn({ deliveryId, reason: "inner_event_authentication_failed" }, - "GitHub connector event retained for retry"); + try { + const companies = new Map(); + for (const binding of matched) companies.set(binding.companyId, [...(companies.get(binding.companyId) ?? []), binding]); + for (const [companyId, companyBindings] of companies) { + const status = await processForCompany(companyId, companyBindings, event); + result[status] += 1; + } + acknowledge.push(event.id); + if (event.event === "installation" && (event.action === "deleted" || event.action === "suspend")) { + await Promise.all(matched.map((binding) => connector.setWebhookBinding({ + subject: binding.subject, + companyId: binding.companyId, + id: binding.id, + installationId: binding.installationId, + connectionId: binding.connectionId, + grantId: binding.grantId, + active: false, + }))); + } + } catch { + result.failed += 1; } - return result; + } + if (acknowledge.length > 0) { + await connector.acknowledgeEvents({ + subject: first.subject, + companyId: first.companyId, + leaseId: lease.leaseId, + deliveryIds: acknowledge, + }); + } + for (const deliveryId of lease.failedEventIds ?? []) { + logger.warn({ deliveryId, reason: "inner_event_authentication_failed" }, + "GitHub connector event retained for retry"); + } + return result; + } + + return { + pollOnce(input: { forSetup?: boolean } = {}): Promise { + // A manager actively awaiting a dedicated App's signed ping must not + // inherit a personal connection's old empty-queue backoff. This is a + // bounded readiness probe, not an additional background polling loop. + if (inFlight) return inFlight; + const time = now().getTime(); + if (time < (input.forSetup ? nextSetupPollAt : nextPollAt)) { + return Promise.resolve({ leased: 0, processed: 0, duplicate: 0, ignored: 0, failed: 0 }); + } + nextSetupPollAt = time + 5_000; + nextPollAt = Math.max(nextPollAt, nextSetupPollAt); + const work = runPoll().finally(() => { if (inFlight === work) inFlight = null; }); + inFlight = work; + return work; }, }; } diff --git a/server/src/services/github-operation-credentials.ts b/server/src/services/github-operation-credentials.ts index b82e4be4211..c24aeabb5c1 100644 --- a/server/src/services/github-operation-credentials.ts +++ b/server/src/services/github-operation-credentials.ts @@ -112,7 +112,7 @@ export async function resolveGitHubOperationCredentials( runId: string; }, ) { - const { run, context } = await captureRunIdentity(db, input); + const { run, context } = await captureRunIdentity(db, input, { readScope: "github_credentials" }); if (!context) throw forbidden("This run predates managed GitHub credentials"); let summary: GitHubCredentialSummary; let env: Record = {}; diff --git a/server/src/services/heartbeat-run-events.ts b/server/src/services/heartbeat-run-events.ts index ecb7ff5864a..efefbc63b3c 100644 --- a/server/src/services/heartbeat-run-events.ts +++ b/server/src/services/heartbeat-run-events.ts @@ -202,18 +202,36 @@ export async function appendHeartbeatRunEvent( } async function contiguousCursor(db: Db, runId: string, sourceInstanceId: string): Promise { - const rows = await db + const scope = and( + eq(heartbeatRunEvents.runId, runId), + eq(heartbeatRunEvents.sourceInstanceId, sourceInstanceId), + sql`${heartbeatRunEvents.sourceSeq} >= 1`, + ); + const [range] = await db + .select({ + count: sql`count(*)`.mapWith(Number), + max: sql`max(${heartbeatRunEvents.sourceSeq})`.mapWith(Number), + }) + .from(heartbeatRunEvents) + .where(scope); + if (!range || !range.count) return 0; + if (range.max === range.count) return range.count; + const [first] = await db .select({ sourceSeq: heartbeatRunEvents.sourceSeq }) .from(heartbeatRunEvents) - .where(and( - eq(heartbeatRunEvents.runId, runId), - eq(heartbeatRunEvents.sourceInstanceId, sourceInstanceId), - )) - .orderBy(asc(heartbeatRunEvents.sourceSeq)); - let cursor = 0; - for (const row of rows) { - if (row.sourceSeq === cursor + 1) cursor += 1; - else if (row.sourceSeq !== null && row.sourceSeq > cursor + 1) break; - } - return cursor; + .where(and(scope, eq(heartbeatRunEvents.sourceSeq, 1))) + .limit(1); + if (!first) return 0; + const [last] = await db + .select({ sourceSeq: heartbeatRunEvents.sourceSeq }) + .from(heartbeatRunEvents) + .where(and(scope, sql`not exists ( + select 1 from heartbeat_run_events next_event + where next_event.run_id = heartbeat_run_events.run_id + and next_event.source_instance_id = heartbeat_run_events.source_instance_id + and next_event.source_seq = heartbeat_run_events.source_seq + 1 + )`)) + .orderBy(asc(heartbeatRunEvents.sourceSeq)) + .limit(1); + return last?.sourceSeq ?? 0; } diff --git a/server/src/services/heartbeat-run-privacy.ts b/server/src/services/heartbeat-run-privacy.ts index 17b61f4cbbd..d1cb41a19a2 100644 --- a/server/src/services/heartbeat-run-privacy.ts +++ b/server/src/services/heartbeat-run-privacy.ts @@ -1,7 +1,7 @@ import type { Db } from "@paperclipai/db"; import { issues, heartbeatRuns } from "@paperclipai/db"; -import { and, eq } from "drizzle-orm"; -import { canActorReadExecutionWorkspace, type AuthorizationActor, type AuthorizationDecision } from "./authorization.js"; +import { and, eq, inArray } from "drizzle-orm"; +import { canActorReadExecutionWorkspace, type AuthorizationActor, type AuthorizationDecision, type authorizationService } from "./authorization.js"; import { retryIdempotentDatabaseOperation } from "../database-retry.js"; type RunIssueBinding = { @@ -13,6 +13,7 @@ type RunIssueBinding = { export async function canActorReadHeartbeatRun( db: Db, access: { + decideReads?: ReturnType["decideReads"]; decide(input: { actor: AuthorizationActor; action: "issue:read"; @@ -66,6 +67,44 @@ export async function getHeartbeatRunReadDecisions( actor: AuthorizationActor, runs: readonly RunIssueBinding[], ): Promise { + const decideReads = access.decideReads; + if (decideReads) { + const bindings = runs.map(run => ({ companyId: run.companyId, scopeKind: run.scopeKind, issueId: run.issueId })); + const key = (companyId: string, issueId: string) => JSON.stringify([companyId, issueId]); + const issuesByCompany = new Map>(); + for (const run of bindings) { + if (run.scopeKind === "company" || !run.issueId) continue; + let ids = issuesByCompany.get(run.companyId); + if (!ids) { ids = new Set(); issuesByCompany.set(run.companyId, ids); } + ids.add(run.issueId); + } + const resources: Parameters[1] = []; + for (const [companyId, issueIds] of issuesByCompany) { + const ids = [...issueIds]; + for (let offset = 0; offset < ids.length; offset += 500) { + const chunk = ids.slice(offset, offset + 500); + const existing = await retryIdempotentDatabaseOperation(() => db + .select({ id: issues.id }) + .from(issues) + .where(and(eq(issues.companyId, companyId), inArray(issues.id, chunk)))); + for (const { id: issueId } of existing) resources.push({ + type: "issue", companyId, issueId, projectId: null, parentIssueId: null, + assigneeAgentId: null, assigneeUserId: null, + }); + } + } + const allowed = new Map(); + if (resources.length) { + const decisions = await retryIdempotentDatabaseOperation(() => decideReads(actor, resources)); + for (let index = 0; index < resources.length; index++) { + const resource = resources[index]; + if (resource.type === "issue" && resource.issueId) { + allowed.set(key(resource.companyId, resource.issueId), decisions[index]?.allowed === true); + } + } + } + return bindings.map(run => run.scopeKind === "company" || Boolean(run.issueId && allowed.get(key(run.companyId, run.issueId)))); + } const decisions = new Map>(); return Promise.all(runs.map(run => { const binding = { companyId: run.companyId, scopeKind: run.scopeKind, issueId: run.issueId }; diff --git a/server/src/services/heartbeat-run-summary.ts b/server/src/services/heartbeat-run-summary.ts index 25e0de11276..97368cb1da6 100644 --- a/server/src/services/heartbeat-run-summary.ts +++ b/server/src/services/heartbeat-run-summary.ts @@ -298,7 +298,7 @@ export function readAcceptedTaskContinuationSummary( const result = record(resultJson); if (result.finalizationPhase !== "committed") return null; const kind = result.finalizationReasonCode === "live_continuation_registered" ? "same_agent" - : ["scheduled_monitor_waiting", "scheduled_monitor_dispatched"].includes(String(result.finalizationReasonCode)) ? "monitor" : null; + : ["scheduled_monitor_waiting", "scheduled_monitor_dispatched", "scheduled_monitor_preserved"].includes(String(result.finalizationReasonCode)) ? "monitor" : null; if (!kind) return null; for (const candidate of semanticResultCandidates(result)) { const continuation = record(candidate.continuation); diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index 543d96a97f8..8cda4456d6c 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -811,6 +811,9 @@ export function heartbeatService( getAgent, resolveSessionBeforeForWakeup, getRun, + getRunCleanupState, + getRunExecutionWait, + getRunWorkspaceSettlement, getRunStatus, ensureRuntimeState, getTaskSession, @@ -823,6 +826,7 @@ export function heartbeatService( hasResolvablePriorSessionWorkspaceForWake, getRunReadAccess, getRunLogAccess, + getRunWorkspaceAccess, listRuns, getRuntimeStateWithSessions, listTaskSessions, @@ -830,6 +834,8 @@ export function heartbeatService( listEvents, getRetryExhaustedReason, getRunIssueSummary, + getRunIssueControl, + getActiveRunIssueControlForAgent, getActiveRunForAgent, getActiveRunIssueSummaryForAgent, } = createHeartbeatRunState(db); @@ -1014,6 +1020,7 @@ export function heartbeatService( environmentRuntime, scheduleNativeSessionResumeDispatch, getRun, + getRunWorkspaceSettlement, executeRun, activeRunExecutionPromises, cancelHeartbeatNativeRun, @@ -1386,6 +1393,8 @@ export function heartbeatService( const runControl = createHeartbeatRunControl(db, { getRun, + getRunCleanupState, + getRunExecutionWait, options, envOrchestrator, activeRunExecutions, @@ -3987,6 +3996,7 @@ export function heartbeatService( getRunReadAccess, getRunLogAccess, + getRunWorkspaceAccess, getRuntimeState: getRuntimeStateWithSessions, @@ -4007,7 +4017,7 @@ export function heartbeatService( logStore: string | null; logRef: string | null; }, - opts?: { offset?: number; limitBytes?: number }, + opts?: { offset?: number; limitBytes?: number; alignStartToRecord?: boolean; signal?: AbortSignal }, ) => { const run = typeof runOrLookup === "string" @@ -4149,6 +4159,10 @@ export function heartbeatService( getRunIssueSummary, + getRunIssueControl, + + getActiveRunIssueControlForAgent, + getActiveRunForAgent, getActiveRunIssueSummaryForAgent, diff --git a/server/src/services/heartbeat/__tests__/run-control-fixtures.ts b/server/src/services/heartbeat/__tests__/run-control-fixtures.ts index 42f6bea5150..2adce9077c1 100644 --- a/server/src/services/heartbeat/__tests__/run-control-fixtures.ts +++ b/server/src/services/heartbeat/__tests__/run-control-fixtures.ts @@ -13,6 +13,7 @@ import type { HeartbeatRunControlDependencies } from "../run-control.js"; type TestCallbacks = HeartbeatRunControlDependencies & ReturnType & { getRun: Mock; getAgent: Mock; + getRunCleanupState: Mock; setRunStatusFromLive: Mock; getSchedulingSuppression: Mock; options: { closeWarmNativeSessionsForRun: Mock> }; @@ -43,6 +44,7 @@ export function callbacks(db: Db): TestCallbacks { ...state, ...lifecycle, getRun: vi.fn(state.getRun), + getRunCleanupState: vi.fn(state.getRunCleanupState), getAgent: vi.fn(state.getAgent), setRunStatusFromLive: vi.fn(lifecycle.setRunStatusFromLive), setRunStatus: vi.fn(lifecycle.setRunStatus), diff --git a/server/src/services/heartbeat/conversation-events.ts b/server/src/services/heartbeat/conversation-events.ts new file mode 100644 index 00000000000..60e830e6644 --- /dev/null +++ b/server/src/services/heartbeat/conversation-events.ts @@ -0,0 +1,30 @@ +import { sql } from "drizzle-orm"; +import { heartbeatRunEvents } from "@paperclipai/db"; + +/** + * Canonical PRP conversation data, excluding historical tool activity. + * Keep request lifecycles so a cold open cannot hide a question or approval. + * Item identity may occur on started/completed rather than each text delta. + */ +export function conversationEventCondition(runId: string) { + return sql`( + ${heartbeatRunEvents.eventType} like 'runtime_request.%' + or ${heartbeatRunEvents.eventType} in ('run.result.accepted', 'run.terminal', 'turn.completed', 'turn.failed', 'turn.cancelled', 'turn.interrupted') + or ( + ${heartbeatRunEvents.eventType} in ('item.started', 'item.delta', 'item.completed') + and (lower(replace(coalesce(${heartbeatRunEvents.payload} #>> '{prpEvent,payload,kind}', ${heartbeatRunEvents.payload} #>> '{prpEvent,payload,item,kind}', ${heartbeatRunEvents.payload} #>> '{prpEvent,payload,item,type}', ''), '_', '')) in ('agentmessage', 'assistantmessage') + or coalesce( + ${heartbeatRunEvents.payload} #>> '{prpEvent,itemId}', + ${heartbeatRunEvents.payload} #>> '{prpEvent,payload,itemId}', + ${heartbeatRunEvents.payload} #>> '{prpEvent,payload,item,id}', + ${heartbeatRunEvents.payload} #>> '{prpEvent,payload,id}' + ) in ( + select coalesce(e.payload #>> '{prpEvent,itemId}', e.payload #>> '{prpEvent,payload,itemId}', e.payload #>> '{prpEvent,payload,item,id}', e.payload #>> '{prpEvent,payload,id}') + from heartbeat_run_events e + where e.run_id = ${runId} + and e.event_type in ('item.started', 'item.delta', 'item.completed') + and lower(replace(coalesce(e.payload #>> '{prpEvent,payload,kind}', e.payload #>> '{prpEvent,payload,item,kind}', e.payload #>> '{prpEvent,payload,item,type}', ''), '_', '')) in ('agentmessage', 'assistantmessage') + )) + ) + )`; +} diff --git a/server/src/services/heartbeat/native-stop-evidence-read.test.ts b/server/src/services/heartbeat/native-stop-evidence-read.test.ts new file mode 100644 index 00000000000..d53a5d4c417 --- /dev/null +++ b/server/src/services/heartbeat/native-stop-evidence-read.test.ts @@ -0,0 +1,93 @@ +import { randomBytes, randomUUID } from "node:crypto"; +import { heartbeatRuns, issues, type Db } from "@paperclipai/db"; +import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture"; +import { eq, sql } from "drizzle-orm"; +import { describe, expect, it, vi } from "vitest"; +import { isAcknowledgedNativeStop } from "../acknowledged-native-stop.js"; +import { callbacks, runControlPostgresSupport, setupRunControlDatabase } from "./__tests__/run-control-fixtures.js"; +import { readNativeStopExecutionEvidence } from "./native-stop-evidence-read.js"; +import { createHeartbeatRunControl } from "./run-control.js"; + +vi.mock("../local-service-supervisor.js", async importOriginal => ({ + ...await importOriginal(), + terminateLocalService: vi.fn(async () => undefined), +})); + +describe.skipIf(!runControlPostgresSupport.supported)("native Stop evidence reads", () => { + let db: Db; + const { fixture } = setupRunControlDatabase("native-stop-evidence-", value => { db = value; }); + + it("reads current receipts below the byte budget without changing saved data", async () => { + const { company, agent } = await fixture(); + const [issue] = await db.insert(issues).values({ companyId: company.id, title: "Stop evidence fixture" }).returning(); + const id = randomUUID(), bulk = randomBytes(128 * 1024).toString("base64"); + const receipt = { schema: "paperclip.native-cancellation.v1", runId: id, companyId: company.id, issueId: issue.id, + scope: "run", reasonCode: "cancellation_run_only", dispatchState: "acknowledged", dispatched: true, + intentAuditId: randomUUID(), acknowledgementAuditId: randomUUID(), futureProof: { version: 2 } }; + const result = { nativeCancellation: receipt, cancelledByActorType: "user", cancelledByUserId: randomUUID(), + stdout: bulk, stderr: bulk, futureExecutionEvidence: { savedInput: bulk } }; + const [before] = await db.insert(heartbeatRuns).values({ id, companyId: company.id, agentId: agent.id, + nativeIssueId: issue.id, runtimeMode: "native", status: "cancelled", processPid: 123, processGroupId: 456, + contextSnapshot: { savedInput: bulk }, runnerProfileJson: { savedInput: bulk }, stdoutExcerpt: bulk, + stderrExcerpt: bulk, resultJson: result }).returning(); + const events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(db, { heartbeatRuns }, capture); + let evidence: Awaited>; + try { evidence = await readNativeStopExecutionEvidence(db, company.id, id); } + finally { capture.flush(); } + expect(evidence!).toEqual({ id, companyId: company.id, nativeIssueId: issue.id, status: "cancelled", processPid: 123, + processGroupId: 456, resultJson: { nativeCancellation: receipt, cancelledByActorType: "user", cancelledByUserId: result.cancelledByUserId } }); + expect(isAcknowledgedNativeStop(evidence!)).toBe(true); + expect(events.at(-1)).toMatchObject({ completedQueries: 1, groups: 1 }); + expect(Number(events.at(-1)?.responseValueBytes)).toBeLessThan(2048); + expect(JSON.stringify(events)).not.toContain(bulk.slice(0, 64)); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, id)))[0]).toEqual(before); + await db.update(heartbeatRuns).set({ resultJson: { ...result, nativeCancellation: { ...receipt, dispatchState: "pending" } } }) + .where(eq(heartbeatRuns.id, id)); + expect(isAcknowledgedNativeStop((await readNativeStopExecutionEvidence(db, company.id, id))!)).toBe(false); + expect(await readNativeStopExecutionEvidence(db, randomUUID(), id)).toBeNull(); + expect(await readNativeStopExecutionEvidence(db, company.id, randomUUID())).toBeNull(); + }); + + it.each([null, [], false, 42, "not a receipt", { nativeCancellation: null }, { nativeCancellation: { dispatched: "true" } }])( + "keeps malformed receipts fail-closed: %j", async result => { + const { company, agent } = await fixture(); + const [run] = await db.insert(heartbeatRuns).values({ companyId: company.id, agentId: agent.id, + runtimeMode: "native", status: "cancelled" }).returning(); + await db.execute(sql`update ${heartbeatRuns} set result_json = ${JSON.stringify(result)}::jsonb where id = ${run.id}`); + expect(isAcknowledgedNativeStop((await readNativeStopExecutionEvidence(db, company.id, run.id))!)).toBe(false); + }, + ); + + it.each(["string", "array"])("does not transfer a large malformed %s result", async kind => { + const { company, agent } = await fixture(); + const bulk = randomBytes(128 * 1024).toString("base64"); + const saved = kind === "array" ? [bulk] : bulk; + const [run] = await db.insert(heartbeatRuns).values({ companyId: company.id, agentId: agent.id, + runtimeMode: "native", status: "cancelled" }).returning(); + await db.execute(sql`update ${heartbeatRuns} set result_json = ${JSON.stringify(saved)}::jsonb where id = ${run.id}`); + const events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(db, { heartbeatRuns }, capture); + try { + const evidence = await readNativeStopExecutionEvidence(db, company.id, run.id); + expect(evidence?.resultJson).toBeNull(); + expect(isAcknowledgedNativeStop(evidence!)).toBe(false); + } finally { capture.flush(); } + expect(events.at(-1)).toMatchObject({ completedQueries: 1, groups: 1 }); + expect(Number(events.at(-1)?.responseValueBytes)).toBeLessThan(1024); + expect((await db.select({ result: heartbeatRuns.resultJson }).from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0].result).toEqual(saved); + }); + + it("uses the scoped receipt reader for the native continuation recheck", async () => { + const { company, agent } = await fixture(); + const [issue] = await db.insert(issues).values({ companyId: company.id, title: "Native continuation fixture" }).returning(); + const [run] = await db.insert(heartbeatRuns).values({ companyId: company.id, agentId: agent.id, + nativeIssueId: issue.id, runtimeMode: "native", status: "succeeded" }).returning(); + const deps = callbacks(db); + const control = createHeartbeatRunControl(db, deps); + await control.resumeRemoteStopComments(run); + expect(deps.getRun).not.toHaveBeenCalled(); + }); +}); diff --git a/server/src/services/heartbeat/native-stop-evidence-read.ts b/server/src/services/heartbeat/native-stop-evidence-read.ts new file mode 100644 index 00000000000..6ee776bf03f --- /dev/null +++ b/server/src/services/heartbeat/native-stop-evidence-read.ts @@ -0,0 +1,20 @@ +import { heartbeatRuns, type Db } from "@paperclipai/db"; +import { and, eq, sql } from "drizzle-orm"; + +export async function readNativeStopExecutionEvidence(db: Db, companyId: string, runId: string) { + const [run] = await db.select({ + id: heartbeatRuns.id, + companyId: heartbeatRuns.companyId, + nativeIssueId: heartbeatRuns.nativeIssueId, + status: heartbeatRuns.status, + processPid: heartbeatRuns.processPid, + processGroupId: heartbeatRuns.processGroupId, + resultJson: sql`case + when jsonb_typeof(${heartbeatRuns.resultJson}) = 'object' then ( + select coalesce(jsonb_object_agg(key, value), '{}'::jsonb) + from jsonb_each(${heartbeatRuns.resultJson}) + where key = any(array['nativeCancellation', 'cancelledByActorType', 'cancelledByUserId']) + ) else null end`, + }).from(heartbeatRuns).where(and(eq(heartbeatRuns.id, runId), eq(heartbeatRuns.companyId, companyId))).limit(1); + return run ?? null; +} diff --git a/server/src/services/heartbeat/queue.test.ts b/server/src/services/heartbeat/queue.test.ts index 5a31b5ee131..ef889747669 100644 --- a/server/src/services/heartbeat/queue.test.ts +++ b/server/src/services/heartbeat/queue.test.ts @@ -1,12 +1,14 @@ import { claimNativeSessionResumptions } from "../native-runtime/native-finalization-reconciler.js"; import type { Db } from "@paperclipai/db"; import { agents, agentWakeupRequests, caseHandoffs, cases, companies, createDb, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, heartbeatRuns, issueComments, issues, nativeRunFinalizations } from "@paperclipai/db"; -import { eq, sql } from "drizzle-orm"; +import { eq, inArray, sql } from "drizzle-orm"; import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; import { createPostgresWakeQueueAdapter } from "../../modules/wake-queue/adapters/postgres.js"; import { createReleaseIssueExecution } from "../../modules/wake-queue/application/use-cases.js"; import { createHeartbeatQueue, type HeartbeatQueueDependencies } from "./queue.js"; +import { createHeartbeatRunControl } from "./run-control.js"; +import { callbacks as runControlCallbacks } from "./__tests__/run-control-fixtures.js"; import { createHeartbeatQueueTestDependencies as callbacks } from "../../__tests__/helpers/heartbeat-queue.js"; @@ -113,6 +115,116 @@ describe.skipIf(!support.supported)("heartbeat queue database wiring", () => { return { ...seeded, recipient, pending }; } + async function addStrandedMessages(seeded: Awaited>) { + const messages = [seeded.pending]; + for (let index = 0; index < 2; index++) { + const [comment] = await db.insert(issueComments).values({ companyId: seeded.company.id, issueId: seeded.issue.id, + body: `Additional saved feedback ${index}`, authorAgentId: seeded.agent.id }).returning(); + const [wake] = await db.insert(agentWakeupRequests).values({ companyId: seeded.company.id, agentId: seeded.recipient.id, + source: "automation", reason: "issue_execution_deferred", status: "deferred_issue_execution", + payload: { issueId: seeded.issue.id, _paperclipWakeContext: { issueId: seeded.issue.id, + wakeReason: "issue_comment_mentioned", wakeCommentIds: [comment.id] } } }).returning(); + messages.push(wake); + } + await db.update(agentWakeupRequests).set({ updatedAt: new Date(0) }).where(inArray(agentWakeupRequests.id, messages.map(row => row.id))); + return messages; + } + + async function expectSavedMessages(messages: Awaited>) { + const rows = await db.select().from(agentWakeupRequests).where(inArray(agentWakeupRequests.id, messages.map(row => row.id))); + expect(rows).toHaveLength(messages.length); + for (const before of messages) { + const row = rows.find(row => row.id === before.id)!; + expect(row).toMatchObject({ payload: before.payload, status: "deferred_issue_execution", runId: null, + finishedAt: null, error: null }); + expect(row.updatedAt.getTime()).toBeGreaterThan(0); + } + } + + it.each(["legacy", "native"])("keeps saved %s messages while avoiding repeated native source release", async runtimeMode => { + const seeded = await strandedMention(runtimeMode); + const messages = await addStrandedMessages(seeded); + const deps = callbacks(db), queue = createHeartbeatQueue(db, deps); + await queue.resumeQueuedRuns(); + expect(deps.releaseIssueExecutionAndPromote).toHaveBeenCalledTimes(runtimeMode === "native" ? 1 : 3); + await expectSavedMessages(messages); + await queue.resumeQueuedRuns(); + expect(deps.releaseIssueExecutionAndPromote).toHaveBeenCalledTimes(runtimeMode === "native" ? 2 : 6); + expect(deps.executeRun).not.toHaveBeenCalled(); + }); + + it("retries a rejected native release on the next saved message", async () => { + const seeded = await strandedMention("native"), messages = await addStrandedMessages(seeded); + const deps = callbacks(db); + deps.releaseIssueExecutionAndPromote.mockRejectedValueOnce(new Error("Fixture release remains pending")); + await createHeartbeatQueue(db, deps).resumeQueuedRuns(); + expect(deps.releaseIssueExecutionAndPromote).toHaveBeenCalledTimes(2); + await expectSavedMessages(messages); + expect(deps.executeRun).not.toHaveBeenCalled(); + }); + + it("joins the native release before it suppresses another message's duplicate check", async () => { + const seeded = await strandedMention("native"), messages = await addStrandedMessages(seeded); + const deps = callbacks(db); + let finish!: () => void; + deps.releaseIssueExecutionAndPromote.mockImplementationOnce(() => new Promise(resolve => { finish = resolve; })); + let settled = false; + const pending = createHeartbeatQueue(db, deps).resumeQueuedRuns().then(() => { settled = true; }); + try { + await vi.waitFor(() => expect(deps.releaseIssueExecutionAndPromote).toHaveBeenCalledTimes(1)); + expect(settled).toBe(false); + } finally { finish?.(); } + await pending; + expect(deps.releaseIssueExecutionAndPromote).toHaveBeenCalledTimes(1); + await expectSavedMessages(messages); + }); + + it("rechecks a different native source that completes during the pass", async () => { + const seeded = await strandedMention("native"), messages = await addStrandedMessages(seeded); + await db.update(heartbeatRuns).set({ createdAt: new Date(0) }).where(eq(heartbeatRuns.id, seeded.run.id)); + const deps = callbacks(db); + let nextId: string | undefined; + deps.releaseIssueExecutionAndPromote.mockImplementationOnce(async () => { + const [next] = await db.insert(heartbeatRuns).values({ companyId: seeded.company.id, agentId: seeded.agent.id, + runtimeMode: "native", nativeIssueId: seeded.issue.id, status: "succeeded", finishedAt: new Date(), + contextSnapshot: { issueId: seeded.issue.id } }).returning(); + nextId = next.id; + }); + await createHeartbeatQueue(db, deps).resumeQueuedRuns(); + expect(deps.releaseIssueExecutionAndPromote.mock.calls.map(([run]) => run.id)).toEqual([seeded.run.id, nextId]); + await expectSavedMessages(messages); + }); + + it("rechecks changed native evidence on the same source run", async () => { + const seeded = await strandedMention("native"), messages = await addStrandedMessages(seeded); + const deps = callbacks(db); + deps.releaseIssueExecutionAndPromote.mockImplementationOnce(async () => { + await db.update(heartbeatRuns).set({ resultJson: { observedReceipt: "new evidence" } }).where(eq(heartbeatRuns.id, seeded.run.id)); + }); + await createHeartbeatQueue(db, deps).resumeQueuedRuns(); + expect(deps.releaseIssueExecutionAndPromote).toHaveBeenCalledTimes(2); + expect(deps.releaseIssueExecutionAndPromote.mock.calls.map(([run]) => run.id)).toEqual([seeded.run.id, seeded.run.id]); + await expectSavedMessages(messages); + }); + + it("checks each unrelated native task in the same pass", async () => { + const seeded = await strandedMention("native"), firstMessages = await addStrandedMessages(seeded); + const [issue] = await db.insert(issues).values({ companyId: seeded.company.id, title: "Another saved task", status: "todo", assigneeAgentId: seeded.agent.id }).returning(); + const [run] = await db.insert(heartbeatRuns).values({ companyId: seeded.company.id, agentId: seeded.agent.id, + runtimeMode: "native", nativeIssueId: issue.id, status: "succeeded", finishedAt: new Date(), contextSnapshot: { issueId: issue.id } }).returning(); + const [comment] = await db.insert(issueComments).values({ companyId: seeded.company.id, issueId: issue.id, + body: "Other task feedback", authorAgentId: seeded.agent.id }).returning(); + const [pending] = await db.insert(agentWakeupRequests).values({ companyId: seeded.company.id, agentId: seeded.recipient.id, + source: "automation", reason: "issue_execution_deferred", status: "deferred_issue_execution", + payload: { issueId: issue.id, _paperclipWakeContext: { issueId: issue.id, wakeReason: "issue_comment_mentioned", wakeCommentIds: [comment.id] } } }).returning(); + const secondMessages = await addStrandedMessages({ ...seeded, issue, run, pending }); + const deps = callbacks(db); + await createHeartbeatQueue(db, deps).resumeQueuedRuns(); + expect(deps.releaseIssueExecutionAndPromote).toHaveBeenCalledTimes(2); + expect(new Set(deps.releaseIssueExecutionAndPromote.mock.calls.map(([row]) => row.id))).toEqual(new Set([seeded.run.id, run.id])); + await expectSavedMessages([...firstMessages, ...secondMessages]); + }); + it("rechecks a handoff's assignment version under the issue claim lock", async () => { const { company, agent, issue, wake, run } = await fixture("queued"); const [record] = await db.insert(cases).values({ companyId: company.id, caseNumber: 1, identifier: "QM-CASE-1", @@ -137,7 +249,9 @@ describe.skipIf(!support.supported)("heartbeat queue database wiring", () => { }); it.each(["legacy", "native"])("periodic recovery preserves a paused recipient through the real %s release adapter", async runtimeMode => { - const { company, issue, recipient, pending } = await strandedMention(runtimeMode); + const seeded = await strandedMention(runtimeMode); + const { company, issue, recipient } = seeded; + const messages = await addStrandedMessages(seeded); await db.update(agents).set({ status: "paused" }).where(eq(agents.id, recipient.id)); await db.update(issues).set({ status: "blocked" }).where(eq(issues.id, issue.id)); const deps = callbacks(db); @@ -146,16 +260,16 @@ describe.skipIf(!support.supported)("heartbeat queue database wiring", () => { getRoutineEnv: async () => ({ routineId: null, env: null, responsibleUserId: null }), resolveSessionBeforeForWakeup: async () => null, }), recovery: { escalateStrandedAssignedIssue: async () => undefined, escalateStrandedRecoveryIssueInPlace: async () => undefined } }); - deps.releaseIssueExecutionAndPromote.mockImplementation(async (run, options) => { - await release({ companyId: run.companyId, runId: run.id, now: new Date(), ...options }); - }); + const controlDeps = runControlCallbacks(db); + vi.mocked(controlDeps.wakeQueue.releaseIssueExecution).mockImplementation(release); + const control = createHeartbeatRunControl(db, controlDeps); + deps.releaseIssueExecutionAndPromote.mockImplementation(control.releaseIssueExecutionAndPromote); for (let pass = 0; pass < 2; pass++) { await createHeartbeatQueue(db, deps).resumeQueuedRuns(); - expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, pending.id)))[0]) - .toMatchObject({ status: "deferred_issue_execution", runId: null, finishedAt: null, error: null, payload: pending.payload }); + await expectSavedMessages(messages); expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, company.id))).toHaveLength(1); } - expect(deps.releaseIssueExecutionAndPromote).toHaveBeenCalledTimes(2); + expect(deps.releaseIssueExecutionAndPromote).toHaveBeenCalledTimes(runtimeMode === "native" ? 2 : 6); expect(deps.executeRun).not.toHaveBeenCalled(); }); diff --git a/server/src/services/heartbeat/queue.ts b/server/src/services/heartbeat/queue.ts index 515bd683fb3..35ff18e6e88 100644 --- a/server/src/services/heartbeat/queue.ts +++ b/server/src/services/heartbeat/queue.ts @@ -1834,6 +1834,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende AND live_issue_run.status IN ('queued', 'running', 'scheduled_retry'))`, cutoff ? gte(agentWakeupRequests.requestedAt, cutoff) : undefined)) .orderBy(asc(agentWakeupRequests.updatedAt)).limit(50); + const releasedNativeSources = new Set(); for (const { wake } of strandedQueues) { if (!queuedCommentIdsFromWakePayload(wake.payload).length && !await readQueuedInteractionResponse(db, wake.companyId, String(wake.payload?.issueId), wake.payload)) continue; @@ -1845,6 +1846,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende startedAt: heartbeatRuns.startedAt, runtimeMode: heartbeatRuns.runtimeMode, errorCode: heartbeatRuns.errorCode, + sourceVersion: sql`${heartbeatRuns}.xmin::text`, }).from(heartbeatRuns).where(and( eq(heartbeatRuns.companyId, wake.companyId), sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${String(wake.payload?.issueId)}`, @@ -1855,6 +1857,8 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende )); if (!latest || !latest.finishedAt || !isHeartbeatRunTerminalStatus(latest.status)) continue; if (latest.runtimeMode !== "legacy" && latest.runtimeMode !== "native") continue; + const releaseSource = `${latest.companyId}:${String(wake.payload?.issueId)}:${latest.id}:${latest.sourceVersion}`; + if (latest.runtimeMode === "native" && releasedNativeSources.has(releaseSource)) continue; if (latest.runtimeMode === "legacy") { const cancelledAdmission = latest.status === "cancelled" && !latest.startedAt && latest.errorCode === "execution_reconciliation_required"; @@ -1864,7 +1868,9 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende }); } } - await releaseIssueExecutionAndPromote(latest, { suppressImmediateRecovery: true, resumeStrandedQueue: true }).catch(err => { + await releaseIssueExecutionAndPromote(latest, { suppressImmediateRecovery: true, resumeStrandedQueue: true }).then(() => { + if (latest.runtimeMode === "native") releasedNativeSources.add(releaseSource); + }).catch(err => { logger.warn({ err, queueId: wake.id }, "failed to promote stranded comments"); }); } diff --git a/server/src/services/heartbeat/recovery.test.ts b/server/src/services/heartbeat/recovery.test.ts index 8faa8c7e865..34139757468 100644 --- a/server/src/services/heartbeat/recovery.test.ts +++ b/server/src/services/heartbeat/recovery.test.ts @@ -7,6 +7,7 @@ import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } fro import { classifyEnvironmentCapabilities, environmentRuntimeService } from "../environment-runtime.js"; import { environmentService } from "../environments.js"; import { createHeartbeatRecovery, type HeartbeatRecoveryDependencies } from "./recovery.js"; +import { createHeartbeatRunState } from "./run-state.js"; import type { HotRestartIntent } from "../hot-restart.js"; import type { NativeRestartRecoveryClaim, NativeRestartRecoveryDisposition } from "../native-runtime/index.js"; import { NATIVE_OWNERSHIP_UNVERIFIED_ERROR_CODE } from "../native-runtime/native-runner-ownership.js"; @@ -48,6 +49,7 @@ function callbacks(db: Db) { return { enterShutdown: vi.fn(), getRun: vi.fn(async () => null), + getRunWorkspaceSettlement: vi.fn(createHeartbeatRunState(db).getRunWorkspaceSettlement), getAgent: vi.fn(async () => null), appendRunEvent: vi.fn(async () => undefined), setRunStatusIfRunning: vi.fn(async () => ({ updated: false, run: null })), diff --git a/server/src/services/heartbeat/recovery.ts b/server/src/services/heartbeat/recovery.ts index 73c586bdff3..a5197485881 100644 --- a/server/src/services/heartbeat/recovery.ts +++ b/server/src/services/heartbeat/recovery.ts @@ -101,6 +101,7 @@ export interface HeartbeatRecoveryDependencies extends Pick["getRun"]>[1]) => Promise< Awaited["getRun"]>> | null >; + getRunWorkspaceSettlement: ReturnType["getRunWorkspaceSettlement"]; environmentRuntime: ReturnType; pluginWorkerManager?: WorkspaceWorkers; environmentsSvc: Pick, "getById" | "getLeaseById" | "releaseLease">; @@ -360,6 +361,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD environmentRuntime, scheduleNativeSessionResumeDispatch, getRun, + getRunWorkspaceSettlement, executeRun, activeRunExecutionPromises, cancelHeartbeatNativeRun, @@ -1858,7 +1860,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD agentId: string; succeeded: boolean; }) { - const settledRun = await getRun(input.runId); + const settledRun = await getRunWorkspaceSettlement(input.runId); const workspaceSyncReference = readNativeWorkspaceSyncReference( parseObject(settledRun?.runnerProfileJson).nativeWorkspaceSync, ); diff --git a/server/src/services/heartbeat/run-completion.test.ts b/server/src/services/heartbeat/run-completion.test.ts index 39df53e1d52..1dc15b06e9c 100644 --- a/server/src/services/heartbeat/run-completion.test.ts +++ b/server/src/services/heartbeat/run-completion.test.ts @@ -9,6 +9,7 @@ import { createAgentIdentityRedactor } from "../agent-identity-redaction.js"; import { UnresolvedWorkspaceBaseRefError } from "../workspace-runtime.js"; import { LegacyControllerLeaseLostError } from "../legacy-controller-lease.js"; import { requestedRunCancellation } from "../run-cancellation.js"; +import { ProjectAccessDeniedError } from "../project-access-denial.js"; import { createHeartbeatLifecycle, type HeartbeatLifecycleDependencies } from "./run-lifecycle.js"; import { createHeartbeatRunPreparation, ConfigurationIncompleteFailure } from "./run-preparation.js"; import { createHeartbeatRunState, getAdapterSessionCodec } from "./run-state.js"; @@ -77,15 +78,19 @@ it("constructs completion handlers without querying or starting work", () => { }); const support = await getEmbeddedPostgresTestSupport(); -describe.skipIf(!support.supported)("heartbeat run completion boundary", () => { - let database: Awaited>; +describe.skipIf(!support.supported && !process.env.PAPERCLIP_TEST_DATABASE_URL)("heartbeat run completion boundary", () => { + let database: Awaited> | undefined; let db: Db; beforeAll(async () => { // Keep the accounting transaction real without reading provider receipt files. vi.spyOn(usageReceipts, "indexPendingUsageReceipts").mockResolvedValue(new Map()); vi.spyOn(usageReceipts, "recoverPendingRunUsageReceipts").mockResolvedValue([]); - database = await startEmbeddedPostgresTestDatabase("paperclip-heartbeat-completion"); - db = createDb(database.connectionString); + const external = process.env.PAPERCLIP_TEST_DATABASE_URL; + if (external) db = createDb(external); + else { + database = await startEmbeddedPostgresTestDatabase("paperclip-heartbeat-completion"); + db = createDb(database.connectionString); + } await db.execute(sql`set client_min_messages = warning`); }, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS); afterAll(async () => { await db?.$client.end(); await database?.cleanup(); vi.restoreAllMocks(); }); @@ -334,6 +339,18 @@ describe.skipIf(!support.supported)("heartbeat run completion boundary", () => { expect(f.deps.releaseIssueExecutionAndPromote).toHaveBeenCalledWith(expect.objectContaining({ id: f.run.id }), { suppressImmediateRecovery: true }); }); + it("persists a project denial without provider work or an automatic retry", async () => { + const f = await fixture(); + const projectId = randomUUID(); + await f.completion.failRunSetup(f.setup(new ProjectAccessDeniedError(projectId))); + expect(await read(f.run.id)).toMatchObject({ status: "failed", errorCode: "project_access_denied", + resultJson: { projectAccessDenied: { projectId }, executionRecovery: { kind: "bootstrap", providerWorkStarted: false } } }); + expect(f.deps.scheduleBoundedRetryForRun).not.toHaveBeenCalled(); + expect(f.deps.enqueueWakeup).not.toHaveBeenCalled(); + expect(f.deps.scheduleInteractionContinuationInfrastructureRetryIfEligible).not.toHaveBeenCalled(); + expect(f.deps.finalizeIssueCommentPolicy).not.toHaveBeenCalled(); + }); + it("records a setup configuration gap without inventing provider work or applying comment policy", async () => { const f = await fixture(); await f.completion.failRunSetup(f.setup(new ConfigurationIncompleteFailure("missing credential", { configurationIncomplete: { reason: "missing_binding" } }))); diff --git a/server/src/services/heartbeat/run-completion.ts b/server/src/services/heartbeat/run-completion.ts index 7f2b486dfe7..95026919171 100644 --- a/server/src/services/heartbeat/run-completion.ts +++ b/server/src/services/heartbeat/run-completion.ts @@ -1,4 +1,5 @@ import { measureRunPhase } from "../execution-preparation-timing.js"; +import { PROJECT_ACCESS_DENIED, ProjectAccessDeniedError } from "../project-access-denial.js"; import { pendingCaseHandoffForRun } from "./case-handoff-lifecycle.js"; import { NON_RETRYABLE_PREFLIGHT_FAILURE_CODES, isHeartbeatRunTerminalStatus } from "./run-lifecycle.js"; import { ChatControlRecoveryUnresolvedError } from "./queue.js"; @@ -145,6 +146,7 @@ export interface FailHeartbeatRunSetupInput { } function nonRetryablePreflightFailureCode(error: unknown): string | null { + if (error instanceof ProjectAccessDeniedError) return PROJECT_ACCESS_DENIED; if (error instanceof ChatControlRecoveryUnresolvedError) return CHAT_CONTROL_RECOVERY_UNRESOLVED_CODE; if ( @@ -1344,6 +1346,8 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC // action, so it is persisted even when the agent lookup failed and the // agent-scoped stop metadata cannot be merged in. const setupFailureDetails = + (outerErr instanceof ProjectAccessDeniedError + ? { projectAccessDenied: { projectId: outerErr.projectId } } : null) ?? (workspaceGitScanFailure ? { workspaceGitScan: { code: workspaceGitScanFailure.code, @@ -1453,7 +1457,8 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC ).catch(() => undefined); if ( !isWorkspaceValidationFailedRun(livenessRun) && - !isConfigurationIncompleteFailedRun(livenessRun) + !isConfigurationIncompleteFailedRun(livenessRun) && + livenessRun.errorCode !== PROJECT_ACCESS_DENIED ) { await finalizeIssueCommentPolicy(livenessRun, failedAgent).catch( () => undefined, @@ -1462,15 +1467,17 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC // No provider work began. Retry temporary host scan failures with // the existing durable failure budget, before releasing execution. // Generic recovery must not grant a second budget on exhaustion. - await (isTransientWorkspaceGitScanCode(livenessRun.errorCode) - ? scheduleBoundedRetryForRun(livenessRun, failedAgent) - : scheduleInteractionContinuationInfrastructureRetryIfEligible(livenessRun, failedAgent) - ).catch((retryError) => { - logger.warn( - { err: retryError, runId: livenessRun.id }, - "failed to schedule interaction continuation retry after setup failure", - ); - }); + if (livenessRun.errorCode !== PROJECT_ACCESS_DENIED) { + await (isTransientWorkspaceGitScanCode(livenessRun.errorCode) + ? scheduleBoundedRetryForRun(livenessRun, failedAgent) + : scheduleInteractionContinuationInfrastructureRetryIfEligible(livenessRun, failedAgent) + ).catch((retryError) => { + logger.warn( + { err: retryError, runId: livenessRun.id }, + "failed to schedule interaction continuation retry after setup failure", + ); + }); + } } await releaseIssueExecutionAndPromote(livenessRun, { suppressImmediateRecovery: diff --git a/server/src/services/heartbeat/run-control.test.ts b/server/src/services/heartbeat/run-control.test.ts index 7f3991eaa0c..d0d08896538 100644 --- a/server/src/services/heartbeat/run-control.test.ts +++ b/server/src/services/heartbeat/run-control.test.ts @@ -98,7 +98,7 @@ describe("heartbeat run-control module boundary", () => { it("retains leases while native execution ownership is unverified", async () => { const database = guardedDatabase(); const deps = callbacks(database.db); - deps.getRun.mockResolvedValue({ runtimeMode: "native", status: "running", nativePhase: "terminal_failure", errorCode: "native_execution_ownership_unverified" } as Run); + deps.getRunCleanupState.mockResolvedValue({ runtimeMode: "native", status: "running", nativePhase: "terminal_failure", errorCode: "native_execution_ownership_unverified" } as Run); await createHeartbeatRunControl(database.db, deps).releaseEnvironmentLeasesForRun({ runId: "held", companyId: "company", agentId: "agent", status: "failed", providerResourceDisposition: "destroy" }); expect(database.access).not.toHaveBeenCalled(); expect(deps.options.closeWarmNativeSessionsForRun).not.toHaveBeenCalled(); @@ -108,7 +108,7 @@ describe("heartbeat run-control module boundary", () => { it("uses the persisted terminal outcome when choosing warm resource retention", async () => { const database = guardedDatabase(); const deps = callbacks(database.db); - deps.getRun.mockResolvedValue({ status: "failed", runtimeMode: "legacy" } as Run); + deps.getRunCleanupState.mockResolvedValue({ status: "failed", runtimeMode: "legacy" } as Run); await createHeartbeatRunControl(database.db, deps).releaseEnvironmentLeasesForRun({ runId: "failed", companyId: "company", agentId: "agent", status: "succeeded", providerResourceDisposition: "keep_running" }); expect(deps.envOrchestrator.releaseForRun).toHaveBeenCalledWith(expect.objectContaining({ heartbeatRunId: "failed", status: "failed", providerResourceDisposition: "stop_and_retain" })); expect(database.access).not.toHaveBeenCalled(); @@ -117,7 +117,7 @@ describe("heartbeat run-control module boundary", () => { it("defers destruction when a warm session remains busy", async () => { const database = guardedDatabase(); const deps = callbacks(database.db); - deps.getRun.mockResolvedValue({ status: "failed", runtimeMode: "legacy" } as Run); + deps.getRunCleanupState.mockResolvedValue({ status: "failed", runtimeMode: "legacy" } as Run); deps.options.closeWarmNativeSessionsForRun.mockResolvedValue({ closed: 0, busy: 1, failed: 0 }); await createHeartbeatRunControl(database.db, deps).releaseEnvironmentLeasesForRun({ runId: "busy", companyId: "company", agentId: "agent", status: "failed", providerResourceDisposition: "destroy" }); expect(deps.options.closeWarmNativeSessionsForRun).toHaveBeenCalledWith({ runId: "busy", reason: "terminal heartbeat run destroyed its environment lease" }); diff --git a/server/src/services/heartbeat/run-control.ts b/server/src/services/heartbeat/run-control.ts index 7f80f99321a..7387da1f74d 100644 --- a/server/src/services/heartbeat/run-control.ts +++ b/server/src/services/heartbeat/run-control.ts @@ -1,4 +1,5 @@ import { notifyEnvironmentCleanup } from "../environment-cleanup-notifications.js"; +import { readNativeStopExecutionEvidence } from "./native-stop-evidence-read.js"; import { readTaskRetryIntent, reconcileTaskChatRetryReceipts } from "../task-retry.js"; import { PROCESS_IDENTITY_RECORDED, PROCESS_START_REQUESTED, readNativeLocalProcessStop, recordedLocalExecutionStopped } from "../native-local-process-stop.js"; import { isHeartbeatRunTerminalStatus } from "./run-lifecycle.js"; @@ -164,7 +165,7 @@ async function withBudgetStopDelivery(db: Db, work: () => Promise): Promis /** Cancellation shares executor ownership and delegates lifecycle and admission effects. */ export interface HeartbeatRunControlDependencies extends Pick, - "getRun" | "getAgent">, + "getRun" | "getAgent" | "getRunCleanupState" | "getRunExecutionWait">, Pick, "mergeRunStopMetadataForAgent" | "setRunStatusFromLive" | "setRunStatus" | "setWakeupStatus" | "appendRunEvent">, Pick { @@ -324,6 +325,8 @@ function runCancellationFenceSql(input: { export function createHeartbeatRunControl(db: Db, dependencies: HeartbeatRunControlDependencies) { const { getRun, + getRunCleanupState, + getRunExecutionWait, options, envOrchestrator, activeRunExecutions, @@ -359,7 +362,11 @@ export function createHeartbeatRunControl(db: Db, dependencies: HeartbeatRunCont sandboxResource: "keep_running" | "stop_and_reuse" | "destroy_after_turn"; }; }) { - const leaseOwnerRun = await getRun(input.runId); + let leaseOwnerRun = await getRunCleanupState(input.runId); + if (leaseOwnerRun && isNativeRunnerOwnershipHeld(leaseOwnerRun)) return; + const cancelledRun = leaseOwnerRun?.runtimeMode === "native" && leaseOwnerRun.status === "cancelled" + ? await getRun(input.runId) : null; + if (cancelledRun) leaseOwnerRun = cancelledRun; if (leaseOwnerRun && isNativeRunnerOwnershipHeld(leaseOwnerRun)) return; // Recovery can finish workspace copy-back outside the executor's finally. // Successful copy-back does not earn warm retention for a failed turn. @@ -368,11 +375,11 @@ export function createHeartbeatRunControl(db: Db, dependencies: HeartbeatRunCont input.providerResourceDisposition, status, ); - if (leaseOwnerRun?.runtimeMode === "native" && status === "cancelled" - && readNativeWorkspaceSyncReference(leaseOwnerRun.runnerProfileJson?.nativeWorkspaceSync)?.state === "prepared" - && !nativeRunHasNoWorkspaceAccess(leaseOwnerRun)) { + if (cancelledRun?.runtimeMode === "native" && status === "cancelled" + && readNativeWorkspaceSyncReference(cancelledRun.runnerProfileJson?.nativeWorkspaceSync)?.state === "prepared" + && !nativeRunHasNoWorkspaceAccess(cancelledRun)) { try { - const preserved = await preserveNativeFailedWorkspace(db, leaseOwnerRun, { allowPreterminal: true }); + const preserved = await preserveNativeFailedWorkspace(db, cancelledRun, { allowPreterminal: true }); if (!preserved || !hasRequiredWorkspaceRecovery(preserved.resultJson)) return; } catch { // Keep the active source eligible for the bounded retention sweep. @@ -478,13 +485,14 @@ export function createHeartbeatRunControl(db: Db, dependencies: HeartbeatRunCont async function acknowledgeRemoteStop(runId: string, companyId: string) { // The provider receipt arrives after adapter settlement. A remote ACP child // has no host PID, so only this target-aware boundary can acknowledge Stop. - const stopped = await getRun(runId); - if (stopped?.runtimeMode === "native") { + const state = await getRunCleanupState(runId); + if (state?.runtimeMode === "native") { const scopes = await stoppedRemoteCleanupScopes(db, companyId, runId); for (const remoteCleanupScope of scopes ?? []) { completeTerminatedRemoteNativeSessionCleanup({ companyId, runId, remoteCleanupScope }); } } + const stopped = state?.runtimeMode === "legacy" && state.status === "cancelled" ? await getRun(runId) : null; if (stopped?.runtimeMode === "legacy" && stopped.status === "cancelled" && parseObject(stopped.resultJson?.executionCancellation).state === "requested" && await runUsedConversationAdapter(db, stopped) && @@ -502,7 +510,7 @@ export function createHeartbeatRunControl(db: Db, dependencies: HeartbeatRunCont } } - async function resumeRemoteStopComments(run: typeof heartbeatRuns.$inferSelect, requestId?: string) { + async function resumeRemoteStopComments(run: NonNullable>>, requestId?: string) { if (!isHeartbeatRunTerminalStatus(run.status) || adapterExecutionControls.has(run.id)) return; const [preparationCoordinator] = run.runtimeMode === "legacy" && run.status === "cancelled" && !run.runtimeModeResolvedAt ? await db.select().from(nativeRunFinalizations).where(and( @@ -516,7 +524,7 @@ export function createHeartbeatRunControl(db: Db, dependencies: HeartbeatRunCont !(await recordedLocalExecutionStopped(db, run))) return; const issueId = run.nativeIssueId ?? (typeof run.contextSnapshot?.issueId === "string" ? run.contextSnapshot.issueId : null); if (!issueId) return; - const currentRun = run.runtimeMode === "native" ? await getRun(run.id, { includeExecutionEvidence: true }) : null; + const currentRun = run.runtimeMode === "native" ? await readNativeStopExecutionEvidence(db, run.companyId, run.id) : null; const [coordinator] = currentRun ? await db.select({ phase: nativeRunFinalizations.phase, leaseOwner: nativeRunFinalizations.leaseOwner }).from(nativeRunFinalizations).where(and( eq(nativeRunFinalizations.companyId, run.companyId), eq(nativeRunFinalizations.runId, run.id), @@ -529,7 +537,7 @@ export function createHeartbeatRunControl(db: Db, dependencies: HeartbeatRunCont await acknowledgedNativeStopExecutionHasStopped(db, currentRun) && !(await getExecutionBlocker(db, run.companyId, issueId)); const legacyContinuation = run.runtimeMode === "legacy" && - hasConversationContinuationPolicy((await getRun(run.id, { includeExecutionEvidence: true }))?.resultJson) && + hasConversationContinuationPolicy((await getRunExecutionWait(run.companyId, run.id))?.resultJson) && !(await getExecutionBlocker(db, run.companyId, issueId)); if (run.runtimeMode !== "native" && run.runtimeMode !== "legacy") return; const pending = await db.select().from(agentWakeupRequests).where(and( @@ -796,7 +804,7 @@ export function createHeartbeatRunControl(db: Db, dependencies: HeartbeatRunCont const blocker = await getExecutionBlocker(db, wake.companyId, issueId); const sourceId = blocker?.runId; if (!sourceId || !isUuidLike(sourceId)) continue; - const run = await getRun(sourceId, { includeExecutionEvidence: true }); + const run = await getRunExecutionWait(wake.companyId, sourceId); if (!run || run.companyId !== wake.companyId || run.agentId !== wake.agentId) continue; if (canContinueCancelledRun(run)) { await resumeSavedLegacyComments(wake.companyId, wake.id).catch(err => { diff --git a/server/src/services/heartbeat/run-lifecycle.ts b/server/src/services/heartbeat/run-lifecycle.ts index 5f299e855fa..3bbb6f30b40 100644 --- a/server/src/services/heartbeat/run-lifecycle.ts +++ b/server/src/services/heartbeat/run-lifecycle.ts @@ -244,6 +244,7 @@ const LIVENESS_BOOKKEEPING_EVENT_TYPES = [ ]; export const NON_RETRYABLE_PREFLIGHT_FAILURE_CODES = new Set([ + "project_access_denied", "low_trust_isolation_unavailable", "low_trust_requires_isolated_workspace", "low_trust_boundary_mismatch", diff --git a/server/src/services/heartbeat/run-state.test.ts b/server/src/services/heartbeat/run-state.test.ts index 4a729328027..1b9c4276a61 100644 --- a/server/src/services/heartbeat/run-state.test.ts +++ b/server/src/services/heartbeat/run-state.test.ts @@ -1,4 +1,5 @@ import { randomBytes, randomUUID } from "node:crypto"; +import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture"; import type { Db } from "@paperclipai/db"; import { agents, @@ -19,6 +20,7 @@ import { import * as legacy from "../heartbeat.js"; import * as extracted from "./run-state.js"; import { readRunCancellation } from "../run-cancellation.js"; +import { decideQueuedCommentQueueSteering } from "../issue-queued-comment-queue.js"; // SQL_ASCII needs a separate database cluster. Inspect its selected SQL here; // the PostgreSQL tests below exercise the session queries and transactions. @@ -86,12 +88,26 @@ describe("heartbeat run state module", () => { expect(projectedSql(utf8.columns[0].resultJson)).toContain("pg_column_size"); await asciiState.getRun("evidence", { includeExecutionEvidence: true }); expect(projectedSql(ascii.columns[2].resultJson)).toContain("startupPreparationSettledAt"); + await asciiState.getRunWorkspaceSettlement("settlement"); + expect(projectedSql(ascii.columns[3].error)).toBe("NULL"); + expect(ascii.execute).toHaveBeenCalledTimes(1); const raw = readingDatabase("SQL_ASCII"); await extracted.createHeartbeatRunState(raw.db).getRun("raw", { unsafeFullResultJson: true }); expect(raw.execute).not.toHaveBeenCalled(); expect(raw.columns[0].resultJson).toBe(heartbeatRuns.resultJson); }); + it.each(["UTF8", "SQL_ASCII"])("keeps execution-wait reads company scoped with %s text handling", async encoding => { + const reader = readingDatabase(encoding), state = extracted.createHeartbeatRunState(reader.db); + expect(await state.getRunExecutionWait("company", "run")).toBeNull(); + expect(await state.getRunExecutionWait("other-company", "run")).toBeNull(); + expect(reader.scopes).toEqual([["company", "run"], ["other-company", "run"]]); + expect(reader.execute).toHaveBeenCalledTimes(1); + if (encoding === "UTF8") expect(reader.columns[0].error).toBe(heartbeatRuns.error); + else expect(new PgDialect().sqlToQuery((reader.columns[0].error as SQL.Aliased).sql).sql).toBe("NULL"); + expect(new PgDialect().sqlToQuery((reader.columns[0].resultJson as SQL.Aliased).sql).sql).toContain("startupPreparationSettledAt"); + }); + it("loads current run status without large fields or an encoding probe", async () => { const binding = { id: "run", companyId: "company", agentId: "agent", status: "running" }; const reader = readingDatabase("SQL_ASCII", [binding], []); @@ -154,6 +170,195 @@ describePostgres("heartbeat run state database wiring", () => { return { companyId, agent, issue, state: extracted.createHeartbeatRunState(db) }; } + it("reads current execution-wait evidence without transferring saved input or output", async () => { + const { companyId, agent, issue } = await fixture(); + const payload = randomBytes(128 * 1024).toString("base64"); + const context = { issueId: issue.id, connectionIntentResolution: "existing_connection", source: "connection_intent.resolved", + interactionKind: "connection_intent", interactionStatus: "expired", mutation: "interaction", interactionId: randomUUID() }; + const result = { startupCancellation: { beforeNativeSelection: true, futureProof: { version: 2 } }, + startupPreparationSettledAt: "2020-01-01T00:00:00Z", cancellation: { source: "provider", expected: false, + initiator: { type: "provider" }, reason: "Cancelled", recordedAt: "2020-01-01T00:00:00Z" }, + acpToolInventoryComplete: true, acpPendingToolCount: 0, stdout: payload }; + const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId: agent.id, status: "cancelled", runtimeMode: "legacy", + startedAt: new Date("2020-01-01T00:00:00Z"), finishedAt: new Date("2020-01-01T00:01:00Z"), + contextSnapshot: { ...context, prompt: payload }, resultJson: result, + runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner", adapterConfig: { prompt: payload } }, savedInput: payload }, + stdoutExcerpt: payload, stderrExcerpt: payload, + }).returning(); + const observed = createDb(database.connectionString), events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(observed, { heartbeatRuns }, capture); + try { + const state = extracted.createHeartbeatRunState(observed); + const read = await state.getRunExecutionWait(companyId, run.id); + capture.flush(); + const summary = events.find(event => event.event === "paperclip.db_response_capture.summary.v1")!; + expect(summary.completedQueries).toBe(2); + expect(summary.responseValueBytes).toBeLessThan(4096); + expect(read).toMatchObject({ id: run.id, companyId, agentId: agent.id, status: "cancelled", contextSnapshot: context, + runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" } }, + resultJson: { startupCancellation: result.startupCancellation, cancellation: result.cancellation, + acpToolInventoryComplete: true, acpPendingToolCount: 0 } }); + expect(JSON.stringify(read)).not.toContain(payload.slice(0, 64)); + expect(read).not.toHaveProperty("stdoutExcerpt"); + expect(read).not.toHaveProperty("stderrExcerpt"); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0]).toEqual(run); + await db.update(heartbeatRuns).set({ status: "running", resultJson: { ...result, acpPendingToolCount: 1 }, + runnerProfileJson: { adapterDispatch: { adapterType: "process" }, savedInput: payload }, + }).where(eq(heartbeatRuns.id, run.id)); + expect(await state.getRunExecutionWait(companyId, run.id)).toMatchObject({ status: "running", + resultJson: { acpPendingToolCount: 1 }, runnerProfileJson: { adapterDispatch: { adapterType: "process" } } }); + expect(await state.getRunExecutionWait(randomUUID(), run.id)).toBeNull(); + expect(await state.getRunExecutionWait(companyId, randomUUID())).toBeNull(); + for (const value of [null, false, 42, [], {}, { nested: true }, "task"]) { + await db.update(heartbeatRuns).set({ contextSnapshot: { ...context, issueId: value }, + runnerProfileJson: { adapterDispatch: { adapterType: value } }, + }).where(eq(heartbeatRuns.id, run.id)); + expect(await state.getRunExecutionWait(companyId, run.id)).toMatchObject({ contextSnapshot: { issueId: value }, + runnerProfileJson: { adapterDispatch: { adapterType: value } } }); + } + } finally { capture.flush(); await observed.$client.end(); } + }); + + it("reads current workspace access without execution snapshots and retains JSON types", async () => { + const { companyId, agent, issue } = await fixture(); + const payload = randomBytes(128 * 1024).toString("base64"), workspaceId = randomUUID(); + const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId: agent.id, scopeKind: "issue", issueId: issue.id, + contextSnapshot: { executionWorkspaceId: workspaceId, prompt: payload }, runnerProfileJson: { instructions: payload }, + resultJson: { stdout: payload }, stdoutExcerpt: payload, stderrExcerpt: payload, + }).returning(); + const observed = createDb(database.connectionString), events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(observed, { heartbeatRuns }, capture); + try { + const state = extracted.createHeartbeatRunState(observed); + expect(await state.getRunWorkspaceAccess(run.id)).toEqual({ + id: run.id, companyId, scopeKind: "issue", issueId: issue.id, executionWorkspaceId: workspaceId, + }); + capture.flush(); + const summary = events.find(event => event.event === "paperclip.db_response_capture.summary.v1")!; + expect(summary.completedQueries).toBe(1); + expect(summary.responseValueBytes).toBeLessThan(1024); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0]).toEqual(run); + for (const value of [null, true, 42, " workspace-two ", [workspaceId], { id: workspaceId }]) { + await db.update(heartbeatRuns).set({ contextSnapshot: { executionWorkspaceId: value, prompt: payload } }).where(eq(heartbeatRuns.id, run.id)); + expect((await state.getRunWorkspaceAccess(run.id))?.executionWorkspaceId).toEqual(value); + } + expect(await state.getRunWorkspaceAccess(randomUUID())).toBeNull(); + } finally { capture.flush(); await observed.$client.end(); } + }); + + it("reads fresh issue control facts without transferring execution snapshots", async () => { + const { companyId, agent, issue } = await fixture(); + const payload = randomBytes(128 * 1024).toString("base64"); + const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId: agent.id, status: "running", + runtimeMode: "legacy", runtimeModeResolvedAt: null, + startedAt: new Date("2020-01-01T00:00:00Z"), + contextSnapshot: { issueId: issue.id, prompt: payload }, + runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner", adapterConfig: { prompt: payload } }, savedInput: payload }, + resultJson: { stdout: payload }, stdoutExcerpt: payload, stderrExcerpt: payload, + }).returning(); + const observed = createDb(database.connectionString); + const events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(observed, { heartbeatRuns }, capture); + try { + const state = extracted.createHeartbeatRunState(observed); + const direct = await state.getRunIssueControl(run.id), active = await state.getActiveRunIssueControlForAgent(agent.id); + expect(active).toEqual(direct); + expect(direct).toEqual({ id: run.id, companyId, agentId: agent.id, status: "running", runtimeMode: "legacy", + runtimeModeResolvedAt: null, startedAt: run.startedAt, createdAt: run.createdAt, + contextSnapshot: { issueId: issue.id }, runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" } } }); + capture.flush(); + const summary = events.find(event => event.event === "paperclip.db_response_capture.summary.v1")!; + expect(summary.completedQueries).toBe(2); + expect(summary.responseValueBytes).toBeLessThan(2048); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0]).toEqual(run); + await db.update(heartbeatRuns).set({ status: "failed" }).where(eq(heartbeatRuns.id, run.id)); + expect((await state.getRunIssueControl(run.id))?.status).toBe("failed"); + expect(await state.getActiveRunIssueControlForAgent(agent.id)).toBeNull(); + expect(await state.getActiveRunIssueControlForAgent(randomUUID())).toBeNull(); + expect(await state.getRunIssueControl(randomUUID())).toBeNull(); + } finally { capture.flush(); await observed.$client.end(); } + }); + + it("preserves native preparation, legacy steering, and typed issue context in issue control reads", async () => { + const { companyId, agent, issue, state } = await fixture(); + for (const mode of ["native", "legacy"] as const) { + for (const value of [null, false, 42, [], {}, { nested: true }, "paperclip_runner", "process"]) { + const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId: agent.id, status: "running", runtimeMode: mode, + contextSnapshot: { issueId: value }, runnerProfileJson: { adapterDispatch: { adapterType: value } }, + }).returning(); + const narrow = await state.getRunIssueControl(run.id), full = await state.getRun(run.id); + expect(narrow?.contextSnapshot?.issueId).toEqual(value); + expect(narrow?.runnerProfileJson).toEqual({ adapterDispatch: { adapterType: value } }); + for (const queueState of ["queued", "deferred", null] as const) { + const facts = { state: queueState, queueRunRuntimeMode: mode, assignedAgentAdapterType: "paperclip_runner", queuedCommentCount: 1 }; + expect(decideQueuedCommentQueueSteering({ ...facts, activeRun: narrow })) + .toEqual(decideQueuedCommentQueueSteering({ ...facts, activeRun: full })); + } + await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: issue.id }, runtimeModeResolvedAt: new Date("2020-01-01T00:00:00Z") }) + .where(eq(heartbeatRuns.id, run.id)); + const updated = await state.getRunIssueControl(run.id); + expect(updated?.contextSnapshot).toEqual({ issueId: issue.id }); + expect(updated?.runtimeModeResolvedAt).toEqual(new Date("2020-01-01T00:00:00Z")); + } + } + }); + + it("reads cleanup and workspace settlement without transferring saved input or changing the run", async () => { + const { companyId, agent, issue, state } = await fixture(); + const bulk = "cleanup-saved-input-sentinel🙂".repeat(40_000); + const sync = { schema: "paperclip.native-workspace-sync.v1", state: "prepared", resourceDisposition: "destroy" }; + const [run] = await db.insert(heartbeatRuns).values({ + companyId, agentId: agent.id, issueId: issue.id, runtimeMode: "native", status: "failed", + errorCode: "provider_transport_failed", error: "Retain saved files", nativePhase: "terminal_failure", + runnerProfileJson: { nativeWorkspaceSync: sync, savedInput: bulk }, + resultJson: { workspaceExportRetry: { requestId: "retry-source" }, stdout: bulk }, + contextSnapshot: { timerClaimWasFirstHeartbeat: true, savedInput: bulk }, + stdoutExcerpt: bulk, stderrExcerpt: bulk, + }).returning(); + await state.getRunWorkspaceSettlement(run.id); + const events: Array> = []; + const now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(db, { heartbeatRuns }, capture); + expect(await state.getRunCleanupState(run.id)).toEqual({ + id: run.id, companyId, agentId: agent.id, runtimeMode: run.runtimeMode, + status: run.status, errorCode: run.errorCode, nativePhase: run.nativePhase, + }); + expect(await state.getRunWorkspaceSettlement(run.id)).toEqual({ + id: run.id, status: run.status, error: run.error, + runnerProfileJson: { nativeWorkspaceSync: sync }, + resultJson: { workspaceExportRetry: { requestId: "retry-source" } }, + contextSnapshot: { timerClaimWasFirstHeartbeat: true }, + }); + capture.flush(); + const summary = events.find(event => event.event === "paperclip.db_response_capture.summary.v1")!; + expect(summary.completedQueries).toBe(2); + expect(summary.responseValueBytes).toBeLessThan(2000); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0]).toEqual(run); + expect(JSON.stringify(events)).not.toContain("cleanup-saved-input-sentinel"); + expect(await state.getRunCleanupState(randomUUID())).toBeNull(); + expect(await state.getRunWorkspaceSettlement(randomUUID())).toBeNull(); + }); + + it("preserves JSON types in workspace settlement decisions", async () => { + const { companyId, agent, state } = await fixture(); + for (const value of [null, false, true, "true", 0, [], { future: [1, "retained"] }]) { + const [run] = await db.insert(heartbeatRuns).values({ + companyId, agentId: agent.id, + runnerProfileJson: { nativeWorkspaceSync: value }, + resultJson: { workspaceExportRetry: value }, + contextSnapshot: { timerClaimWasFirstHeartbeat: value }, + }).returning({ id: heartbeatRuns.id }); + const actual = await state.getRunWorkspaceSettlement(run.id); + expect(actual?.runnerProfileJson?.nativeWorkspaceSync).toEqual(value); + expect(actual?.resultJson?.workspaceExportRetry).toEqual(value); + expect(actual?.contextSnapshot?.timerClaimWasFirstHeartbeat).toEqual(value); + } + }); + it("retains the consumed queue identity in issue polling summaries", async () => { const { companyId, agent, issue, state } = await fixture(); const [wake] = await db.insert(agentWakeupRequests).values({ diff --git a/server/src/services/heartbeat/run-state.ts b/server/src/services/heartbeat/run-state.ts index 30f9ef0b001..63834fd5584 100644 --- a/server/src/services/heartbeat/run-state.ts +++ b/server/src/services/heartbeat/run-state.ts @@ -1,3 +1,4 @@ +import { conversationEventCondition } from "./conversation-events.js"; import { deriveTaskKey } from "./run-preparation.js"; import { retryIdempotentDatabaseOperation } from "../../database-retry.js"; import { isConversation } from "../agent-conversations.js"; @@ -354,6 +355,18 @@ const heartbeatRunLogAccessColumns = { logRef: heartbeatRuns.logRef, } as const; +const heartbeatRunIssueControlColumns = { + id: heartbeatRuns.id, companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId, + status: heartbeatRuns.status, runtimeMode: heartbeatRuns.runtimeMode, + runtimeModeResolvedAt: heartbeatRuns.runtimeModeResolvedAt, + startedAt: heartbeatRuns.startedAt, createdAt: heartbeatRuns.createdAt, + contextSnapshot: sql | null>`case when jsonb_typeof(${heartbeatRuns.contextSnapshot}) = 'object' + then jsonb_build_object('issueId', ${heartbeatRuns.contextSnapshot}->'issueId') else null end`, + runnerProfileJson: sql | null>`case when jsonb_typeof(${heartbeatRuns.runnerProfileJson}) = 'object' + then jsonb_build_object('adapterDispatch', jsonb_build_object('adapterType', ${heartbeatRuns.runnerProfileJson}->'adapterDispatch'->'adapterType')) + else null end`, +} as const; + const heartbeatRunIssueSummaryColumns = { id: heartbeatRuns.id, runtimeMode: heartbeatRuns.runtimeMode, @@ -1119,6 +1132,55 @@ export function createHeartbeatRunState(db: Db) { .then((rows) => rows[0] ?? null); } + async function getRunExecutionWait(companyId: string, runId: string) { + const safeForLegacyEncoding = await hasUnsafeTextProjectionDatabase(); + return db.select({ + ...heartbeatRunIssueControlColumns, + finishedAt: heartbeatRuns.finishedAt, + errorCode: heartbeatRuns.errorCode, + error: safeForLegacyEncoding ? heartbeatRunSqlAsciiSafeColumns.error : heartbeatRuns.error, + nativeIssueId: heartbeatRuns.nativeIssueId, + nativeSessionId: heartbeatRuns.nativeSessionId, + processPid: heartbeatRuns.processPid, + processGroupId: heartbeatRunProcessGroupIdColumn, + processStartedAt: heartbeatRuns.processStartedAt, + sessionIdAfter: heartbeatRuns.sessionIdAfter, + executionStage: heartbeatRuns.executionStage, + controllerBootId: heartbeatRuns.controllerBootId, + controllerLeaseExpiresAt: heartbeatRuns.controllerLeaseExpiresAt, + resultJson: heartbeatRunExecutionEvidenceColumn, + contextSnapshot: sql | null>`case when jsonb_typeof(${heartbeatRuns.contextSnapshot}) = 'object' + then (select coalesce(jsonb_object_agg(key, value), '{}'::jsonb) + from jsonb_each(${heartbeatRuns.contextSnapshot}) + where key = any(array['issueId', 'connectionIntentResolution', 'source', 'interactionKind', + 'interactionStatus', 'mutation', 'interactionId'])) else null end`, + }).from(heartbeatRuns).where(and( + eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.id, runId), + )).then(rows => rows[0] ?? null); + } + + async function getRunCleanupState(runId: string) { + return db.select({ + id: heartbeatRuns.id, companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId, + runtimeMode: heartbeatRuns.runtimeMode, status: heartbeatRuns.status, + errorCode: heartbeatRuns.errorCode, nativePhase: heartbeatRuns.nativePhase, + }).from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)).then(rows => rows[0] ?? null); + } + + async function getRunWorkspaceSettlement(runId: string) { + const safeForLegacyEncoding = await hasUnsafeTextProjectionDatabase(); + return db.select({ + id: heartbeatRuns.id, status: heartbeatRuns.status, + error: safeForLegacyEncoding ? heartbeatRunSqlAsciiSafeColumns.error : heartbeatRuns.error, + runnerProfileJson: sql | null>`case when ${heartbeatRuns.runnerProfileJson} is null then null + else jsonb_build_object('nativeWorkspaceSync', ${heartbeatRuns.runnerProfileJson}->'nativeWorkspaceSync') end`, + resultJson: sql | null>`case when ${heartbeatRuns.resultJson} is null then null + else jsonb_build_object('workspaceExportRetry', ${heartbeatRuns.resultJson}->'workspaceExportRetry') end`, + contextSnapshot: sql | null>`case when ${heartbeatRuns.contextSnapshot} is null then null + else jsonb_build_object('timerClaimWasFirstHeartbeat', ${heartbeatRuns.contextSnapshot}->'timerClaimWasFirstHeartbeat') end`, + }).from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)).then(rows => rows[0] ?? null); + } + async function getRunStatus(runId: string) { return db .select({ @@ -1148,6 +1210,13 @@ export function createHeartbeatRunState(db: Db) { .then((rows) => rows[0] ?? null); } + async function getRunWorkspaceAccess(runId: string) { + return db.select({ + ...heartbeatRunReadAccessColumns, + executionWorkspaceId: sql`${heartbeatRuns.contextSnapshot}->'executionWorkspaceId'`, + }).from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)).then(rows => rows[0] ?? null); + } + async function getRuntimeState(agentId: string) { return db .select() @@ -1688,6 +1757,9 @@ export function createHeartbeatRunState(db: Db) { getAgent, resolveSessionBeforeForWakeup, getRun, + getRunCleanupState, + getRunExecutionWait, + getRunWorkspaceSettlement, ensureRuntimeState, getTaskSession, getLatestAgentConfigRevision, @@ -1700,6 +1772,7 @@ export function createHeartbeatRunState(db: Db) { getRunStatus, getRunReadAccess, getRunLogAccess, + getRunWorkspaceAccess, listRuns: async ( companyId: string, @@ -1890,7 +1963,7 @@ export function createHeartbeatRunState(db: Db) { }; }, - listEvents: (runId: string, afterSeq = 0, limit = 200) => + listEvents: (runId: string, afterSeq = 0, limit = 200, conversationOnly = false) => db .select() .from(heartbeatRunEvents) @@ -1898,6 +1971,7 @@ export function createHeartbeatRunState(db: Db) { and( eq(heartbeatRunEvents.runId, runId), gt(heartbeatRunEvents.seq, afterSeq), + conversationOnly ? conversationEventCondition(runId) : undefined, ), ) .orderBy(asc(heartbeatRunEvents.seq)) @@ -1931,6 +2005,19 @@ export function createHeartbeatRunState(db: Db) { return run ?? null; }, + getRunIssueControl: async (runId: string) => { + const [run] = await db.select(heartbeatRunIssueControlColumns).from(heartbeatRuns) + .where(eq(heartbeatRuns.id, runId)).limit(1); + return run ?? null; + }, + + getActiveRunIssueControlForAgent: async (agentId: string) => { + const [run] = await db.select(heartbeatRunIssueControlColumns).from(heartbeatRuns) + .where(and(eq(heartbeatRuns.agentId, agentId), eq(heartbeatRuns.status, "running"))) + .orderBy(desc(heartbeatRuns.startedAt)).limit(1); + return run ?? null; + }, + getActiveRunForAgent: async (agentId: string) => { const [run] = await db .select() diff --git a/server/src/services/isolated-repository-rehome.ts b/server/src/services/isolated-repository-rehome.ts index c8c02f85578..6314f5edcc7 100644 --- a/server/src/services/isolated-repository-rehome.ts +++ b/server/src/services/isolated-repository-rehome.ts @@ -23,7 +23,7 @@ import { withExecutionWorkspaceLifecycleLock } from "./execution-workspace-lifec import { loadGitRestorePoint } from "./git-restore-point-storage.js"; import { nestedWorktreeBranchPrefix, readLinkedNestedCheckout, repairMovedNestedRepositoryWorktree, withNestedRepositoryBaseLock } from "./nested-repository-worktrees.js"; import { getStorageService } from "../storage/index.js"; -import { createGitRemoteAuthProvider } from "./git-credentials.js"; +import { createIssueGitRemoteAuthProvider } from "./git-credentials.js"; import { resolvePaperclipInstanceRoot } from "../home-paths.js"; const execFile = promisify(execFileCallback); @@ -598,7 +598,7 @@ async function worktreeOfClone(worktree: string, cloneReal: string, signal?: Abo } function upstreamCheck(db: Db, companyId: string, issueId: string, clone: string, repoUrl: string, signal?: AbortSignal): MissingUpstream { - const auth = withAnonymousGitFallback(createGitRemoteAuthProvider(db, companyId, { issueId })); + const auth = withAnonymousGitFallback(createIssueGitRemoteAuthProvider(db, companyId, issueId)); return (commits, covered) => hasCommitsMissingUpstream(clone, commits, repoUrl, auth, signal, covered); } diff --git a/server/src/services/issue-creation-source.ts b/server/src/services/issue-creation-source.ts index 42f5ce816a7..c40c898fd39 100644 --- a/server/src/services/issue-creation-source.ts +++ b/server/src/services/issue-creation-source.ts @@ -1,14 +1,17 @@ -import { and, asc, eq, inArray, isNotNull } from "drizzle-orm"; +import { and, asc, eq, inArray, isNotNull, sql } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; import { activityLog, agents, heartbeatRuns, issues } from "@paperclipai/db"; import { isUuidLike } from "@paperclipai/shared"; type IssueRow = typeof issues.$inferSelect; type HeartbeatRunRow = typeof heartbeatRuns.$inferSelect; +type CreationSourceRun = Pick & { + contextSnapshot: { issueId: unknown; taskId: unknown; taskKey: unknown; paperclipIssue: { id: unknown } } | null; +}; /** The task a run was executing when it created another task. */ export interface IssueCreationSourceRecord { - run: HeartbeatRunRow; + run: CreationSourceRun; sourceIssue: IssueRow; } @@ -86,7 +89,22 @@ export async function resolveIssueCreationSource( } if (!runId || !isUuidLike(runId)) return null; const [run] = await db - .select() + .select({ + id: heartbeatRuns.id, + companyId: heartbeatRuns.companyId, + agentId: heartbeatRuns.agentId, + scopeKind: heartbeatRuns.scopeKind, + nativeIssueId: heartbeatRuns.nativeIssueId, + issueId: heartbeatRuns.issueId, + contextSnapshot: sql`case + when coalesce(${heartbeatRuns.nativeIssueId}, ${heartbeatRuns.issueId}) is not null then null + else jsonb_build_object( + 'issueId', ${heartbeatRuns.contextSnapshot}->'issueId', + 'taskId', ${heartbeatRuns.contextSnapshot}->'taskId', + 'taskKey', ${heartbeatRuns.contextSnapshot}->'taskKey', + 'paperclipIssue', jsonb_build_object('id', ${heartbeatRuns.contextSnapshot}->'paperclipIssue'->'id') + ) end`, + }) .from(heartbeatRuns) .where(and(eq(heartbeatRuns.id, runId), eq(heartbeatRuns.companyId, issue.companyId))) .limit(1); diff --git a/server/src/services/issue-tree-control.ts b/server/src/services/issue-tree-control.ts index ecf8a07c6fd..1d8af92fd5b 100644 --- a/server/src/services/issue-tree-control.ts +++ b/server/src/services/issue-tree-control.ts @@ -71,6 +71,7 @@ type RestoreTreeStatusResult = TreeStatusUpdateResult & { }; const TERMINAL_ISSUE_STATUSES = new Set(["done", "cancelled"]); +export const ISSUE_TREE_RESUME_EXECUTABLE_STATUSES = ["todo", "in_progress", "in_review"]; const ACTIVE_RUN_STATUSES = ["queued", "running"] as const; const DEFAULT_RELEASE_POLICY: IssueTreeHoldReleasePolicy = { strategy: "manual" }; const MAX_PAUSE_HOLD_ANCESTOR_DEPTH = 100; @@ -325,13 +326,21 @@ function issueSkipReason(input: { return null; } -function buildAffectedAgents(issuesToPreview: IssueTreePreviewIssue[]): IssueTreePreviewAgent[] { +function canWakeOnResume(issue: IssueTreePreviewIssue): boolean { + return !!issue.assigneeAgentId && ISSUE_TREE_RESUME_EXECUTABLE_STATUSES.includes(issue.status); +} + +function buildAffectedAgents( + issuesToPreview: IssueTreePreviewIssue[], + mode: IssueTreeControlMode, +): IssueTreePreviewAgent[] { const byAgentId = new Map(); for (const issue of issuesToPreview) { if (issue.skipped) continue; + if (mode === "resume" && !canWakeOnResume(issue)) continue; const agentIds = new Set(); if (issue.assigneeAgentId) agentIds.add(issue.assigneeAgentId); - if (issue.activeRun) agentIds.add(issue.activeRun.agentId); + if (mode !== "resume" && issue.activeRun) agentIds.add(issue.activeRun.agentId); for (const agentId of agentIds) { const current = byAgentId.get(agentId) ?? { agentId, issueCount: 0, activeRunCount: 0 }; current.issueCount += 1; @@ -388,6 +397,17 @@ function buildWarnings(input: { }); } + if (input.mode === "resume") { + const idleIssueIds = affectedIssues.filter((issue) => !canWakeOnResume(issue)).map((issue) => issue.id); + if (idleIssueIds.length > 0) { + warnings.push({ + code: "resume_tasks_remain_idle", + message: "Blocked, backlog, and unassigned tasks will be unpaused but will not start.", + issueIds: idleIssueIds.sort(), + }); + } + } + if (input.mode === "restore") { const changedIssueIds = input.issuesToPreview .filter((issue) => issue.skipReason === "changed_after_cancel") @@ -676,7 +696,7 @@ export function issueTreeControlService(db: Db) { const activeRuns = activeRunRows .map(toPreviewRun) .sort((a, b) => a.issueId.localeCompare(b.issueId) || a.id.localeCompare(b.id)); - const affectedAgents = buildAffectedAgents(issuesToPreview); + const affectedAgents = buildAffectedAgents(issuesToPreview, input.mode); return { companyId, diff --git a/server/src/services/issues.ts b/server/src/services/issues.ts index 7f3f78e9936..7e820583f18 100644 --- a/server/src/services/issues.ts +++ b/server/src/services/issues.ts @@ -12342,6 +12342,7 @@ export function issueService(db: Db) { afterCommentId?: string | null; order?: "asc" | "desc"; limit?: number | null; + humanOnly?: boolean; }, ) => { const order = opts?.order === "asc" ? "asc" : "desc"; @@ -12352,6 +12353,10 @@ export function issueService(db: Db) { : null; const conditions = [eq(issueComments.issueId, issueId)]; + if (opts?.humanOnly) conditions.push(or( + eq(issueComments.authorType, "user"), + and(isNull(issueComments.authorType), isNotNull(issueComments.authorUserId)), + )!); if (afterCommentId) { // Guard: reject non-UUID cursors before hitting the DB to avoid Postgres type errors. if (!isUuidLike(afterCommentId)) return []; diff --git a/server/src/services/legacy-execution-recovery.ts b/server/src/services/legacy-execution-recovery.ts index f1b16bd0e49..9a3041c88d8 100644 --- a/server/src/services/legacy-execution-recovery.ts +++ b/server/src/services/legacy-execution-recovery.ts @@ -8,7 +8,7 @@ import { normalizeMaxTurnStopReason } from "./heartbeat-stop-metadata.js"; import { claimedAdapterType, hasConversationContinuationPolicy } from "./conversation-continuation.js"; import { canonicalExecutionUuid } from "./execution-identity.js"; import { randomUUID } from "node:crypto"; -import { and, desc, eq, inArray, isNull, or, sql, type SQL } from "drizzle-orm"; +import { and, desc, eq, getTableColumns, inArray, isNull, or, sql, type SQL } from "drizzle-orm"; import { environmentLeases, heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations, type Db } from "@paperclipai/db"; import { issueRecoveryActionService } from "./issue-recovery-actions.js"; import { parseIssueExecutionState } from "./issue-execution-policy.js"; @@ -20,6 +20,24 @@ import { issueService } from "./issues.js"; type Run = typeof heartbeatRuns.$inferSelect; export const LEGACY_RECOVERY_CAUSE = "legacy_execution_requires_reconciliation"; +const unusedReconciliationContextKeys = [ + "executionContinuation", "paperclipWake", "paperclipWakeComment", + "paperclipTaskMarkdown", "paperclipTaskMarkdownCompact", "paperclipTaskAssignment", + "paperclipTaskAssignmentCompact", "paperclipWorkspace", "paperclipEnvironment", "paperclipContinuationSummary", +] as const; + +export const legacyReconciliationRunFields = { + ...getTableColumns(heartbeatRuns), + contextSnapshot: sql`case + when jsonb_typeof(${heartbeatRuns.contextSnapshot}) = 'object' + then ${heartbeatRuns.contextSnapshot} - ARRAY[${sql.join(unusedReconciliationContextKeys.map(key => sql`${key}`), sql`, `)}]::text[] + else ${heartbeatRuns.contextSnapshot} end`, + runnerProfileJson: sql`case + when ${heartbeatRuns.runtimeMode} = 'native' and jsonb_typeof(${heartbeatRuns.runnerProfileJson}) = 'object' + then ${heartbeatRuns.runnerProfileJson} - 'nativeExecutionInput' + else ${heartbeatRuns.runnerProfileJson} end`, +}; + /** Error families describe availability, not whether earlier actions happened. */ export function legacyExecutionNeedsReconciliation( run: Pick & Partial>, @@ -100,7 +118,9 @@ export async function terminalizeLegacyExecution(input: { ) .for("update") : []; - const [current] = await tx.select().from(heartbeatRuns).where(and( + const [current] = await tx.select({ runtimeMode: heartbeatRuns.runtimeMode, status: heartbeatRuns.status, + resultJson: heartbeatRuns.resultJson, executionStatusDeliveryId: heartbeatRuns.executionStatusDeliveryId, + }).from(heartbeatRuns).where(and( eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId), )).for("update"); if (!current || (current.runtimeMode !== "legacy" && !(input.recordRestoreFailureOnly && current.runtimeMode === "native")) || @@ -115,7 +135,8 @@ export async function terminalizeLegacyExecution(input: { .set({ status: input.recordRestoreFailureOnly ? current.status : status, ...patch, - executionStatusDeliveryId: randomUUID(), + executionStatusDeliveryId: input.recordRestoreFailureOnly || current.status !== status || Object.keys(patch ?? {}).length > 0 + ? randomUUID() : current.executionStatusDeliveryId, updatedAt: new Date(), }) .where( @@ -131,10 +152,16 @@ export async function terminalizeLegacyExecution(input: { const retainedLeaseIds = await preserveLegacyWorkspaceRestoreSources(tx as unknown as Db, input.recordRestoreFailureOnly ? { ...updated, status: "failed" } : updated, input.workspaceRestoreSource); if (retainedLeaseIds.length) { - [updated] = await tx.update(heartbeatRuns).set({ resultJson: { + const retainedResult = { ...updated.resultJson, - workspaceRestoreRecovery: { schema: LEGACY_WORKSPACE_RECOVERY_SCHEMA, leaseIds: retainedLeaseIds }, - } }).where(and(eq(heartbeatRuns.id, updated.id), eq(heartbeatRuns.companyId, updated.companyId))).returning(); + workspaceRestoreRecovery: { schema: LEGACY_WORKSPACE_RECOVERY_SCHEMA, leaseIds: [...retainedLeaseIds].sort() }, + }; + const [retained] = await tx.update(heartbeatRuns).set({ + resultJson: retainedResult, + executionStatusDeliveryId: randomUUID(), + }).where(and(eq(heartbeatRuns.id, updated.id), eq(heartbeatRuns.companyId, updated.companyId), + sql`${heartbeatRuns.resultJson} is distinct from ${JSON.stringify(retainedResult)}::jsonb`)).returning(); + if (retained) updated = retained; } const needsRecovery = input.recordRestoreFailureOnly ? retainedLeaseIds.length > 0 || hasRequiredWorkspaceRecovery(updated.resultJson) diff --git a/server/src/services/native-local-process-stop.ts b/server/src/services/native-local-process-stop.ts index 7e74b8eca1c..07ff696ec59 100644 --- a/server/src/services/native-local-process-stop.ts +++ b/server/src/services/native-local-process-stop.ts @@ -1,4 +1,4 @@ -import { and, asc, desc, eq, gt, inArray, isNotNull, isNull, or } from "drizzle-orm"; +import { and, asc, desc, eq, gt, inArray, isNotNull, isNull, or, sql } from "drizzle-orm"; import { environmentLeases, heartbeatRunEvents, heartbeatRuns, nativeRunFinalizations, type Db } from "@paperclipai/db"; import { appendHeartbeatRunEvent } from "./heartbeat-run-events.js"; import { nativeSessionProviderEvidence } from "./native-runtime/native-session-resume.js"; @@ -22,7 +22,8 @@ function absent(pid: number): boolean { /** A fresh workspace is not process containment. Bind an OS observation to * the original host, process birth, and launch event before accepting it. */ -export async function recordedLocalExecutionStopped(db: Db, run: typeof heartbeatRuns.$inferSelect) { +export async function recordedLocalExecutionStopped(db: Db, run: Pick) { const [identity] = await db.select({ payload: heartbeatRunEvents.payload }).from(heartbeatRunEvents).where(and( eq(heartbeatRunEvents.companyId, run.companyId), eq(heartbeatRunEvents.runId, run.id), eq(heartbeatRunEvents.eventType, PROCESS_IDENTITY_RECORDED), isNull(heartbeatRunEvents.sourceEventId), @@ -40,7 +41,8 @@ export async function recordedLocalExecutionStopped(db: Db, run: typeof heartbea * Call in the same transaction as that compare-and-set. Never inspect remote * process IDs in the control-plane host's process table. */ -export async function recordNativeLocalProcessStop(db: Db, run: typeof heartbeatRuns.$inferSelect) { +export async function recordNativeLocalProcessStop(db: Db, run: Pick) { if (run.runtimeMode !== "native" || (!run.processPid && !run.processGroupId)) return false; const leases = await db.select({ provider: environmentLeases.provider }) .from(environmentLeases) @@ -143,13 +145,17 @@ type SettledProcessRelease = "released" | "pending" | "skipped"; export async function releaseSettledNativeRunProcess(db: Db, runId: string): Promise { return db.transaction(async transaction => { const tx = transaction as unknown as Db; - const [run] = await tx.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)).for("update").limit(1); + const [run] = await tx.select({ id: heartbeatRuns.id, companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId, + runtimeMode: heartbeatRuns.runtimeMode, status: heartbeatRuns.status, finishedAt: heartbeatRuns.finishedAt, + processPid: heartbeatRuns.processPid, processGroupId: heartbeatRuns.processGroupId, processStartedAt: heartbeatRuns.processStartedAt, + hasSessionCheckpoint: sql`coalesce(${heartbeatRuns.runnerProfileJson}->'sessionCheckpoint', 'null'::jsonb) <> 'null'::jsonb`, + }).from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)).for("update").limit(1); if (!run || run.runtimeMode !== "native" || !SETTLED_RUN_STATUSES.includes(run.status) || !run.finishedAt || (run.processPid === null && run.processGroupId === null && run.processStartedAt === null)) return "skipped"; const [coordinator] = await tx.select({ phase: nativeRunFinalizations.phase }).from(nativeRunFinalizations) .where(eq(nativeRunFinalizations.runId, run.id)).limit(1); if (coordinator && !SETTLED_COORDINATOR_PHASES.includes(coordinator.phase)) return "skipped"; - if (run.runnerProfileJson?.sessionCheckpoint == null && + if (!run.hasSessionCheckpoint && !(await nativeSessionProviderEvidence(tx, [run.id])).has(run.id)) return "skipped"; const leases = await tx.select().from(environmentLeases).where(and( eq(environmentLeases.companyId, run.companyId), eq(environmentLeases.heartbeatRunId, run.id), diff --git a/server/src/services/native-runtime/__tests__/helpers/committed-replay.ts b/server/src/services/native-runtime/__tests__/helpers/committed-replay.ts new file mode 100644 index 00000000000..7caef7cf216 --- /dev/null +++ b/server/src/services/native-runtime/__tests__/helpers/committed-replay.ts @@ -0,0 +1,34 @@ +import { heartbeatRuns, type Db } from "@paperclipai/db"; +import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture"; +import { eq } from "drizzle-orm"; +import { expect } from "vitest"; + +export async function verifyCommittedReplayPayloadReads( + db: Db, + binding: { runId: string; companyId: string; agentId: string }, + replay: () => Promise, +) { + const { runId, companyId, agentId } = binding; + const bulk = "saved-native-input-sentinel🙂".repeat(40_000); + const [original] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); + await db.update(heartbeatRuns).set({ + contextSnapshot: { ...original.contextSnapshot, savedInput: bulk }, + runnerProfileJson: { savedInput: bulk, nativeExecutionInput: { + schema: "paperclip.native-execution-input.v8", savedInput: bulk, + provider: { kind: "muse", binding: { companyId, agentId, bindingId: runId } }, + } }, stdoutExcerpt: bulk, stderrExcerpt: bulk, + }).where(eq(heartbeatRuns.id, runId)); + const [before] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); + const events: Array> = []; + const now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(db, { heartbeatRuns }, capture); + try { await replay(); } + finally { capture.flush(); } + const summary = events.find(event => event.event === "paperclip.db_response_capture.summary.v1")!; + expect(summary.completedQueries).toBeGreaterThan(0); + expect(summary.responseValueBytes).toBeLessThan(100_000); + const [after] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); + expect(after).toEqual(before); + expect(JSON.stringify(events)).not.toContain("saved-native-input-sentinel"); +} diff --git a/server/src/services/native-runtime/chat-attachment-reuse.test.ts b/server/src/services/native-runtime/chat-attachment-reuse.test.ts index 324dab123b3..d82c6404b59 100644 --- a/server/src/services/native-runtime/chat-attachment-reuse.test.ts +++ b/server/src/services/native-runtime/chat-attachment-reuse.test.ts @@ -1,4 +1,4 @@ -import { createHash, randomUUID } from "node:crypto"; +import { createHash, randomBytes, randomUUID } from "node:crypto"; import { mkdir, mkdtemp } from "node:fs/promises"; import { tmpdir } from "node:os"; import path from "node:path"; @@ -28,6 +28,7 @@ import { toolApplications, toolConnections, } from "@paperclipai/db"; +import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture"; import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; import { createLocalDiskStorageProvider } from "../../storage/local-disk-provider.js"; @@ -36,6 +37,7 @@ import type { StorageService } from "../../storage/types.js"; import { issueService } from "../issues.js"; import { mergeHeartbeatRunResultJson } from "../heartbeat-run-summary.js"; import { + authorizeChatConversationForBoundRun, prepareReusedChatAttachment, resolveExternalChatResponseWaitAuthorization, type ChatAttachmentReuseSource, @@ -307,6 +309,94 @@ describe("native same-conversation chat attachment reuse", () => { }); } + it("bounds current authorization reads without loading unrelated chat content", async () => { + const binding = { companyId, agentId, issueId, runId }; + const [run] = await db.select({ contextSnapshot: heartbeatRuns.contextSnapshot }) + .from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); + const [endpoint] = await db.select().from(chatEndpoints).where(eq(chatEndpoints.id, endpointId)); + const [conversation] = await db.select().from(chatConversations).where(eq(chatConversations.id, conversationId)); + const [resource] = await db.select().from(chatEndpointResources).where(eq(chatEndpointResources.id, resourceId)); + const deliveries = await db.select().from(chatDeliveries).where(eq(chatDeliveries.endpointId, endpointId)); + const bulk = randomBytes(64 * 1024).toString("base64"); + const events: Array> = []; + let capture: ReturnType | undefined; + try { + await db.update(chatEndpoints).set({ communicationInstructions: bulk, + setup: { ...endpoint.setup, unusedContent: bulk } as typeof endpoint.setup, + }).where(eq(chatEndpoints.id, endpointId)); + await db.update(chatConversations).set({ communicationGuidance: bulk }).where(eq(chatConversations.id, conversationId)); + await db.update(chatEndpointResources).set({ metadata: { ...resource.metadata, unusedContent: bulk } }) + .where(eq(chatEndpointResources.id, resourceId)); + for (const delivery of deliveries) { + await db.update(chatDeliveries).set({ normalizedEvent: { ...delivery.normalizedEvent, unusedContent: bulk } }) + .where(eq(chatDeliveries.id, delivery.id)); + } + const now = Date.now(); + capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(db, { chatMessageLinks, chatDeliveries, chatConversations, chatEndpoints, + chatEndpointResources, chatExternalPrincipals, chatIdentityLinks, companyMemberships }, capture); + for (const lockMode of ["read", "blocking", "nonblocking"] as const) { + await expect(db.transaction(tx => authorizeChatConversationForBoundRun(tx as unknown as typeof db, + binding, run.contextSnapshot, lockMode))).resolves.toEqual({ conversationId, endpointId }); + } + capture.flush(); + expect(Number(events.at(-1)?.responseValueBytes)).toBeLessThan(12_000); + expect(Number(events.at(-1)?.completedQueries)).toBeGreaterThan(12); + expect(events.filter(event => event.queryErrors !== undefined).every(event => event.queryErrors === 0)).toBe(true); + expect((await db.select({ setup: chatEndpoints.setup }).from(chatEndpoints) + .where(eq(chatEndpoints.id, endpointId)))[0].setup).toEqual({ ...endpoint.setup, unusedContent: bulk }); + expect((await db.select({ normalizedEvent: chatDeliveries.normalizedEvent }).from(chatDeliveries) + .where(eq(chatDeliveries.id, deliveries[0].id)))[0].normalizedEvent) + .toEqual({ ...deliveries[0].normalizedEvent, unusedContent: bulk }); + } finally { + capture?.flush(); + await db.update(chatEndpoints).set({ communicationInstructions: endpoint.communicationInstructions, setup: endpoint.setup }) + .where(eq(chatEndpoints.id, endpointId)); + await db.update(chatConversations).set({ communicationGuidance: conversation.communicationGuidance }) + .where(eq(chatConversations.id, conversationId)); + await db.update(chatEndpointResources).set({ metadata: resource.metadata }).where(eq(chatEndpointResources.id, resourceId)); + for (const delivery of deliveries) { + await db.update(chatDeliveries).set({ normalizedEvent: delivery.normalizedEvent }).where(eq(chatDeliveries.id, delivery.id)); + } + } + }); + + it.each([ + { label: "matching numeric generation", generation: 7, fingerprint: "a".repeat(64), allowed: true }, + { label: "string generation", generation: "7", fingerprint: "a".repeat(64), allowed: false }, + { label: "wrong generation", generation: 8, fingerprint: "a".repeat(64), allowed: false }, + { label: "missing generation", generation: undefined, fingerprint: "a".repeat(64), allowed: false }, + { label: "invalid fingerprint", generation: 7, fingerprint: "not-a-fingerprint", allowed: false }, + { label: "numeric fingerprint", generation: 7, fingerprint: 7, allowed: false }, + ])("preserves setup-test authorization for $label", async ({ generation, fingerprint, allowed }) => { + const [endpoint] = await db.select().from(chatEndpoints).where(eq(chatEndpoints.id, endpointId)); + const deliveries = await db.select().from(chatDeliveries).where(eq(chatDeliveries.endpointId, endpointId)); + const [run] = await db.select({ contextSnapshot: heartbeatRuns.contextSnapshot }).from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); + try { + const setup = { step: "test" as const, testStartedAt: "2000-01-01T00:00:00.000Z", runtimeGeneration: 7 }; + await db.update(chatEndpoints).set({ status: "verifying", setup }).where(eq(chatEndpoints.id, endpointId)); + for (const delivery of deliveries) { + await db.update(chatDeliveries).set({ + normalizedEvent: { runtimeContext: { generation, credentialFingerprint: fingerprint } }, + receivedAt: new Date("2000-01-01T00:00:01.000Z"), + processedAt: new Date("2000-01-01T00:00:02.000Z"), + }) + .where(eq(chatDeliveries.id, delivery.id)); + } + const result = db.transaction(tx => authorizeChatConversationForBoundRun(tx as unknown as typeof db, + { companyId, agentId, issueId, runId }, run.contextSnapshot)); + if (allowed) await expect(result).resolves.toEqual({ conversationId, endpointId }); + else await expect(result).rejects.toThrow("paperclip_runner_chat_attachment_binding_denied"); + } finally { + await db.update(chatEndpoints).set({ status: endpoint.status, setup: endpoint.setup }).where(eq(chatEndpoints.id, endpointId)); + for (const delivery of deliveries) { + await db.update(chatDeliveries).set({ normalizedEvent: delivery.normalizedEvent, + receivedAt: delivery.receivedAt, processedAt: delivery.processedAt, + }).where(eq(chatDeliveries.id, delivery.id)); + } + } + }); + it("authorizes an external-chat response wait only from current durable policy", async () => { const binding = { companyId, agentId, issueId, runId }; const alternateRunId = randomUUID(); diff --git a/server/src/services/native-runtime/chat-attachment-reuse.ts b/server/src/services/native-runtime/chat-attachment-reuse.ts index db3ca63d9e9..3d30cf42f74 100644 --- a/server/src/services/native-runtime/chat-attachment-reuse.ts +++ b/server/src/services/native-runtime/chat-attachment-reuse.ts @@ -112,6 +112,17 @@ export type ExternalChatResponseWaitAuthorization = type AuthorizationLockMode = "blocking" | "nonblocking" | "read"; +type AuthorizationEndpoint = Pick; +type AuthorizationConversation = Pick; +type AuthorizationResource = Pick; + class ExternalChatWaitAuthorizationContentionError extends Error { constructor() { super("paperclip_external_chat_wait_authorization_contended"); @@ -267,9 +278,9 @@ function decodeListCursor( } function destinationAllowed( - endpoint: typeof chatEndpoints.$inferSelect, - conversation: typeof chatConversations.$inferSelect, - resource: typeof chatEndpointResources.$inferSelect | null, + endpoint: AuthorizationEndpoint, + conversation: AuthorizationConversation, + resource: AuthorizationResource | null, ): boolean { if (conversation.isDirectMessage) return endpoint.allowDirectMessages; if (!resource || resource.availability !== "available") return false; @@ -283,7 +294,7 @@ function destinationAllowed( } function currentSetupTestAcceptsBoundDeliveries( - endpoint: typeof chatEndpoints.$inferSelect, + endpoint: AuthorizationEndpoint, deliveries: Array<{ normalizedEvent: unknown; processedAt: Date | null; @@ -326,7 +337,7 @@ function currentSetupTestAcceptsBoundDeliveries( async function principalAuthorized( tx: Db, - endpoint: typeof chatEndpoints.$inferSelect, + endpoint: AuthorizationEndpoint, principalId: string, lockMode: AuthorizationLockMode = "blocking", ): Promise { @@ -445,10 +456,32 @@ export async function authorizeChatConversationForBoundRun( const linksQuery = tx .select({ commentId: chatMessageLinks.commentId, - conversation: chatConversations, - endpoint: chatEndpoints, + conversation: { + id: chatConversations.id, + resourceId: chatConversations.resourceId, + isDirectMessage: chatConversations.isDirectMessage, + }, + endpoint: { + id: chatEndpoints.id, + companyId: chatEndpoints.companyId, + provider: chatEndpoints.provider, + providerAccountId: chatEndpoints.providerAccountId, + status: chatEndpoints.status, + allowDirectMessages: chatEndpoints.allowDirectMessages, + allowGroupChats: chatEndpoints.allowGroupChats, + allowUnlinkedPeople: chatEndpoints.allowUnlinkedPeople, + sponsorUserId: chatEndpoints.sponsorUserId, + setup: sql`jsonb_build_object( + 'step', ${chatEndpoints.setup}->'step', + 'testStartedAt', ${chatEndpoints.setup}->'testStartedAt', + 'runtimeGeneration', ${chatEndpoints.setup}->'runtimeGeneration' + )`, + }, principalId: chatDeliveries.principalId, - normalizedEvent: chatDeliveries.normalizedEvent, + normalizedEvent: sql`jsonb_build_object('runtimeContext', jsonb_build_object( + 'generation', ${chatDeliveries.normalizedEvent}->'runtimeContext'->'generation', + 'credentialFingerprint', ${chatDeliveries.normalizedEvent}->'runtimeContext'->'credentialFingerprint' + ))`, processedAt: chatDeliveries.processedAt, receivedAt: chatDeliveries.receivedAt, }) @@ -518,7 +551,11 @@ export async function authorizeChatConversationForBoundRun( const resource = conversation.resourceId ? await (() => { const query = tx - .select() + .select({ + availability: chatEndpointResources.availability, + type: chatEndpointResources.type, + enabled: chatEndpointResources.enabled, + }) .from(chatEndpointResources) .where( and( diff --git a/server/src/services/native-runtime/external-chat-wait.integration.test.ts b/server/src/services/native-runtime/external-chat-wait.integration.test.ts index b0ee8e0c5f6..70a6933e80a 100644 --- a/server/src/services/native-runtime/external-chat-wait.integration.test.ts +++ b/server/src/services/native-runtime/external-chat-wait.integration.test.ts @@ -4,6 +4,7 @@ import { tmpdir } from "node:os"; import path from "node:path"; import { and, eq, ne, sql } from "drizzle-orm"; +import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture"; import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; import { agentWakeupRequests, @@ -71,6 +72,7 @@ import { voiceCredentialFingerprint } from "../voice/voice-session-store.js"; import { nativeSha256 } from "./canonical.js"; import { questionResponseDeliveryValues } from "../question-response-delivery.js"; import { resolveExternalChatQuestionResponse } from "./external-chat-question-response.js"; +import { listExternalReviewWaits } from "./external-review-wait.js"; import { materializeExternalChatQuestionResponseInput } from "./external-chat-question-response-input.js"; import * as nativeInteractionBridge from "./native-interaction-bridge.js"; import { defaultGitHubReviewPolicy, type AskUserQuestionsInteraction } from "@paperclipai/shared"; @@ -866,6 +868,50 @@ describe("native external-chat response wait", () => { }; } + it.each(["telegram", "github", "github_board_answer", "slack_board_answer"] as const)( + "reads small current external review waits without saved prompts (%s)", async source => { + const fixture = await (async () => { + if (source === "github_board_answer" || source === "slack_board_answer") { + const answered = await seedBoardAnsweredTurn(true, source === "github_board_answer" ? "github" : "slack"); + await attestAnswer(answered); + return answered; + } + return seedWaitTurn(source); + })(); + const finalized = await finalizeNativeRun({ db, runId: fixture.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true }); + expect(finalized.phase).toBe("committed"); + await db.update(statusDecisions).set({ toStatus: "in_review" }).where(eq(statusDecisions.id, finalized.decisionId!)); + await db.update(issues).set({ status: "in_review" }).where(eq(issues.id, fixture.issueId)); + const [decision] = await db.select({ reason: statusDecisions.reasonCode, state: statusDecisions.applicationState }) + .from(statusDecisions).where(eq(statusDecisions.id, finalized.decisionId!)); + const governed = source === "github_board_answer" || source === "slack_board_answer"; + expect(decision).toEqual({ reason: governed ? "governed_response_waiting" : "external_chat_response_waiting", state: "applied" }); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId)); + expect(run!.status).toBe("succeeded"); + const contextSnapshot = { ...run!.contextSnapshot, savedPrompt: "external-wait-saved-input🙂".repeat(30_000) }; + await db.update(heartbeatRuns).set({ contextSnapshot }).where(eq(heartbeatRuns.id, fixture.runId)); + await authorizeChatConversationForBoundRun(db, fixture, contextSnapshot, "read"); + const observed = createDb(temporary?.connectionString ?? externalTestDatabaseUrl!); + const events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(observed, { issues, heartbeatRuns }, capture); + try { + expect(await listExternalReviewWaits(observed, fixture.companyId, [fixture.issueId])).toEqual(governed ? [] : [ + expect.objectContaining({ companyId: fixture.companyId, issueId: fixture.issueId }), + ]); + capture.flush(); + const reads = events.filter(event => event.table === "issues" && event.operation === "select"); + expect(reads.length).toBeGreaterThan(0); + expect(reads.reduce((bytes, event) => bytes + Number(event.responseValueBytes), 0)).toBeLessThan(4096); + expect(await listExternalReviewWaits(observed, randomUUID(), [fixture.issueId])).toEqual([]); + expect(await listExternalReviewWaits(observed, fixture.companyId, [])).toEqual([]); + await db.update(companyMemberships).set({ status: "suspended" }).where(eq(companyMemberships.companyId, fixture.companyId)); + expect(await listExternalReviewWaits(observed, fixture.companyId, [fixture.issueId])).toEqual([]); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId)))[0]?.contextSnapshot).toEqual(contextSnapshot); + } finally { capture.flush(); await observed.$client.end(); } + }, + ); + it("resumes an oversized Slack multi-select answered through its Paperclip handoff", async () => { const fixture = await seedBoardAnsweredTurn(true, "slack", true); const [publication] = await db.select().from(chatPublications).where(eq(chatPublications.id, fixture.publicationId)); diff --git a/server/src/services/native-runtime/external-review-wait.ts b/server/src/services/native-runtime/external-review-wait.ts index fe5347a9cea..4a63ad6510d 100644 --- a/server/src/services/native-runtime/external-review-wait.ts +++ b/server/src/services/native-runtime/external-review-wait.ts @@ -1,6 +1,23 @@ -import { and, eq, inArray } from "drizzle-orm"; +import { and, eq, inArray, sql } from "drizzle-orm"; import { chatConversations, chatEndpoints, heartbeatRuns, issues, statusDecisions, type Db } from "@paperclipai/db"; import { authorizeChatConversationForBoundRun } from "./chat-attachment-reuse.js"; +import { EXTERNAL_CHAT_QUESTION_RESPONSE_KEY } from "./external-chat-question-response.js"; + +const authorizationContextKeys = [ + "source", "issueId", "wakeReason", "interactionKind", "interactionStatus", "externalChatContinuation", + "interactionId", "sourceRunId", "sourceCommentId", "wakeCommentId", "wakeCommentIds", + "paperclipHarnessCheckedOut", "paperclipExternalChatExecutionBound", + EXTERNAL_CHAT_QUESTION_RESPONSE_KEY, +] as const; + +const authorizationContext = sql | null>` + case when ${heartbeatRuns.contextSnapshot} is null then null else ( + select coalesce(jsonb_object_agg(field.key, field.value), '{}'::jsonb) + from jsonb_each(case when jsonb_typeof(${heartbeatRuns.contextSnapshot}) = 'object' + then ${heartbeatRuns.contextSnapshot} else '{}'::jsonb end) as field(key, value) + where field.key in (${sql.join(authorizationContextKeys.map(key => sql`${key}`), sql`, `)}) + ) end +`; export async function listExternalReviewWaits(db: Db, companyId: string, issueIds: string[]) { if (!issueIds.length) return []; @@ -12,7 +29,7 @@ export async function listExternalReviewWaits(db: Db, companyId: string, issueId createdAt: statusDecisions.createdAt, runId: heartbeatRuns.id, agentId: heartbeatRuns.agentId, - contextSnapshot: heartbeatRuns.contextSnapshot, + contextSnapshot: authorizationContext, }).from(issues) .innerJoin(statusDecisions, and( eq(statusDecisions.id, issues.lastStatusDecisionId), diff --git a/server/src/services/native-runtime/native-finalization-reconciler.ts b/server/src/services/native-runtime/native-finalization-reconciler.ts index ec391457b07..283aac76b2d 100644 --- a/server/src/services/native-runtime/native-finalization-reconciler.ts +++ b/server/src/services/native-runtime/native-finalization-reconciler.ts @@ -742,8 +742,8 @@ async function reconcileNativeFinalizationsPass( resultId: nativeRunFinalizations.resultId, assessmentId: nativeRunFinalizations.assessmentId, decisionId: nativeRunFinalizations.decisionId, - runnerProfileJson: heartbeatRuns.runnerProfileJson, - contextSnapshot: heartbeatRuns.contextSnapshot, + nativeWorkspaceSync: sql`${heartbeatRuns.runnerProfileJson}->'nativeWorkspaceSync'`, + nativeStatusWakeIntentId: sql`${heartbeatRuns.contextSnapshot}->'nativeStatusWakeIntentId'`, }) .from(heartbeatRuns) .innerJoin(nativeRunFinalizations, eq(nativeRunFinalizations.runId, heartbeatRuns.id)) @@ -955,8 +955,7 @@ async function reconcileNativeFinalizationsPass( if (reviewBarrier && reviewBarrier.status !== "succeeded") continue; const childCompletionRecipient = { companyId: row.companyId, issueId: row.issueId, agentId: row.agentId, runId: row.runId, - sourceIntentId: typeof record(row.contextSnapshot).nativeStatusWakeIntentId === "string" - ? record(row.contextSnapshot).nativeStatusWakeIntentId as string : null, + sourceIntentId: typeof row.nativeStatusWakeIntentId === "string" ? row.nativeStatusWakeIntentId : null, }; const hasPendingChildCompletion = await hasPendingNativeChildCompletion(db, childCompletionRecipient); const reassessmentRow = await recordNativeWorkAssessment({ @@ -1095,7 +1094,7 @@ async function reconcileNativeFinalizationsPass( projectRunStatus: true, preserveProviderAttempt: Boolean( readNativeWorkspaceSyncReference( - record(row.runnerProfileJson).nativeWorkspaceSync, + row.nativeWorkspaceSync, ), ), }); @@ -1153,7 +1152,7 @@ async function reconcileNativeFinalizationsPass( projectRunStatus: true, preserveProviderAttempt: Boolean( readNativeWorkspaceSyncReference( - record(row.runnerProfileJson).nativeWorkspaceSync, + row.nativeWorkspaceSync, ), ), }); diff --git a/server/src/services/native-runtime/native-harness-backup-expiration.ts b/server/src/services/native-runtime/native-harness-backup-expiration.ts index 448997670e1..d4c0975f0b5 100644 --- a/server/src/services/native-runtime/native-harness-backup-expiration.ts +++ b/server/src/services/native-runtime/native-harness-backup-expiration.ts @@ -17,10 +17,15 @@ import { nativeSessionScopeDigestIsBusy } from "./native-session-executor.js"; export const NATIVE_HARNESS_BACKUP_EXPIRATION_INTERVAL_MS = 60 * 60 * 1000; export const NATIVE_HARNESS_BACKUP_EXPIRATION_STARTUP_DELAY_MS = 2 * 60 * 1000; export const NATIVE_HARNESS_BACKUP_EXPIRATION_DEFAULT_DAYS = 7; +export const NATIVE_RUNNER_EVIDENCE_RETENTION_DEFAULT_DAYS = 7; const DAY_MS = 24 * 60 * 60 * 1000; const SCOPE_PATTERN = /^[0-9a-f]{64}$/; const UUID_PATTERN = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i; +const CONTINUITY_ARCHIVE_PATTERN = /^(\d{1,15})-[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i; +const QUARANTINE_PATTERN = + /^([0-9a-f]{64})\.(?:identity_indeterminate|identity_mismatch)\.(\d{1,15})\.\d{1,10}\.[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i; +const QUARANTINE_CLEANUP_PATTERN = /^[0-9a-f]{64}\.identity_indeterminate\.cleanup\./; const EXPIRED_PREFIX = ".expired-"; const RETIRING_PREFIX = ".retiring-"; const LIVE_RUN_STATUSES = ["queued", "scheduled_retry", "running"]; @@ -31,6 +36,7 @@ const MAX_CONTROL_PLANE_STATE_BYTES = 4 * 1024 * 1024; export type NativeHarnessBackupExpirationOptions = { closedForDays?: number; + evidenceRetentionDays?: number; dryRun?: boolean; maxEntries?: number; maxRemovals?: number; @@ -47,11 +53,25 @@ export type NativeHarnessBackupExpirationResult = { expired: number; reclaimedBytes: number; clearedTaskSessions: number; + evidenceCandidates: number; + evidenceCandidateBytes: number; + evidenceRemoved: number; + evidenceReclaimedBytes: number; failed: number; truncated: boolean; skipped: Record; }; +type EvidenceContext = { + dryRun: boolean; + isSessionBusy: (scopeDigest: string) => boolean; + now: number; + retentionMs: number; + retainedBefore: number; + canInspect: () => boolean; + dueAt: (at: number) => void; +}; + type SessionIdentity = { runId: string; normalizedSessionId: string; @@ -66,6 +86,10 @@ function emptyResult(): NativeHarnessBackupExpirationResult { expired: 0, reclaimedBytes: 0, clearedTaskSessions: 0, + evidenceCandidates: 0, + evidenceCandidateBytes: 0, + evidenceRemoved: 0, + evidenceReclaimedBytes: 0, failed: 0, truncated: false, skipped: {}, @@ -223,6 +247,13 @@ export function nativeHarnessBackupExpirationService( defaults: NativeHarnessBackupExpirationOptions = {}, ) { let cursor: string | null = null; + let resume: { scope: string; archive: string } | null = null; + let quarantineCursor: string | null = null; + let phase: "sessions" | "quarantine" = "sessions"; + let passActive = false; + let evidencePass = false; + let evidenceDueAt = 0; + let passDueAt = Number.POSITIVE_INFINITY; async function settleLeftovers(backupRoot: string) { const current = path.join(backupRoot, "current"); @@ -310,46 +341,247 @@ export function nativeHarnessBackupExpirationService( result.reclaimedBytes += bytes; } + async function removeExpiredLeftovers(parent: string, context: EvidenceContext, result: NativeHarnessBackupExpirationResult) { + for (const name of await fs.readdir(parent).catch(() => [] as string[])) { + if (!name.startsWith(EXPIRED_PREFIX) || !UUID_PATTERN.test(name.slice(EXPIRED_PREFIX.length))) continue; + if (!context.canInspect()) { + context.dueAt(context.now); + return; + } + result.checked++; + await removeBackupTree(path.join(parent, name)); + } + } + + async function retireEvidence( + parent: string, + name: string, + scopeDigest: string, + stampedAt: number, + context: EvidenceContext, + result: NativeHarnessBackupExpirationResult, + ) { + const skip = (reason: string) => { + result.skipped[reason] = (result.skipped[reason] ?? 0) + 1; + }; + const target = path.join(parent, name); + const stat = await fs.lstat(target).catch(() => null); + if (path.dirname(target) !== parent || !stat?.isDirectory()) { + return skip("unsafe_path"); + } + const createdAt = stampedAt > 0 && stampedAt <= context.now ? stampedAt : stat.mtimeMs; + if (createdAt >= context.retainedBefore) { + context.dueAt(createdAt + context.retentionMs); + return skip("recent_evidence"); + } + if (context.isSessionBusy(scopeDigest)) { + context.dueAt(context.now); + return skip("session_active"); + } + const bytes = await allocatedBytes(target); + result.evidenceCandidates++; + result.evidenceCandidateBytes += bytes; + if (context.dryRun) return; + if (context.isSessionBusy(scopeDigest)) { + context.dueAt(context.now); + return skip("session_active"); + } + const trash = path.join(parent, `${EXPIRED_PREFIX}${randomUUID()}`); + await fs.rename(target, trash); + await removeBackupTree(trash); + result.evidenceRemoved++; + result.evidenceReclaimedBytes += bytes; + } + + async function retireContinuityBreaks( + root: string, + scopeDigest: string, + resumeAfter: string | null, + context: EvidenceContext, + result: NativeHarnessBackupExpirationResult, + ): Promise { + const parent = path.join(root, "continuity-breaks"); + const stat = await fs.lstat(parent).catch(() => null); + if (!stat) return null; + if (!stat.isDirectory()) { + result.skipped.unsafe_path = (result.skipped.unsafe_path ?? 0) + 1; + return null; + } + if (!context.dryRun) await removeExpiredLeftovers(parent, context, result); + let last = resumeAfter ?? ""; + const names = (await fs.readdir(parent)).filter((name) => name > last).sort(); + for (const name of names) { + const match = CONTINUITY_ARCHIVE_PATTERN.exec(name); + if (!match) continue; + if (!context.canInspect()) return last; + result.checked++; + last = name; + try { + await retireEvidence(parent, name, scopeDigest, Number(match[1]), context, result); + } catch (err) { + result.failed++; + context.dueAt(context.now); + logger.warn({ err, scope: scopeDigest }, "native runner evidence retention skipped a continuity archive"); + } + } + if (!context.dryRun) await fs.rmdir(parent).catch(() => undefined); + return null; + } + + async function retireQuarantine( + base: string, + context: EvidenceContext, + result: NativeHarnessBackupExpirationResult, + ): Promise { + const parent = path.join(base, "quarantine"); + const stat = await fs.lstat(parent).catch(() => null); + if (!stat) return true; + if (!stat.isDirectory()) { + result.skipped.unsafe_path = (result.skipped.unsafe_path ?? 0) + 1; + return true; + } + if (!context.dryRun && quarantineCursor === null) await removeExpiredLeftovers(parent, context, result); + const names = (await fs.readdir(parent)).sort(); + const pending = quarantineCursor ? names.filter((name) => name > quarantineCursor!) : names; + for (const name of pending) { + const match = QUARANTINE_PATTERN.exec(name); + const cleanupArchive = QUARANTINE_CLEANUP_PATTERN.test(name); + if (!match && !cleanupArchive) continue; + if (!context.canInspect()) return false; + result.checked++; + quarantineCursor = name; + if (!match) { + result.skipped.cleanup_archive = (result.skipped.cleanup_archive ?? 0) + 1; + continue; + } + try { + await retireEvidence(parent, name, match[1]!, Number(match[2]), context, result); + } catch (err) { + result.failed++; + context.dueAt(context.now); + logger.warn({ err, scope: match[1] }, "native runner evidence retention skipped a quarantine entry"); + } + } + quarantineCursor = null; + return true; + } + + function noteEvidence(writtenAt: number) { + evidenceDueAt = Math.min(evidenceDueAt, writtenAt + evidenceRetentionMs()); + } + + function evidenceRetentionMs(options: NativeHarnessBackupExpirationOptions = {}) { + return Math.max( + 1, + options.evidenceRetentionDays ?? defaults.evidenceRetentionDays ?? NATIVE_RUNNER_EVIDENCE_RETENTION_DEFAULT_DAYS, + ) * DAY_MS; + } + async function tick(options: NativeHarnessBackupExpirationOptions = {}) { const closedForDays = Math.max( 1, options.closedForDays ?? defaults.closedForDays ?? NATIVE_HARNESS_BACKUP_EXPIRATION_DEFAULT_DAYS, ); + const retentionMs = evidenceRetentionMs(options); const dryRun = options.dryRun ?? defaults.dryRun ?? false; const maxEntries = Math.max(1, options.maxEntries ?? defaults.maxEntries ?? 200); const maxRemovals = Math.max(1, options.maxRemovals ?? defaults.maxRemovals ?? 50); const shouldContinue = options.shouldContinue ?? defaults.shouldContinue ?? (() => true); const isSessionBusy = options.isSessionBusy ?? defaults.isSessionBusy ?? nativeSessionScopeDigestIsBusy; const base = path.resolve(options.stateBase ?? defaults.stateBase ?? nativeHarnessStateBase()); - const closedBefore = new Date((options.now ?? defaults.now ?? new Date()).getTime() - closedForDays * DAY_MS); + const now = (options.now ?? defaults.now ?? new Date()).getTime(); + const closedBefore = new Date(now - closedForDays * DAY_MS); const result = emptyResult(); - - const names = (await fs.readdir(base).catch(() => [] as string[])) - .filter((name) => SCOPE_PATTERN.test(name)) - .sort(); - const pending = cursor ? names.filter((name) => name > cursor!) : names; - for (const name of pending) { - if (!shouldContinue() || result.checked >= maxEntries || result.candidates >= maxRemovals) { - result.truncated = true; - break; - } - result.checked++; - cursor = name; - const root = path.join(base, name); - if (path.dirname(root) !== base || !(await isRealDirectory(root))) { - result.skipped.unsafe_path = (result.skipped.unsafe_path ?? 0) + 1; - continue; - } - try { - await expireSession(root, name, { dryRun, isSessionBusy, closedBefore }, result); - } catch (err) { - result.failed++; - logger.warn({ err, scope: name }, "native harness backup expiration skipped a session"); + const hasBudget = () => shouldContinue() && result.candidates + result.evidenceCandidates < maxRemovals; + if (!passActive) { + passActive = true; + evidencePass = evidenceDueAt <= now; + if (evidencePass) { + evidenceDueAt = Number.POSITIVE_INFINITY; + passDueAt = Number.POSITIVE_INFINITY; } } - if (!result.truncated) cursor = null; + const context: EvidenceContext = { + dryRun, + isSessionBusy, + now, + retentionMs, + retainedBefore: now - retentionMs, + canInspect: () => hasBudget() && result.checked < maxEntries, + dueAt: (at) => { + passDueAt = Math.min(passDueAt, at); + }, + }; + + if (phase === "sessions") { + const names = (await fs.readdir(base).catch(() => [] as string[])) + .filter((name) => SCOPE_PATTERN.test(name)) + .sort(); + const pending = cursor ? names.filter((name) => name > cursor!) : names; + for (const name of pending) { + if (!hasBudget() || result.checked >= maxEntries) { + result.truncated = true; + break; + } + const resumeAfter = resume?.scope === name ? resume.archive : null; + if (resume && name >= resume.scope) resume = null; + if (resumeAfter === null) result.checked++; + const previous = cursor; + cursor = name; + const root = path.join(base, name); + if (path.dirname(root) !== base || !(await isRealDirectory(root))) { + result.skipped.unsafe_path = (result.skipped.unsafe_path ?? 0) + 1; + continue; + } + if (resumeAfter === null) { + try { + await expireSession(root, name, { dryRun, isSessionBusy, closedBefore }, result); + } catch (err) { + result.failed++; + logger.warn({ err, scope: name }, "native harness backup expiration skipped a session"); + } + } + if (!evidencePass) continue; + let stoppedAt: string | null = null; + try { + stoppedAt = await retireContinuityBreaks(root, name, resumeAfter, context, result); + } catch (err) { + result.failed++; + context.dueAt(now); + logger.warn({ err, scope: name }, "native runner evidence retention skipped a session"); + } + if (stoppedAt !== null) { + resume = { scope: name, archive: stoppedAt }; + cursor = previous; + result.truncated = true; + break; + } + } + if (result.truncated) return result; + cursor = null; + if (!evidencePass) { + passActive = false; + return result; + } + phase = "quarantine"; + } + try { + if (!(await retireQuarantine(base, context, result))) { + result.truncated = true; + return result; + } + } catch (err) { + result.failed++; + context.dueAt(now); + quarantineCursor = null; + logger.warn({ err }, "native runner evidence retention skipped quarantine"); + } + phase = "sessions"; + passActive = false; + evidencePass = false; + evidenceDueAt = Math.min(evidenceDueAt, passDueAt); return result; } - return { tick }; + return { tick, noteEvidence }; } diff --git a/server/src/services/native-runtime/native-monitor-delivery.ts b/server/src/services/native-runtime/native-monitor-delivery.ts index d38e41d8bc1..b2af0e80605 100644 --- a/server/src/services/native-runtime/native-monitor-delivery.ts +++ b/server/src/services/native-runtime/native-monitor-delivery.ts @@ -1,7 +1,8 @@ -import { agentWakeupRequests, heartbeatRuns, type Db } from "@paperclipai/db"; -import { and, desc, eq, inArray, ne, sql } from "drizzle-orm"; +import { agentWakeupRequests, heartbeatRuns, nativeRunFinalizations, nativeRunResults, type Db, type issues } from "@paperclipai/db"; +import { and, desc, eq, inArray, isNotNull, ne, sql } from "drizzle-orm"; import { alias } from "drizzle-orm/pg-core"; import { boundRunIssueIdSql } from "../run-task-binding.js"; +import { scheduledIssueMonitorWait } from "../issue-monitors.js"; function record(value: unknown): Record { return value !== null && typeof value === "object" && !Array.isArray(value) @@ -9,6 +10,39 @@ function record(value: unknown): Record { } export type NativeMonitorDelivery = { wakeId: string; successorRunId: string }; +export type NativePreservedMonitorWait = { agentId: string; nextCheckAt: string; resultId: string }; + +export async function readNativePreservedMonitorWait( + db: Db, + binding: { companyId: string; issueId: string; runId: string; agentId: string }, + issue: typeof issues.$inferSelect, +): Promise { + if (issue.companyId !== binding.companyId || issue.id !== binding.issueId || issue.workMode !== "standard" || + issue.executionRunId !== null || issue.checkoutRunId !== null) return null; + const nextCheckAt = scheduledIssueMonitorWait(issue, binding.agentId); + if (!nextCheckAt) return null; + const [source] = await db.select({ resultId: nativeRunResults.id, hasReceipt: sql`exists ( + select 1 from jsonb_each(case when jsonb_typeof(${heartbeatRuns.resultJson}->'semanticToolReceipts') = 'object' + then ${heartbeatRuns.resultJson}->'semanticToolReceipts' else '{}'::jsonb end) receipt + where receipt.value->>'operationId' = 'set_task_monitor' + and receipt.value->'result'->>'taskId' = ${binding.issueId} + and receipt.value->'result'->'monitor'->>'status' = 'scheduled' + and jsonb_typeof(receipt.value->'result'->'monitor'->'nextCheckAt') = 'string' + )` }).from(heartbeatRuns).innerJoin(nativeRunFinalizations, and( + eq(nativeRunFinalizations.companyId, binding.companyId), eq(nativeRunFinalizations.issueId, binding.issueId), + eq(nativeRunFinalizations.runId, heartbeatRuns.id), + )).innerJoin(nativeRunResults, and( + eq(nativeRunResults.id, nativeRunFinalizations.resultId), eq(nativeRunResults.companyId, binding.companyId), + eq(nativeRunResults.issueId, binding.issueId), eq(nativeRunResults.runId, heartbeatRuns.id), + eq(nativeRunResults.schemaStatus, "accepted"), + )).where(and(eq(heartbeatRuns.id, binding.runId), eq(heartbeatRuns.companyId, binding.companyId), + eq(heartbeatRuns.agentId, binding.agentId), eq(heartbeatRuns.nativeIssueId, binding.issueId), + eq(heartbeatRuns.runtimeMode, "native"), eq(heartbeatRuns.status, "succeeded"), isNotNull(heartbeatRuns.finishedAt), + ...["cancellation", "nativeCancellation", "executionCancellation", "startupCancellation"].map(key => + sql`coalesce(${heartbeatRuns.resultJson}->${key}, 'null'::jsonb) = 'null'::jsonb`), + )).limit(1); + return source?.hasReceipt ? { agentId: binding.agentId, nextCheckAt, resultId: source.resultId } : null; +} export async function readNativeMonitorSource( db: Pick, diff --git a/server/src/services/native-runtime/native-preserved-monitor-wait.test.ts b/server/src/services/native-runtime/native-preserved-monitor-wait.test.ts new file mode 100644 index 00000000000..1cedcb28be8 --- /dev/null +++ b/server/src/services/native-runtime/native-preserved-monitor-wait.test.ts @@ -0,0 +1,234 @@ +import { randomUUID } from "node:crypto"; +import { and, eq } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; +import { agents, agentWakeupRequests, companies, completionContracts, createDb, heartbeatRuns, + issueComments, issueRecoveryActions, issues, nativeRunFinalizations, nativeRunResults, statusDecisions, subscribeDatabaseWork } from "@paperclipai/db"; +import { PROVIDER_QUOTA_MONITOR_SERVICE_NAME } from "@paperclipai/shared"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; +import { PaperclipRunnerToolAuthority } from "./paperclip-runner-tool-authority.js"; +import { finalizeNativeRun, repairCommittedNativeChatResponse } from "./native-run-finalizer.js"; +import { nativeSha256 } from "./canonical.js"; +import * as monitorDelivery from "./native-monitor-delivery.js"; +import { DELIVERY_QUEUES } from "../delivery-work-notifications.js"; +import { hasPendingRunCleanup } from "../execution-blocker.js"; + +const support = await getEmbeddedPostgresTestSupport(); +describe.skipIf(!support.supported)("released native monitor results", () => { + let temporary: Awaited>; + let db: ReturnType; + beforeAll(async () => { + temporary = await startEmbeddedPostgresTestDatabase("native-preserved-monitor-"); + db = createDb(temporary.connectionString); + }); + afterAll(async () => { await db?.$client.end(); await temporary?.cleanup(); }); + + async function fixture() { + const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), runId = randomUUID(); + const contractId = randomUUID(), resultId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Saved monitor", issuePrefix: `P${companyId.slice(0, 7)}` }); + await db.insert(agents).values({ id: agentId, companyId, name: "Worker", adapterType: "paperclip_runner", status: "active" }); + await db.insert(issues).values({ id: issueId, companyId, title: "Saved follow-up", status: "in_progress", assigneeAgentId: agentId }); + await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, nativeIssueId: issueId, + runtimeMode: "native", status: "running", contextSnapshot: { issueId } }); + await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId)); + const nextCheckAt = new Date(Date.now() + 3_600_000).toISOString(); + const authority = new PaperclipRunnerToolAuthority(db, { companyId, agentId, issueId, runId }); + await authority.execute({ tool: "set_task_monitor", callId: randomUUID(), arguments: { + idempotencyKey: "saved-monitor", monitor: { nextCheckAt, notes: "Check the saved result." }, + } }); + await db.insert(completionContracts).values({ id: contractId, companyId, issueId, revision: 1, + schemaVersion: "paperclip.completion-contract.v1", policyVersion: "test", risk: "standard", + completionAuthority: "server_arbiter", incompleteCriteriaPolicy: "preserve_non_terminal", + contractJson: { revision: "test", criteria: [] }, canonicalSha256: randomUUID(), + createdByActorType: "system", createdByActorId: "test" }); + const result = { result: { schema: "paperclip.run_result.v1", reportedWorkDisposition: "yielded", + summary: "Wait for the saved check.", completionClaim: { contractRevision: "test", objectiveSatisfied: false, + criteria: [], remainingWork: [{ description: "Check again", blocksCompletion: true }] }, + continuation: { kind: "monitor", summary: "Wait for the check", idempotencyKey: "provider-selected" }, + evidence: [], verification: [], attentionRequests: [], artifacts: [] }, + terminal: { schema: "paperclip.prp.terminal.v1", runTerminalState: "succeeded", turnTerminalState: "completed", reportedWorkDisposition: "yielded" } }; + const finishedAt = new Date("2020-01-01T00:00:00Z"); + await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt, completionContractId: contractId }).where(eq(heartbeatRuns.id, runId)); + await db.insert(nativeRunResults).values({ id: resultId, companyId, issueId, runId, completionContractId: contractId, + serverFingerprint: randomUUID(), canonicalSha256: randomUUID(), schemaStatus: "accepted", resultJson: result }); + await db.insert(nativeRunFinalizations).values({ companyId, issueId, runId, phase: "retryable_failure", resultId }); + await db.update(issues).set({ executionRunId: null, checkoutRunId: null }).where(eq(issues.id, issueId)); + const readIssue = async () => (await db.select().from(issues).where(eq(issues.id, issueId)))[0]!; + const finalize = (workspaceFinalizeStatus: "succeeded" | "failed" = "succeeded") => finalizeNativeRun({ + db, runId, workspaceFinalizeStatus, projectRunStatus: true, + }); + return { companyId, agentId, issueId, runId, resultId, result, finishedAt, nextCheckAt, readIssue, finalize }; + } + + it.each(["original", "rescheduled", "past_due"])("keeps the saved monitor and settles a released successful result (%s)", async state => { + const f = await fixture(); + if (state !== "original") { + const issue = await f.readIssue(); + const nextCheckAt = state === "past_due" ? "2020-01-01T01:00:00Z" : new Date(Date.now() + 7_200_000).toISOString(); + await db.update(issues).set({ monitorNextCheckAt: new Date(nextCheckAt), executionPolicy: { + ...issue.executionPolicy, monitor: { ...(issue.executionPolicy?.monitor as Record), nextCheckAt }, + } }).where(eq(issues.id, f.issueId)); + } + const issueBefore = await f.readIssue(); + const [hold] = await db.insert(issueRecoveryActions).values({ companyId: f.companyId, sourceIssueId: f.issueId, + kind: "active_run_watchdog", cause: "legacy_execution_requires_reconciliation", fingerprint: randomUUID(), + status: "active", ownerType: "board", nextAction: "Preserve independent recovery.", evidence: { runId: randomUUID() }, + }).returning(); + await f.finalize(); + const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)); + expect(coordinator.phase).toBe("committed"); + const decisions = await db.select().from(statusDecisions).where(eq(statusDecisions.runId, f.runId)); + expect(decisions.at(-1)).toMatchObject({ reasonCode: "scheduled_monitor_preserved", toStatus: issueBefore.status }); + expect(decisions.at(-1)?.decisionJson).toMatchObject({ effects: [] }); + expect(await f.readIssue()).toMatchObject({ status: issueBefore.status, executionPolicy: issueBefore.executionPolicy, + monitorNextCheckAt: issueBefore.monitorNextCheckAt, monitorAttemptCount: issueBefore.monitorAttemptCount, + executionRunId: null, checkoutRunId: null }); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0].finishedAt).toEqual(f.finishedAt); + expect((await db.select().from(nativeRunResults).where(eq(nativeRunResults.id, f.resultId)))[0].resultJson).toEqual(f.result); + expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, hold!.id))).toEqual([hold]); + expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId))).toEqual([]); + expect(await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId))).toEqual([{ id: f.runId }]); + await f.finalize(); + expect(await db.select().from(statusDecisions).where(eq(statusDecisions.runId, f.runId))).toEqual(decisions); + }); + + it.each(["cleared", "owner", "checkout", "other_agent", "no_receipt", "foreign_receipt", "unfinished", "failed_source", "quota", "workspace_failed", + "cancellation", "nativeCancellation", "executionCancellation", "startupCancellation"])( + "does not settle a released monitor without current proof (%s)", async state => { + const f = await fixture(); + const issue = await f.readIssue(); + if (state === "cleared") await db.update(issues).set({ monitorNextCheckAt: null, executionPolicy: {} }).where(eq(issues.id, f.issueId)); + if (state === "quota") await db.update(issues).set({ executionPolicy: { ...issue.executionPolicy, + monitor: { ...(issue.executionPolicy?.monitor as Record), serviceName: PROVIDER_QUOTA_MONITOR_SERVICE_NAME }, + } }).where(eq(issues.id, f.issueId)); + if (state === "other_agent") await db.update(issues).set({ assigneeAgentId: null }).where(eq(issues.id, f.issueId)); + if (state === "owner" || state === "checkout") { + const [owner] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" }).returning(); + await db.update(issues).set(state === "owner" ? { executionRunId: owner!.id } : { checkoutRunId: owner!.id }).where(eq(issues.id, f.issueId)); + } + if (state === "unfinished") await db.update(heartbeatRuns).set({ finishedAt: null }).where(eq(heartbeatRuns.id, f.runId)); + if (state === "failed_source") await db.update(heartbeatRuns).set({ status: "failed" }).where(eq(heartbeatRuns.id, f.runId)); + if (state === "no_receipt") await db.update(heartbeatRuns).set({ resultJson: {} }).where(eq(heartbeatRuns.id, f.runId)); + if (["cancellation", "nativeCancellation", "executionCancellation", "startupCancellation"].includes(state)) { + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)); + await db.update(heartbeatRuns).set({ resultJson: { ...run.resultJson, [state]: { state: "pending" } } }).where(eq(heartbeatRuns.id, f.runId)); + } + if (state === "foreign_receipt") await db.update(heartbeatRuns).set({ resultJson: { semanticToolReceipts: { + invalid: { operationId: "set_task_monitor", result: { taskId: randomUUID(), monitor: { status: "scheduled", nextCheckAt: f.nextCheckAt } } }, + } } }).where(eq(heartbeatRuns.id, f.runId)); + const before = await f.readIssue(); + await f.finalize(state === "workspace_failed" ? "failed" : "succeeded"); + const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)); + expect(coordinator.phase).toBe("retryable_failure"); + expect(await f.readIssue()).toMatchObject({ executionPolicy: before.executionPolicy, monitorNextCheckAt: before.monitorNextCheckAt, + executionRunId: before.executionRunId, checkoutRunId: before.checkoutRunId, assigneeAgentId: before.assigneeAgentId }); + expect(await db.select().from(statusDecisions).where(and(eq(statusDecisions.runId, f.runId), eq(statusDecisions.reasonCode, "scheduled_monitor_preserved")))).toEqual([]); + }, + ); + + it.each(["schedule", "receipt", "finished_time"] as const)("supersedes an earlier authority error after restoring %s without replaying the result", async state => { + const f = await fixture(), scheduled = await f.readIssue(); + const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)); + await db.update(nativeRunFinalizations).set({ controllerBootId: randomUUID(), attempt: 1 }) + .where(eq(nativeRunFinalizations.runId, f.runId)); + if (state === "schedule") await db.update(issues).set({ executionPolicy: {}, monitorNextCheckAt: null }).where(eq(issues.id, f.issueId)); + if (state === "receipt") await db.update(heartbeatRuns).set({ resultJson: {} }).where(eq(heartbeatRuns.id, f.runId)); + if (state === "finished_time") await db.update(heartbeatRuns).set({ finishedAt: null }).where(eq(heartbeatRuns.id, f.runId)); + await f.finalize(); + const old = await db.select().from(statusDecisions).where(eq(statusDecisions.runId, f.runId)); + expect(old.some(decision => decision.reasonCode === "monitor_wait_authority_lost")).toBe(true); + if (state === "schedule") await db.update(issues).set({ executionPolicy: scheduled.executionPolicy, monitorNextCheckAt: scheduled.monitorNextCheckAt }) + .where(eq(issues.id, f.issueId)); + if (state === "receipt") { + const [current] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)); + await db.update(heartbeatRuns).set({ resultJson: { ...current.resultJson, + semanticToolReceipts: source.resultJson?.semanticToolReceipts } }).where(eq(heartbeatRuns.id, f.runId)); + } + if (state === "finished_time") await db.update(heartbeatRuns).set({ finishedAt: f.finishedAt }).where(eq(heartbeatRuns.id, f.runId)); + await f.finalize(); + const [settled] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)); + expect(settled).toMatchObject({ phase: "committed", nextAttemptAt: null, attempt: 1 }); + const after = await db.select().from(statusDecisions).where(eq(statusDecisions.runId, f.runId)); + for (const before of old) expect(after.find(decision => decision.id === before.id)).toEqual(before); + expect(after.some(decision => decision.reasonCode === "scheduled_monitor_preserved")).toBe(true); + expect((await db.select().from(nativeRunResults).where(eq(nativeRunResults.id, f.resultId)))[0].resultJson).toEqual(f.result); + expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId))).toEqual([]); + expect(await f.readIssue()).toMatchObject({ monitorNextCheckAt: scheduled.monitorNextCheckAt, + executionRunId: null, checkoutRunId: null }); + await f.finalize(); + expect(await db.select().from(statusDecisions).where(eq(statusDecisions.runId, f.runId))).toEqual(after); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0].finishedAt).toEqual(f.finishedAt); + }); + + it("notifies the existing monitor scheduler once when finalization stops blocking it", async () => { + const f = await fixture(), events: string[] = []; + expect(await hasPendingRunCleanup(db, f.companyId, f.runId)).toBe(true); + const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.issueMonitors, event => { events.push(event); }); + try { + await f.finalize(); + expect(events).toEqual(["intent", "settled"]); + expect(await hasPendingRunCleanup(db, f.companyId, f.runId)).toBe(false); + expect((await f.readIssue()).monitorNextCheckAt?.toISOString()).toBe(f.nextCheckAt); + await f.finalize(); + expect(events).toEqual(["intent", "settled"]); + } finally { unsubscribe(); } + }); + + it.each(["live", "recovered"])("publishes the accepted monitor explanation once (%s)", async mode => { + const f = await fixture(), before = await f.readIssue(); + const completionContractSha256 = `sha256:${nativeSha256({ revision: "test", criteria: [] })}`; + const canonicalSha256 = `sha256:${nativeSha256({ ...f.result, turnId: null })}`; + const serverFingerprint = `sha256:${nativeSha256({ runId: f.runId, completionContractSha256, canonicalSha256 })}`; + await db.update(nativeRunResults).set({ canonicalSha256, serverFingerprint }).where(eq(nativeRunResults.id, f.resultId)); + await db.update(heartbeatRuns).set({ completionContractSha256, + contextSnapshot: { issueId: f.issueId, ...(mode === "recovered" ? { skipIssueComment: true } : {}) }, + }).where(eq(heartbeatRuns.id, f.runId)); + const comments = () => db.select({ id: issueComments.id, body: issueComments.body }).from(issueComments) + .where(and(eq(issueComments.companyId, f.companyId), eq(issueComments.createdByRunId, f.runId))); + const repair = () => repairCommittedNativeChatResponse(db, { companyId: f.companyId, issueId: f.issueId, runId: f.runId }); + await f.finalize(); + if (mode === "recovered") { + expect(await comments()).toEqual([]); + await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: f.issueId } }).where(eq(heartbeatRuns.id, f.runId)); + expect(await repair()).toBe(true); + } + const published = await comments(); + expect(published).toEqual([{ id: expect.any(String), body: f.result.result.summary }]); + expect(await repair()).toBe(false); + await f.finalize(); + expect(await comments()).toEqual(published); + expect((await f.readIssue()).monitorNextCheckAt).toEqual(before.monitorNextCheckAt); + expect((await db.select().from(nativeRunResults).where(eq(nativeRunResults.id, f.resultId)))[0].resultJson).toEqual(f.result); + expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId))).toEqual([]); + }); + + it.each(["cleared", "owner", "receipt", "rescheduled"])("rechecks monitor proof under the task lock (%s)", async state => { + const f = await fixture(), scheduled = await f.readIssue(); + const [owner] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" }).returning(); + const nextCheckAt = new Date(Date.now() + 7_200_000).toISOString(); + const original = monitorDelivery.readNativePreservedMonitorWait; + const events: string[] = []; + const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.issueMonitors, event => { events.push(event); }); + const race = vi.spyOn(monitorDelivery, "readNativePreservedMonitorWait").mockImplementationOnce(async (...args) => { + const proof = await original(...args); + expect(proof).not.toBeNull(); + if (state === "cleared") await db.update(issues).set({ executionPolicy: {}, monitorNextCheckAt: null }).where(eq(issues.id, f.issueId)); + if (state === "owner") await db.update(issues).set({ executionRunId: owner!.id }).where(eq(issues.id, f.issueId)); + if (state === "receipt") await db.update(heartbeatRuns).set({ resultJson: {} }).where(eq(heartbeatRuns.id, f.runId)); + if (state === "rescheduled") await db.update(issues).set({ monitorNextCheckAt: new Date(nextCheckAt), executionPolicy: { + ...scheduled.executionPolicy, monitor: { ...(scheduled.executionPolicy?.monitor as Record), nextCheckAt }, + } }).where(eq(issues.id, f.issueId)); + return proof; + }); + try { await f.finalize(); } finally { race.mockRestore(); unsubscribe(); } + const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)); + expect(coordinator.phase).toBe(state === "rescheduled" ? "committed" : "retryable_failure"); + expect(events).toEqual(state === "rescheduled" ? ["intent", "settled"] : []); + const task = await f.readIssue(); + expect(task.executionRunId).toBe(state === "owner" ? owner!.id : null); + expect(task.monitorNextCheckAt?.toISOString() ?? null).toBe(state === "cleared" ? null : state === "rescheduled" ? nextCheckAt : f.nextCheckAt); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, owner!.id)))[0].status).toBe("running"); + expect((await db.select().from(nativeRunResults).where(eq(nativeRunResults.id, f.resultId)))[0].resultJson).toEqual(f.result); + expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId))).toEqual([]); + }); +}); diff --git a/server/src/services/native-runtime/native-run-finalizer-telemetry.test.ts b/server/src/services/native-runtime/native-run-finalizer-telemetry.test.ts index 0a8b6ce4ed1..5d1ed177987 100644 --- a/server/src/services/native-runtime/native-run-finalizer-telemetry.test.ts +++ b/server/src/services/native-runtime/native-run-finalizer-telemetry.test.ts @@ -1,5 +1,6 @@ import { enableRunFailureReporting } from "../../__tests__/helpers/run-failure-reporting.js"; import { randomUUID } from "node:crypto"; +import { verifyCommittedReplayPayloadReads } from "./__tests__/helpers/committed-replay.js"; import { eq } from "drizzle-orm"; import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; import { @@ -34,6 +35,12 @@ const mockTelemetryClient = vi.hoisted(() => ({ })); vi.mock("../../telemetry.ts", () => ({ getTelemetryClient: () => mockTelemetryClient })); +const reconcileMuseAssignments = vi.hoisted(() => vi.fn(async () => undefined)); +vi.mock("../muse-runner-broker.js", async importOriginal => ({ + ...await importOriginal(), + museRunnerBroker: () => ({ reconcileTerminalAssignments: reconcileMuseAssignments }), +})); + const mockCaptureRunFailure = vi.hoisted(() => vi.fn()); vi.mock("../../sentry.ts", async () => { const actual = await vi.importActual("../../sentry.ts"); @@ -261,18 +268,23 @@ describeEmbeddedPostgres("native run finalizer / status decision committer — a projectRunStatus: true, }); - const callsBefore = mockTelemetryClient.track.mock.calls.length; - // The coordinator is already "committed", so this second call takes the - // projectCommittedRun short-circuit. A succeeded row is eligible so stale - // cleanup errors can be projected separately; an unchanged terminal state - // must still never emit a second agent.task_run event. - await finalizeNativeRun({ - db, - runId: fixture.runId, - workspaceFinalizeStatus: "succeeded", - projectRunStatus: true, + await verifyCommittedReplayPayloadReads(db, { runId: fixture.runId, companyId, agentId }, async () => { + const callsBefore = mockTelemetryClient.track.mock.calls.length; + reconcileMuseAssignments.mockClear(); + await finalizeNativeRun({ + db, + runId: fixture.runId, + workspaceFinalizeStatus: "succeeded", + projectRunStatus: true, + }); + expect(agentTaskRunCalls(callsBefore)).toHaveLength(0); + expect(reconcileMuseAssignments).toHaveBeenCalledExactlyOnceWith(companyId, agentId, fixture.runId); + await db.insert(workspaceOperations).values({ companyId, issueId: fixture.issueId, + heartbeatRunId: fixture.runId, phase: "workspace_finalize", status: "succeeded" }); + const reconciled = await reconcileNativeFinalizations(db, [fixture.runId]); + expect(reconciled).toContainEqual(expect.objectContaining({ runId: fixture.runId, phase: "committed" })); + expect(agentTaskRunCalls(callsBefore)).toHaveLength(0); }); - expect(agentTaskRunCalls(callsBefore)).toHaveLength(0); }); it("repairs a stale succeeded-run error once without duplicate telemetry or destroying retained owner evidence", async () => { diff --git a/server/src/services/native-runtime/native-run-finalizer.ts b/server/src/services/native-runtime/native-run-finalizer.ts index ff5d3267a9d..5931662d7db 100644 --- a/server/src/services/native-runtime/native-run-finalizer.ts +++ b/server/src/services/native-runtime/native-run-finalizer.ts @@ -4,7 +4,7 @@ import { reconcileMuseNativeTerminalRun } from "./muse-terminal-reconciliation.j import { assertNativeCaseHandoffResult } from "./native-case-handoff.js"; import { scheduledIssueMonitorWait } from "../issue-monitors.js"; import { hasPublishedGitHubReview } from "../chat-channels/github/review-wait.js"; -import { readNativeMonitorDelivery } from "./native-monitor-delivery.js"; +import { readNativeMonitorDelivery, readNativePreservedMonitorWait } from "./native-monitor-delivery.js"; import { isNativePlanWaitResult, readNativePlanWait } from "./native-plan-wait.js"; import { hasPendingNativeChildCompletion } from "./native-child-completion-delivery.js"; import { settleExternalConversation } from "../slack-conversation-lifecycle.js"; @@ -568,7 +568,7 @@ export async function recordNativeFinalizationFailure(input: { }); } -async function resolveCommittedFinalizationRecovery(db: Db, run: typeof heartbeatRuns.$inferSelect, issueId: string) { +async function resolveCommittedFinalizationRecovery(db: Db, run: Pick, issueId: string) { const actions = await db.select().from(issueRecoveryActions).where(and( eq(issueRecoveryActions.companyId, run.companyId), eq(issueRecoveryActions.sourceIssueId, issueId), eq(issueRecoveryActions.kind, "active_run_watchdog"), inArray(issueRecoveryActions.status, ["active", "escalated"]), @@ -596,7 +596,7 @@ function recoveredExecutionFailureMetadata() { async function projectCommittedRun(input: { db: Db; - run: typeof heartbeatRuns.$inferSelect; + run: Pick; coordinator: typeof nativeRunFinalizations.$inferSelect; }) { if (!input.coordinator.resultId) return; @@ -704,7 +704,8 @@ async function projectCommittedRun(input: { async function materializeCommittedReviewResponse(db: Db, runId: string) { try { const [run] = await db - .select() + .select({ id: heartbeatRuns.id, companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId, + nativeIssueId: heartbeatRuns.nativeIssueId, resultJson: heartbeatRuns.resultJson }) .from(heartbeatRuns) .where(eq(heartbeatRuns.id, runId)) .limit(1); @@ -764,7 +765,9 @@ async function materializeCommittedReviewResponse(db: Db, runId: string) { async function acceptedResponseDigestMatches( db: Db, - run: typeof heartbeatRuns.$inferSelect, + run: Pick, accepted: typeof nativeRunResults.$inferSelect, ): Promise { const envelope = record(accepted.resultJson); @@ -842,7 +845,30 @@ export async function repairCommittedNativeChatResponse( .for("update"); if (!issue) return false; const [run] = await tx - .select() + .select({ + id: heartbeatRuns.id, + companyId: heartbeatRuns.companyId, + agentId: heartbeatRuns.agentId, + nativeIssueId: heartbeatRuns.nativeIssueId, + nativeSessionId: heartbeatRuns.nativeSessionId, + completionContractId: heartbeatRuns.completionContractId, + completionContractSha256: heartbeatRuns.completionContractSha256, + runnerInstanceId: heartbeatRuns.runnerInstanceId, + runtimeMode: heartbeatRuns.runtimeMode, + status: heartbeatRuns.status, + errorCode: heartbeatRuns.errorCode, + nativePhase: heartbeatRuns.nativePhase, + finishedAt: heartbeatRuns.finishedAt, + resultJson: heartbeatRuns.resultJson, + contextSnapshot: sql | null>`case when ${heartbeatRuns.contextSnapshot} is null then null + else jsonb_build_object( + 'skipIssueComment', ${heartbeatRuns.contextSnapshot}->'skipIssueComment', + 'source', ${heartbeatRuns.contextSnapshot}->'source', + 'externalChatContinuation', ${heartbeatRuns.contextSnapshot}->'externalChatContinuation', + 'paperclipWake', jsonb_build_object('externalInteractionContinuation', + ${heartbeatRuns.contextSnapshot} #> '{paperclipWake,externalInteractionContinuation}') + ) end`, + }) .from(heartbeatRuns) .where( and( @@ -1121,13 +1147,15 @@ export async function finalizeNativeRun(input: { preserveProviderAttempt?: boolean; failpoint?: NativeStatusCommitFailpoint; }) { - const run = await input.db - .select() + const admittedRun = await input.db + .select({ id: heartbeatRuns.id, companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId, + runtimeMode: heartbeatRuns.runtimeMode, status: heartbeatRuns.status, errorCode: heartbeatRuns.errorCode, + nativePhase: heartbeatRuns.nativePhase, finishedAt: heartbeatRuns.finishedAt }) .from(heartbeatRuns) .where(eq(heartbeatRuns.id, input.runId)) .limit(1) .then((rows) => rows[0] ?? null); - if (!run || run.runtimeMode !== "native") + if (!admittedRun || admittedRun.runtimeMode !== "native") throw new Error("native_finalization_run_missing"); const claim = await claimCoordinator({ db: input.db, @@ -1136,12 +1164,12 @@ export async function finalizeNativeRun(input: { }); const coordinator = claim.coordinator; if (!claim.leaseOwner && coordinator.phase === "committed") { - if (isNativeRunnerOwnershipHeld(run)) return coordinator; + if (isNativeRunnerOwnershipHeld(admittedRun)) return coordinator; const presentationAlreadyMaterialized = coordinator.decisionId ? await hasMaterializedNativeReviewResponse(input.db, { - companyId: run.companyId, + companyId: admittedRun.companyId, issueId: coordinator.issueId, - runId: run.id, + runId: admittedRun.id, decisionId: coordinator.decisionId, }) : false; @@ -1153,9 +1181,9 @@ export async function finalizeNativeRun(input: { ) { await input.db.transaction((tx) => restoreNativeChatReviewPresentationInTransaction(tx as unknown as Db, { - companyId: run.companyId, + companyId: admittedRun.companyId, issueId: coordinator.issueId, - runId: run.id, + runId: admittedRun.id, decisionId: coordinator.decisionId!, resultId: coordinator.resultId!, assessmentId: coordinator.assessmentId!, @@ -1163,27 +1191,45 @@ export async function finalizeNativeRun(input: { ); } if (input.projectRunStatus) - await projectCommittedRun({ db: input.db, run, coordinator }); + await projectCommittedRun({ db: input.db, run: admittedRun, coordinator }); if (input.projectRunStatus && !presentationAlreadyMaterialized) await materializeCommittedReviewResponse(input.db, input.runId); if (input.projectRunStatus) await repairCommittedNativeChatResponse(input.db, { - companyId: run.companyId, + companyId: admittedRun.companyId, issueId: coordinator.issueId, - runId: run.id, + runId: admittedRun.id, }); if (input.projectRunStatus) { - await settleExternalConversation(input.db, run.companyId, coordinator.issueId).catch((err) => { - logger.warn({ err, runId: run.id }, "Slack conversation settlement deferred to reconciliation"); + await settleExternalConversation(input.db, admittedRun.companyId, coordinator.issueId).catch((err) => { + logger.warn({ err, runId: admittedRun.id }, "Slack conversation settlement deferred to reconciliation"); }); } - await resolveCommittedFinalizationRecovery(input.db, run, coordinator.issueId); + await resolveCommittedFinalizationRecovery(input.db, admittedRun, coordinator.issueId); if (input.projectRunStatus) { - const [projectedRun] = await input.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)); + const [projectedRun] = await input.db.select({ + id: heartbeatRuns.id, companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId, + runtimeMode: heartbeatRuns.runtimeMode, status: heartbeatRuns.status, + runnerProfileJson: sql | null>`case when ${heartbeatRuns.runnerProfileJson} is null then null + else jsonb_build_object('nativeExecutionInput', jsonb_build_object( + 'schema', ${heartbeatRuns.runnerProfileJson} #> '{nativeExecutionInput,schema}', + 'provider', jsonb_build_object( + 'kind', ${heartbeatRuns.runnerProfileJson} #> '{nativeExecutionInput,provider,kind}', + 'binding', jsonb_build_object( + 'companyId', ${heartbeatRuns.runnerProfileJson} #> '{nativeExecutionInput,provider,binding,companyId}', + 'agentId', ${heartbeatRuns.runnerProfileJson} #> '{nativeExecutionInput,provider,binding,agentId}', + 'bindingId', ${heartbeatRuns.runnerProfileJson} #> '{nativeExecutionInput,provider,binding,bindingId}' + ) + ) + )) end`, + }).from(heartbeatRuns).where(eq(heartbeatRuns.id, admittedRun.id)); if (projectedRun) await reconcileMuseNativeTerminalRun(input.db, projectedRun); } return coordinator; } + const run = await input.db.select().from(heartbeatRuns) + .where(eq(heartbeatRuns.id, input.runId)).limit(1).then(rows => rows[0] ?? null); + if (!run || run.runtimeMode !== "native") throw new Error("native_finalization_run_missing"); const [resultRow, contractRow] = await Promise.all([ input.db .select() @@ -1285,10 +1331,16 @@ export async function finalizeNativeRun(input: { }) : "not_applicable"; const monitorWaitAt = scheduledIssueMonitorWait(authoritativeIssue, run.agentId); + const preservedMonitorWait = terminalState === "succeeded" && input.workspaceFinalizeStatus === "succeeded" && + assessment.reportedDisposition === "yielded" && assessment.continuation?.kind === "monitor" + ? await readNativePreservedMonitorWait(input.db, { + companyId: run.companyId, issueId: authoritativeIssue.id, runId: run.id, agentId: run.agentId, + }, authoritativeIssue) : null; + if (preservedMonitorWait && preservedMonitorWait.resultId !== resultRow.id) throw new NativeStatusRaceError(); const supersededReviews = terminalState === "succeeded" && input.workspaceFinalizeStatus === "succeeded" && assessment.attentionRequests.length === 0 && (assessment.reportedDisposition === "done" || (assessment.reportedDisposition === "yielded" && (externalChatResponseWaitAuthorization === "authorized" || - (assessment.continuation?.kind === "monitor" && monitorWaitAt !== null)))) + (assessment.continuation?.kind === "monitor" && monitorWaitAt !== null && !preservedMonitorWait)))) ? await findSupersededExternalReviewRequests(input.db, { companyId: run.companyId, issueId: authoritativeIssue.id, runId: run.id, agentId: run.agentId, }) : []; @@ -1373,6 +1425,7 @@ export async function finalizeNativeRun(input: { const proposedDecision = resolveNativeFinalizerStatus({ caseHandoffAuthorized: Boolean(await assertNativeCaseHandoffResult(input.db, run.id, authoritativeIssue.id, result)), monitorWaitDispatched: monitorDelivery !== null, + monitorWaitPreserved: preservedMonitorWait !== null, monitorWaitAuthorized: authoritativeIssue.workMode === "standard" && authoritativeIssue.executionRunId === run.id && monitorWaitAt !== null, planWaitAuthorized: planWait !== null, hasPendingChildCompletion, @@ -1435,6 +1488,7 @@ export async function finalizeNativeRun(input: { workspaceFinalizeStatus: input.workspaceFinalizeStatus, monitorDelivery, monitorWaitAt: assessment.continuation?.kind === "monitor" ? monitorWaitAt : null, + preservedMonitorWait, }); await input.db .update(nativeRunFinalizations) @@ -1476,6 +1530,8 @@ export async function finalizeNativeRun(input: { ? { agentId: run.agentId, nextCheckAt: monitorWaitAt } : undefined, requireMonitorDelivery: decision.reasonCode === "scheduled_monitor_dispatched" ? monitorDelivery ?? undefined : undefined, + requirePreservedMonitorWait: decision.reasonCode === "scheduled_monitor_preserved" + ? preservedMonitorWait ?? undefined : undefined, requirePlanWaitSource: decision.reasonCode === "native_plan_accepted_waiting_for_continuation" ? planWait?.source : undefined, @@ -1562,33 +1618,42 @@ export async function finalizeNativeRun(input: { statusVersionAfter: Number(committed.issue.statusVersion), workspaceFinalizeStatus: input.workspaceFinalizeStatus, }; + const projectedStatus = terminalState === "succeeded" ? "succeeded" as const + : terminalState === "cancelled" ? "cancelled" as const : "failed" as const; + const projectedResult = sql`coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) + || ${JSON.stringify(finalizationMetadata)}::jsonb + || ${clearExecutionFailure ? recoveredExecutionFailureMetadata() : sql`'{}'::jsonb`}`; + const projectedError = providerFailure ? sql`coalesce(${heartbeatRuns.error}, ${providerFailure.errorMessage})` + : clearExecutionFailure ? sql`null` : heartbeatRuns.error; + const projectedErrorCode = providerFailure ? sql`coalesce(${heartbeatRuns.errorCode}, ${providerFailure.errorCode})` + : clearExecutionFailure ? sql`null` : heartbeatRuns.errorCode; + const projectionChanged = or( + sql`${heartbeatRuns.nativePhase} is distinct from ${finalizationPhase}`, + sql`${heartbeatRuns.resultJson} is distinct from ${projectedResult}`, + sql`${heartbeatRuns.error} is distinct from ${projectedError}`, + sql`${heartbeatRuns.errorCode} is distinct from ${projectedErrorCode}`, + ...(input.projectRunStatus ? [ + sql`${heartbeatRuns.status} is distinct from ${projectedStatus}`, + isNull(heartbeatRuns.finishedAt), + ] : []), + ); const projectRun = (tx: Db) => tx .update(heartbeatRuns) .set({ - executionStatusDeliveryId: randomUUID(), + executionStatusDeliveryId: sql`case when ${projectionChanged} + then ${randomUUID()}::uuid else ${heartbeatRuns.executionStatusDeliveryId} end`, ...(input.projectRunStatus ? { - status: - terminalState === "succeeded" - ? "succeeded" - : terminalState === "cancelled" - ? "cancelled" - : "failed", - finishedAt: now, + status: projectedStatus, + finishedAt: sql`case when ${heartbeatRuns.status} is distinct from ${projectedStatus} + then ${now.toISOString()}::timestamptz else coalesce(${heartbeatRuns.finishedAt}, ${now.toISOString()}::timestamptz) end`, } : {}), nativePhase: finalizationPhase, nativePhaseUpdatedAt: now, ...(clearExecutionFailure ? { error: null, errorCode: null } : {}), - ...(providerFailure ? { - // Recovery can finalize before heartbeat saves the adapter result. - // Retain any independently recorded execution/cleanup failure. - error: sql`coalesce(${heartbeatRuns.error}, ${providerFailure.errorMessage})`, - errorCode: sql`coalesce(${heartbeatRuns.errorCode}, ${providerFailure.errorCode})`, - } : {}), - resultJson: sql`coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) - || ${JSON.stringify(finalizationMetadata)}::jsonb - || ${clearExecutionFailure ? recoveredExecutionFailureMetadata() : sql`'{}'::jsonb`}`, + ...(providerFailure ? { error: projectedError, errorCode: projectedErrorCode } : {}), + resultJson: projectedResult, updatedAt: now, }) .where(eq(heartbeatRuns.id, run.id)) diff --git a/server/src/services/native-runtime/native-runner-evidence-signal.ts b/server/src/services/native-runtime/native-runner-evidence-signal.ts new file mode 100644 index 00000000000..39d51bee146 --- /dev/null +++ b/server/src/services/native-runtime/native-runner-evidence-signal.ts @@ -0,0 +1,12 @@ +const listeners = new Set<(writtenAt: number) => void>(); + +export function onNativeRunnerEvidence(listener: (writtenAt: number) => void): () => void { + listeners.add(listener); + return () => { + listeners.delete(listener); + }; +} + +export function noteNativeRunnerEvidence(writtenAt: number): void { + for (const listener of listeners) listener(writtenAt); +} diff --git a/server/src/services/native-runtime/native-session-executor.test.ts b/server/src/services/native-runtime/native-session-executor.test.ts index 51e0c3af818..39862333f8c 100644 --- a/server/src/services/native-runtime/native-session-executor.test.ts +++ b/server/src/services/native-runtime/native-session-executor.test.ts @@ -33,6 +33,7 @@ import { dirname, join } from "node:path"; import { heartbeatRuns, heartbeatRunEvents, + issueThreadInteractions, issues, nativeRunFinalizations, nativeRunResults, @@ -49,14 +50,17 @@ import { validatePrpStructuredRunResult, validatePrpEvent, parseNativeExecutionInput, + executeNativeSession, type NativeExecutionInputV1, type NativeExecutionInputV3, type NativeExecutionInput, type PrpEvent, } from "@paperclipai/paperclip-runner"; +import { existsSync } from "node:fs"; import { createHash, generateKeyPairSync, randomUUID } from "node:crypto"; import { DatabaseSync } from "node:sqlite"; import { nativeSha256 } from "./canonical.js"; +import { resolveRemoteProviderPackAuthority } from "./remote-provider-pack-authority.js"; import { RemoteRunnerRecoveryError, readRemoteRunnerRecoveryError } from "./remote-runner-recovery-diagnostics.js"; import { agentDirectoryWorkingCopyService } from "../agent-directory-working-copies.js"; import { probeAgentDirectory } from "../agent-directory-probe.js"; @@ -341,6 +345,7 @@ import { executePaperclipNativeSession, detachNativeSessionsForRestart, NativeControllerDetachedForRestartError, + NativeCancellationPendingRecoveryError, getNativeSessionSteeringState, NativeSessionSteeringError, assertRemoteRunnerBuildMetadata, @@ -389,6 +394,7 @@ import { verifyNativeHarnessBackup, shouldRestoreNativeHarnessBackupIntoSandbox, } from "./native-session-executor.js"; +import { onNativeRunnerEvidence } from "./native-runner-evidence-signal.js"; beforeEach(() => { usageRecorder.capture.mockReset().mockImplementation(async receipt => receipt); @@ -1432,7 +1438,7 @@ describe("remote provider pack manifest", () => { productionLock: "pnpm-lock.yaml", opencodeCommand: "node_modules/.bin/opencode", opencodeExecutable: "node_modules/opencode-ai/bin/opencode.exe", opencodeProxy: "dist/cli/opencode-app-server-proxy.cjs", acpxSidecar: "dist/cli/acpx-runtime-sidecar.cjs", - }).map(([key, path]) => [key, { path, sha256: digest }])); + }).map(([key, path]) => [key, { path, sha256: digest }])) as Record<"nodeCommand" | "opencodeCommand", { path: string; sha256: string }>; const payload = { target: { platform: "linux", architecture }, runnerSourceRevision: "a".repeat(40), pins: { nodeMinimum: "24.11.0", codex: QUALIFIED_ACPX_PROFILES.codex.agentRuntimeVersion, @@ -1484,6 +1490,43 @@ describe("remote provider pack manifest", () => { sourcePath: state.bundledRunnerBinary, targetPath: "/workspace/.paperclip-runtime/paperclip-runner/bin/paperclip-runnerd", })] })]); expect(execute.mock.calls.some(([call]) => call.args?.[1] === "uname -s; uname -m")).toBe(false); + expect(existsSync(join(transport.stateDirectory!, "provider-pack-authority.json"))).toBe(false); + if (provider === "claude-x64") { + const identity = transport.prpIdentity!; + const artifact = { version: "0.0.0", digest: "sha256:" + createHash("sha256").update(daemon).digest("hex") }; + resolveRemoteProviderPackAuthority({ stateDirectory: transport.stateDirectory!, + binding: { companyId: remoteExecution.binding.companyId, agentId: remoteExecution.binding.agentId, + workspaceId: remoteExecution.binding.executionWorkspaceId, sessionId: remoteExecution.session.normalizedSessionId!, + runnerInstanceId: identity.runnerInstanceId, environmentLeaseId: identity.environmentLeaseId, + remoteCwd: "/workspace", providerDigest: nativeSha256(remoteExecution.provider) }, + reattach: false, current: () => manifest, runnerArtifact: () => artifact }).verified(); + await mkdir(join(transport.stateDirectory!, "control-plane"), { recursive: true }); + await writeFile(join(transport.stateDirectory!, "control-plane", "control-plane-state.json"), JSON.stringify(durableControlPlaneState(identity))); + const next = structuredClone(manifest); + next.payload.runnerSourceRevision = "b".repeat(40); + next.digest = "sha256:" + createHash("sha256").update(canonical(next.payload)).digest("hex"); + await writeFile(state.bundledPackManifestPath!, JSON.stringify(next)); + await writeFile(state.bundledRunnerBinary!, Buffer.concat([daemon, Buffer.from("new release")])); + const recoveryExecute = vi.fn(async (request: { args: string[] }) => { + const script = request.args[1]; + const stdout = script.includes("base64") ? Buffer.from(JSON.stringify({ ...identity, lifecycle: "running" })).toString("base64") + : script.includes("cat --") ? `nonce\n123\n2026-09-19T10:00:00.000Z\n${identity.runnerInstanceId}\nlinux:abcd-1234:100\n` : ""; + return { stdout, stderr: "", exitCode: 0, timedOut: false }; + }); + await createRunnerdBackend({ db: leaseDb(remoteExecution), execution: remoteExecution, + runnerInstanceId: identity.runnerInstanceId, + runnerExecutionTarget: { kind: "remote", transport: "sandbox", remoteCwd: "/workspace", + environmentId: "environment", leaseId: "lease", providerKey: "daytona", runner: { execute: recoveryExecute }, + sandboxLeaseAcquisition: { outcome: "resumed", providerLeaseId: "sandbox" } } as never, + restartRecovery: { kind: "reattach_remote_runner", runId: remoteExecution.binding.runId, + leaseOwner: "controller", controllerGeneration: 2, providerAttempt: 1, restartKind: "graceful", recoveryRequestId: null, + remote: { providerLeaseId: "sandbox", remoteCwd: "/workspace" } }, + }); + state.createBackend.mock.calls.at(-1)![1].codexTransportFactory!(); + expect(state.createTransport.mock.calls.at(-1)![0]).toMatchObject({ + providerPackAuthorityDigest: manifest.digest, runnerArtifact: artifact, adoptExistingRunner: { pid: 123 }, + }); + } } finally { state.bundledPackManifestPath = null; state.bundledRunnerBinary = null; await rm(root, { recursive: true, force: true }); @@ -1838,13 +1881,35 @@ describe("native harness persistence profiles", () => { }); }); - it("excludes only nested Codex launch state from ACPX recovery", () => { + it("excludes per-launch OpenCode homes and re-installable plugins from OpenCode recovery", () => { + const opencode = resolveNativeHarnessPersistenceProfile({ + provider: { kind: "opencode" }, + session: { + driverKind: "opencode_server", + normalizedSessionId: "company:agent/session 1", + protocolVersion: 1, + lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null }, + }, + } as unknown as NativeExecutionInputV1); + expect( + opencode.directories.find((directory) => directory.name === "opencode"), + ).toMatchObject({ + excludeEntries: [ + "company_agent_session_1/homes", + "company_agent_session_1/config/opencode/node_modules", + ], + }); + }); + + it("excludes only nested Codex launch state and package caches from ACPX recovery", () => { const acpx = profile({ kind: "acpx", agent: "codex" }, "acpx_runtime"); const sessionDirectory = acpxRuntimeSessionDirectoryName("session"); expect( acpx.directories.find((directory) => directory.name === "acpx"), ).toMatchObject({ excludeEntries: [ + `acpx/${sessionDirectory}/home/.npm`, + `acpx/${sessionDirectory}/data/pnpm/store`, `acpx/${sessionDirectory}/codex-home/tmp`, `acpx/${sessionDirectory}/codex-home/.tmp`, `acpx/${sessionDirectory}/codex-home/auth.json`, @@ -1865,7 +1930,12 @@ describe("native harness persistence profiles", () => { { kind: "acpx", agent: "claude" }, "acpx_runtime", ).directories.find((directory) => directory.name === "acpx"), - ).toMatchObject({ excludeEntries: [] }); + ).toMatchObject({ + excludeEntries: [ + `acpx/${sessionDirectory}/home/.npm`, + `acpx/${sessionDirectory}/data/pnpm/store`, + ], + }); }); }); @@ -2581,6 +2651,100 @@ describe("remote provider checkpoint snapshots", () => { expect(await readFile(join(home, "logs", "unified.jsonl"), "utf8")).toBe("AUTH_SENTINEL"); } finally { await rm(root, { recursive: true, force: true }); } }); + it.each<{ + name: string; + provider: Record; + driverKind: string; + directory: string; + sessionPath: string; + kept: Record; + omitted: Record; + }>([ + { + name: "OpenCode", + provider: { kind: "opencode" }, + driverKind: "opencode_server", + directory: "opencode", + sessionPath: "company_agent_session_1", + kept: { + "workspace": "/workspace\n", + "data/opencode/opencode.db": "history", + "config/opencode/opencode.json": "{}", + "config/opencode/package.json": "{}", + "home-legacy/.local/state/opencode/locks/lock": "legacy", + }, + omitted: { + "homes/home-AbC123/.npm/_cacache/index": "npm-cache", + "homes/home-AbC123/.claude/skills/assigned/SKILL.md": "skill", + "config/opencode/node_modules/@opencode-ai/plugin/package.json": "plugin", + }, + }, + { + name: "ACPX", + provider: { kind: "acpx", agent: "pi" }, + driverKind: "acpx_runtime", + directory: "acpx", + sessionPath: `acpx/${acpxRuntimeSessionDirectoryName("company:agent/session 1")}`, + kept: { + "workspace": "/workspace\n", + "pi-home/sessions/session.jsonl": "history", + "data/pnpm/config.json": "{}", + "home/.profile": "profile", + }, + omitted: { + "home/.npm/_cacache/index": "npm-cache", + "data/pnpm/store/v10/files/00/blob": "pnpm-store", + }, + }, + ])("omits disposable $name package caches from the checkpoint and restores the session without them", async ({ + provider, driverKind, directory, sessionPath, kept, omitted, + }) => { + const root = await mkdtemp(join(tmpdir(), "provider-cache-checkpoint-")); + try { + const live = join(root, "live", directory); + const backup = join(root, "backup", directory); + const replacement = join(root, "replacement", directory); + const files: Record = { ...kept, ...omitted }; + for (const [path, contents] of Object.entries(files)) { + await mkdir(join(live, sessionPath, path, ".."), { recursive: true }); + await writeFile(join(live, sessionPath, path), contents); + } + const persistence = resolveNativeHarnessPersistenceProfile({ + provider, + session: { normalizedSessionId: "company:agent/session 1", driverKind }, + } as unknown as NativeExecutionInputV1); + const excludeEntries = persistence.directories.find(entry => entry.name === directory)!.excludeEntries; + const execute = async ({ command, args }: { command: string; args: string[] }) => ({ + exitCode: 0, timedOut: false, stdout: execFileSync(command, args, { encoding: "utf8" }), stderr: "", + }); + const copy = async (operations: Array<{ files: Array<{ sourcePath: string; targetPath: string }> }>) => { + for (const operation of operations) for (const file of operation.files) + await cp(file.sourcePath, file.targetPath, { recursive: true }); + }; + await syncRemoteRunnerDirectoryOut({ runner: { execute, syncOut: copy } as never, + sourcePath: live, targetPath: backup, mode: 0o700, excludeEntries }); + for (const [path, contents] of Object.entries(kept)) + expect(await readFile(join(backup, sessionPath, path), "utf8")).toBe(contents); + for (const path of Object.keys(omitted)) + await expect(access(join(backup, sessionPath, path))).rejects.toMatchObject({ code: "ENOENT" }); + for (const [path, contents] of Object.entries(omitted)) + expect(await readFile(join(live, sessionPath, path), "utf8")).toBe(contents); + + await stageRemoteRunnerDirectory({ + target: { kind: "remote", transport: "provider", remoteCwd: root, runner: { syncIn: copy } } as never, + runner: { syncIn: copy } as never, + sourcePath: backup, + targetPath: replacement, + mode: 0o700, + excludeEntries, + }); + for (const [path, contents] of Object.entries(kept)) + expect(await readFile(join(replacement, sessionPath, path), "utf8")).toBe(contents); + for (const path of Object.keys(omitted)) + await expect(access(join(replacement, sessionPath, path))).rejects.toMatchObject({ code: "ENOENT" }); + } finally { await rm(root, { recursive: true, force: true }); } + }); + it("keeps Codex session state while omitting re-downloadable remote plugins from the checkpoint", async () => { const root = await mkdtemp(join(tmpdir(), "codex-plugin-checkpoint-")); try { @@ -5656,6 +5820,10 @@ function leaseDb( agentId: boundExecution.binding.agentId, companyId: boundExecution.binding.companyId, nativeIssueId: boundExecution.binding.issueId, + nativeSessionId: boundExecution.session.normalizedSessionId ?? `session-${boundExecution.binding.runId}`, + runnerInstanceId: "runner", + completionContractId: boundExecution.completionContract.id, + completionContractSha256: boundExecution.completionContract.sha256, resultJson: runResultJson, runnerProfileJson, runtimeMode: "native", @@ -5853,6 +6021,306 @@ describe("native startup cancellation fence", () => { }); describe("native startup restart detachment", () => { + it("uses one admission deadline for the whole restart sweep", async () => { + const inputs = ["first", "second"].map(label => { + const input = structuredClone(execution); + input.binding.runId = `restart-shared-deadline-${label}`; + input.binding.agentId = `restart-shared-deadline-agent-${label}`; + return input; + }); + const releases: Array<() => void> = []; + const ready: Array> = []; + state.execute.mockReset(); + for (const _input of inputs) { + let started!: () => void; + ready.push(new Promise(resolve => { started = resolve; })); + const opening = new Promise(resolve => { releases.push(resolve); }); + state.execute.mockImplementationOnce(async () => { + started(); await opening; + throw new Error("startup fixture released"); + }); + } + const outcomes = inputs.map(input => executePaperclipNativeSession({ + db: leaseDb(input), execution: input, runnerInstanceId: "runner", + }).catch(error => error)); + await Promise.all(ready); + vi.useFakeTimers({ toFake: ["setTimeout", "clearTimeout", "Date"] }); + const finished = vi.fn(); + const runIds = inputs.map(input => input.binding.runId); + const shutdown = detachNativeSessionsForRestart(runIds).then(finished); + try { + await vi.advanceTimersByTimeAsync(30_000); + expect(finished).toHaveBeenCalledWith({ detachedRunIds: [], inactiveRunIds: [], + unsupportedRunIds: [], pendingRunIds: runIds }); + } finally { + vi.useRealTimers(); releases.forEach(release => release()); + await Promise.all(outcomes); await shutdown; + } + }); + + it.each(["authority", "questions"])("keeps a post-publication Stop owner through %s attachment and restart", async (seam) => { + const restarting = structuredClone(execution); + restarting.binding.runId = `restart-stop-after-publication-${seam}`; + let published!: () => void, releaseRead!: () => void, acknowledgeStop!: () => void; + let cleaning!: () => void, releaseCleanup!: () => void; + const publication = new Promise(resolve => { published = resolve; }); + const read = new Promise(resolve => { releaseRead = resolve; }); + const receipt = new Promise(resolve => { acknowledgeStop = resolve; }); + const cleanupStarted = new Promise(resolve => { cleaning = resolve; }); + const cleanup = new Promise(resolve => { releaseCleanup = resolve; }); + const detach = vi.fn(async () => undefined); + const cancel = vi.fn(() => ({ cleanup: receipt })); + const submit = vi.fn(); + const db = leaseDb(restarting); + const select = db.select.bind(db); + db.select = ((fields?: Record) => { + if (seam === "authority" && fields && Object.keys(fields).length === 2 && "status" in fields && "resultJson" in fields) { + return { from: () => ({ where: () => ({ limit: () => { + published(); + return read.then(() => [{ status: "running", resultJson: {} }]); + } }) }) }; + } + if (seam === "questions" && !fields) { + const query = select(); + return { from: (table: unknown) => table === issueThreadInteractions + ? { where: () => ({ orderBy: () => { published(); return read.then(() => []); } }) } + : query.from(table as never) }; + } + return select(fields as never); + }) as typeof db.select; + state.execute.mockReset().mockImplementationOnce(async options => { + try { + await options.onSession({ detachControllerForRestart: detach, cancel, resolveRuntimeRequest: vi.fn() }); + submit(); + } finally { + cleaning(); + await cleanup; + await options.onSession(null); + } + }); + const outcome = executePaperclipNativeSession({ db, execution: restarting, runnerInstanceId: "runner" }).catch(error => error); + await publication; + const stopping = cancelNativeSession(restarting.binding.runId, "operator Stop after publication"); + const completed = vi.fn(); + const shutdown = detachNativeSessionsForRestart([restarting.binding.runId]).then(completed); + try { + releaseRead(); + await new Promise(resolve => setImmediate(resolve)); + expect(cancel).toHaveBeenCalledOnce(); + expect(detach).not.toHaveBeenCalled(); + expect(completed).not.toHaveBeenCalled(); + // Admission must not unwind into generic close while Stop awaits its ACK. + const cleanupObserved = vi.fn(); + void cleanupStarted.then(cleanupObserved); + await new Promise(resolve => setImmediate(resolve)); + expect(cleanupObserved).not.toHaveBeenCalled(); + acknowledgeStop(); + expect(await stopping).toBe(true); + await cleanupStarted; + expect(completed).not.toHaveBeenCalled(); + releaseCleanup(); + await outcome; + await shutdown; + expect(completed).toHaveBeenCalledWith(expect.objectContaining({ + detachedRunIds: [], inactiveRunIds: [restarting.binding.runId], pendingRunIds: [], + })); + expect(detach).not.toHaveBeenCalled(); + expect(submit).not.toHaveBeenCalled(); + } finally { + releaseRead(); acknowledgeStop(); releaseCleanup(); + await stopping; await outcome; await shutdown; + } + }); + + it.each(["startup", "active"])("keeps healthy hot-restart adoption when another %s Stop cleanup exceeds the sweep deadline", async (mode) => { + const { coordinateHeartbeatSchedulerShutdown } = await import("../../shutdown.js"); + const { createHeartbeatRecovery } = await import("../heartbeat/recovery.js"); + const { writeHotRestartIntent, readHotRestartIntent } = await import("../hot-restart.js"); + const home = await mkdtemp(join(tmpdir(), "native-restart-stop-sweep-")); + const previousHome = process.env.PAPERCLIP_HOME; + process.env.PAPERCLIP_HOME = home; + await writeHotRestartIntent({ previousServerPid: process.pid, previousServerIdentity: "test-controller" }); + const stoppingExecution = structuredClone(execution); + stoppingExecution.binding.runId = `restart-${mode}-stop-timeout`; + stoppingExecution.session.normalizedSessionId = "d371f9bb-6c46-44c6-9e3a-026fa9be0153"; + const healthyExecution = structuredClone(execution); + healthyExecution.binding.runId = `restart-healthy-with-${mode}-stopping-peer`; + healthyExecution.session.normalizedSessionId = "a8fa0cb8-9cf1-46b0-96f0-78518b15847e"; + healthyExecution.binding.agentId = "healthy-restart-agent"; + let stoppedReady!: () => void, healthyReady!: () => void; + let releaseStopped!: () => void, releaseHealthy!: () => void, opening!: () => void, open!: () => void; + const openingStarted = new Promise(resolve => { opening = resolve; }); + const openingGate = new Promise(resolve => { open = resolve; }); + const stopPublished = new Promise(resolve => { stoppedReady = resolve; }); + const healthyPublished = new Promise(resolve => { healthyReady = resolve; }); + const stoppedLifetime = new Promise(resolve => { releaseStopped = resolve; }); + const healthyLifetime = new Promise(resolve => { releaseHealthy = resolve; }); + const stoppedDetach = vi.fn(async () => undefined); + const healthyDetach = vi.fn(async () => undefined); + state.execute.mockReset() + .mockImplementationOnce(async options => { + opening(); await openingGate; + try { + await options.onSession({ detachControllerForRestart: stoppedDetach, cancel: () => ({ cleanup: Promise.resolve() }) }); + } finally { + stoppedReady(); await stoppedLifetime; + await options.onSession(null); + } + throw new Error("stopped owner settled"); + }) + .mockImplementationOnce(async options => { + await options.onSession({ detachControllerForRestart: healthyDetach }); + healthyReady(); await healthyLifetime; + await options.onSession(null); + throw new Error("healthy controller detached"); + }); + const stoppedOutcome = executePaperclipNativeSession({ db: leaseDb(stoppingExecution), execution: stoppingExecution, runnerInstanceId: "runner" }).catch(error => error); + await openingStarted; + const stopping = mode === "startup" ? cancelNativeSession(stoppingExecution.binding.runId, "Stop while opening") : null; + open(); + await stopPublished; + await stopping; + const healthyOutcome = executePaperclipNativeSession({ db: leaseDb(healthyExecution), execution: healthyExecution, runnerInstanceId: "runner" }).catch(error => error); + await healthyPublished; + if (mode === "active") await cancelNativeSession(stoppingExecution.binding.runId, "operator Stop after active admission"); + const snapshotRows = [stoppingExecution, healthyExecution].map(item => ({ + adapterType: "paperclip_runner", adapterConfig: {}, + run: { id: item.binding.runId, companyId: item.binding.companyId, agentId: item.binding.agentId, + runtimeMode: "native", nativeSessionId: item.session.normalizedSessionId, status: "running", contextSnapshot: {} }, + })); + const snapshotDb = { select: () => ({ from: () => ({ innerJoin: () => ({ where: async () => snapshotRows }) }) }) } as unknown as Db; + const { logger } = await import("../../middleware/logger.js"); + const logged = vi.spyOn(logger, "info").mockImplementation(() => undefined); + const recovery = createHeartbeatRecovery(snapshotDb, { + enterShutdown: vi.fn(), appendRunEvent: vi.fn(async () => undefined), + } as unknown as import("../heartbeat/recovery.js").HeartbeatRecoveryDependencies); + vi.useFakeTimers({ toFake: ["setTimeout", "clearTimeout", "Date"] }); + const shutdown = coordinateHeartbeatSchedulerShutdown({ signal: "SIGTERM", + waitForHeartbeatSchedulerIdle: async () => undefined, + prepareHotRestartShutdown: recovery.prepareHotRestartShutdown, + }); + try { + // Wait for the real snapshot I/O before advancing its bounded sweep. + await vi.waitFor(() => expect(vi.getTimerCount()).toBeGreaterThan(0)); + await vi.advanceTimersByTimeAsync(30_000); + expect(await shutdown).toMatchObject({ hotRestart: { mode: "hot_restart", skipDrain: true }, preparationError: null }); + expect(logged).toHaveBeenCalledWith(expect.objectContaining({ detachedNativeSessions: { + detachedRunIds: [healthyExecution.binding.runId], inactiveRunIds: [], unsupportedRunIds: [], + pendingRunIds: [stoppingExecution.binding.runId], + } }), expect.any(String)); + expect((await readHotRestartIntent())?.shutdownSnapshot?.activeRuns.map(run => run.runId)) + .toEqual([stoppingExecution.binding.runId, healthyExecution.binding.runId]); + expect(stoppedDetach).not.toHaveBeenCalled(); + expect(healthyDetach).toHaveBeenCalledOnce(); + } finally { + vi.useRealTimers(); releaseStopped(); releaseHealthy(); + await stoppedOutcome; await healthyOutcome; await shutdown; + logged.mockRestore(); + if (previousHome === undefined) delete process.env.PAPERCLIP_HOME; + else process.env.PAPERCLIP_HOME = previousHome; + await rm(home, { recursive: true, force: true }); + } + }); + + it.each(["detach", "expired", "stop"])("bounds read-only startup admission during drain: %s", async (mode) => { + const expired = mode === "expired"; + const stopped = mode === "stop"; + const restarting = structuredClone(execution); + restarting.binding.runId = `restart-real-admission-${mode}`; + restarting.session.normalizedSessionId = "restart-real-admission-session"; + const identity = { ...restarting.binding, sessionId: restarting.session.normalizedSessionId }; + let release!: () => void, started!: () => void; + const opening = new Promise(resolve => { release = resolve; }); + const pending = new Promise(resolve => { started = resolve; }); + const detach = vi.fn(async () => undefined); + const close = vi.fn(async () => undefined); + const startTurn = vi.fn(); + let acknowledgeStop!: () => void; + const stopReceipt = new Promise(resolve => { acknowledgeStop = resolve; }); + const cancel = vi.fn(() => ({ cleanup: stopReceipt })); + const backend = { + descriptor: async () => ({ kind: "mock" as const, name: "startup-drain", version: "1", + capabilities: { resume: true, typedEvents: true, steering: false, interruption: true, structuredResult: true }, + runtimeContextCapabilities: { instructions: "native", skills: "native", mcp: "native" } }), + openSession: vi.fn(async () => { + started(); await opening; + return { identity: () => identity, detachControllerForRestart: detach, close, startTurn, cancel, + snapshot: async () => ({ sessionId: identity.sessionId, identity, backendKind: "mock" as const }), + }; + }), + }; + let runtimeError: unknown; + state.execute.mockReset().mockImplementationOnce(async options => { + try { return await executeNativeSession({ ...options, onSessionAdmission: undefined }); } catch (error) { runtimeError = error; throw error; } + }); + const outcome = executePaperclipNativeSession({ db: leaseDb(restarting), execution: restarting, + runnerInstanceId: "runner", backend: backend as never }).catch(error => error); + await Promise.race([pending, outcome.then(error => { throw runtimeError ?? error; })]); + const stopping = stopped ? cancelNativeSession(restarting.binding.runId, "operator Stop during startup") : null; + if (expired) vi.useFakeTimers({ toFake: ["setTimeout", "clearTimeout"] }); + const shutdown = detachNativeSessionsForRestart([restarting.binding.runId]).catch(error => error); + try { + if (expired) { + await vi.advanceTimersByTimeAsync(30_000); + expect(await shutdown).toMatchObject({ pendingRunIds: [restarting.binding.runId], detachedRunIds: [] }); + vi.useRealTimers(); + } else { + await new Promise(resolve => setImmediate(resolve)); + } + release(); + if (stopped) { + await vi.waitFor(() => expect(cancel).toHaveBeenCalledOnce()); + expect(detach).not.toHaveBeenCalled(); + acknowledgeStop(); + expect(await stopping).toBe(true); + } + expect(await outcome).toBeInstanceOf(NativeControllerDetachedForRestartError); + if (stopped) { + expect(await shutdown).toMatchObject({ detachedRunIds: [], inactiveRunIds: [restarting.binding.runId] }); + expect(detach).not.toHaveBeenCalled(); + } else if (expired) { + expect(detach).not.toHaveBeenCalled(); + expect(close).toHaveBeenCalledWith({ reason: "native control-plane run admission failed" }); + } else { + expect(await shutdown).toMatchObject({ detachedRunIds: [restarting.binding.runId], inactiveRunIds: [] }); + expect(detach).toHaveBeenCalledOnce(); + expect(close).not.toHaveBeenCalledWith({ reason: "native control-plane run admission failed" }); + } + expect(backend.openSession).toHaveBeenCalledOnce(); + expect(startTurn).not.toHaveBeenCalled(); + } finally { vi.useRealTimers(); release(); acknowledgeStop(); await stopping; await outcome; await shutdown; } + }); + it("bounds a pending startup Stop without detaching its cleanup owner", async () => { + const restarting = structuredClone(execution); + restarting.binding.runId = "restart-startup-stop-cleanup-timeout"; + let open!: () => void, started!: () => void, releaseCleanup!: () => void, cleaning!: () => void; + const opening = new Promise(resolve => { open = resolve; }); + const pending = new Promise(resolve => { started = resolve; }); + const cleanup = new Promise(resolve => { releaseCleanup = resolve; }); + const cleanupStarted = new Promise(resolve => { cleaning = resolve; }); + const detach = vi.fn(async () => undefined); + const cancel = vi.fn(() => ({ cleanup: Promise.resolve() })); + state.execute.mockReset().mockImplementationOnce(async options => { + started(); await opening; + try { await options.onSession({ detachControllerForRestart: detach, cancel }); } + finally { cleaning(); await cleanup; } + }); + const outcome = executePaperclipNativeSession({ db: leaseDb(restarting), execution: restarting, runnerInstanceId: "runner" }).catch(error => error); + await pending; + const stopping = cancelNativeSession(restarting.binding.runId, "operator Stop before restart"); + vi.useFakeTimers({ toFake: ["setTimeout", "clearTimeout"] }); + const shutdown = detachNativeSessionsForRestart([restarting.binding.runId]).catch(error => error); + try { + open(); + await cleanupStarted; + expect(await stopping).toBe(true); + await vi.advanceTimersByTimeAsync(30_000); + expect(await shutdown).toMatchObject({ pendingRunIds: [restarting.binding.runId], detachedRunIds: [] }); + expect(detach).not.toHaveBeenCalled(); + expect(cancel).toHaveBeenCalledOnce(); + } finally { vi.useRealTimers(); open(); releaseCleanup(); await stopping; await outcome; await shutdown; } + }); + it("waits for a checkpointed governed settlement before relinquishing its controller", async () => { const restarting = structuredClone(execution); restarting.binding.runId = "restart-governed-settlement"; @@ -5994,7 +6462,7 @@ describe("native startup restart detachment", () => { const detached = detachNativeSessionsForRestart([restarting.binding.runId]).catch(error => error); if (exceedDeadline) { await vi.advanceTimersByTimeAsync(30_000); - expect(await detached).toMatchObject({ message: "native_restart_startup_not_ready" }); + expect(await detached).toMatchObject({ pendingRunIds: [restarting.binding.runId], detachedRunIds: [] }); vi.useRealTimers(); } else { release(); @@ -6010,18 +6478,13 @@ describe("native startup restart detachment", () => { } }); - it("remembers shutdown while the session is still opening and detaches its late publication", async () => { + it("rejects an entirely new startup after the run has been fenced for shutdown", async () => { const restarting = structuredClone(execution); - restarting.binding.runId = "restart-during-session-open"; - const detach = vi.fn(async () => undefined); + restarting.binding.runId = "restart-new-startup-after-drain"; await expect(detachNativeSessionsForRestart([restarting.binding.runId])).resolves.toMatchObject({ inactiveRunIds: [restarting.binding.runId] }); - state.execute.mockReset().mockImplementationOnce(async (options) => { - await options.onSession({ detachControllerForRestart: detach }); - expect(detach).toHaveBeenCalledOnce(); - await options.onSession(null); - throw new Error("detachment closed the old event stream"); - }); + state.execute.mockReset(); await expect(executePaperclipNativeSession({ db: leaseDb(restarting), execution: restarting, runnerInstanceId: "runner" })).rejects.toBeInstanceOf(NativeControllerDetachedForRestartError); + expect(state.execute).not.toHaveBeenCalled(); }); }); @@ -11980,6 +12443,8 @@ describe("runnerd provider runtime wiring", () => { await writeRunnerAuthority(legacyRoot, identity, "ready"); state.createBackend.mockClear(); state.createTransport.mockClear(); + const evidence: number[] = []; + const stopEvidence = onNativeRunnerEvidence((writtenAt) => evidence.push(writtenAt)); await expect( createRunnerdBackend({ @@ -11987,11 +12452,12 @@ describe("runnerd provider runtime wiring", () => { execution: currentExecution, runnerInstanceId: "runner-after-legacy-terminal-unsuspended", }), - ).rejects.toThrow("runner_state_identity_mismatch"); + ).rejects.toThrow("runner_state_identity_mismatch").finally(stopEvidence); await expect(access(legacyRoot)).rejects.toThrow(); const quarantineEntries = await readdir(join(stateBase, "quarantine")); expect(quarantineEntries).toHaveLength(1); expect(quarantineEntries[0]).toContain(".identity_indeterminate."); + expect(evidence).toEqual([Number(quarantineEntries[0]!.split(".")[2])]); expect(state.createBackend).not.toHaveBeenCalled(); expect(state.createTransport).not.toHaveBeenCalled(); }); diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index fe6d4732e76..234f673e664 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -1,3 +1,5 @@ +import { providerPackWorkspaceId, recordAdmittedRemoteProviderPack, resolveRemoteProviderPackAuthority } from "./remote-provider-pack-authority.js"; +import { verifyRemoteProviderPack as verifyRemoteProviderPackIdentity } from "./remote-provider-pack-verification.js"; import { writeEnvironmentCleanup } from "../environment-cleanup-notifications.js"; import { DELIVERY_QUEUES } from "../delivery-work-notifications.js"; import { writeExecutionControl } from "../execution-control-notifications.js"; @@ -47,7 +49,7 @@ import { copyBackCodexAuth } from "@paperclipai/adapter-codex-local/server"; import { nativeCompletionFeedback } from "./native-completion-feedback.js"; import { hasAcknowledgedNativeReassignmentStopIntent, hasAcknowledgedNativeStopIntent } from "../acknowledged-native-stop.js"; import { stoppedCodexTurnIsTextOnly } from "./stopped-codex-turn.js"; -import { prepareVerifiedRemoteProviderPack, useVerifiedPreinstalledRemoteProviderPack, runRemoteProviderVerification, shouldUseBundledAcpxImageAssets } from "./remote-provider-pack.js"; +import { prepareVerifiedRemoteProviderPack, useVerifiedPreinstalledRemoteProviderPack, shouldUseBundledAcpxImageAssets } from "./remote-provider-pack.js"; import { readProviderPackArchive, stageProviderPackArchive } from "./provider-pack-archive.js"; import { selectRemotePiCompanion } from "./remote-pi-companion.js"; import { readNativeLocalProcessStop, PROCESS_START_REQUESTED, withReleasedNativeRunProcess } from "../native-local-process-stop.js"; @@ -107,6 +109,8 @@ import { completeRetainedNativeSessionCleanup, completeTerminatedLocalNativeSessionCleanup, acpxRuntimeSessionDirectoryName, + OPENCODE_ISOLATED_HOMES_DIRECTORY, + opencodeRuntimeSessionDirectoryName, createNativeSessionBackend, createRunnerdCodexTransport, executeNativeSession, @@ -205,6 +209,7 @@ import { type NativeRunTrace, } from "./native-run-trace.js"; import { createNativeHarnessBackupStamp } from "./native-harness-backup-stamp.js"; +import { noteNativeRunnerEvidence } from "./native-runner-evidence-signal.js"; import { removeNativeHarnessBackup } from "./native-harness-backup-cleanup.js"; import { registerLiveRunnerGoalController } from "../runner-goal-control-broker.js"; import { applyRunnerGoalPrpEvent } from "../runner-goals.js"; @@ -225,6 +230,7 @@ import { type ActiveNativeSession = { session: NativeSession; cancelRequested: boolean; + cancellationSettled?: Promise; restartDetach?: Promise; }; @@ -262,6 +268,7 @@ type NativeSessionStartup = { promise: Promise; resolve: (session: ActiveNativeSession | null) => void; stopRequested?: boolean; + restartAdmissionPending?: boolean; cancellationSettled?: Promise; governedWait?: boolean; settled: Promise; @@ -270,13 +277,14 @@ type NativeSessionStartup = { const nativeSessionStartups = new Map(); function detachActiveNativeSessionForRestart(active: ActiveNativeSession) { - active.restartDetach ??= Promise.resolve().then(() => active.session.detachControllerForRestart!()); + active.restartDetach ??= active.session.detachControllerForRestart!(); return active.restartDetach; } async function waitForNativeSessionStartup( runId: string, timeoutError: () => Error = () => new Error("native_restart_startup_not_ready"), + timeoutMs = 30_000, ) { const startup = nativeSessionStartups.get(runId); if (!startup) return null; @@ -285,7 +293,7 @@ async function waitForNativeSessionStartup( return await Promise.race([ startup.promise, new Promise((_resolve, reject) => { - timer = setTimeout(() => reject(timeoutError()), 30_000); + timer = setTimeout(() => reject(timeoutError()), timeoutMs); }), ]); } finally { @@ -299,10 +307,13 @@ export async function detachNativeSessionsForRestart( detachedRunIds: string[]; inactiveRunIds: string[]; unsupportedRunIds: string[]; + pendingRunIds: string[]; }> { const detachedRunIds: string[] = []; const inactiveRunIds: string[] = []; const unsupportedRunIds: string[] = []; + const pendingRunIds: string[] = []; + const restartDeadline = Date.now() + 30_000; const settlementDeadline = Date.now() + 20_000; for (const runId of new Set(runIds)) { // A governed stop already revoked new work and is collecting terminal @@ -318,11 +329,48 @@ export async function detachNativeSessionsForRestart( } finally { clearTimeout(timer); } } nativeRunsDetachingForRestart.add(runId); - const active = activeNativeSessions.get(runId) ?? await waitForNativeSessionStartup(runId); + let active = activeNativeSessions.get(runId); + if (!active) { + // Only a startup registered before this fence may complete read-only + // admission. It must publish its exact session for detachment; writes and + // new turns remain revoked, including after the bounded wait expires. + const startup = nativeSessionStartups.get(runId); + if (startup && Date.now() >= restartDeadline) { + pendingRunIds.push(runId); + continue; + } + if (startup) startup.restartAdmissionPending = true; + try { + active = await waitForNativeSessionStartup(runId, undefined, Math.max(0, restartDeadline - Date.now())) ?? undefined; + } catch (error) { + if (!(error instanceof Error) || error.message !== "native_restart_startup_not_ready") throw error; + pendingRunIds.push(runId); + continue; + } finally { + if (startup) startup.restartAdmissionPending = false; + } + } if (!active) { inactiveRunIds.push(runId); continue; } + const stopping = nativeSessionStartups.get(runId); + if (stopping?.stopRequested || active.cancelRequested) { + // Stop owns cancellation and final cleanup, including after publication. + // A deadline leaves this run pending; it must not abort healthy adoption + // or claim that the transport/cleanup owner has settled. + let timer: ReturnType | undefined; + try { + const settled = stopping && await Promise.race([ + stopping.settled.then(() => true), + new Promise(resolve => { + timer = setTimeout(() => resolve(false), Math.max(0, restartDeadline - Date.now())); + }), + ]); + (settled ? inactiveRunIds : pendingRunIds).push(runId); + } finally { clearTimeout(timer); } + continue; + } if (active.session.detachControllerForRestart === undefined) { unsupportedRunIds.push(runId); continue; @@ -330,7 +378,7 @@ export async function detachNativeSessionsForRestart( await detachActiveNativeSessionForRestart(active); detachedRunIds.push(runId); } - return { detachedRunIds, inactiveRunIds, unsupportedRunIds }; + return { detachedRunIds, inactiveRunIds, unsupportedRunIds, pendingRunIds }; } const MAX_REMOTE_CHECKPOINT_ARCHIVE_BYTES = 64 * 1024 * 1024; const MAX_REMOTE_CHECKPOINT_EXPANDED_BYTES = 64 * 1024 * 1024; @@ -365,6 +413,14 @@ const CODEX_HOME_CHECKPOINT_EXCLUDED_ENTRIES = [ ...CODEX_HOME_NON_PERSISTENT_ENTRIES, ...CODEX_HOME_REDOWNLOADABLE_ENTRIES, ] as const; +const OPENCODE_RUNTIME_CHECKPOINT_EXCLUDED_ENTRIES = [ + OPENCODE_ISOLATED_HOMES_DIRECTORY, + "config/opencode/node_modules", +] as const; +const ACPX_RUNTIME_CHECKPOINT_EXCLUDED_ENTRIES = [ + "home/.npm", + "data/pnpm/store", +] as const; const RUNNERD_CONTROL_PLANE_STATE_SCHEMA = "paperclip.runner.durable.control-plane-state.v1"; const RUNNERD_STATE_SCHEMA = "paperclip.runner.durable.state.v1"; @@ -1897,11 +1953,13 @@ function quarantineRunnerdStateRoot( if (!isSafeNativeStateDirectory(quarantineRoot)) { throw new Error("runner_state_directory_unsafe"); } + const quarantinedAt = Date.now(); const destination = resolve( quarantineRoot, - `${stateKey}.${reason}.${Date.now()}.${process.pid}.${randomUUID()}`, + `${stateKey}.${reason}.${quarantinedAt}.${process.pid}.${randomUUID()}`, ); renameSync(resolvedRoot, destination); + noteNativeRunnerEvidence(quarantinedAt); return destination; } @@ -5601,7 +5659,10 @@ export function resolveNativeHarnessPersistenceProfile( ? { name: "opencode", location: "filesystem", - excludeEntries: [], + excludeEntries: OPENCODE_RUNTIME_CHECKPOINT_EXCLUDED_ENTRIES.map( + (entry) => + `${opencodeRuntimeSessionDirectoryName(nativeSessionKey(execution))}/${entry}`, + ), } : execution.provider.kind === "acpx" ? { @@ -5613,8 +5674,10 @@ export function resolveNativeHarnessPersistenceProfile( // nor necessary to restore. Credentials and launch-time config // are also re-materialized in the replacement sandbox. Grok // diagnostics can contain auth fields and are not session state. - excludeEntries: - execution.provider.agent === "grok" + excludeEntries: [ + ...ACPX_RUNTIME_CHECKPOINT_EXCLUDED_ENTRIES.map((entry) => + `acpx/${acpxRuntimeSessionDirectoryName(nativeSessionKey(execution))}/${entry}`), + ...(execution.provider.agent === "grok" ? ["auth.json", "auth-refresh.json", "auth-refresh.json.tmp", "config.toml", "logs"].map((entry) => `acpx/${acpxRuntimeSessionDirectoryName(nativeSessionKey(execution))}/grok-home/${entry}`) : execution.provider.agent === "codex" @@ -5622,7 +5685,8 @@ export function resolveNativeHarnessPersistenceProfile( (entry) => `acpx/${acpxRuntimeSessionDirectoryName(nativeSessionKey(execution))}/codex-home/${entry}`, ) - : [], + : []), + ], } : null; return { @@ -7228,6 +7292,7 @@ export async function cancelNativeSession( // Do not acknowledge a no-op Stop and let that owner submit a turn afterwards. const startup = runStop && !activeNativeSessions.has(runId) ? nativeSessionStartups.get(runId) : undefined; let settleStartupCancellation: (() => void) | undefined; + let settleActiveCancellation: (() => void) | undefined; if (startup) { startup.stopRequested = true; startup.cancellationSettled = new Promise(resolve => { settleStartupCancellation = resolve; }); @@ -7240,6 +7305,7 @@ export async function cancelNativeSession( dispatched = true; if (!active.cancelRequested) { active.cancelRequested = true; + active.cancellationSettled = new Promise(resolve => { settleActiveCancellation = resolve; }); try { if (active.session.cancel) { const cancellationAbort = new AbortController(); @@ -7449,6 +7515,7 @@ export async function cancelNativeSession( return { dispatched, decision, decisionId, auditId }; } finally { settleStartupCancellation?.(); + settleActiveCancellation?.(); } } @@ -7672,6 +7739,9 @@ export async function executePaperclipNativeSession(input: { } } const runId = input.execution.binding.runId; + if (nativeRunsDetachingForRestart.has(runId)) { + throw new NativeControllerDetachedForRestartError(); + } if (nativeSessionStartups.has(runId)) { throw new Error("native_session_supervisor_busy"); } @@ -8360,6 +8430,12 @@ async function executePaperclipNativeSessionWithinScope( throw new NativeControllerDetachedForRestartError(); } }; + const startupForAdmission = nativeSessionStartups.get(input.execution.binding.runId); + const assertRunAdmissionActive = () => { + if (startupForAdmission?.restartAdmissionPending + && nativeSessionStartups.get(input.execution.binding.runId) === startupForAdmission) return; + assertControllerActive(); + }; const controlPlane = new PaperclipControlPlanePort( input.db, { @@ -8376,6 +8452,7 @@ async function executePaperclipNativeSessionWithinScope( { privateKeyPem: input.runnerEnvironment?.PAPERCLIP_AGENT_PRIVATE_KEY, assertControllerActive, + assertRunAdmissionActive, onCommittedEvent: async (event) => { const textKind = !firstTokenRecorded ? observeDeliverableText(event) : null; if (textKind) { @@ -9218,6 +9295,13 @@ async function executePaperclipNativeSessionWithinScope( eq(heartbeatRuns.agentId, input.execution.binding.agentId), )).limit(1); const cancellation = record(currentRun?.resultJson?.nativeCancellation); + if (active.cancelRequested) { + // A Stop can acquire the published handle while the authority + // read above is in flight. Keep its ACK/cleanup owner before + // unwinding admission; neither detach nor generic close wins. + await active.cancellationSettled; + throw new NativeCancellationPendingRecoveryError(); + } if (!currentRun || currentRun.status !== "running" || currentRun.resultJson?.startupCancellation || (cancellation.scope === "run" && @@ -9227,11 +9311,15 @@ async function executePaperclipNativeSessionWithinScope( // callback fails; no provider turn may follow publication. throw new NativeCancellationPendingRecoveryError(); } - if (session.resolveRuntimeRequest) await liveQuestions.attach(); if (nativeRunsDetachingForRestart.has(input.execution.binding.runId)) { if (session.detachControllerForRestart) await detachActiveNativeSessionForRestart(active); + throw new NativeControllerDetachedForRestartError(); + } + if (session.resolveRuntimeRequest) await liveQuestions.attach(); + if (active.cancelRequested) { + await active.cancellationSettled; + throw new NativeCancellationPendingRecoveryError(); } - if (active.cancelRequested) throw new NativeCancellationPendingRecoveryError(); } else { liveQuestions.close(); activeNativeSessions.delete(input.execution.binding.runId); @@ -11930,7 +12018,8 @@ async function createRunnerdBackendWithinSessionClaim( claudeEffort: "claudeEffort" in provider ? provider.claudeEffort : undefined, acpxPermissionModePinned: true } : {}), ...(provider.kind === "opencode" ? { opencodePermissionMode: provider.permissionMode } : {}), - runnerReconnectGraceMs: 120_000, lifecyclePolicy: input.execution.session.lifecyclePolicy, + runnerReconnectGraceMs: 120_000, runnerProcessReconnectGraceMs: null, + lifecyclePolicy: input.execution.session.lifecyclePolicy, runtimeContext: "runtimeContext" in input.execution ? input.execution.runtimeContext : null, externallySandboxed: true, baseInstructions: recovery?.baseInstructions, resumeDynamicTools: dynamicTools, resumeCompletionContract: { revision: input.execution.completionContract.contract.revision, @@ -12023,7 +12112,6 @@ async function createRunnerdBackendWithinSessionClaim( })); const configuredProviderPackRoot = input.runnerRemoteProviderPackPath?.trim() || remotePiCompanion?.providerPack || bundledClaudeRoot || null; - let expectedProviderPackManifest: RemoteProviderPackManifest | null = null; const useBundledRemoteImageAssets = (requiresRemoteProviderPack && !configuredProviderPackRoot && (input.execution.provider.kind === "opencode" || remoteClaude)) || @@ -12031,24 +12119,6 @@ async function createRunnerdBackendWithinSessionClaim( requiresRemoteProviderPack, configuredProviderPackRoot, provider: input.execution.provider, }); - if (useBundledRemoteImageAssets) { - expectedProviderPackManifest = bundledClaudeManifest && (remoteClaudeTarget === "linux-x64" || remoteClaudeTarget === "linux-arm64") - ? readBundledRemoteProviderPackManifest(bundledClaudeManifest, remoteClaudeTarget) - : readBundledRemoteProviderPackManifest(); - } else if (requiresRemoteProviderPack) { - if ( - !configuredProviderPackRoot || - !existsSync(configuredProviderPackRoot) || - !lstatSync(configuredProviderPackRoot).isDirectory() - ) { - throw new Error( - "runner_remote_provider_artifact_incompatible: configure PAPERCLIP_RUNNER_REMOTE_PROVIDER_PACK_PATH with the build-owned provider pack", - ); - } - expectedProviderPackManifest = readRemoteProviderPackManifest( - configuredProviderPackRoot, - ); - } const stagedRemoteProviderPackRoot = remoteRuntimeRoot ? posix.join(remoteRuntimeRoot, "provider-pack") : null; @@ -12056,6 +12126,25 @@ async function createRunnerdBackendWithinSessionClaim( const remoteBinary = remoteRuntimeRoot ? posix.join(remoteRuntimeRoot, "bin", "paperclip-runnerd") : null; + let currentProviderPackManifest: RemoteProviderPackManifest | undefined; + const resolveCurrentProviderPackManifest = () => { + if (currentProviderPackManifest) return currentProviderPackManifest; + currentProviderPackManifest = (() => { + if (useBundledRemoteImageAssets) { + return bundledClaudeManifest && (remoteClaudeTarget === "linux-x64" || remoteClaudeTarget === "linux-arm64") + ? readBundledRemoteProviderPackManifest(bundledClaudeManifest, remoteClaudeTarget) + : readBundledRemoteProviderPackManifest(); + } + if (!configuredProviderPackRoot || !existsSync(configuredProviderPackRoot) + || !lstatSync(configuredProviderPackRoot).isDirectory()) { + throw new Error("runner_remote_provider_artifact_incompatible: configure PAPERCLIP_RUNNER_REMOTE_PROVIDER_PACK_PATH with the build-owned provider pack"); + } + return readRemoteProviderPackManifest(configuredProviderPackRoot); + })(); + return currentProviderPackManifest; + }; + const reattachProviderPack = input.restartRecovery?.kind === "reattach_remote_runner" || input.remoteWarmRecovery !== undefined; + if (requiresRemoteProviderPack && !reattachProviderPack) resolveCurrentProviderPackManifest(); // The transport hashes runnerBinary on the controller before an external // launcher starts runnerd. Keep that artifact identity in the controller's // filesystem; the remote launcher separately owns the sandbox command path. @@ -12067,6 +12156,19 @@ async function createRunnerdBackendWithinSessionClaim( : remoteTarget ? remoteClaudeRunnerBinary || input.runnerRemoteBinaryPath?.trim() || remotePiCompanion?.runnerBinary || (useBundledRemoteImageAssets ? bundledRemoteRunnerBinary() : resolvePaperclipRunnerBinary()) : resolvePaperclipRunnerBinary(); + const selectProviderPackAuthority = (reattach: boolean) => requiresRemoteProviderPack ? resolveRemoteProviderPackAuthority({ + stateDirectory: root, + binding: { companyId: input.execution.binding.companyId, agentId: input.execution.binding.agentId, + workspaceId: providerPackWorkspaceId(input.execution.binding.executionWorkspaceId, input.execution.binding.runId, effectiveEnvironmentLeaseId), sessionId: nativeSessionKey(input.execution), + runnerInstanceId: effectiveRunnerInstanceId, environmentLeaseId: effectiveEnvironmentLeaseId, + remoteCwd: remoteTarget!.remoteCwd, providerDigest: nativeSha256(input.execution.provider) }, + reattach, + runnerArtifact: () => readRunnerdArtifactBinding(controllerRunnerBinary), + current: resolveCurrentProviderPackManifest, + }) : null; + let providerPackAuthority = selectProviderPackAuthority(reattachProviderPack); + let expectedProviderPackManifest = providerPackAuthority?.manifest ?? null; + let providerPackVerified = false; const explicitRemoteCodex = input.runnerRemoteCodexPath?.trim() || null; const remoteCodexNpmSpec = input.runnerRemoteCodexNpmSpec?.trim() || null; if (explicitRemoteCodex && remoteCodexNpmSpec) { @@ -12159,62 +12261,9 @@ async function createRunnerdBackendWithinSessionClaim( const verifyRemoteProviderPack = async (packRoot: string) => { if (!remoteTarget || !remoteCommandRunner || !expectedProviderPackManifest) return; - const expected = Buffer.from( - canonicalJson(expectedProviderPackManifest), - "utf8", - ).toString("base64"); - const providerNodeCommand = posix.join( - packRoot, - expectedProviderPackManifest.payload.artifacts.nodeCommand.path, - ); - const verifyScript = [ - "const fs=require('node:fs')", - "const crypto=require('node:crypto')", - "const path=require('node:path')", - "const root=process.argv[1]", - "const expected=Buffer.from(process.argv[2],'base64').toString('utf8')", - "const actual=fs.readFileSync(path.join(root,'provider-pack.json'),'utf8').trim()", - "const canonical=(v)=>Array.isArray(v)?'['+v.map(canonical).join(',')+']':v&&typeof v==='object'?'{'+Object.keys(v).sort().map(k=>JSON.stringify(k)+':'+canonical(v[k])).join(',')+'}':JSON.stringify(v)", - "const manifest=JSON.parse(actual)", - "if(canonical(manifest)!==expected)throw new Error('manifest mismatch')", - "const hash=(p)=>'sha256:'+crypto.createHash('sha256').update(fs.readFileSync(path.join(root,p))).digest('hex')", - "const tree=(treeRoot)=>{const digest=crypto.createHash('sha256');const visit=(directory,prefix='')=>{for(const entry of fs.readdirSync(directory,{withFileTypes:true}).sort((a,b)=>a.name.localeCompare(b.name))){const relative=prefix?prefix+'/'+entry.name:entry.name;const absolute=path.join(directory,entry.name);if(entry.isDirectory()){digest.update('directory\\0'+relative+'\\n');visit(absolute,relative)}else if(entry.isFile()){digest.update('file\\0'+relative+'\\0'+'sha256:'+crypto.createHash('sha256').update(fs.readFileSync(absolute)).digest('hex')+'\\n')}else if(entry.isSymbolicLink()){digest.update('symlink\\0'+relative+'\\0'+fs.readlinkSync(absolute)+'\\n')}else throw new Error('unsupported dist entry '+relative)}};visit(treeRoot);return 'sha256:'+digest.digest('hex')}", - "for(const name of ['nodeCommand','productionLock','opencodeCommand','opencodeExecutable','opencodeProxy','acpxSidecar','grokLauncher']){const artifact=manifest.payload.artifacts[name];if(hash(artifact.path)!==artifact.sha256)throw new Error(name+' digest mismatch')}", - "if(tree(path.join(root,'dist'))!==manifest.payload.distDigest)throw new Error('dist tree digest mismatch')", - "for(const candidate of Object.values({...manifest.payload.providers,...manifest.payload.candidateProviders})){if(tree(path.join(root,candidate.path))!==candidate.sha256)throw new Error('candidate asset tree digest mismatch')}", - "const version=process.versions.node.split('.').map(Number)", - "const minimum=manifest.payload.pins.nodeMinimum.split('.').map(Number)", - "if(version[0]JSON.parse(fs.readFileSync(path.join(root,'node_modules',...pkg.split('/'),'package.json'),'utf8')).version", - "const expectedPackages={acpx:manifest.payload.pins.acpx,'@agentclientprotocol/claude-agent-acp':manifest.payload.pins.claudeAcp,'@agentclientprotocol/codex-acp':manifest.payload.pins.codexAcp,'opencode-ai':manifest.payload.pins.opencode}", - "for(const [pkg,version] of Object.entries(expectedPackages))if(packageVersion(pkg)!==version)throw new Error(pkg+' version mismatch')", - ].join(";"); - await runRemoteProviderVerification(() => remoteCommandRunner.execute({ - command: providerNodeCommand, - args: ["-e", verifyScript, packRoot, expected], - cwd: remoteTarget.remoteCwd, - bypassSession: true, - timeoutMs: 30_000, - }), "provider pack verification"); - const opencodeCommand = posix.join( - packRoot, - expectedProviderPackManifest.payload.artifacts.opencodeCommand.path, - ); - const opencodeVersion = await runRemoteProviderVerification(() => remoteCommandRunner.execute({ - command: opencodeCommand, - args: ["--version"], - cwd: remoteTarget.remoteCwd, - bypassSession: true, - timeoutMs: 30_000, - }), "OpenCode version verification"); - if ( - opencodeVersion.stdout.trim() !== REMOTE_PROVIDER_PACK_PINS.opencode - ) { - throw new Error( - "runner_remote_provider_artifact_incompatible: OpenCode version mismatch", - ); - } + await verifyRemoteProviderPackIdentity({ runner: remoteCommandRunner, + cwd: remoteTarget.remoteCwd, packRoot, manifest: expectedProviderPackManifest }); + providerPackVerified = true; }; const discoverPreinstalledProviderPack = async () => { @@ -12333,7 +12382,7 @@ async function createRunnerdBackendWithinSessionClaim( ); } const archive = await measureNativeRunnerSpan(input.trace, "provider_pack.archive.verify", () => - readProviderPackArchive(configuredProviderPackRoot, expectedProviderPackManifest.digest), + readProviderPackArchive(configuredProviderPackRoot, expectedProviderPackManifest!.digest), ); const escapedPackRoot = stagedRemoteProviderPackRoot.replaceAll( "'", @@ -13230,9 +13279,11 @@ async function createRunnerdBackendWithinSessionClaim( if (hasRetainedWarmTransitionEvidence(root)) { throw new Error("native_runner_warm_transition_recovery_unproven"); } - const archiveToken = `${Date.now()}-${randomUUID()}`; + const archivedAt = Date.now(); + const archiveToken = `${archivedAt}-${randomUUID()}`; const archiveRoot = resolve(root, "continuity-breaks", archiveToken); mkdirSync(archiveRoot, { recursive: true, mode: 0o700 }); + noteNativeRunnerEvidence(archivedAt); for (const name of [ "control-plane", "runner", @@ -13243,6 +13294,7 @@ async function createRunnerdBackendWithinSessionClaim( // makes the fresh replacement look like ambiguous lost harness state. // Keep their evidence inside the same continuity-break archive. "failover-backups", + "provider-pack-authority.json", ]) { const source = resolve(root, name); if (existsSync(source)) renameSync(source, resolve(archiveRoot, name)); @@ -13265,6 +13317,9 @@ async function createRunnerdBackendWithinSessionClaim( } } remotePrepared = false; + providerPackAuthority = selectProviderPackAuthority(false); + expectedProviderPackManifest = providerPackAuthority?.manifest ?? null; + providerPackVerified = false; }; const localRecoveryProcess = input.restartRecovery?.kind === "reattach_existing_runner" @@ -13455,11 +13510,16 @@ async function createRunnerdBackendWithinSessionClaim( : {}), stateDirectory: root, runnerBinary: controllerRunnerBinary, + ...(providerPackAuthority ? { runnerArtifact: providerPackAuthority.runnerArtifact } : {}), codexCommand: remoteCodexBinary ?? undefined, sourceCodexHome: remoteTarget ? resolveSourceCodexHome(resolveNativeProviderEnvironment(input.execution.provider, runnerEnvironment)) : undefined, runnerReconnectGraceMs: remoteTarget ? 120_000 : undefined, + // A controller's bounded reconnect attempt is not the remote owner's + // lifetime. Retain it through restarts until its authenticated lease + // expires or explicit Stop/suspension proves that execution ended. + runnerProcessReconnectGraceMs: remoteTarget ? null : undefined, environment: effectiveRunnerEnvironment, observability: input.observability, onDiagnostic: (message) => { @@ -13660,12 +13720,22 @@ async function createRunnerdBackendWithinSessionClaim( priorAuthorityEpoch.revoke(); } sessionToolAuthorityEpochs.set(sessionScopeId, authorityEpoch); + const recordAdmittedProviderPack = async () => { + if (providerPackVerified && providerPackAuthority) { + await recordAdmittedRemoteProviderPack(providerPackAuthority, message => input.onLog?.("stderr", message)); + } + }; return { bindManagedSession: wrapManagedSession, descriptor: () => backend.descriptor(), - openSession: async (sessionInput) => wrapManagedSession(await backend.openSession(sessionInput)), + openSession: async (sessionInput) => { + const session = await backend.openSession(sessionInput); + await recordAdmittedProviderPack(); + return wrapManagedSession(session); + }, recoverSession: async (snapshot, options) => { const result = backend.recoverSession ? await backend.recoverSession(snapshot, options) : { recovered: false, reason: "driver does not support recovery" }; + if (result.session) await recordAdmittedProviderPack(); return result.session ? { ...result, session: wrapManagedSession(result.session) } : result; }, openReplacementSession: async (sessionInput) => { @@ -13675,7 +13745,9 @@ async function createRunnerdBackendWithinSessionClaim( archiveContinuityState, { parentName: "native.session.execute" }, ); - return wrapManagedSession(await backend.openSession(sessionInput)); + const session = await backend.openSession(sessionInput); + await recordAdmittedProviderPack(); + return wrapManagedSession(session); }, } satisfies NativeSessionBackend & { bindManagedSession(session: NativeSession): NativeSession }; } diff --git a/server/src/services/native-runtime/native-terminal-status-projection.test.ts b/server/src/services/native-runtime/native-terminal-status-projection.test.ts new file mode 100644 index 00000000000..ec23c45f1b6 --- /dev/null +++ b/server/src/services/native-runtime/native-terminal-status-projection.test.ts @@ -0,0 +1,132 @@ +import { randomUUID } from "node:crypto"; +import { eq } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; +import { agents, companies, completionContracts, createDb, heartbeatRuns, issues, + nativeRunFinalizations, nativeRunResults } from "@paperclipai/db"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; +import { deliverExecutionStatuses } from "../execution-status-delivery.js"; +import { buildIssueMonitorTriggeredPatch, normalizeIssueExecutionPolicy } from "../issue-execution-policy.js"; +import { PaperclipRunnerToolAuthority } from "./paperclip-runner-tool-authority.js"; +import { finalizeNativeRun } from "./native-run-finalizer.js"; + +const support = await getEmbeddedPostgresTestSupport(); +describe.skipIf(!support.supported)("native terminal status projection", () => { + let temporary: Awaited>; + let db: ReturnType; + beforeAll(async () => { + temporary = await startEmbeddedPostgresTestDatabase("native-terminal-status-"); + db = createDb(temporary.connectionString); + }); + afterAll(async () => { await temporary?.cleanup(); }); + + async function fixture() { + await deliverExecutionStatuses(db, { publish: vi.fn() }); + const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), runId = randomUUID(); + const contractId = randomUUID(), resultId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Monitor status", issuePrefix: `S${companyId.slice(0, 7)}` }); + await db.insert(agents).values({ id: agentId, companyId, name: "Worker", adapterType: "paperclip_runner", status: "active" }); + await db.insert(issues).values({ id: issueId, companyId, title: "Saved check", status: "in_progress", assigneeAgentId: agentId }); + await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, nativeIssueId: issueId, + runtimeMode: "native", status: "running", contextSnapshot: { issueId } }); + await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId)); + const authority = new PaperclipRunnerToolAuthority(db, { companyId, agentId, issueId, runId }); + await authority.execute({ tool: "set_task_monitor", callId: randomUUID(), arguments: { + idempotencyKey: "saved-monitor", monitor: { nextCheckAt: new Date(Date.now() + 60_000).toISOString(), notes: "Check the saved result." }, + } }); + await db.insert(completionContracts).values({ id: contractId, companyId, issueId, revision: 1, + schemaVersion: "paperclip.completion-contract.v1", policyVersion: "test", risk: "standard", + completionAuthority: "server_arbiter", incompleteCriteriaPolicy: "preserve_non_terminal", + contractJson: { revision: "test", criteria: [] }, canonicalSha256: randomUUID(), + createdByActorType: "system", createdByActorId: "test" }); + const accepted = { result: { schema: "paperclip.run_result.v1", reportedWorkDisposition: "yielded", + summary: "Wait for the saved check.", completionClaim: { contractRevision: "test", objectiveSatisfied: false, + criteria: [], remainingWork: [{ description: "Check again", blocksCompletion: true }] }, + continuation: { kind: "monitor", summary: "Wait for the check", idempotencyKey: "provider-selected" }, + evidence: [], verification: [], attentionRequests: [], artifacts: [] }, + terminal: { schema: "paperclip.prp.terminal.v1", runTerminalState: "succeeded", turnTerminalState: "completed", reportedWorkDisposition: "yielded" } }; + const finishedAt = new Date("2020-01-01T00:00:00Z"); + await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt, completionContractId: contractId }).where(eq(heartbeatRuns.id, runId)); + await db.insert(nativeRunResults).values({ id: resultId, companyId, issueId, runId, completionContractId: contractId, + serverFingerprint: randomUUID(), canonicalSha256: randomUUID(), schemaStatus: "accepted", resultJson: accepted }); + await db.insert(nativeRunFinalizations).values({ companyId, issueId, runId, phase: "retryable_failure", resultId }); + const [task] = await db.select().from(issues).where(eq(issues.id, issueId)); + await db.update(issues).set({ ...buildIssueMonitorTriggeredPatch({ issue: task!, + policy: normalizeIssueExecutionPolicy(task!.executionPolicy), triggeredAt: new Date() }), executionRunId: null, checkoutRunId: null, + }).where(eq(issues.id, issueId)); + const finalize = () => finalizeNativeRun({ db, runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true }); + const read = async () => (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)))[0]!; + return { runId, companyId, issueId, finishedAt, resultId, accepted, finalize, read }; + } + + it("preserves the original completion time while retrying an accepted monitor result", async () => { + const f = await fixture(); + await f.finalize(); + expect((await f.read()).finishedAt).toEqual(f.finishedAt); + await f.finalize(); + expect((await f.read()).finishedAt).toEqual(f.finishedAt); + const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)); + expect(coordinator).toMatchObject({ phase: "retryable_failure", failureCode: "monitor_wait_authority_lost", resultId: f.resultId }); + expect(coordinator.nextAttemptAt!.getTime()).toBeGreaterThan(Date.now()); + expect((await db.select().from(nativeRunResults).where(eq(nativeRunResults.id, f.resultId)))[0].resultJson).toEqual(f.accepted); + }); + + it("retains an unacknowledged status delivery across unchanged finalization retries", async () => { + const f = await fixture(); + await f.finalize(); + const before = await f.read(); + expect(before.executionStatusDeliveryId).toBeTruthy(); + await f.finalize(); + expect((await f.read()).executionStatusDeliveryId).toBe(before.executionStatusDeliveryId); + const publish = vi.fn(); + await expect(deliverExecutionStatuses(db, { publish, failpoint: () => { throw new Error("after publish"); } })).rejects.toThrow("after publish"); + await f.finalize(); + expect((await f.read()).executionStatusDeliveryId).toBe(before.executionStatusDeliveryId); + await deliverExecutionStatuses(db, { publish }); + const deliveries = publish.mock.calls.filter(([event]) => event.payload.runId === f.runId); + expect(deliveries).toHaveLength(2); + expect(deliveries[0][0].payload.deliveryId).toBe(deliveries[1][0].payload.deliveryId); + }); + + it("preserves a pending status notification created by another producer", async () => { + const f = await fixture(); + await f.finalize(); + const newer = randomUUID(); + await db.update(heartbeatRuns).set({ executionStatusDeliveryId: newer }).where(eq(heartbeatRuns.id, f.runId)); + await f.finalize(); + expect((await f.read()).executionStatusDeliveryId).toBe(newer); + }); + + it.each(["metadata", "status", "missing_completion_time"])("broadcasts a changed terminal projection after acknowledgement (%s)", async change => { + const f = await fixture(); + await f.finalize(); + const publish = vi.fn(); + await deliverExecutionStatuses(db, { publish }); + if (change === "metadata") { + await finalizeNativeRun({ db, runId: f.runId, workspaceFinalizeStatus: "failed", projectRunStatus: true }); + expect((await f.read()).resultJson?.workspaceFinalizeStatus).toBe("failed"); + } else { + await db.update(heartbeatRuns).set(change === "status" ? { status: "failed" } : { finishedAt: null }) + .where(eq(heartbeatRuns.id, f.runId)); + await f.finalize(); + expect(await f.read()).toMatchObject({ status: "succeeded", finishedAt: expect.any(Date) }); + } + await deliverExecutionStatuses(db, { publish }); + expect(publish.mock.calls.filter(([event]) => event.payload.runId === f.runId)).toHaveLength(2); + }); + + it("does not broadcast an unchanged terminal projection after acknowledgement or a new DB client", async () => { + const f = await fixture(); + await f.finalize(); + const publish = vi.fn(); + await deliverExecutionStatuses(db, { publish }); + await f.finalize(); + await deliverExecutionStatuses(db, { publish }); + const reopened = createDb(temporary.connectionString); + try { + await finalizeNativeRun({ db: reopened, runId: f.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true }); + await deliverExecutionStatuses(reopened, { publish }); + } finally { await reopened.$client.end(); } + expect(publish.mock.calls.filter(([event]) => event.payload.runId === f.runId)).toHaveLength(1); + expect((await f.read()).executionStatusDeliveryId).toBeNull(); + }); +}); diff --git a/server/src/services/native-runtime/paperclip-control-plane-port.test.ts b/server/src/services/native-runtime/paperclip-control-plane-port.test.ts index 5321a764f8b..f8b9eb1d022 100644 --- a/server/src/services/native-runtime/paperclip-control-plane-port.test.ts +++ b/server/src/services/native-runtime/paperclip-control-plane-port.test.ts @@ -300,6 +300,38 @@ describe("PaperclipControlPlanePort conformance", () => { } }); + it("permits only read-only admission while an existing startup drains", async () => { + const identity = { ...CONTROL_PLANE_CONFORMANCE_OPEN.identity, runId: randomUUID(), sessionId: randomUUID() }; + const runnerId = randomUUID(); + await db.insert(heartbeatRuns).values({ + id: identity.runId, companyId: identity.companyId, agentId: identity.agentId, + status: "running", runtimeMode: "native", nativeIssueId: identity.issueId, + nativeSessionId: identity.sessionId, runnerInstanceId: runnerId, + completionContractId: contractId, completionContractSha256: contractSha, + }); + let awaitingStartup = true; + const port = new PaperclipControlPlanePort(db, { + ...identity, completionContractId: contractId, completionContractSha256: contractSha, + sourceInstanceId: runnerId, controlPlaneSourceInstanceId: `draining-${identity.runId}`, + }, { + assertControllerActive: () => { throw new Error("controller_detached"); }, + assertRunAdmissionActive: () => { if (!awaitingStartup) throw new Error("controller_detached"); }, + }); + const open = { identity, backendKind: "mock" as const, sourceInstanceId: runnerId }; + await expect(port.openRun({ ...open, sourceInstanceId: randomUUID() })).rejects.toThrow("native_open_run_binding_mismatch"); + await db.update(heartbeatRuns).set({ runnerInstanceId: randomUUID() }).where(eq(heartbeatRuns.id, identity.runId)); + await expect(port.openRun(open)).rejects.toThrow("native_open_run_not_authorized"); + await db.update(heartbeatRuns).set({ runnerInstanceId: runnerId }).where(eq(heartbeatRuns.id, identity.runId)); + await expect(port.openRun(open)).resolves.toBeUndefined(); + await expect(port.checkpointSession({ backendKind: "mock", sessionId: identity.sessionId, identity })).rejects.toThrow("controller_detached"); + await expect(port.completeRun({ result: CONTROL_PLANE_CONFORMANCE_RESULT, terminal: CONTROL_PLANE_CONFORMANCE_TERMINAL })).rejects.toThrow("controller_detached"); + awaitingStartup = false; + await expect(port.openRun(open)).rejects.toThrow("controller_detached"); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, identity.runId)); + expect(run.runnerProfileJson?.sessionCheckpoint).toBeUndefined(); + expect(await db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, identity.runId))).toEqual([]); + }); + it.each([ ["checkpoint", "detach"], ["result", "detach"], ["event", "detach"], ["checkpoint", "abort"], ["result", "abort"], ["event", "abort"], diff --git a/server/src/services/native-runtime/paperclip-control-plane-port.ts b/server/src/services/native-runtime/paperclip-control-plane-port.ts index acb02e19b63..d6c6413d060 100644 --- a/server/src/services/native-runtime/paperclip-control-plane-port.ts +++ b/server/src/services/native-runtime/paperclip-control-plane-port.ts @@ -83,6 +83,7 @@ export class PaperclipControlPlanePort implements ControlPlanePort { readonly #binding: PaperclipControlPlaneBinding; #sessionId: string | null = null; readonly #assertControllerActive?: () => void; + readonly #assertRunAdmissionActive?: () => void; readonly #onCommittedEvent?: (event: PrpEvent) => Promise; readonly #onDuplicateEvent?: (event: PrpEvent) => Promise; @@ -94,12 +95,15 @@ export class PaperclipControlPlanePort implements ControlPlanePort { privateKeyPem?: string; /** Synchronous revocation fence for a controller handing off on restart. */ assertControllerActive?: () => void; + /** Read-only admission may finish an existing startup solely for restart detachment. */ + assertRunAdmissionActive?: () => void; onCommittedEvent?: (event: PrpEvent) => Promise; onDuplicateEvent?: (event: PrpEvent) => Promise; } = {}, ) { this.#db = db; this.#assertControllerActive = options.assertControllerActive; + this.#assertRunAdmissionActive = options.assertRunAdmissionActive ?? options.assertControllerActive; this.#identityRedactor = createAgentIdentityRedactor(options.privateKeyPem); this.#binding = structuredClone(binding); this.#onCommittedEvent = options.onCommittedEvent; @@ -123,7 +127,7 @@ export class PaperclipControlPlanePort implements ControlPlanePort { } async openRun(input: OpenControlPlaneRunInput): Promise { - this.#assertControllerActive?.(); + this.#assertRunAdmissionActive?.(); const identity = input.identity; if ( identity.companyId !== this.#binding.companyId @@ -146,7 +150,7 @@ export class PaperclipControlPlanePort implements ControlPlanePort { if (!run || !this.#matchesPersistedBinding(run)) { throw new Error("native_open_run_not_authorized"); } - this.#assertControllerActive?.(); + this.#assertRunAdmissionActive?.(); this.#sessionId = identity.sessionId; } diff --git a/server/src/services/native-runtime/paperclip-runner-tool-authority.test.ts b/server/src/services/native-runtime/paperclip-runner-tool-authority.test.ts index 04a9b79a2cc..74524c733f3 100644 --- a/server/src/services/native-runtime/paperclip-runner-tool-authority.test.ts +++ b/server/src/services/native-runtime/paperclip-runner-tool-authority.test.ts @@ -1,4 +1,4 @@ -import { connectorToolDefinitionsForSkills } from "../connector-runtime.js"; +import * as connectorRuntime from "../connector-runtime.js"; import * as cloudIdentity from "../cloud-runtime-identity.js"; import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; import { randomUUID } from "node:crypto"; @@ -20,6 +20,7 @@ import { issues, } from "@paperclipai/db"; import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; +import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture"; import { initializeRunIdentity, reserveSteeredIdentity, reconcileSteeredIdentity } from "../run-identity.js"; import { documentService } from "../documents.js"; import { issueService } from "../issues.js"; @@ -100,7 +101,7 @@ describe("PaperclipRunnerToolAuthority", () => { }); it("registers pinned connector definitions without discovery and rejects revoked assignments at use", async () => { - const connectorAssignments = connectorToolDefinitionsForSkills(new Set([ + const connectorAssignments = connectorRuntime.connectorToolDefinitionsForSkills(new Set([ "paperclipai/paperclip/slack", "paperclipai/paperclip/browser-use-cloud", "custom/agentmail", ])); expect(connectorAssignments).toHaveLength(1); @@ -198,6 +199,126 @@ describe("PaperclipRunnerToolAuthority", () => { ).rejects.toThrow("paperclip_runner_tool_not_advertised"); }); + it("reads current tool authority without transferring saved profiles or output", async () => { + const binding = { companyId, agentId, issueId, runId }; + const authority = new PaperclipRunnerToolAuthority(db, binding); + const contextCall = { tool: "get_task_context", callId: "context-read-width", arguments: {} }; + const apiCall = { tool: "search_api", callId: "api-read-width", arguments: { query: "read task" } }; + const expectedContext = await authority.execute(contextCall); + const expectedApi = await authority.execute(apiCall); + const [original] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); + const [actor] = await db.select({ status: agents.status }).from(agents).where(eq(agents.id, agentId)); + const payload = "saved-profile-and-output🙂".repeat(32_768); + await db.update(heartbeatRuns).set({ + runnerProfileJson: { ...original.runnerProfileJson, retainedPayload: payload }, + stdoutExcerpt: payload, + stderrExcerpt: payload, + }).where(eq(heartbeatRuns.id, runId)); + const [expanded] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); + const observed = createDb(temporary!.connectionString), events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(observed, { heartbeatRuns, issues, agents, companies }, capture); + try { + const reader = new PaperclipRunnerToolAuthority(observed, binding); + const context = await reader.execute(contextCall), api = await reader.execute(apiCall); + expect(context).toEqual(expectedContext); + expect(api).toEqual(expectedApi); + expect(JSON.stringify([context, api])).not.toContain("saved-profile-and-output"); + capture.flush(); + const reads = events.filter(event => event.table === "heartbeat_runs" && event.operation === "select"); + expect(reads.length).toBeGreaterThan(0); + expect(reads.reduce((bytes, event) => bytes + Number(event.responseValueBytes), 0)).toBeLessThan(16 * 1024); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)))[0]).toEqual(expanded); + await db.update(agents).set({ status: "paused" }).where(eq(agents.id, agentId)); + await expect(reader.execute(contextCall)).rejects.toThrow("paperclip_runner_tool_binding_not_authorized"); + await expect(reader.execute(apiCall)).rejects.toThrow("paperclip_runner_tool_binding_not_authorized"); + } finally { + capture.flush(); + await observed.$client.end(); + await db.update(agents).set({ status: actor.status }).where(eq(agents.id, agentId)); + await db.update(heartbeatRuns).set({ + runnerProfileJson: original.runnerProfileJson, + stdoutExcerpt: original.stdoutExcerpt, + stderrExcerpt: original.stderrExcerpt, + }).where(eq(heartbeatRuns.id, runId)); + } + }); + + it("reads tool authority without transferring unused launch renderings or wake payloads", async () => { + const binding = { companyId, agentId, issueId, runId }; + const authority = new PaperclipRunnerToolAuthority(db, binding); + const contextCall = { tool: "get_task_context", callId: "launch-context-read", arguments: {} }; + const apiCall = { tool: "search_api", callId: "launch-api-read", arguments: { query: "read task" } }; + const expectedContext = await authority.execute(contextCall), expectedApi = await authority.execute(apiCall); + const [original] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); + const payload = "retained-launch-rendering🙂".repeat(8192); + const context = { ...original.contextSnapshot, executionContinuation: { savedInput: payload }, paperclipTaskMarkdown: payload, + paperclipTaskMarkdownCompact: payload, paperclipTaskMarkdownAssignment: payload, paperclipTaskMarkdownAssignmentCompact: payload, + paperclipWorkspace: { savedInput: payload }, paperclipEnvironment: { savedInput: payload }, + paperclipContinuationSummary: payload, paperclipWakeComment: payload, paperclipWake: { retainedPayload: payload } }; + await db.update(heartbeatRuns).set({ contextSnapshot: context }).where(eq(heartbeatRuns.id, runId)); + const [expanded] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); + const observed = createDb(temporary!.connectionString), events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(observed, { heartbeatRuns, issues, agents, companies }, capture); + try { + const reader = new PaperclipRunnerToolAuthority(observed, binding); + const actualContext = await reader.execute(contextCall), actualApi = await reader.execute(apiCall); + capture.flush(); + const reads = events.filter(event => event.table === "heartbeat_runs" && event.operation === "select"); + expect(reads.length).toBeGreaterThan(0); + expect(reads.reduce((bytes, event) => bytes + Number(event.responseValueBytes), 0)).toBeLessThan(16 * 1024); + expect(actualContext).toEqual(expectedContext); + expect(actualApi).toEqual(expectedApi); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)))[0]).toEqual(expanded); + await db.update(heartbeatRuns).set({ status: "cancelled" }).where(eq(heartbeatRuns.id, runId)); + await expect(reader.execute(contextCall)).rejects.toThrow("paperclip_runner_tool_binding_not_authorized"); + await expect(reader.execute(apiCall)).rejects.toThrow("paperclip_runner_tool_binding_not_authorized"); + } finally { + capture.flush(); await observed.$client.end(); + await db.update(heartbeatRuns).set({ contextSnapshot: original.contextSnapshot, status: original.status }).where(eq(heartbeatRuns.id, runId)); + } + }); + + it("checks the current full wake before mail actions while ordinary tools omit its payload", async () => { + const connectorAssignments = connectorRuntime.connectorToolDefinitionsForSkills(new Set(["paperclipai/paperclip/agentmail"])); + const name = connectorAssignments.flatMap(assignment => assignment.tools).find(tool => tool.name.startsWith("agentmail_"))!.name; + const authority = new PaperclipRunnerToolAuthority(db, { companyId, agentId, issueId, runId, connectorAssignments }); + const [original] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); + const dispatch = vi.spyOn(connectorRuntime, "executeConnectorTool").mockResolvedValue({ outcome: "mail-dispatched" }); + const wake = { + reason: "External chat message received", externalChatProvider: "github", checkedOutByHarness: true, + issue: { id: issueId, identifier: "RNT-1", title: "Conversation", description: "Task", descriptionTruncated: false, + status: "in_progress", workMode: "standard" }, + commentWindow: { requestedCount: 1, includedCount: 1, missingCount: 0 }, commentIds: ["wake-comment"], + latestCommentId: "wake-comment", comments: [{ id: "wake-comment", issueId, body: "Reply here", bodyTruncated: false, authorType: "user" }], + fallbackFetchNeeded: false, + }; + const call = { tool: name, callId: "current-mail-wake", arguments: {} }; + try { + await db.update(heartbeatRuns).set({ contextSnapshot: { issueId, paperclipWake: wake } }).where(eq(heartbeatRuns.id, runId)); + await expect(authority.execute(call)).rejects.toThrow("Restricted chat runs cannot use email actions"); + expect(dispatch).not.toHaveBeenCalled(); + await expect(authority.execute({ tool: "get_task_context", callId: "chat-task-context", arguments: {} })) + .resolves.toMatchObject({ activeTask: { id: issueId } }); + await db.update(heartbeatRuns).set({ contextSnapshot: { issueId } }).where(eq(heartbeatRuns.id, runId)); + await expect(authority.execute(call)).resolves.toEqual({ outcome: "mail-dispatched" }); + expect(dispatch).toHaveBeenCalledTimes(1); + await db.update(heartbeatRuns).set({ contextSnapshot: { issueId, paperclipWake: { ...wake, checkedOutByHarness: false, + externalChatExecutionBound: true, issue: { ...wake.issue, status: "in_review" } } } }).where(eq(heartbeatRuns.id, runId)); + await expect(authority.execute(call)).rejects.toThrow("Restricted chat runs cannot use email actions"); + expect(dispatch).toHaveBeenCalledTimes(1); + for (const context of [{ issueId, source: "chat:github" }, { issueId, paperclipExternalChatQuestionResponse: true }]) { + await db.update(heartbeatRuns).set({ contextSnapshot: context }).where(eq(heartbeatRuns.id, runId)); + await expect(authority.execute(call)).rejects.toThrow("Restricted chat runs cannot use email actions"); + expect(dispatch).toHaveBeenCalledTimes(1); + } + } finally { + dispatch.mockRestore(); + await db.update(heartbeatRuns).set({ contextSnapshot: original.contextSnapshot }).where(eq(heartbeatRuns.id, runId)); + } + }); + it("preserves platform receipt provenance in both mixed task history readers", async () => { const at = new Date("2026-10-10T12:00:00Z"); const commentIds = Array.from({ length: 4 }, () => randomUUID()); diff --git a/server/src/services/native-runtime/paperclip-runner-tool-authority.ts b/server/src/services/native-runtime/paperclip-runner-tool-authority.ts index 56bccadf0b6..31f65f12fcd 100644 --- a/server/src/services/native-runtime/paperclip-runner-tool-authority.ts +++ b/server/src/services/native-runtime/paperclip-runner-tool-authority.ts @@ -449,7 +449,7 @@ export class PaperclipRunnerToolAuthority { } if (isConnectorTool(call.tool)) { if (!(this.binding.connectorAssignments ?? []).some((assignment) => assignment.tools.some((tool) => tool.name === call.tool))) throw forbidden("Connector tool is not available to this run"); - const { run } = await this.#boundContext(); + const { run } = await this.#boundContext({ includeChatWake: call.tool.startsWith("agentmail_") }); const snapshot = record(run.contextSnapshot); if (call.tool.startsWith("agentmail_") && (isPaperclipExternalChatContractTurn(snapshot.paperclipWake) || String(snapshot.source ?? "").startsWith("chat:") || snapshot.paperclipExternalChatQuestionResponse)) throw forbidden("Restricted chat runs cannot use email actions"); return executeConnectorTool(this.db, this.binding, call.tool, call.arguments); @@ -1225,9 +1225,24 @@ export class PaperclipRunnerToolAuthority { ...(this.binding.museRuntime ? { authorizingUserId: this.binding.externalAuthorizingUserId } : {}) }; } - async #boundContext() { + async #boundContext(options: { includeChatWake?: boolean } = {}) { if (this.binding.museRuntime && !this.binding.externalAuthorizingUserId) throw forbidden("Muse authorizer is unavailable"); - const [row] = await this.db.select({ issue: issues, actor: agents, run: heartbeatRuns }) + const [row] = await this.db.select({ issue: issues, actor: agents, run: { + id: heartbeatRuns.id, + companyId: heartbeatRuns.companyId, + agentId: heartbeatRuns.agentId, + runtimeMode: heartbeatRuns.runtimeMode, + status: heartbeatRuns.status, + invocationSource: heartbeatRuns.invocationSource, + responsibleUserId: heartbeatRuns.responsibleUserId, + contextSnapshot: sql | null>`case when jsonb_typeof(${heartbeatRuns.contextSnapshot}) = 'object' + then ${heartbeatRuns.contextSnapshot} - array['executionContinuation', 'paperclipTaskMarkdown', + 'paperclipTaskMarkdownCompact', 'paperclipTaskMarkdownAssignment', 'paperclipTaskMarkdownAssignmentCompact', + 'paperclipWorkspace', 'paperclipEnvironment', 'paperclipContinuationSummary', 'paperclipWakeComment'] + ${options.includeChatWake ? sql`` : sql`- 'paperclipWake'`} + else ${heartbeatRuns.contextSnapshot} end`, + resultJson: heartbeatRuns.resultJson, + } }) .from(heartbeatRuns) .innerJoin(issues, eq(issues.id, this.binding.issueId)) .innerJoin(agents, eq(agents.id, this.binding.agentId)) @@ -2168,7 +2183,7 @@ export class PaperclipRunnerToolAuthority { }); } - async #sourcePlanApproval(context: { issue: typeof issues.$inferSelect; run: typeof heartbeatRuns.$inferSelect }) { + async #sourcePlanApproval(context: { issue: typeof issues.$inferSelect; run: Pick }) { const plan = await handoffPlanContext(this.db, context.issue, this.#privacyActor(context.run)); if (!plan || !this.binding.museRuntime) return plan; const document = await documentService(this.db).getIssueDocumentByKey(plan.sourceIssueId, "plan"); diff --git a/server/src/services/native-runtime/remote-provider-pack-authority.test.ts b/server/src/services/native-runtime/remote-provider-pack-authority.test.ts new file mode 100644 index 00000000000..f5f2039c4cf --- /dev/null +++ b/server/src/services/native-runtime/remote-provider-pack-authority.test.ts @@ -0,0 +1,121 @@ +import { createHash } from "node:crypto"; +import { mkdtempSync, mkdirSync, rmSync, readFileSync, writeFileSync, symlinkSync, statSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterEach, expect, it, vi } from "vitest"; +import { providerPackWorkspaceId, recordAdmittedRemoteProviderPack, resolveRemoteProviderPackAuthority } from "./remote-provider-pack-authority.js"; +import { canonicalProviderPackJson } from "./remote-provider-pack-verification.js"; + +const directories: string[] = []; +afterEach(() => { for (const dir of directories.splice(0)) rmSync(dir, { recursive: true, force: true }); }); +const binding = { companyId: "company", agentId: "agent", workspaceId: "workspace", sessionId: "session", + runnerInstanceId: "runner", environmentLeaseId: "lease", remoteCwd: "/workspace", providerDigest: "provider" }; +const runnerArtifact = () => ({ version: "0.0.0", digest: "sha256:" + "c".repeat(64) }); +const manifest = (revision: string) => { + const payload = { runnerSourceRevision: revision.repeat(40), target: { platform: "linux", architecture: "x64" }, + pins: { opencode: "1.18.34" }, artifacts: { nodeCommand: { path: "node", sha256: "sha256:" + "1".repeat(64) }, + opencodeCommand: { path: "opencode", sha256: "sha256:" + "2".repeat(64) } } }; + return { schema: "paperclip-runner/remote-provider-pack/v1", payload, + digest: "sha256:" + createHash("sha256").update(canonicalProviderPackJson(payload)).digest("hex") }; +}; +it("reattaches with the verified launch authority after the controller release changes", () => { + const stateDirectory = mkdtempSync(join(tmpdir(), "provider-pack-authority-")); directories.push(stateDirectory); + const original = manifest("a"), next = manifest("b"); + const launched = resolveRemoteProviderPackAuthority({ stateDirectory, binding, reattach: false, runnerArtifact, current: () => original }); + launched.verified(); + const current = vi.fn(() => next); + const recovered = resolveRemoteProviderPackAuthority({ stateDirectory, binding, reattach: true, current, runnerArtifact }); + expect(recovered.manifest).toEqual(original); + expect(recovered.runnerArtifact).toEqual(runnerArtifact()); + expect(current).not.toHaveBeenCalled(); +}); + +function fixture() { + const stateDirectory = mkdtempSync(join(tmpdir(), "provider-pack-authority-")); directories.push(stateDirectory); + const current = vi.fn(() => manifest("a")); + const input = { stateDirectory, binding, reattach: false, current, runnerArtifact }; + return { input, path: join(stateDirectory, "provider-pack-authority.json") }; +} +it("uses the new release for a fresh launch without changing the old receipt until verification", () => { + const { input, path } = fixture(); + resolveRemoteProviderPackAuthority(input).verified(); + input.current.mockReturnValue(manifest("b")); + const fresh = resolveRemoteProviderPackAuthority(input); + expect(fresh.manifest).toEqual(manifest("b")); + expect(JSON.parse(readFileSync(path, "utf8")).manifest).toEqual(manifest("a")); + fresh.verified(); + expect(JSON.parse(readFileSync(path, "utf8")).manifest).toEqual(manifest("b")); + expect(statSync(path).mode & 0o777).toBe(0o600); +}); +it("keeps historical sessions without a receipt at the current release integrity gate", () => { + const { input } = fixture(); + const recovered = resolveRemoteProviderPackAuthority({ ...input, reattach: true }); + expect(recovered.manifest).toEqual(manifest("a")); + expect(input.current).toHaveBeenCalledOnce(); +}); +it.each(Object.keys(binding) as (keyof typeof binding)[])("rejects a saved authority for a different %s", field => { + const { input } = fixture(); + resolveRemoteProviderPackAuthority(input).verified(); + expect(() => resolveRemoteProviderPackAuthority({ ...input, reattach: true, + binding: { ...binding, [field]: "other" } })).toThrow("does not match this execution"); + expect(input.current).toHaveBeenCalledOnce(); +}); +it.each(["digest", "schema", "symlink", "oversized", "invalid_json"])("fails closed on %s instead of trusting remote or current bytes", kind => { + const { input, path } = fixture(); + resolveRemoteProviderPackAuthority(input).verified(); + const saved = JSON.parse(readFileSync(path, "utf8")); + if (kind === "digest") saved.manifest.payload.runnerSourceRevision = "c".repeat(40); + if (kind === "schema") saved.schema = "unknown"; + writeFileSync(path, JSON.stringify(saved)); + if (kind === "invalid_json") writeFileSync(path, "{"); + if (kind === "oversized") writeFileSync(path, " ".repeat(512 * 1024 + 1)); + if (kind === "symlink") { const copy = path + ".copy"; writeFileSync(copy, JSON.stringify(saved)); rmSync(path); symlinkSync(copy, path); } + expect(() => resolveRemoteProviderPackAuthority({ ...input, reattach: true })).toThrow("saved provider pack authority"); + expect(input.current).toHaveBeenCalledOnce(); +}); + +it("uses current authority for a replacement lease instead of restoring the prior pack", () => { + const { input, path } = fixture(); + resolveRemoteProviderPackAuthority(input).verified(); + const currentRunner = { version: "0.0.1", digest: "sha256:" + "d".repeat(64) }; + const replacement = resolveRemoteProviderPackAuthority({ ...input, binding: { ...binding, environmentLeaseId: "replacement" }, + current: () => manifest("b"), runnerArtifact: () => currentRunner }); + expect(replacement.manifest).toEqual(manifest("b")); + expect(replacement.runnerArtifact).toEqual(currentRunner); + replacement.verified(); + expect(JSON.parse(readFileSync(path, "utf8")).binding.environmentLeaseId).toBe("replacement"); +}); +it("retains the authenticated daemon identity across controller daemon replacement", () => { + const { input } = fixture(); + resolveRemoteProviderPackAuthority(input).verified(); + const changedRunner = vi.fn(() => ({ version: "new", digest: "sha256:" + "d".repeat(64) })); + const recovered = resolveRemoteProviderPackAuthority({ ...input, reattach: true, runnerArtifact: changedRunner }); + expect(recovered.runnerArtifact).toEqual(runnerArtifact()); + expect(changedRunner).not.toHaveBeenCalled(); +}); +it.each([{}, { version: "0.0.0", digest: "not-a-sha256" }])("rejects an invalid saved daemon identity: %j", runnerArtifact => { + const { input, path } = fixture(); + resolveRemoteProviderPackAuthority(input).verified(); + const saved = JSON.parse(readFileSync(path, "utf8")); + writeFileSync(path, JSON.stringify({ ...saved, runnerArtifact })); + expect(() => resolveRemoteProviderPackAuthority({ ...input, reattach: true })).toThrow("saved provider pack authority"); +}); + +it("keeps admitted session ownership with the caller if receipt storage or its warning fails", async () => { + const { input, path } = fixture(); + mkdirSync(path); + const authority = resolveRemoteProviderPackAuthority(input); + const warning = vi.fn(() => { throw new Error("log unavailable"); }); + await expect(recordAdmittedRemoteProviderPack(authority, warning)).resolves.toBe(false); + expect(warning).toHaveBeenCalledWith(expect.stringContaining("could not save remote release authority")); + expect(() => resolveRemoteProviderPackAuthority({ ...input, reattach: true })).toThrow("saved provider pack authority"); +}); +it("retains the original projectless lease across warm run placeholders but rejects a different real workspace", () => { + const { input } = fixture(); + const first = { ...binding, workspaceId: providerPackWorkspaceId("run-1", "run-1", "original-lease") }; + resolveRemoteProviderPackAuthority({ ...input, binding: first }).verified(); + const next = { ...binding, workspaceId: providerPackWorkspaceId("run-2", "run-2", "original-lease") }; + expect(resolveRemoteProviderPackAuthority({ ...input, binding: next, reattach: true }).manifest).toEqual(manifest("a")); + const changed = { ...binding, workspaceId: providerPackWorkspaceId("project-workspace", "run-2", "original-lease") }; + expect(() => resolveRemoteProviderPackAuthority({ ...input, binding: changed, reattach: true })).toThrow("does not match this execution"); +}); diff --git a/server/src/services/native-runtime/remote-provider-pack-authority.ts b/server/src/services/native-runtime/remote-provider-pack-authority.ts new file mode 100644 index 00000000000..5516ad6d959 --- /dev/null +++ b/server/src/services/native-runtime/remote-provider-pack-authority.ts @@ -0,0 +1,107 @@ +import { createHash, randomUUID } from "node:crypto"; +import { closeSync, constants, fstatSync, lstatSync, openSync, readFileSync, renameSync, rmSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; +import { redactSensitiveText } from "../../redaction.js"; +import { canonicalProviderPackJson, type ProviderPackIdentity } from "./remote-provider-pack-verification.js"; + +export type ProviderPackBinding = { + companyId: string; + agentId: string; + workspaceId: string; + sessionId: string; + runnerInstanceId: string; + environmentLeaseId: string; + remoteCwd: string; + providerDigest: string; +}; +export type RunnerArtifactIdentity = { version: string; digest: string }; + +const schema = "paperclip.remote-provider-pack-authority.v1"; +const filename = "provider-pack-authority.json"; +const maximumBytes = 512 * 1024; + +function assertManifest(manifest: ProviderPackIdentity): void { + if (manifest?.schema !== "paperclip-runner/remote-provider-pack/v1" || !manifest.payload + || manifest.digest !== `sha256:${createHash("sha256").update(canonicalProviderPackJson(manifest.payload)).digest("hex")}`) { + throw new Error("runner_remote_provider_artifact_incompatible: saved provider pack authority is invalid"); + } +} + +export function resolveRemoteProviderPackAuthority(input: { + stateDirectory: string; + binding: ProviderPackBinding; + reattach: boolean; + current: () => T; + runnerArtifact: () => RunnerArtifactIdentity; +}): { manifest: T; runnerArtifact: RunnerArtifactIdentity; verified: () => void } { + const root = lstatSync(input.stateDirectory); + if (!root.isDirectory() || root.isSymbolicLink()) { + throw new Error("runner_remote_provider_artifact_incompatible: provider pack authority directory is unsafe"); + } + const path = join(input.stateDirectory, filename); + let manifest: T | undefined; + let runnerArtifact: RunnerArtifactIdentity | undefined; + if (input.reattach) { + let fd: number | undefined; + try { + fd = openSync(path, constants.O_RDONLY | constants.O_NOFOLLOW); + const stat = fstatSync(fd); + if (!stat.isFile() || stat.size > maximumBytes) throw new Error("invalid authority file"); + const saved = JSON.parse(readFileSync(fd, "utf8")); + if (saved.schema !== schema || canonicalProviderPackJson(saved.binding) !== canonicalProviderPackJson(input.binding)) { + throw new Error("authority binding mismatch"); + } + assertManifest(saved.manifest); + if (typeof saved.runnerArtifact?.version !== "string" || !saved.runnerArtifact.version + || !/^sha256:[a-f0-9]{64}$/.test(saved.runnerArtifact.digest)) throw new Error("invalid runner artifact identity"); + manifest = saved.manifest as T; + runnerArtifact = saved.runnerArtifact; + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== "ENOENT") { + throw new Error("runner_remote_provider_artifact_incompatible: saved provider pack authority is invalid or does not match this execution", { cause: error }); + } + } finally { + if (fd !== undefined) closeSync(fd); + } + } + manifest ??= input.current(); + assertManifest(manifest); + runnerArtifact ??= input.runnerArtifact(); + const selectedRunner = structuredClone(runnerArtifact); + const selected = structuredClone(manifest); + const receipt = JSON.stringify({ schema, binding: input.binding, manifest: selected, runnerArtifact: selectedRunner }); + if (Buffer.byteLength(receipt) > maximumBytes) throw new Error("runner_remote_provider_artifact_incompatible: provider pack authority exceeds its size limit"); + return { + manifest: selected, + runnerArtifact: selectedRunner, + verified: () => { + const temporary = `${path}.${randomUUID()}.tmp`; + try { + writeFileSync(temporary, receipt, { flag: "wx", mode: 0o600 }); + renameSync(temporary, path); + } finally { + rmSync(temporary, { force: true }); + } + }, + }; +} + +export function providerPackWorkspaceId(workspaceId: string, runId: string, environmentLeaseId: string): string { + return workspaceId === runId ? environmentLeaseId : workspaceId; +} + +export async function recordAdmittedRemoteProviderPack( + authority: { verified: () => void }, + onWarning: (message: string) => void | Promise, +): Promise { + try { + authority.verified(); + return true; + } catch (error) { + const reason = redactSensitiveText(error instanceof Error ? error.message : String(error)).slice(-1_024); + try { + await onWarning(`[paperclip-runner] could not save remote release authority; recovery still requires a verified receipt or current release artifacts: ${reason}\n`); + } catch {} + return false; + } +} diff --git a/server/src/services/native-runtime/remote-provider-pack-verification.ts b/server/src/services/native-runtime/remote-provider-pack-verification.ts new file mode 100644 index 00000000000..3f504a636e9 --- /dev/null +++ b/server/src/services/native-runtime/remote-provider-pack-verification.ts @@ -0,0 +1,85 @@ +import { posix } from "node:path"; +import type { CommandManagedRuntimeRunner } from "@paperclipai/adapter-utils/command-managed-runtime"; +import { runRemoteProviderVerification } from "./remote-provider-pack.js"; + +export function canonicalProviderPackJson(value: unknown): string { + if (Array.isArray(value)) return `[${value.map(canonicalProviderPackJson).join(",")}]`; + if (value && typeof value === "object") return `{${Object.entries(value).sort(([a], [b]) => a.localeCompare(b)) + .map(([key, entry]) => `${JSON.stringify(key)}:${canonicalProviderPackJson(entry)}`).join(",")}}`; + return JSON.stringify(value) ?? "null"; +} + +export type ProviderPackIdentity = { + schema: string; + digest: string; + payload: { + pins: { opencode: string; [key: string]: string | null }; + target: { platform: string; architecture: string }; + runnerSourceRevision: string; + artifacts: { nodeCommand: { path: string; sha256: string }; opencodeCommand: { path: string; sha256: string }; [key: string]: { path: string; sha256: string } }; + }; +}; + +export async function verifyRemoteProviderPack({ runner, cwd, packRoot, manifest }: { + runner: CommandManagedRuntimeRunner; + cwd: string; + packRoot: string; + manifest: ProviderPackIdentity; +}): Promise { + const expected = Buffer.from( + canonicalProviderPackJson(manifest), + "utf8", + ).toString("base64"); + const providerNodeCommand = posix.join( + packRoot, + manifest.payload.artifacts.nodeCommand.path, + ); + const verifyScript = [ + "const fs=require('node:fs')", + "const crypto=require('node:crypto')", + "const path=require('node:path')", + "const root=process.argv[1]", + "const expected=Buffer.from(process.argv[2],'base64').toString('utf8')", + "const actual=fs.readFileSync(path.join(root,'provider-pack.json'),'utf8').trim()", + "const canonical=(v)=>Array.isArray(v)?'['+v.map(canonical).join(',')+']':v&&typeof v==='object'?'{'+Object.keys(v).sort().map(k=>JSON.stringify(k)+':'+canonical(v[k])).join(',')+'}':JSON.stringify(v)", + "const manifest=JSON.parse(actual)", + "if(canonical(manifest)!==expected)throw new Error('manifest mismatch')", + "const hash=(p)=>'sha256:'+crypto.createHash('sha256').update(fs.readFileSync(path.join(root,p))).digest('hex')", + "const tree=(treeRoot)=>{const digest=crypto.createHash('sha256');const visit=(directory,prefix='')=>{for(const entry of fs.readdirSync(directory,{withFileTypes:true}).sort((a,b)=>a.name.localeCompare(b.name))){const relative=prefix?prefix+'/'+entry.name:entry.name;const absolute=path.join(directory,entry.name);if(entry.isDirectory()){digest.update('directory\\0'+relative+'\\n');visit(absolute,relative)}else if(entry.isFile()){digest.update('file\\0'+relative+'\\0'+'sha256:'+crypto.createHash('sha256').update(fs.readFileSync(absolute)).digest('hex')+'\\n')}else if(entry.isSymbolicLink()){digest.update('symlink\\0'+relative+'\\0'+fs.readlinkSync(absolute)+'\\n')}else throw new Error('unsupported dist entry '+relative)}};visit(treeRoot);return 'sha256:'+digest.digest('hex')}", + "for(const name of ['nodeCommand','productionLock','opencodeCommand','opencodeExecutable','opencodeProxy','acpxSidecar','grokLauncher']){const artifact=manifest.payload.artifacts[name];if(hash(artifact.path)!==artifact.sha256)throw new Error(name+' digest mismatch')}", + "if(tree(path.join(root,'dist'))!==manifest.payload.distDigest)throw new Error('dist tree digest mismatch')", + "for(const candidate of Object.values({...manifest.payload.providers,...manifest.payload.candidateProviders})){if(tree(path.join(root,candidate.path))!==candidate.sha256)throw new Error('candidate asset tree digest mismatch')}", + "const version=process.versions.node.split('.').map(Number)", + "const minimum=manifest.payload.pins.nodeMinimum.split('.').map(Number)", + "if(version[0]JSON.parse(fs.readFileSync(path.join(root,'node_modules',...pkg.split('/'),'package.json'),'utf8')).version", + "const expectedPackages={acpx:manifest.payload.pins.acpx,'@agentclientprotocol/claude-agent-acp':manifest.payload.pins.claudeAcp,'@agentclientprotocol/codex-acp':manifest.payload.pins.codexAcp,'opencode-ai':manifest.payload.pins.opencode}", + "for(const [pkg,version] of Object.entries(expectedPackages))if(packageVersion(pkg)!==version)throw new Error(pkg+' version mismatch')", + ].join(";"); + await runRemoteProviderVerification(() => runner.execute({ + command: providerNodeCommand, + args: ["-e", verifyScript, packRoot, expected], + cwd: cwd, + bypassSession: true, + timeoutMs: 30_000, + }), "provider pack verification"); + const opencodeCommand = posix.join( + packRoot, + manifest.payload.artifacts.opencodeCommand.path, + ); + const opencodeVersion = await runRemoteProviderVerification(() => runner.execute({ + command: opencodeCommand, + args: ["--version"], + cwd: cwd, + bypassSession: true, + timeoutMs: 30_000, + }), "OpenCode version verification"); + if ( + opencodeVersion.stdout.trim() !== manifest.payload.pins.opencode + ) { + throw new Error( + "runner_remote_provider_artifact_incompatible: OpenCode version mismatch", + ); + } +} diff --git a/server/src/services/native-runtime/remote-provider-pack.test.ts b/server/src/services/native-runtime/remote-provider-pack.test.ts index a4ec1f60d97..5d5991abca4 100644 --- a/server/src/services/native-runtime/remote-provider-pack.test.ts +++ b/server/src/services/native-runtime/remote-provider-pack.test.ts @@ -25,6 +25,14 @@ describe("remote provider verification", () => { await expect(runRemoteProviderVerification(execute, "pack")).rejects.toThrow("runner_remote_provider_artifact_incompatible: pack failed (exitCode=1"); expect(execute).toHaveBeenCalledOnce(); }); + it("retains a bounded redacted rejection from a combined-output transport", async () => { + const execute = vi.fn().mockResolvedValue({ ...ok, exitCode: 1, + stdout: "Error: manifest mismatch\nANTHROPIC_API_KEY=secret-fixture-value", stderr: "" }); + const failure = await runRemoteProviderVerification(execute, "pack").catch(error => error); + expect(failure.message).toContain("manifest mismatch"); + expect(failure.message).not.toContain("secret-fixture-value"); + expect(execute).toHaveBeenCalledOnce(); + }); it("recovers a transport exception and propagates cancellation", async () => { const execute = vi.fn().mockRejectedValueOnce(new Error("connection unavailable")).mockResolvedValueOnce(ok); await expect(runRemoteProviderVerification(execute, "pack")).resolves.toEqual(ok); diff --git a/server/src/services/native-runtime/remote-provider-pack.ts b/server/src/services/native-runtime/remote-provider-pack.ts index b7045ad42d7..f6621e1e1ea 100644 --- a/server/src/services/native-runtime/remote-provider-pack.ts +++ b/server/src/services/native-runtime/remote-provider-pack.ts @@ -1,3 +1,4 @@ +import { redactSensitiveText } from "../../redaction.js"; import type { NativeRunTrace } from "./native-run-trace.js"; import type { NativeExecutionInput } from "../../vendor/paperclip-runner/index.js"; @@ -37,10 +38,10 @@ export async function runRemoteProviderVerification( result = await execute(); } catch (error) { if (error instanceof Error && error.name === "AbortError") throw error; - diagnostic = error instanceof Error ? error.message : String(error); + diagnostic = redactSensitiveText(error instanceof Error ? error.message : String(error)).slice(-1_024); continue; } - diagnostic = `exitCode=${result.exitCode}; timedOut=${result.timedOut}; signal=${result.signal ?? "none"}; ${result.stderr.trim().slice(-1_024)}`; + diagnostic = `exitCode=${result.exitCode}; timedOut=${result.timedOut}; signal=${result.signal ?? "none"}; ${redactSensitiveText([result.stderr, result.stdout].filter(Boolean).join("\n")).trim().slice(-1_024)}`; if (result.timedOut || result.exitCode === null || result.signal) continue; if (result.exitCode !== 0) { throw new Error(`runner_remote_provider_artifact_incompatible: ${label} failed (${diagnostic})`); diff --git a/server/src/services/native-runtime/status-arbiter.test.ts b/server/src/services/native-runtime/status-arbiter.test.ts index 6262e01adaf..9a3609f9a2c 100644 --- a/server/src/services/native-runtime/status-arbiter.test.ts +++ b/server/src/services/native-runtime/status-arbiter.test.ts @@ -560,7 +560,7 @@ describe("native status authority", () => { expect.objectContaining({ statusAction: "blocked", toStatus: "blocked", - policyVersion: "phase6-v12", + policyVersion: "phase6-v13", reasonCode: "current_track_blocker_waiting", unblockDescriptor: { owner: "board", diff --git a/server/src/services/native-runtime/status-arbiter.ts b/server/src/services/native-runtime/status-arbiter.ts index 5c6e4679e12..4fdc5496cc1 100644 --- a/server/src/services/native-runtime/status-arbiter.ts +++ b/server/src/services/native-runtime/status-arbiter.ts @@ -1,7 +1,7 @@ import type { NativeEvidenceAssessment } from "./evidence-classifier.js"; import { NATIVE_MODEL_REJECTION_MESSAGE, NATIVE_PROVIDER_CAPACITY_MAX_RETRIES, NATIVE_PROVIDER_OVERLOADED_CODE, NATIVE_PROVIDER_OVERLOADED_MESSAGE } from "./native-provider-failure.js"; -export const NATIVE_STATUS_ARBITER_POLICY_VERSION = "phase6-v12"; +export const NATIVE_STATUS_ARBITER_POLICY_VERSION = "phase6-v13"; export type NativeAuthoritativeIssueStatus = | "backlog" @@ -118,6 +118,7 @@ export function arbitrateNativeStatus(input: { caseHandoffAuthorized?: boolean; /** The saved timer already admitted another run and this run no longer owns checkout. */ monitorWaitDispatched?: boolean; + monitorWaitPreserved?: boolean; boardResponseWaitAuthorized?: boolean; boardResponseWaitOrigin?: boolean; isConversation?: boolean; @@ -483,6 +484,13 @@ export function arbitrateNativeStatus(input: { effects: [], }; } + if (input.monitorWaitPreserved) { + return { + policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION, + statusAction: "preserve", toStatus: input.priorIssueStatus, + reasonCode: "scheduled_monitor_preserved", unblockDescriptor: null, effects: [], + }; + } if (input.monitorWaitAuthorized) { return { policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION, diff --git a/server/src/services/native-runtime/status-decision-committer.ts b/server/src/services/native-runtime/status-decision-committer.ts index eafd64245c9..c7684448cbf 100644 --- a/server/src/services/native-runtime/status-decision-committer.ts +++ b/server/src/services/native-runtime/status-decision-committer.ts @@ -3,7 +3,7 @@ import { hasPublishedGitHubReview } from "../chat-channels/github/review-wait.js import { pendingNativeGovernance } from "./native-governance.js"; import { findSupersededExternalReviewRequests, supersedeExternalReviewRequestsInTransaction } from "./superseded-external-review-requests.js"; import { normalizePrpResultSignals, type PrpStructuredRunResult } from "../../vendor/paperclip-runner/index.js"; -import { readNativeMonitorDelivery, type NativeMonitorDelivery } from "./native-monitor-delivery.js"; +import { readNativeMonitorDelivery, readNativePreservedMonitorWait, type NativeMonitorDelivery, type NativePreservedMonitorWait } from "./native-monitor-delivery.js"; import { readNativePlanWait, type NativePlanWaitSource } from "./native-plan-wait.js"; import { activeIssueInteractionCondition } from "../issue-question-context.js"; import { isConversation } from "../agent-conversations.js"; @@ -1565,6 +1565,7 @@ export async function commitNativeStatusDecision(input: { requirePlanWaitSource?: NativePlanWaitSource; requireMonitorWait?: { agentId: string; nextCheckAt: string }; requireMonitorDelivery?: NativeMonitorDelivery; + requirePreservedMonitorWait?: NativePreservedMonitorWait; requireNoPendingChildCompletion?: NativeChildCompletionRecipient; requireModelRejectionOwner?: { agentId: string; reviewContext: NativeReviewAssignmentContext | null }; requireProviderFailureOwner?: { agentId: string; reviewContext: NativeReviewAssignmentContext | null }; @@ -1675,6 +1676,18 @@ export async function commitNativeStatusDecision(input: { input.decision.statusAction !== "preserve" || input.decision.toStatus !== issue.status || input.decision.effects.length !== 0) throw new NativeStatusRaceError(); } + if (reasonCode === "scheduled_monitor_preserved") { + const expected = input.requirePreservedMonitorWait; + const actual = expected ? await readNativePreservedMonitorWait(tx as unknown as Db, { + companyId: input.companyId, issueId: input.issueId, runId: input.runId, agentId: expected.agentId, + }, issue) : null; + if (!expected || !actual || actual.nextCheckAt !== expected.nextCheckAt || actual.resultId !== expected.resultId || + input.decision.statusAction !== "preserve" || input.decision.toStatus !== issue.status || + input.decision.effects.length !== 0 || await pendingNativeGovernance({ db: tx as unknown as Db, + companyId: input.companyId, issueId: input.issueId, runId: input.runId, executionState: record(issue.executionState), + })) throw new NativeStatusRaceError(); + await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors); + } // Recheck under the parent status lock: a child can finish between the // finalizer's read and this commit. Re-arbitrate instead of discarding its wake. if (input.requireNoPendingChildCompletion && await hasPendingNativeChildCompletion( diff --git a/server/src/services/native-runtime/work-assessments.ts b/server/src/services/native-runtime/work-assessments.ts index 829b497040b..58975d83b6d 100644 --- a/server/src/services/native-runtime/work-assessments.ts +++ b/server/src/services/native-runtime/work-assessments.ts @@ -3,7 +3,7 @@ import type { Db } from "@paperclipai/db"; import { workAssessments } from "@paperclipai/db"; import type { NativeEvidenceAssessment } from "./evidence-classifier.js"; import { nativeSha256 } from "./canonical.js"; -import type { NativeMonitorDelivery } from "./native-monitor-delivery.js"; +import type { NativeMonitorDelivery, NativePreservedMonitorWait } from "./native-monitor-delivery.js"; export async function recordNativeWorkAssessment(input: { db: Db; @@ -26,12 +26,14 @@ export async function recordNativeWorkAssessment(input: { monitorDelivery?: NativeMonitorDelivery | null; /** A persisted schedule can regain settlement authority after a policy repair. */ monitorWaitAt?: string | null; + preservedMonitorWait?: NativePreservedMonitorWait | null; supersedesAssessmentId?: string | null; }) { const assessmentJson = { ...input.assessment, ...(input.monitorDelivery ? { monitorDelivery: input.monitorDelivery } : {}), ...(input.monitorWaitAt ? { monitorWaitAt: input.monitorWaitAt } : {}), + ...(input.preservedMonitorWait ? { preservedMonitorWait: input.preservedMonitorWait } : {}), } as unknown as Record; const inputDigest = nativeSha256({ contractId: input.contractId, diff --git a/server/src/services/nested-clone-migration.ts b/server/src/services/nested-clone-migration.ts index eb0e3a87eda..09b4f6663a9 100644 --- a/server/src/services/nested-clone-migration.ts +++ b/server/src/services/nested-clone-migration.ts @@ -13,7 +13,7 @@ import { GIT_CACHE_RESTORATION_KEY, readGitCacheRestoration, type GitCacheRestor import { readManagedCheckoutContents, selectRestoredProjectRepositories } from "./git-workspace-contents.js"; import { applyRestorePoint, captureRestorePoint, currentRestorePointSnapshot, importRestorePoint } from "./git-restore-point.js"; import { managedProjectCheckoutName } from "./managed-project-checkout-name.js"; -import { createGitRemoteAuthProvider, type GitRemoteAuthProvider } from "./git-credentials.js"; +import { createIssueGitRemoteAuthProvider, type GitRemoteAuthProvider } from "./git-credentials.js"; import { isNestedWorktreeBranchName, isVerifiedNestedRepositoryBase, moveNestedRepositoryWorktree, nestedRepositoryBaseCandidates, nestedWorktreeBranchPrefix, readLinkedNestedCheckout, removeStaleNestedRegistrations, trackNestedWorktreeUpstream, withNestedRepositoryBaseLock } from "./nested-repository-worktrees.js"; import { assertSafeBaseConfig, assertTransferredItems, historyKey, linkedCheckoutBases, moveTransferItems, presentCommits, returnTransferItems, @@ -442,7 +442,7 @@ export async function migrateNestedRepositoryClone(db: Db, snapshot: Workspace, const base = source.base; await assertSafeBaseConfig(base, signal); await fetchBase(base, source.repoUrl, options.resolveGitAuth - ?? withAnonymousGitFallback(createGitRemoteAuthProvider(db, row!.companyId, { issueId: row!.sourceIssueId! })), signal); + ?? withAnonymousGitFallback(createIssueGitRemoteAuthProvider(db, row!.companyId, row!.sourceIssueId!)), signal); await assertSafeBaseConfig(base, signal); await assertNoBranchOnlyReflogHistory(inspected, signal); const planned = await planRefs(inspected, base, prefix, signal); diff --git a/server/src/services/nested-repository-worktrees.ts b/server/src/services/nested-repository-worktrees.ts index 8a2969ec2f3..ad907159cc3 100644 --- a/server/src/services/nested-repository-worktrees.ts +++ b/server/src/services/nested-repository-worktrees.ts @@ -15,6 +15,7 @@ import { sanitizeRuntimeServiceBaseEnv } from "./runtime-service-env.js"; const execFile = promisify(execFileCallback); const NESTED_WORKTREE_GIT_TIMEOUT_MS = 10 * 60 * 1000; const NESTED_WORKTREE_BRANCH = /^paperclip\/[a-f0-9]{12}\/./; +export const OWNED_NESTED_REF_LIMIT = 64; export const DETACHED_REPOSITORIES_DIR = path.join(".paperclip-runtime", "detached-repositories"); export type LinkedNestedCheckout = { checkout: string; base: string; gitDir: string; admin: string; branchPrefix: string }; diff --git a/server/src/services/plugin-agent-runtime-state.ts b/server/src/services/plugin-agent-runtime-state.ts index a1a4ce22ca7..0770246ef14 100644 --- a/server/src/services/plugin-agent-runtime-state.ts +++ b/server/src/services/plugin-agent-runtime-state.ts @@ -36,7 +36,7 @@ async function readState(db: Db, pluginId: string, companyId: string, agentId: s }; const instructions = await agentInstructionsService().getBundleLocation(agent); const records = agentRecordQueries(tx as unknown as Db); - const companyAgents = await records.listCompanyAgentRows(companyId); + const companyAgents = await records.listCompanyAgentEligibilityRows(companyId); return { revision: String(latest?.id ?? 0), agent: records.normalizeAgentRow(agent, companyAgents) as PluginAgentRuntimeState["agent"], instructionsRootPath: instructions.rootPath, instructionsEntryFile: instructions.entryFile, diff --git a/server/src/services/plugin-environment-driver.ts b/server/src/services/plugin-environment-driver.ts index d7f56f2b197..f26f32483ff 100644 --- a/server/src/services/plugin-environment-driver.ts +++ b/server/src/services/plugin-environment-driver.ts @@ -1,4 +1,5 @@ -import type { Db } from "@paperclipai/db"; +import { plugins, type Db } from "@paperclipai/db"; +import { sql } from "drizzle-orm"; import type { EnvironmentProbeResult, PluginEnvironmentConfig, @@ -162,6 +163,62 @@ export async function resolvePluginEnvironmentDriverByKey(input: { }); } +type CachedProviderSchema = { pluginId: string; revision: string; schema: Record | null }; +const providerSchemas = new WeakMap>(); +const MAX_CACHED_PROVIDER_SCHEMAS = 32; +const MAX_CACHED_PROVIDER_SCHEMA_BYTES = 64 * 1024; + +export async function resolvePluginSandboxProviderConfigSchema(input: { + db: Db; driverKey: string; +}): Promise | null> { + const { db, driverKey } = input; + let cache = providerSchemas.get(db); + const cached = cache?.get(driverKey); + const unchanged = sql`coalesce(plugin_id = ${cached?.pluginId ?? null}::uuid + and revision = ${cached?.revision ?? null}, false)`; + const [row] = await db.execute<{ + pluginId: string; revision: string; unchanged: boolean; configSchema: unknown; + }>(sql`with selected_provider as materialized ( + select ${plugins.id} as plugin_id, ( + select declaration.value->'configSchema' + from jsonb_array_elements(${plugins.manifestJson}->'environmentDrivers') + with ordinality as declaration(value, position) + where declaration.value->'driverKey' = to_jsonb(${driverKey}::text) + and declaration.value->>'kind' = 'sandbox_provider' + order by declaration.position limit 1 + ) as schema_value + from ${plugins} + where ${plugins.manifestJson} @> ${JSON.stringify({ environmentDrivers: [{ driverKey, kind: "sandbox_provider" }] })}::jsonb + order by ${plugins.installOrder} limit 1 + ), versioned_provider as materialized ( + select plugin_id, schema_value, + encode(sha256(convert_to(coalesce(schema_value::text, 'null'), 'UTF8')), 'hex') as revision + from selected_provider + ) + select plugin_id as "pluginId", revision, ${unchanged} as unchanged, + case when ${unchanged} then null else schema_value end as "configSchema" + from versioned_provider`); + if (!row) { + cache?.delete(driverKey); + return null; + } + if (row.unchanged) { + if (!cached || cached.pluginId !== row.pluginId || cached.revision !== row.revision) { + throw new Error("Sandbox provider schema revision did not match the retained definition"); + } + return structuredClone(cached.schema); + } + const schema = row.configSchema && typeof row.configSchema === "object" && !Array.isArray(row.configSchema) + ? row.configSchema as Record : null; + cache?.delete(driverKey); + if (Buffer.byteLength(JSON.stringify(schema)) <= MAX_CACHED_PROVIDER_SCHEMA_BYTES) { + if (!cache) { cache = new Map(); providerSchemas.set(db, cache); } + if (cache.size >= MAX_CACHED_PROVIDER_SCHEMAS) cache.delete(cache.keys().next().value!); + cache.set(driverKey, { pluginId: row.pluginId, revision: row.revision, schema: structuredClone(schema) }); + } + return schema; +} + export async function resolvePluginSandboxProviderDriverByKey(input: { db: Db; driverKey: string; diff --git a/server/src/services/presentation-ownership-read.test.ts b/server/src/services/presentation-ownership-read.test.ts new file mode 100644 index 00000000000..08ee36f558e --- /dev/null +++ b/server/src/services/presentation-ownership-read.test.ts @@ -0,0 +1,214 @@ +import { randomBytes, randomUUID } from "node:crypto"; +import { agents, companies, completionContracts, createDb, environmentLeases, heartbeatRunEvents, heartbeatRuns, issueComments, issues, nativeRunFinalizations, nativeRunResults, type Db } from "@paperclipai/db"; +import { eq, type SQL } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js"; +import { NativeRunCoordinatorStore } from "./native-runtime/native-run-coordinator-store.js"; +import { getConversationOwnershipBlocker } from "./conversation-continuation.js"; +import { isExternalChatPresentationContext } from "./heartbeat-run-summary.js"; +import { finalizeNativeRun, repairCommittedNativeChatResponse } from "./native-runtime/native-run-finalizer.js"; + +const support = await getEmbeddedPostgresTestSupport(); +describe.skipIf(!support.supported)("presentation and ownership PostgreSQL read projections", () => { + let database: Awaited>; + let db: ReturnType; + beforeAll(async () => { + database = await startEmbeddedPostgresTestDatabase("presentation-ownership-read-"); + db = createDb(database.connectionString); + }, 60_000); + afterAll(async () => { await db?.$client.end(); await database?.cleanup(); }); + + async function fixture(patch: Partial = {}) { + const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Read fixture", issuePrefix: companyId.slice(0, 8) }); + await db.insert(agents).values({ id: agentId, companyId, name: "Read agent", adapterType: "codex_local" }); + await db.insert(issues).values({ id: issueId, companyId, title: "Read task", status: "blocked", assigneeAgentId: agentId }); + const payload = randomBytes(128 * 1024).toString("base64"); + const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId, nativeIssueId: issueId, + status: "failed", runtimeMode: "legacy", finishedAt: new Date("2020-01-01T00:00:00Z"), + contextSnapshot: { issueId, source: "chat:fixture", prompt: payload, + paperclipWake: { externalInteractionContinuation: true, prompt: payload } }, + runnerProfileJson: { adapterDispatch: { adapterType: "codex_local", instructions: payload }, nativeExecutionInput: { prompt: payload } }, + resultJson: { unknownFutureRecoveryReceipt: { preserved: true } }, stdoutExcerpt: payload, stderrExcerpt: payload, + ...patch, + }).returning(); + return { companyId, agentId, issueId, run }; + } + + function captureReads(fullRows = false) { + const rows: unknown[][] = []; + function recordSelects>(executor: T): T { + return new Proxy(executor, { get(target, key) { + if (key !== "select") return Reflect.get(target, key); + return (...fields: Parameters) => { + const builder = target.select(...fields), from = builder.from.bind(builder); + builder.from = ((...args: Parameters) => { + const selection = fields[0]; + const nested = selection && "run" in selection && "activeLease" in selection; + const repairSource = selection && "nativeSessionId" in selection && "completionContractId" in selection && "finishedAt" in selection; + const sourceRead = args[0] === heartbeatRuns && (nested || repairSource); + const query = fullRows && sourceRead + ? (nested ? target.select({ run: heartbeatRuns, activeLease: selection.activeLease as SQL }) : target.select()).from(heartbeatRuns) + : from(...args); + if (sourceRead) { + const execute = query.execute.bind(query); + query.execute = async (...executeArgs: Parameters) => { + const result = await execute(...executeArgs); rows.push(result); return result; + }; + } + return query; + }) as typeof builder.from; + return builder; + }; + }}); + } + const root = recordSelects(db); + const transaction: Db["transaction"] = (callback, config) => db.transaction(tx => callback(recordSelects(tx)), config); + const recorded = new Proxy(root, { get(target, key) { return key === "transaction" ? transaction : Reflect.get(target, key); } }); + return { recorded, rows }; + } + + async function compareOwnership(f: Awaited>) { + const baseline = captureReads(true), projected = captureReads(); + const expected = await getConversationOwnershipBlocker(baseline.recorded, f.companyId, f.issueId); + const actual = await getConversationOwnershipBlocker(projected.recorded, f.companyId, f.issueId); + expect(actual).toEqual(expected); + expect(projected.rows.flat()).toHaveLength(1); + expect(Buffer.byteLength(JSON.stringify(projected.rows))).toBeLessThan(4096); + expect(Buffer.byteLength(JSON.stringify(baseline.rows))).toBeGreaterThan(128 * 1024); + return actual; + } + + it("keeps current process authority without reading execution payloads", async () => { + const f = await fixture({ processPid: process.pid }); + expect(await compareOwnership(f)).toMatchObject({ runId: f.run.id, cause: "execution_owner_active" }); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.run.id)))[0]).toEqual(f.run); + expect(await getConversationOwnershipBlocker(db, randomUUID(), f.issueId)).toBeNull(); + await db.update(heartbeatRuns).set({ processPid: null }).where(eq(heartbeatRuns.id, f.run.id)); + expect(await getConversationOwnershipBlocker(db, f.companyId, f.issueId)).toBeNull(); + }); + + it("retains an active lease until the current database row releases it", async () => { + const f = await fixture(); + const [lease] = await db.insert(environmentLeases).values({ companyId: f.companyId, heartbeatRunId: f.run.id, + issueId: f.issueId, provider: "daytona", providerLeaseId: "fixture-lease", status: "active" }).returning(); + expect(await compareOwnership(f)).toMatchObject({ runId: f.run.id, cause: "execution_owner_active" }); + expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, lease.id)))[0]).toEqual(lease); + await db.update(environmentLeases).set({ status: "released", releasedAt: new Date("2020-01-01T00:00:00Z") }).where(eq(environmentLeases.id, lease.id)); + expect(await getConversationOwnershipBlocker(db, f.companyId, f.issueId)).toBeNull(); + }); + + it("keeps unconfirmed remote cleanup and all recovery evidence", async () => { + const f = await fixture(); + const [lease] = await db.insert(environmentLeases).values({ companyId: f.companyId, heartbeatRunId: f.run.id, + issueId: f.issueId, provider: "daytona", providerLeaseId: "retained-fixture", status: "released", + releasedAt: new Date("2020-01-01T00:00:00Z"), cleanupStatus: "success" }).returning(); + const result = { ...f.run.resultJson, workspaceRestoreFailure: "restore_unsafe_archive", + workspaceRestoreRecovery: { schema: "paperclip.workspace-restore-recovery.v1", leaseIds: [lease.id] } }; + await db.update(heartbeatRuns).set({ resultJson: result }).where(eq(heartbeatRuns.id, f.run.id)); + expect(await compareOwnership(f)).toMatchObject({ runId: f.run.id, cause: "execution_owner_active" }); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.run.id)))[0].resultJson).toEqual(result); + }); + + async function compareRepair(f: Awaited>) { + const baseline = captureReads(true), projected = captureReads(); + const input = { companyId: f.companyId, issueId: f.issueId, runId: f.run.id }; + expect(await repairCommittedNativeChatResponse(projected.recorded, input)) + .toBe(await repairCommittedNativeChatResponse(baseline.recorded, input)); + expect(projected.rows.flat()).toHaveLength(1); + expect(Buffer.byteLength(JSON.stringify(projected.rows))).toBeLessThan(4096); + expect(Buffer.byteLength(JSON.stringify(baseline.rows))).toBeGreaterThan(128 * 1024); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.run.id)))[0]).toEqual(f.run); + return projected.rows.flat()[0] as { contextSnapshot: Record | null; resultJson: unknown }; + } + + it("preserves selected answers without downloading prompts, profiles, or excerpts", async () => { + const f = await fixture({ runtimeMode: "native", status: "succeeded", resultJson: { + presentationDecision: { commentId: randomUUID() }, unknownFutureReceipt: { preserved: true } } }); + const row = await compareRepair(f); + expect(row.resultJson).toEqual(f.run.resultJson); + expect(await repairCommittedNativeChatResponse(db, { companyId: randomUUID(), issueId: f.issueId, runId: f.run.id })).toBe(false); + const [other] = await db.insert(issues).values({ companyId: f.companyId, title: "Other task" }).returning(); + expect(await repairCommittedNativeChatResponse(db, { companyId: f.companyId, issueId: other.id, runId: f.run.id })).toBe(false); + }); + + async function unpublishedAcceptedReply() { + const payload = randomBytes(128 * 1024).toString("base64"); + const f = await fixture({ runtimeMode: "native", status: "running", finishedAt: null, + contextSnapshot: { source: "internal", prompt: payload, paperclipWake: { prompt: payload } } }); + const [contract] = await db.insert(completionContracts).values({ companyId: f.companyId, issueId: f.issueId, + revision: 1, schemaVersion: "paperclip.completion-contract.v1", policyVersion: "test", risk: "standard", + completionAuthority: "server_arbiter", incompleteCriteriaPolicy: "preserve_non_terminal", + contractJson: { revision: "test", objective: "Finish the fixture task", criteria: [] }, + canonicalSha256: "read-fixture-contract", createdByActorType: "system", createdByActorId: "test" }).returning(); + const sessionId = randomUUID(), runnerId = randomUUID(); + await db.update(heartbeatRuns).set({ nativeSessionId: sessionId, runnerInstanceId: runnerId, + completionContractId: contract.id, completionContractSha256: contract.canonicalSha256 }).where(eq(heartbeatRuns.id, f.run.id)); + await db.update(issues).set({ status: "in_progress", executionRunId: f.run.id }).where(eq(issues.id, f.issueId)); + const port = new NativeRunCoordinatorStore(db, { companyId: f.companyId, issueId: f.issueId, + runId: f.run.id, agentId: f.agentId, normalizedSessionId: sessionId, completionContractId: contract.id, + completionContractSha256: contract.canonicalSha256, runnerSourceInstanceId: runnerId, + completionContractRevision: "test", completionContractCriterionIds: [] }); + const reply = "The accepted provider reply is preserved exactly."; + await db.insert(heartbeatRunEvents).values({ companyId: f.companyId, runId: f.run.id, agentId: f.agentId, + seq: 1_000_000, eventType: "item.completed", payload: { prpEvent: { turnId: "reply-turn", sourceEventId: "fixture-final-answer", + payload: { kind: "agentMessage", channel: "final", text: reply } } } }); + await port.completeRun({ result: { schema: "paperclip.run_result.v1", reportedWorkDisposition: "done", + summary: "The accepted summary differs from its final answer.", completionClaim: { contractRevision: "test", + objectiveSatisfied: true, criteria: [], remainingWork: [] }, evidence: [], verification: [], attentionRequests: [], artifacts: [] }, + terminal: { schema: "paperclip.prp.terminal.v1", runTerminalState: "succeeded", turnTerminalState: "completed", reportedWorkDisposition: "done" }, + turnId: "reply-turn" }); + await finalizeNativeRun({ db, runId: f.run.id, workspaceFinalizeStatus: "succeeded", projectRunStatus: false }); + expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.run.id)))[0]).toMatchObject({ phase: "committed" }); + await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date("2020-01-01T00:00:00Z") }).where(eq(heartbeatRuns.id, f.run.id)); + expect(await db.select().from(issueComments).where(eq(issueComments.createdByRunId, f.run.id))).toEqual([]); + const [before] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.run.id)); + expect(before.resultJson?.presentationDecision).toBeUndefined(); + return { ...f, before, reply }; + } + + it("publishes the same accepted reply and decision from equivalent full-row and projected starting states", async () => { + const outcomes: unknown[] = []; + for (const fullRows of [true, false]) { + const f = await unpublishedAcceptedReply(), captured = captureReads(fullRows); + const input = { companyId: f.companyId, issueId: f.issueId, runId: f.run.id }; + expect(await repairCommittedNativeChatResponse(captured.recorded, input)).toBe(true); + const comments = await db.select().from(issueComments).where(eq(issueComments.createdByRunId, f.run.id)); + expect(comments).toHaveLength(1); expect(comments[0].body).toBe(f.reply); + const [after] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.run.id)); + const decision = after.resultJson!.presentationDecision as Record; + expect(decision.commentId).toBe(comments[0].id); + const { commentId: _, ...stableDecision } = decision; + outcomes.push({ body: comments[0].body, decision: stableDecision }); + expect(after.resultJson).toMatchObject(f.before.resultJson!); + expect(after.contextSnapshot).toEqual(f.before.contextSnapshot); + expect(after.runnerProfileJson).toEqual(f.before.runnerProfileJson); + expect(after.stdoutExcerpt).toBe(f.before.stdoutExcerpt); + expect(after.stderrExcerpt).toBe(f.before.stderrExcerpt); + expect(await db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, f.run.id))).toHaveLength(1); + expect(await repairCommittedNativeChatResponse(db, input)).toBe(false); + expect(await db.select().from(issueComments).where(eq(issueComments.createdByRunId, f.run.id))).toHaveLength(1); + expect(captured.rows.flat()).toHaveLength(1); + if (fullRows) expect(Buffer.byteLength(JSON.stringify(captured.rows))).toBeGreaterThan(128 * 1024); + else expect(Buffer.byteLength(JSON.stringify(captured.rows))).toBeLessThan(4096); + } + expect(outcomes[1]).toEqual(outcomes[0]); + }); + + it.each([null, [], "scalar", 42, { source: 42, skipIssueComment: "true", paperclipWake: [] }, + { source: "issue.comment.created", externalChatContinuation: true }, + { source: "chat:fixture", skipIssueComment: true, paperclipWake: { externalInteractionContinuation: "true" } }, + { source: "internal", paperclipWake: { externalInteractionContinuation: true } }, + ].map(context => ({ context })))("preserves typed publication flags for $context", async ({ context }) => { + const f = await fixture({ runtimeMode: "native", status: "succeeded", contextSnapshot: context as Record | null }); + const row = await compareRepair(f); + expect(isExternalChatPresentationContext(row.contextSnapshot)).toBe(isExternalChatPresentationContext(context)); + const original = context && typeof context === "object" && !Array.isArray(context) ? context : {}; + expect(row.contextSnapshot?.skipIssueComment === true).toBe((original as Record).skipIssueComment === true); + }); + + it.each([null, ["stdout", { receipt: true }], "scalar", 42].map(result => ({ result })))("preserves non-object result evidence $result", async ({ result }) => { + const f = await fixture({ runtimeMode: "native", status: "succeeded", resultJson: result as Record | null }); + expect((await compareRepair(f)).resultJson).toEqual(result); + }); +}); diff --git a/server/src/services/project-access-denial.ts b/server/src/services/project-access-denial.ts new file mode 100644 index 00000000000..f9e699d1bdc --- /dev/null +++ b/server/src/services/project-access-denial.ts @@ -0,0 +1,10 @@ +import { HttpError } from "../errors.js"; + +export const PROJECT_ACCESS_DENIED = "project_access_denied"; + +/** Carries a safe, typed setup failure without changing the authorization decision. */ +export class ProjectAccessDeniedError extends HttpError { + constructor(readonly projectId: string) { + super(403, "Project is outside this low-trust boundary."); + } +} diff --git a/server/src/services/recovery/disposition-repair.ts b/server/src/services/recovery/disposition-repair.ts index ed315f19e1f..b3ac840187a 100644 --- a/server/src/services/recovery/disposition-repair.ts +++ b/server/src/services/recovery/disposition-repair.ts @@ -30,6 +30,17 @@ type DispositionRepairIssue = Pick< | "monitorNextCheckAt" >; +export const dispositionRepairIssueFields = { + id: issues.id, + companyId: issues.companyId, + status: issues.status, + assigneeAgentId: issues.assigneeAgentId, + assigneeUserId: issues.assigneeUserId, + executionPolicy: issues.executionPolicy, + executionState: issues.executionState, + monitorNextCheckAt: issues.monitorNextCheckAt, +}; + export type DispositionRepairSourceState = { fingerprint: string; dependencyIssueIds: string[]; diff --git a/server/src/services/recovery/issue-state-read.test.ts b/server/src/services/recovery/issue-state-read.test.ts new file mode 100644 index 00000000000..5a8ad5acf28 --- /dev/null +++ b/server/src/services/recovery/issue-state-read.test.ts @@ -0,0 +1,117 @@ +import { randomBytes, randomUUID } from "node:crypto"; +import { agents, companies, createDb, issueRecoveryActions, issues } from "@paperclipai/db"; +import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture"; +import { eq } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; +import { recoveryService } from "./service.js"; + +const support = await getEmbeddedPostgresTestSupport(); + +describe.skipIf(!support.supported)("recovery issue state reads", () => { + let database: Awaited>; + let db: ReturnType; + + beforeAll(async () => { + database = await startEmbeddedPostgresTestDatabase("recovery-issue-state-"); + db = createDb(database.connectionString); + }); + afterAll(async () => { + await db?.$client.end(); + await database?.cleanup(); + }); + + it.each([false, true])("does not load unused source and child descriptions (child=%s)", async (withChild) => { + const [company] = await db.insert(companies).values({ + name: "Recovery state reads", + issuePrefix: `R${randomUUID().slice(0, 7).toUpperCase()}`, + }).returning(); + const description = randomBytes(128 * 1024).toString("base64"); + const [parent] = await db.insert(issues).values({ + companyId: company.id, + title: "Await the board decision", + description, + status: "blocked", + }).returning(); + const child = withChild + ? (await db.insert(issues).values({ + companyId: company.id, + title: "A child that also needs a decision", + description, + status: "blocked", + parentId: parent.id, + }).returning())[0] + : null; + const [action] = await db.insert(issueRecoveryActions).values({ + companyId: company.id, + sourceIssueId: parent.id, + kind: "stranded_assigned_issue", + cause: "stranded_assigned_issue", + fingerprint: randomUUID(), + status: "active", + ownerType: "board", + wakePolicy: { type: "board_escalation" }, + nextAction: "Choose the next disposition", + evidence: { retainedControl: { proof: "unchanged" } }, + }).returning(); + const enqueueWakeup = vi.fn(async () => { + throw new Error("Board-owned work must remain pending"); + }); + const service = recoveryService(db, { enqueueWakeup }); + const events: Array> = []; + const now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(db, { issues, issueRecoveryActions }, capture); + try { + await service.reconcileStrandedAssignedIssues(); + } finally { + capture.flush(); + } + expect(Number(events.at(-1)?.completedQueries)).toBeGreaterThan(5); + expect(Number(events.at(-1)?.responseValueBytes)).toBeLessThan(16_384); + expect(enqueueWakeup).not.toHaveBeenCalled(); + expect((await db.select().from(issues).where(eq(issues.id, parent.id)))[0]).toEqual(parent); + if (child) { + expect((await db.select().from(issues).where(eq(issues.id, child.id)))[0]).toEqual(child); + } + expect((await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, action.id)))[0]).toEqual(action); + }); + + it("keeps a candidate waiting for agent setup without reading its description", async () => { + const [company] = await db.insert(companies).values({ + name: "Recovery candidate read", + issuePrefix: `R${randomUUID().slice(0, 7).toUpperCase()}`, + }).returning(); + const [agent] = await db.insert(agents).values({ + companyId: company.id, + name: "Preparing agent", + lifecycleState: "preparing", + }).returning(); + const [issue] = await db.insert(issues).values({ + companyId: company.id, + title: "Wait for agent setup", + description: randomBytes(128 * 1024).toString("base64"), + status: "todo", + assigneeAgentId: agent.id, + }).returning(); + const enqueueWakeup = vi.fn(async () => { + throw new Error("Setup still owns this task"); + }); + const service = recoveryService(db, { enqueueWakeup }); + const events: Array> = []; + const now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(db, { agents, issues, issueRecoveryActions }, capture); + try { + expect(await service.reconcileStrandedAssignedIssues()).toMatchObject({ skipped: 1 }); + } finally { + capture.flush(); + } + expect(Number(events.at(-1)?.completedQueries)).toBeGreaterThan(5); + expect(Number(events.at(-1)?.responseValueBytes)).toBeLessThan(16_384); + expect(enqueueWakeup).not.toHaveBeenCalled(); + expect((await db.select().from(issues).where(eq(issues.id, issue.id)))[0]).toEqual(issue); + expect((await db.select().from(agents).where(eq(agents.id, agent.id)))[0]).toEqual(agent); + expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.companyId, company.id))).toEqual([]); + }); +}); diff --git a/server/src/services/recovery/legacy-disposition-read.test.ts b/server/src/services/recovery/legacy-disposition-read.test.ts new file mode 100644 index 00000000000..749a7991879 --- /dev/null +++ b/server/src/services/recovery/legacy-disposition-read.test.ts @@ -0,0 +1,88 @@ +import { randomBytes, randomUUID } from "node:crypto"; +import { agents, companies, createDb, heartbeatRuns } from "@paperclipai/db"; +import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture"; +import { eq } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; +import { readLegacyDispositionSource } from "./legacy-disposition-read.js"; +import { recoveryService } from "./service.js"; + +const support = await getEmbeddedPostgresTestSupport(); + +describe.skipIf(!support.supported)("legacy disposition source reads", () => { + let database: Awaited>; + let db: ReturnType; + let companyId: string; + let agentId: string; + + beforeAll(async () => { + database = await startEmbeddedPostgresTestDatabase("legacy-disposition-read-"); + db = createDb(database.connectionString); + const [company] = await db.insert(companies).values({ name: "Legacy disposition fixture", issuePrefix: "LDF" }).returning(); + const [agent] = await db.insert(agents).values({ companyId: company.id, name: "Disposition agent", adapterType: "process" }).returning(); + companyId = company.id; + agentId = agent.id; + }, 60_000); + + afterAll(async () => { + await db?.$client.end(); + await database?.cleanup(); + }); + + it("declines native input below the byte budget and keeps complete legacy evidence", async () => { + const issueId = randomUUID(), bulk = randomBytes(128 * 1024).toString("base64"); + const context = { issueId, savedInput: bulk, executionRetryAccounting: { version: 1, failureRetries: 2, maxTurnContinuations: 1 } }; + const [before] = await db.insert(heartbeatRuns).values({ companyId, agentId, runtimeMode: "native", status: "succeeded", + contextSnapshot: context, runnerProfileJson: { savedInput: bulk }, stdoutExcerpt: bulk, stderrExcerpt: bulk, + errorCode: "adapter_failed", resultJson: { executionRecovery: { providerWorkStarted: true }, stdout: bulk } }).returning(); + const events: Array> = []; + const now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(db, { heartbeatRuns }, capture); + try { + expect(await readLegacyDispositionSource(db, { companyId, issueId, runId: before.id })) + .toEqual({ kind: "skip", decision: { kind: "skip", reason: "native_finalization" } }); + } finally { capture.flush(); } + expect(events.at(-1)).toMatchObject({ completedQueries: 1, groups: 1 }); + expect(Number(events.at(-1)?.responseValueBytes)).toBeLessThan(1024); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, before.id)))[0]).toEqual(before); + expect(JSON.stringify(events)).not.toContain(bulk.slice(0, 64)); + await db.update(heartbeatRuns).set({ runtimeMode: "legacy" }).where(eq(heartbeatRuns.id, before.id)); + const legacy = await readLegacyDispositionSource(db, { companyId, issueId, runId: before.id }); + expect(legacy.kind).toBe("full"); + if (legacy.kind !== "full") throw new Error("expected complete legacy input"); + expect(legacy.run).toEqual({ ...before, runtimeMode: "legacy" }); + expect((await readLegacyDispositionSource(db, { companyId: randomUUID(), issueId, runId: before.id }))) + .toMatchObject({ kind: "full", run: undefined }); + }); + + it.each([ + { context: { issueId: " target " }, issueId: "target", reason: "invalid_issue_binding" }, + { context: { issueId: " ", taskId: "target" }, issueId: "target", reason: "native_finalization" }, + { context: { taskId: " target " }, issueId: "target", reason: "invalid_issue_binding" }, + { context: { issueId: 42, taskId: "target" }, issueId: "target", reason: "native_finalization" }, + { context: { issueId: "other", taskId: "target" }, issueId: "target", reason: "invalid_issue_binding" }, + { context: { issueId: null, taskId: false }, issueId: "target", reason: "invalid_issue_binding" }, + { context: null, issueId: "target", reason: "invalid_issue_binding" }, + ])("keeps native binding decisions for $context", async ({ context, issueId, reason }) => { + const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId, runtimeMode: "native", contextSnapshot: context }).returning(); + expect(await readLegacyDispositionSource(db, { companyId, issueId, runId: run.id })) + .toEqual({ kind: "skip", decision: { kind: "skip", reason } }); + }); + + it("skips native continuation in SQL without loading or rewriting its saved input", async () => { + const bulk = randomBytes(128 * 1024).toString("base64"); + const [before] = await db.insert(heartbeatRuns).values({ companyId, agentId, runtimeMode: "native", status: "succeeded", + contextSnapshot: { issueId: randomUUID(), savedInput: bulk }, runnerProfileJson: { savedInput: bulk } }).returning(); + const enqueueWakeup = vi.fn(async () => { throw new Error("native work must not enter legacy recovery"); }); + const service = recoveryService(db, { enqueueWakeup }); + const events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(db, { heartbeatRuns }, capture); + try { expect(await service.reconcileLegacyContinuation(before.id)).toBe("skipped"); } + finally { capture.flush(); } + expect(events.at(-1)).toMatchObject({ completedQueries: 1, responseValueBytes: 0 }); + expect(enqueueWakeup).not.toHaveBeenCalled(); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, before.id)))[0]).toEqual(before); + }); +}); diff --git a/server/src/services/recovery/legacy-disposition-read.ts b/server/src/services/recovery/legacy-disposition-read.ts new file mode 100644 index 00000000000..08f9204f491 --- /dev/null +++ b/server/src/services/recovery/legacy-disposition-read.ts @@ -0,0 +1,24 @@ +import { heartbeatRuns, type Db } from "@paperclipai/db"; +import { and, eq, getTableColumns, sql } from "drizzle-orm"; +import { recoveryRunResultColumn } from "./run-result-read.js"; + +export async function readLegacyDispositionSource( + db: Db, + scope: { companyId: string; issueId: string; runId: string }, +) { + const where = and(eq(heartbeatRuns.id, scope.runId), eq(heartbeatRuns.companyId, scope.companyId)); + const [native] = await db.select({ + issueId: sql`${heartbeatRuns.contextSnapshot}->'issueId'`, + taskId: sql`${heartbeatRuns.contextSnapshot}->'taskId'`, + }).from(heartbeatRuns).where(and(where, eq(heartbeatRuns.runtimeMode, "native"))).limit(1); + if (native) { + const bound = [native.issueId, native.taskId] + .map(value => typeof value === "string" && value.trim().length > 0 ? value : null).find(Boolean) ?? null; + return { kind: "skip" as const, decision: { + kind: "skip" as const, reason: bound === scope.issueId ? "native_finalization" : "invalid_issue_binding", + } }; + } + const [run] = await db.select({ ...getTableColumns(heartbeatRuns), resultJson: recoveryRunResultColumn }) + .from(heartbeatRuns).where(where).limit(1); + return { kind: "full" as const, run }; +} diff --git a/server/src/services/recovery/provider-failure-classification.test.ts b/server/src/services/recovery/provider-failure-classification.test.ts index 31407ac82c3..29dded8d4d6 100644 --- a/server/src/services/recovery/provider-failure-classification.test.ts +++ b/server/src/services/recovery/provider-failure-classification.test.ts @@ -10,6 +10,10 @@ import { legacyExecutionNeedsReconciliation } from "../legacy-execution-recovery import { isConfigurationIncompleteFailedRun } from "../../modules/wake-queue/domain/values.js"; describe("classifyAdapterFailureForRecovery", () => { + it("does not automatically replay a project access denial", () => { + expect(classifyContinuationFailure({ errorCode: "project_access_denied", resultJson: null } as never)) + .toMatchObject({ kind: "non_retryable", maxAttempts: 0 }); + }); it("waits for model configuration changes instead of retrying a typed native model rejection", () => { const run = { errorCode: "native_provider_model_rejected", error: "Fixed model configuration message", resultJson: null }; expect(classifyAdapterFailureForRecovery(run)).toEqual({ kind: "configuration_incomplete" }); diff --git a/server/src/services/recovery/reconciliation-run-read.test.ts b/server/src/services/recovery/reconciliation-run-read.test.ts new file mode 100644 index 00000000000..ff42eb35aac --- /dev/null +++ b/server/src/services/recovery/reconciliation-run-read.test.ts @@ -0,0 +1,107 @@ +import { randomBytes, randomUUID } from "node:crypto"; +import { agents, agentWakeupRequests, companies, createDb, heartbeatRuns, issueRecoveryActions, issues } from "@paperclipai/db"; +import { and, eq } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; +import { executionFailureRetryCount } from "../execution-recovery-attempt.js"; +import { legacyExecutionNeedsReconciliationWithEvidence, legacyReconciliationRunFields, terminalizeLegacyExecution } from "../legacy-execution-recovery.js"; + +const support = await getEmbeddedPostgresTestSupport(); +describe.skipIf(!support.supported)("reconciliation source PostgreSQL reads", () => { + let temporary: Awaited>; + let db: ReturnType; + let companyId: string, agentId: string; + const launchKeys = ["executionContinuation", "paperclipWake", "paperclipWakeComment", "paperclipTaskMarkdown", + "paperclipTaskMarkdownCompact", "paperclipTaskAssignment", "paperclipTaskAssignmentCompact", "paperclipWorkspace", + "paperclipEnvironment", "paperclipContinuationSummary"]; + beforeAll(async () => { + temporary = await startEmbeddedPostgresTestDatabase("reconciliation-source-"); + db = createDb(temporary.connectionString); + companyId = randomUUID(); agentId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Recovery reads", issuePrefix: "RCR" }); + await db.insert(agents).values({ id: agentId, companyId, name: "Recovery worker", adapterType: "process", status: "active" }); + }); + afterAll(async () => { await db?.$client.end(); await temporary?.cleanup(); }); + + async function fixture(patch: Partial = {}) { + const payload = randomBytes(64 * 1024).toString("base64"); + const contextSnapshot = { ...Object.fromEntries(launchKeys.map(key => [key, { prompt: payload }])), + issueId: randomUUID(), taskId: randomUUID(), executionWorkspaceId: randomUUID(), + executionRetryAccounting: { version: 1, failureRetries: 2, maxTurnContinuations: 3 }, + failureRetriesBeforeAiConnectionWait: 1, failureRetriesBeforeWorkspaceWait: 1, + conversationSessionGeneration: 7, connectionIntentResolution: "existing_connection", source: "connection_intent.resolved", + interactionKind: "connection_intent", interactionStatus: "expired", mutation: "interaction", interactionId: randomUUID(), + futureControl: { nonce: "retained", nested: null }, cancellation: { state: "pending" }, + }; + const [full] = await db.insert(heartbeatRuns).values({ companyId, agentId, status: "failed", runtimeMode: "legacy", + errorCode: "process_lost", contextSnapshot, resultJson: { futureOutput: { retained: true } }, + runnerProfileJson: { adapterDispatch: { adapterType: "process" }, futureControl: { retained: true } }, + ...patch, + }).returning(); + const [selected] = await db.select(legacyReconciliationRunFields).from(heartbeatRuns) + .where(and(eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.id, full!.id))); + return { full: full!, selected: selected! }; + } + + it.each(["native", "legacy"])("omits unused launch text and preserves current control fields (%s)", async runtimeMode => { + const profile = { nativeExecutionInput: { prompt: randomBytes(128 * 1024).toString("base64") }, futureControl: { retained: true } }; + const f = await fixture({ runtimeMode, ...(runtimeMode === "native" ? { runnerProfileJson: profile } : {}) }); + const expected = { ...f.full.contextSnapshot }; + for (const key of launchKeys) delete expected[key]; + expect(f.selected).toEqual({ ...f.full, contextSnapshot: expected, + runnerProfileJson: runtimeMode === "native" ? { futureControl: profile.futureControl } : f.full.runnerProfileJson }); + expect(Buffer.byteLength(JSON.stringify(f.full))).toBeGreaterThan(800_000); + expect(Buffer.byteLength(JSON.stringify(f.selected))).toBeLessThan(8192); + expect(executionFailureRetryCount(f.selected)).toBe(executionFailureRetryCount(f.full)); + expect(await legacyExecutionNeedsReconciliationWithEvidence(db, f.selected)) + .toBe(await legacyExecutionNeedsReconciliationWithEvidence(db, f.full)); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.full.id)))[0]).toEqual(f.full); + }); + + it.each([null, [], ["unexpected"], "scalar", 42].map(value => ({ value })))("preserves non-object JSON $value", async ({ value }) => { + for (const runtimeMode of ["native", "legacy"]) { + const f = await fixture({ runtimeMode, + contextSnapshot: value as typeof heartbeatRuns.$inferInsert.contextSnapshot, + runnerProfileJson: value as typeof heartbeatRuns.$inferInsert.runnerProfileJson, + }); + expect(f.selected).toEqual(f.full); + } + }); + + it.each([ + { status: "failed", errorCode: "setup_failed", resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } } }, + { status: "cancelled", errorCode: "cancelled", resultJson: { executionCancellation: { state: "acknowledged" }, + executionRecovery: { kind: "interrupted", providerStopped: true, sessionPreserved: true, actionOutcomes: "settled" } } }, + { status: "cancelled", errorCode: "cancelled", resultJson: { executionCancellation: { state: "requested" }, + executionRecovery: { kind: "interrupted", providerStopped: false, sessionPreserved: true, actionOutcomes: "unknown" } } }, + { status: "cancelled", errorCode: "ai_connection_busy", resultJson: { executionRecovery: { kind: "ai_connection_wait", providerWorkStarted: false } } }, + { status: "cancelled", errorCode: "workspace_busy", resultJson: { executionRecovery: { kind: "workspace_wait", providerWorkStarted: false } } }, + { status: "failed", errorCode: "workspace_restore_failed", resultJson: { workspaceRestoreFailure: "restore_unsafe_archive" } }, + { status: "failed", errorCode: "process_lost", resultJson: { conversationContinuation: "continue_conversation_v1" } }, + { status: "cancelled", errorCode: "issue_waiting_for_response", resultJson: { stopReason: "issue_waiting_for_response", timeoutSource: "stale_queued_run_gate" } }, + ])("keeps reconciliation, cancellation, and retry decisions ($errorCode)", async patch => { + const f = await fixture(patch); + expect(f.selected.resultJson).toEqual(f.full.resultJson); + expect(executionFailureRetryCount(f.selected)).toBe(executionFailureRetryCount(f.full)); + expect(await legacyExecutionNeedsReconciliationWithEvidence(db, f.selected)) + .toBe(await legacyExecutionNeedsReconciliationWithEvidence(db, f.full)); + }); + + it("keeps full stored input and newer ownership when the projected source is terminalized", async () => { + const issueId = randomUUID(); + await db.insert(issues).values({ id: issueId, companyId, title: "Recover the stopped source", status: "in_progress", assigneeAgentId: agentId }); + const f = await fixture({ issueId, contextSnapshot: { issueId, paperclipTaskMarkdown: randomBytes(128 * 1024).toString("base64"), + executionRetryAccounting: { version: 1, failureRetries: 2, maxTurnContinuations: 0 } } }); + const [successor] = await db.insert(heartbeatRuns).values({ companyId, agentId, issueId, status: "running" }).returning(); + await db.update(issues).set({ executionRunId: successor!.id, checkoutRunId: successor!.id }).where(eq(issues.id, issueId)); + await terminalizeLegacyExecution({ db, run: f.selected, status: "failed", fromStatuses: ["failed"] }); + const [stored] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.full.id)); + expect(stored).toMatchObject({ contextSnapshot: f.full.contextSnapshot, runnerProfileJson: f.full.runnerProfileJson, resultJson: f.full.resultJson }); + expect((await db.select().from(issues).where(eq(issues.id, issueId)))[0]) + .toMatchObject({ executionRunId: successor!.id, checkoutRunId: successor!.id }); + expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId))) + .toEqual([expect.objectContaining({ cause: "legacy_execution_requires_reconciliation", ownerType: "board", + evidence: expect.objectContaining({ runId: f.full.id, attempt: 3 }) })]); + expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, companyId))).toEqual([]); + }); +}); diff --git a/server/src/services/recovery/service.ts b/server/src/services/recovery/service.ts index e72811ec8d0..4b21e406c4e 100644 --- a/server/src/services/recovery/service.ts +++ b/server/src/services/recovery/service.ts @@ -1,5 +1,6 @@ import { recoveryRunContextColumn, recoveryRunVersionColumn } from "./run-context-read.js"; import { recoveryRunResultColumn } from "./run-result-read.js"; +import { readLegacyDispositionSource } from "./legacy-disposition-read.js"; import { isAgentAwaitingSetup } from "../../modules/agent-lifecycle/index.js"; import { onboardingFirstTaskAwaitingUserSql } from "./onboarding-first-task-wait.js"; import { isAiAuthenticationBlocked } from "../ai-auth-failure.js"; @@ -20,7 +21,7 @@ import { import { CONTROLLER_LEASE_LOST_FAILURE_CODE, hasLiveLegacyController } from "../legacy-controller-lease.js"; import { isCompletedOnboardingHandoffWake } from "../chat-completion-delivery.js"; import { instanceSettingsService } from "../instance-settings.js"; -import { isWaitingConversation, settleConversationTurn, deliverConversationComments } from "../agent-conversations.js"; +import { isWaitingConversation, settleConversationTurnForRun, deliverConversationComments } from "../agent-conversations.js"; import { and, asc, @@ -105,6 +106,7 @@ import { issueRecoveryActionService } from "../issue-recovery-actions.js"; import { hasUnverifiedNativeWorkspaceFinalizationOwner } from "../native-runtime/native-workspace-finalization-recovery-state.js"; import { legacyExecutionNeedsReconciliationWithEvidence, + legacyReconciliationRunFields, terminalizeLegacyExecution, } from "../legacy-execution-recovery.js"; import { issueTreeControlService } from "../issue-tree-control.js"; @@ -157,6 +159,7 @@ import { withRecoveryContext } from "./status-only-context.js"; import { isAutomaticRecoverySuppressedByPauseHold } from "./pause-hold-guard.js"; import { collectDispositionRepairSourceState, + dispositionRepairIssueFields, dispositionRepairDelayMs, DISPOSITION_REPAIR_MAX_ATTEMPTS, } from "./disposition-repair.js"; @@ -516,6 +519,7 @@ const TRANSIENT_INFRA_CONTINUATION_ERROR_CODES = new Set([ ]); const NON_RETRYABLE_CONTINUATION_ERROR_CODES = new Set([ + "project_access_denied", "native_provider_model_rejected", "provider_tool_definition_invalid", "adapter_engine_unavailable", @@ -860,7 +864,7 @@ function formatIssueLinksForComment( } function isStrandedIssueRecoveryIssue( - issue: Pick, + issue: Pick, ) { return isStrandedIssueRecoveryOriginKind(issue.originKind); } @@ -932,6 +936,10 @@ function isRepeatedProductiveContinuationRecovery( ); } +type RecoveryIssue = Omit; + +const { description: _unusedIssueDescription, ...recoveryIssueFields } = getTableColumns(issues); + export function recoveryService( db: Db, deps: { @@ -1247,7 +1255,7 @@ export function recoveryService( * Keep it separate from the broader monitor/delegated/legacy wait predicate. */ async function hasCurrentNativePassiveWait( - issue: typeof issues.$inferSelect, + issue: RecoveryIssue, latestRun: LatestIssueRun, ): Promise { if ( @@ -1457,7 +1465,7 @@ export function recoveryService( } async function hasPersistedDurableWaitPath( - issue: typeof issues.$inferSelect, + issue: RecoveryIssue, latestRun: LatestIssueRun, ) { if (issue.monitorNextCheckAt) return true; @@ -1522,7 +1530,7 @@ export function recoveryService( } async function nativeBlockedUnblockAction( - issue: typeof issues.$inferSelect, + issue: RecoveryIssue, latestRun: LatestIssueRun, database: Db = db, ) { @@ -1593,7 +1601,7 @@ export function recoveryService( } async function repairNativeBlockedWait( - issue: typeof issues.$inferSelect, + issue: RecoveryIssue, latestRun: NonNullable, candidate: NonNullable< Awaited> @@ -1797,7 +1805,7 @@ export function recoveryService( } async function wasTodoHandedBackDuringOrAfterLatestRun( - issue: typeof issues.$inferSelect, + issue: RecoveryIssue, latestRun: LatestIssueRun, ) { if (issue.status !== "todo" || latestRun?.status !== "succeeded") @@ -2008,7 +2016,7 @@ export function recoveryService( }) { if (input.retryOfRunId) { const [predecessor] = await db - .select() + .select(legacyReconciliationRunFields) .from(heartbeatRuns) .where( and( @@ -2129,7 +2137,7 @@ export function recoveryService( } async function enqueueInitialAssignedTodoDispatch( - issue: typeof issues.$inferSelect, + issue: RecoveryIssue, agentId: string, ) { return deps.enqueueWakeup(agentId, { @@ -2162,7 +2170,7 @@ export function recoveryService( // contract is that nothing runs until the user types. Until a user-authored // comment exists on it, the issue is intentionally idle rather than stranded. async function isOnboardingFirstTaskAwaitingUser( - issue: typeof issues.$inferSelect, + issue: RecoveryIssue, ) { if (issue.originKind !== ONBOARDING_FIRST_TASK_ORIGIN_KIND) return false; const [result] = await db.execute<{ awaiting: boolean }>(sql`SELECT @@ -2171,7 +2179,7 @@ export function recoveryService( } async function isInvocationBudgetBlocked( - issue: typeof issues.$inferSelect, + issue: RecoveryIssue, agentId: string, ) { const budgetBlock = await budgets.getInvocationBlock( @@ -2405,12 +2413,12 @@ export function recoveryService( } } - function isStrandedIssueRecoveryIssue(issue: typeof issues.$inferSelect) { + function isStrandedIssueRecoveryIssue(issue: RecoveryIssue) { return issue.originKind === STRANDED_ISSUE_RECOVERY_ORIGIN_KIND; } async function buildNestedStrandedRecoveryLine( - issue: typeof issues.$inferSelect, + issue: RecoveryIssue, prefix: string, ) { const sourceIssueId = readNonEmptyString(issue.originId); @@ -2441,7 +2449,7 @@ export function recoveryService( } function resolveStrandedRecoveryRouting(input: { - issue: typeof issues.$inferSelect; + issue: RecoveryIssue; latestRun: LatestIssueRun; }) { const originalAgentId = @@ -2464,7 +2472,7 @@ export function recoveryService( } function strandedRecoveryActionFingerprint(input: { - issue: typeof issues.$inferSelect; + issue: RecoveryIssue; recoveryCause: StrandedRecoveryCause; latestRun: LatestIssueRun; }) { @@ -2510,7 +2518,7 @@ export function recoveryService( } function buildStrandedRecoveryActionEvidence(input: { - issue: typeof issues.$inferSelect; + issue: RecoveryIssue; latestRun: LatestIssueRun; previousStatus: StrandedPreviousStatus; recoveryCause: StrandedRecoveryCause; @@ -2545,7 +2553,7 @@ export function recoveryService( } async function ensureSourceScopedStrandedRecoveryAction(input: { - issue: typeof issues.$inferSelect; + issue: RecoveryIssue; latestRun: LatestIssueRun; previousStatus: StrandedPreviousStatus; recoveryCause?: StrandedRecoveryCause; @@ -2676,7 +2684,7 @@ export function recoveryService( } async function ensureProviderQuotaWaitRecoveryMonitor(input: { - issue: typeof issues.$inferSelect; + issue: RecoveryIssue; latestRun: LatestIssueRun; actionId: string; agentId: string; @@ -2776,7 +2784,7 @@ export function recoveryService( } function buildRecoveryIssueInPlaceEscalationComment(input: { - issue: typeof issues.$inferSelect; + issue: RecoveryIssue; previousStatus: StrandedPreviousStatus; latestRun: LatestIssueRun; prefix: string; @@ -2811,7 +2819,7 @@ export function recoveryService( } async function escalateStrandedRecoveryIssueInPlace(input: { - issue: typeof issues.$inferSelect; + issue: RecoveryIssue; previousStatus: StrandedPreviousStatus; latestRun: LatestIssueRun; }, transaction?: { @@ -2950,7 +2958,7 @@ export function recoveryService( ); } - async function openChildIssues(issue: typeof issues.$inferSelect) { + async function openChildIssues(issue: RecoveryIssue) { return db .select({ id: issues.id, identifier: issues.identifier }) .from(issues) @@ -2964,10 +2972,13 @@ export function recoveryService( ); } - async function healthyOpenChildIssues(issue: typeof issues.$inferSelect, sameWorkspaceOnly = false) { + async function healthyOpenChildIssues( + issue: Pick, + sameWorkspaceOnly = false, + ) { if (sameWorkspaceOnly && !issue.projectWorkspaceId) return []; const childCandidates = await db - .select() + .select({ ...dispositionRepairIssueFields, identifier: issues.identifier }) .from(issues) .where( and( @@ -2994,7 +3005,7 @@ export function recoveryService( } async function resolveContinuationWaitingOnReview( - issue: typeof issues.$inferSelect, + issue: RecoveryIssue, ) { const [existingBlockers, openChildren] = await Promise.all([ existingUnresolvedBlockerIssues(issue.companyId, issue.id), @@ -3095,7 +3106,7 @@ export function recoveryService( } async function resolveDispositionRepairActionAsCovered( - issue: typeof issues.$inferSelect, + issue: RecoveryIssue, reason: string, ) { const active = await recoveryActionsSvc.getActiveForIssue( @@ -3129,7 +3140,7 @@ export function recoveryService( } async function ensureDispositionRepairAction(input: { - issue: typeof issues.$inferSelect; + issue: RecoveryIssue; latestRun: LatestIssueRun; fingerprint: string; attemptCount: number; @@ -3207,7 +3218,7 @@ export function recoveryService( } async function scheduleDispositionRepairAttempt(input: { - issue: typeof issues.$inferSelect; + issue: RecoveryIssue; latestRun: LatestIssueRun; action: Awaited>; fingerprint: string; @@ -3480,7 +3491,10 @@ export function recoveryService( async function reconcileActiveRecoveryActions() { const rows = await db - .select({ action: issueRecoveryActions, issue: issues }) + .select({ + action: issueRecoveryActions, + issue: { ...dispositionRepairIssueFields, projectWorkspaceId: issues.projectWorkspaceId }, + }) .from(issueRecoveryActions) .innerJoin( issues, @@ -3641,7 +3655,7 @@ export function recoveryService( } async function escalateDispositionRepair(input: { - issue: typeof issues.$inferSelect; + issue: RecoveryIssue; latestRun: LatestIssueRun; fingerprint: string; attemptCount: number; @@ -3786,15 +3800,15 @@ export function recoveryService( } async function decidePersistedLegacyContinuation( - issue: typeof issues.$inferSelect, + issue: RecoveryIssue, runId: string, state: Awaited>, episode: LegacyDispositionEpisode, dispatchRunId?: string, ) { - const [run] = await db.select(recoveryRunFields).from(heartbeatRuns).where(and( - eq(heartbeatRuns.id, runId), eq(heartbeatRuns.companyId, issue.companyId), - )).limit(1); + const source = await readLegacyDispositionSource(db, { companyId: issue.companyId, issueId: issue.id, runId }); + if (source.kind === "skip") return source.decision; + const run = source.run; if (!run) return { kind: "skip" as const, reason: "missing_run" }; const context = parseObject(run.contextSnapshot); if ((readNonEmptyString(context.issueId) ?? readNonEmptyString(context.taskId)) !== issue.id) { @@ -3853,7 +3867,9 @@ export function recoveryService( } async function reconcileLegacyContinuation(runId: string) { - const [run] = await db.select(recoveryRunFields).from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)).limit(1); + const [run] = await db.select(recoveryRunFields).from(heartbeatRuns).where(and( + eq(heartbeatRuns.id, runId), not(eq(heartbeatRuns.runtimeMode, "native")), eq(heartbeatRuns.status, "succeeded"), + )).limit(1); if (!run || run.runtimeMode === "native" || run.status !== "succeeded") return "skipped" as const; const context = parseObject(run.contextSnapshot); const issueId = readNonEmptyString(context.issueId) ?? readNonEmptyString(context.taskId); @@ -3864,7 +3880,7 @@ export function recoveryService( } async function reconcileDispositionRepair( - issue: typeof issues.$inferSelect, + issue: Pick, latestRun: LatestIssueRun, options: { historicalAttemptCount?: number; legacyEpisode?: LegacyDispositionEpisode } = {}, ): Promise<"queued" | "escalated" | "covered" | "skipped"> { @@ -4011,7 +4027,7 @@ export function recoveryService( } async function escalateStrandedAssignedIssue(input: { - issue: typeof issues.$inferSelect; + issue: RecoveryIssue; previousStatus: StrandedPreviousStatus; latestRun: LatestIssueRun; comment?: string; @@ -4378,7 +4394,7 @@ export function recoveryService( } async function scheduleProviderQuotaRecoveryMonitor(input: { - issue: typeof issues.$inferSelect; + issue: RecoveryIssue; latestRun: NonNullable; classification: Extract< NonNullable, @@ -4461,7 +4477,7 @@ export function recoveryService( } function getAdapterFailureRecoveryTargetAgentId( - issue: typeof issues.$inferSelect, + issue: RecoveryIssue, ) { if (issue.status !== "in_review") return issue.assigneeAgentId; @@ -4476,7 +4492,7 @@ export function recoveryService( } function hasPendingProviderQuotaRecoveryMonitor( - issue: typeof issues.$inferSelect, + issue: RecoveryIssue, latestRun: LatestIssueRun, now: Date, ) { @@ -4498,7 +4514,7 @@ export function recoveryService( issueCreatedAtGte?: Date | null; }) { const candidates = await db - .select() + .select(recoveryIssueFields) .from(issues) .where( and( @@ -4577,7 +4593,7 @@ export function recoveryService( if (issue.conversationAgentId) { const lastRun = await getLatestIssueRun(issue.companyId, issue.id); if (lastRun?.status === "succeeded") { - if (await settleConversationTurn(db, (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, lastRun.id)))[0]!)) { + if (await settleConversationTurnForRun(db, issue.companyId, lastRun.id)) { const [current] = await db.select().from(issues).where(eq(issues.id, issue.id)); if (current) Object.assign(issue, current); } @@ -4805,7 +4821,7 @@ export function recoveryService( ) ) { const [source] = await db - .select() + .select(legacyReconciliationRunFields) .from(heartbeatRuns) .where( and( diff --git a/server/src/services/run-identity.ts b/server/src/services/run-identity.ts index cc75397a991..2f1e9e97d65 100644 --- a/server/src/services/run-identity.ts +++ b/server/src/services/run-identity.ts @@ -1,4 +1,4 @@ -import { and, asc, desc, eq, inArray, sql } from "drizzle-orm"; +import { and, asc, desc, eq, getTableColumns, inArray, sql } from "drizzle-orm"; import { agentWakeupRequests, heartbeatRuns, @@ -434,12 +434,22 @@ export async function acceptSteeredIdentity( export async function captureRunIdentity( db: Db, input: { companyId: string; runId: string; agentId: string }, + options?: { readScope: "github_credentials" | "request_authentication" }, ) { // Lock acquisition serializes with steering delivery and its durable acknowledgement. return db.transaction(async (tx) => { await lockIdentityTask(tx, input.companyId, input.runId); const [run] = await tx - .select() + .select(options?.readScope === "github_credentials" || options?.readScope === "request_authentication" ? { + ...getTableColumns(heartbeatRuns), + contextSnapshot: sql | null>`case when jsonb_typeof(${heartbeatRuns.contextSnapshot}) = 'object' + then jsonb_build_object('issueId', ${heartbeatRuns.contextSnapshot}->'issueId', + 'taskId', ${heartbeatRuns.contextSnapshot}->'taskId', 'projectId', ${heartbeatRuns.contextSnapshot}->'projectId', + 'executionPolicy', ${heartbeatRuns.contextSnapshot}->'executionPolicy') else null end`, + runnerProfileJson: sql | null>`null`, + resultJson: sql | null>`null`, + stdoutExcerpt: sql`null`, stderrExcerpt: sql`null`, + } : getTableColumns(heartbeatRuns)) .from(heartbeatRuns) .where( and( diff --git a/server/src/services/run-log-store.test.ts b/server/src/services/run-log-store.test.ts index c4ad5900fea..aee7e202198 100644 --- a/server/src/services/run-log-store.test.ts +++ b/server/src/services/run-log-store.test.ts @@ -68,6 +68,26 @@ afterEach(async () => { const begin = { companyId: "co1", agentId: "ag1", runId: "run1" }; describe("createDurableRunLogStore", () => { + it.each([false, true])("reads a complete long final record from a compact tail (S3 fallback: %s)", async (remote) => { + const { provider, objects, calls } = createMemoryProvider(); + const store = createDurableRunLogStore({ basePath: baseDir, s3: { provider } }); + const handle = await store.begin(begin); + await store.append(handle, { stream: "stdout", chunk: "older tool output ".repeat(100_000), ts: "t1" }); + const answer = "Complete report 中文🧪\n".repeat(8_000); + await store.append(handle, { stream: "stdout", chunk: answer, ts: "t2" }); + await store.finalize(handle); + const bytes = objects.get(handle.logRef)!; + // Cut inside a multibyte character to exercise byte alignment, not just + // ASCII JSON reconstruction. The earlier large record must stay omitted. + const offset = bytes.lastIndexOf(Buffer.from("🧪"), bytes.length - 32_000) + 1; + if (remote) await fs.rm(baseDir, { recursive: true, force: true }); + const result = await store.read(handle, { offset, limitBytes: bytes.length - offset, alignStartToRecord: true }); + expect(result.content.trim().split("\n").map((line) => JSON.parse(line).chunk)).toEqual([answer]); + expect(result.nextOffset).toBe(bytes.length); + expect(await store.read(handle, { offset: result.nextOffset })).toMatchObject({ content: "" }); + expect(calls.get).toBeLessThan(12); + }); + it("keeps store id 'local_file' so downstream coupling (feedback, casts) is unchanged", async () => { const { provider } = createMemoryProvider(); const store = createDurableRunLogStore({ basePath: baseDir, s3: { provider } }); diff --git a/server/src/services/run-log-store.ts b/server/src/services/run-log-store.ts index 7e26b15261e..2c11aade946 100644 --- a/server/src/services/run-log-store.ts +++ b/server/src/services/run-log-store.ts @@ -19,6 +19,8 @@ export interface RunLogHandle { export interface RunLogReadOptions { offset?: number; limitBytes?: number; + /** Include the start of the first NDJSON record when reading a log tail. */ + alignStartToRecord?: boolean; signal?: AbortSignal; } @@ -27,6 +29,11 @@ export interface RunLogReadResult { nextOffset?: number; } +interface RunLogByteRange { + bytes: Buffer; + nextOffset?: number; +} + export interface RunLogFinalizeSummary { // Null means a stalled write prevented a verified final snapshot. Callers // can still settle the run without recording a false byte count or hash. @@ -230,7 +237,7 @@ export function createDurableRunLogStore(options: DurableRunLogStoreOptions): Ru offset: number, limitBytes: number, signal?: AbortSignal, - ): Promise { + ): Promise { signal?.throwIfAborted(); const start = Math.max(0, offset); // Read one extra byte to discover whether another page exists. A single @@ -249,9 +256,8 @@ export function createDurableRunLogStore(options: DurableRunLogStoreOptions): Ru } signal?.throwIfAborted(); const bytes = Buffer.concat(chunks); - const content = bytes.subarray(0, limitBytes).toString("utf8"); const nextOffset = bytes.length > limitBytes ? start + limitBytes : undefined; - return { content, nextOffset }; + return { bytes: bytes.subarray(0, limitBytes), nextOffset }; } async function readS3Range( @@ -259,7 +265,7 @@ export function createDurableRunLogStore(options: DurableRunLogStoreOptions): Ru offset: number, limitBytes: number, signal?: AbortSignal, - ): Promise { + ): Promise { signal?.throwIfAborted(); if (!s3) throw notFound("Run log not found"); const key = s3Key(logRef); @@ -273,7 +279,7 @@ export function createDurableRunLogStore(options: DurableRunLogStoreOptions): Ru // must short-circuit to an empty read instead of clamping end up to // start and requesting `bytes=total-total`. const end = Math.min(start + limitBytes - 1, total - 1); - if (total === 0 || start > end) return { content: "", nextOffset: start < total ? start : undefined }; + if (total === 0 || start > end) return { bytes: Buffer.alloc(0), nextOffset: start < total ? start : undefined }; const result = await s3.provider.getObject({ objectKey: key, range: { start, end }, signal }); // Destroy a body that stalls after headers arrive, including a body @@ -284,9 +290,8 @@ export function createDurableRunLogStore(options: DurableRunLogStoreOptions): Ru chunks.push(Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk)); } signal?.throwIfAborted(); - const content = Buffer.concat(chunks).toString("utf8"); const nextOffset = end + 1 < total ? end + 1 : undefined; - return { content, nextOffset }; + return { bytes: Buffer.concat(chunks), nextOffset }; } async function sha256File(filePath: string): Promise { @@ -435,12 +440,33 @@ export function createDurableRunLogStore(options: DurableRunLogStoreOptions): Ru opts?.signal?.throwIfAborted(); if (handle.store !== "local_file") throw notFound("Run log not found"); const absPath = resolveWithin(basePath, handle.logRef); - const offset = opts?.offset ?? 0; + const offset = Math.max(0, opts?.offset ?? 0); const limitBytes = opts?.limitBytes ?? 256_000; - const local = await readLocalRange(absPath, offset, limitBytes, opts?.signal); - if (local) return local; - // Local file gone (pod rolled) -> serve from the S3 mirror if configured. - return readS3Range(handle.logRef, offset, limitBytes, opts?.signal); + const readRange = async (start: number, length: number) => + await readLocalRange(absPath, start, length, opts?.signal) + // Local file gone (pod rolled) -> serve from the S3 mirror if configured. + ?? await readS3Range(handle.logRef, start, length, opts?.signal); + const result = await readRange(offset, limitBytes); + if (!opts?.alignStartToRecord || offset === 0 || result.bytes.length === 0) { + return { content: result.bytes.toString("utf8"), nextOffset: result.nextOffset }; + } + // Walk back only to the preceding record boundary. Keep bytes intact + // until decoding so a tail beginning inside UTF-8 text is not corrupted. + const parts = [result.bytes]; + let before = offset; + while (before > 0) { + const start = Math.max(0, before - 32_000); + const prefix = await readRange(start, before - start); + if (prefix.bytes.length !== before - start) throw new Error("Run log changed while reading its tail"); + const boundary = prefix.bytes.lastIndexOf(10); + parts.unshift(prefix.bytes.subarray(boundary + 1)); + if (boundary >= 0) break; + before = start; + } + return { + content: Buffer.concat(parts).toString("utf8"), + nextOffset: result.nextOffset ?? offset + result.bytes.length, + }; }, async flushInflightMirrors() { diff --git a/server/src/services/task-interruption-presentation.test.ts b/server/src/services/task-interruption-presentation.test.ts new file mode 100644 index 00000000000..9c730d1c5fc --- /dev/null +++ b/server/src/services/task-interruption-presentation.test.ts @@ -0,0 +1,55 @@ +import { describe, expect, it } from "vitest"; +import { projectTaskInterruption } from "./task-interruption-presentation.js"; + +const issue = { id: "task", companyId: "11111111-1111-4111-8111-111111111111", status: "blocked", assigneeAgentId: "agent", projectId: "22222222-2222-4222-8222-222222222222" }; +const run = { id: "run", agentId: "agent", status: "failed", scheduledRetryAt: null, + errorCode: "project_access_denied", error: null, livenessState: null, + resultJson: { projectAccessDenied: { projectId: "22222222-2222-4222-8222-222222222222" } } }; +const agent = { status: "idle", pauseReason: null }; + +describe("task project-access interruption", () => { + it("explains a project denial without offering a futile retry", () => { + expect(projectTaskInterruption(issue, { run, agent, paused: false, approval: false })).toMatchObject({ + state: "waiting_for_user", responsibility: "user", userAction: "project_access", canRetry: false, + projectAccess: { agentId: "agent", projectId: "22222222-2222-4222-8222-222222222222", ready: false }, + }); + }); + + it("projects the historical setup denial without requiring another failed run", () => { + expect(projectTaskInterruption(issue, { run: { ...run, errorCode: "setup_failed", + error: "Project is outside this low-trust boundary.", resultJson: null }, agent, paused: false, approval: false })) + .toMatchObject({ userAction: "project_access", canRetry: false }); + }); + + it("offers an explicit retry after the project is added, without changing the failed run", () => { + expect(projectTaskInterruption(issue, { run, agent, projectAccessAllowed: true, paused: false, approval: false })) + .toMatchObject({ state: "failed", canRetry: true, projectAccess: { ready: true } }); + expect(run.status).toBe("failed"); + }); + + it("shows an accepted retry as working after access is repaired", () => { + const input = { run, agent, paused: false, approval: false, wait: { id: "request", requestedAt: new Date() } }; + expect(projectTaskInterruption(issue, { ...input, projectAccessAllowed: true })) + .toMatchObject({ state: "working", canRetry: false }); + expect(projectTaskInterruption(issue, { ...input, projectAccessAllowed: true })).not.toHaveProperty("projectAccess"); + expect(projectTaskInterruption(issue, { ...input, projectAccessAllowed: false })) + .toMatchObject({ state: "waiting_for_user", userAction: "project_access", canRetry: false }); + }); + + it("does not reinterpret unrelated failures or an old assignee's denial", () => { + for (const otherRun of [{ ...run, errorCode: "setup_failed", error: "Disk full", resultJson: null }, + { ...run, agentId: "previous-agent" }]) { + expect(projectTaskInterruption(issue, { run: otherRun, agent, paused: false, approval: false })) + .not.toHaveProperty("projectAccess"); + } + }); + + it("keeps approval, pause, running and completed states authoritative", () => { + expect(projectTaskInterruption(issue, { run, agent, paused: false, approval: true })).toMatchObject({ userAction: "approval" }); + expect(projectTaskInterruption(issue, { run, agent, paused: true, approval: false })).toMatchObject({ userAction: "resume" }); + expect(projectTaskInterruption(issue, { run: { ...run, status: "running" }, agent, paused: false, approval: false })) + .toMatchObject({ state: "working" }); + expect(projectTaskInterruption({ ...issue, status: "done" }, { run, agent, paused: false, approval: false })) + .toMatchObject({ state: "resolved" }); + }); +}); diff --git a/server/src/services/task-interruption-presentation.ts b/server/src/services/task-interruption-presentation.ts index 249ff55d8e5..eb489f1ef6d 100644 --- a/server/src/services/task-interruption-presentation.ts +++ b/server/src/services/task-interruption-presentation.ts @@ -4,21 +4,38 @@ import { agents, agentWakeupRequests, approvals, companies, heartbeatRuns, issue import type { TaskInterruptionPresentation } from "@paperclipai/shared"; import { isAiAuthenticationBlocked } from "./ai-auth-failure.js"; import { readSystemRecovery } from "./system-recovery-state.js"; +import { PROJECT_ACCESS_DENIED } from "./project-access-denial.js"; +import { authorizationService } from "./authorization.js"; +import { taskWorkspaceProjectRequirements } from "./task-workspace-access.js"; type ProjectionIssue = Pick; type ProjectionRun = Pick; + "id" | "status" | "scheduledRetryAt" | "errorCode" | "livenessState" | "resultJson"> & + Partial>; type ProjectionState = { run?: ProjectionRun; action?: typeof issueRecoveryActions.$inferSelect; wait?: { id: string; requestedAt: Date }; agent?: Pick; + projectAccessAllowed?: boolean; + projectAccessProjectId?: string | null; + workspaceSelectionAvailable?: boolean; project?: Pick; company?: Pick; paused: boolean; approval: boolean; }; +/** Recognize old persisted failures without displaying arbitrary run error text. */ +function deniedProjectId(issue: ProjectionIssue, run?: ProjectionRun): string | null | undefined { + const historical = run?.errorCode === "setup_failed" && run.error === "Project is outside this low-trust boundary."; + if (!run || run.status !== "failed" || run.agentId !== issue.assigneeAgentId + || (run.errorCode !== PROJECT_ACCESS_DENIED && !historical)) return undefined; + const detail = run.resultJson?.projectAccessDenied; + return detail && typeof detail === "object" && "projectId" in detail && typeof detail.projectId === "string" + ? detail.projectId : null; +} + /** Presentation never grants authority. Dispatch repeats all gates under its * locks. The same projection is used in detail, lists and attention surfaces. */ export function projectTaskInterruption(issue: ProjectionIssue, input: ProjectionState, now = Date.now()): TaskInterruptionPresentation | null { @@ -36,6 +53,17 @@ export function projectTaskInterruption(issue: ProjectionIssue, input: Projectio ? "budget" : input.paused || project?.pausedAt || agent?.status === "paused" || company?.status === "paused" ? "resume" : null; if (userAction) return { ...base, responsibility: "user", state: "waiting_for_user", userAction }; if (["running", "queued"].includes(run.status)) return { ...base, state: "working" }; + const deniedProject = deniedProjectId(issue, run); + let projectAccess: TaskInterruptionPresentation["projectAccess"]; + if (deniedProject !== undefined) { + // Permissions cannot repair a stale or unavailable workspace selection. + if (input.workspaceSelectionAvailable === false) return { ...base, state: "failed" }; + const ready = input.projectAccessAllowed === true; + const projectId = input.projectAccessProjectId === undefined ? deniedProject : input.projectAccessProjectId; + projectAccess = { agentId: run.agentId!, projectId: projectId === issue.projectId ? projectId : null, ready }; + if (!ready) return { ...base, responsibility: "user", state: "waiting_for_user", + userAction: "project_access", projectAccess }; + } const retryAt = system?.nextAttemptAt ?? run.scheduledRetryAt?.getTime(); if (retryAt && retryAt > now) return { ...base, state: "retry_scheduled", nextRetryAt: new Date(retryAt).toISOString(), canRetry: true }; @@ -44,7 +72,7 @@ export function projectTaskInterruption(issue: ProjectionIssue, input: Projectio activeObservation > now - 45_000) return { ...base, state: "working" }; if (wait && wait.requestedAt.getTime() > now - 10_000 && system?.state !== "exhausted") return { ...base, state: "working" }; - return { ...base, state: "failed", canRetry: true }; + return { ...base, state: "failed", canRetry: true, ...(projectAccess ? { projectAccess } : {}) }; } /** Batch list reads rather than doing seven queries per row. Keep run logs and @@ -61,8 +89,12 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i const projectIds = selected.flatMap(issue => issue.projectId ? [issue.projectId] : []); const [runs, waits, agentRows, projectRows, pauseRows, approvalRows, companyRows] = await Promise.all([ db.selectDistinctOn([runIssue], { issueId: runIssue, id: heartbeatRuns.id, status: heartbeatRuns.status, + agentId: heartbeatRuns.agentId, + error: sql`case when ${heartbeatRuns.errorCode} = 'setup_failed' + and ${heartbeatRuns.error} = 'Project is outside this low-trust boundary.' then ${heartbeatRuns.error} else null end`, errorCode: heartbeatRuns.errorCode, livenessState: heartbeatRuns.livenessState, scheduledRetryAt: heartbeatRuns.scheduledRetryAt, - resultJson: sql>`jsonb_build_object('configurationIncomplete', ${heartbeatRuns.resultJson}->'configurationIncomplete')`, + resultJson: sql>`jsonb_build_object('configurationIncomplete', ${heartbeatRuns.resultJson}->'configurationIncomplete', + 'projectAccessDenied', ${heartbeatRuns.resultJson}->'projectAccessDenied')`, }).from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId), inArray(runIssue, ids))) .orderBy(runIssue, desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id)), db.selectDistinctOn([wakeIssue], { issueId: wakeIssue, id: agentWakeupRequests.id, requestedAt: agentWakeupRequests.requestedAt }) @@ -91,15 +123,45 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i )).orderBy(issueRecoveryActions.sourceIssueId, desc(issueRecoveryActions.createdAt)) : []; const byIssue = (rows: T[]) => new Map(rows.map(row => [row.issueId, row])); const runMap = byIssue(runs), waitMap = byIssue(waits); + // Recheck current authority, including standard trust, root-issue grants and + // project policy. Do not reuse the failed run's frozen policy for a new retry. + // Only denied tasks need these reads; deduplicate projects and batch per agent. + // A task's organizational project can differ from its retained source. Old + // failures did not record the denied project, and even typed failures can + // become obsolete after a workspace selection changes. + const deniedIssues = selected.filter(issue => issue.assigneeAgentId && !["done", "cancelled"].includes(issue.status) + && deniedProjectId(issue, runMap.get(issue.id)) !== undefined); + const requirements = await taskWorkspaceProjectRequirements(db, companyId, deniedIssues.map(issue => issue.id)); + const deniedByAgent = new Map>(); + for (const issue of deniedIssues) { + const ids = deniedByAgent.get(issue.assigneeAgentId!) ?? new Set(); + for (const projectId of requirements.get(issue.id)?.projectIds ?? []) ids.add(projectId); + if (ids.size) deniedByAgent.set(issue.assigneeAgentId!, ids); + } + const currentAccess = new Map(); + const authorization = authorizationService(db); + for (const [agentId, ids] of deniedByAgent) { + const projectIds = [...ids]; + const decisions = await authorization.decideReads({ type: "agent", agentId, companyId, source: "agent_key" }, + projectIds.map(projectId => ({ type: "project" as const, companyId, projectId }))); + projectIds.forEach((projectId, index) => currentAccess.set(`${agentId}:${projectId}`, decisions[index]?.allowed === true)); + } const actionMap = new Map(actions.map(action => [action.sourceIssueId, action])); const agentMap = new Map(agentRows.map(agent => [agent.id, agent])); const projectMap = new Map(projectRows.map(project => [project.id, project])); const pauses = new Set(pauseRows.map(row => row.issueId)), approvalsSet = new Set(approvalRows.map(row => row.issueId)); - for (const issue of selected) result.set(issue.id, projectTaskInterruption(issue, { - run: runMap.get(issue.id), action: actionMap.get(issue.id), wait: waitMap.get(issue.id), - agent: agentMap.get(issue.assigneeAgentId ?? ""), project: projectMap.get(issue.projectId ?? ""), - company: companyRows[0], paused: pauses.has(issue.id), approval: approvalsSet.has(issue.id), - })); + for (const issue of selected) { + const required = requirements.get(issue.id); + const denied = required?.projectIds.filter(projectId => currentAccess.get(`${issue.assigneeAgentId}:${projectId}`) !== true); + result.set(issue.id, projectTaskInterruption(issue, { + run: runMap.get(issue.id), action: actionMap.get(issue.id), wait: waitMap.get(issue.id), + projectAccessAllowed: denied?.length === 0, + projectAccessProjectId: denied?.includes(issue.projectId ?? "") ? issue.projectId : null, + workspaceSelectionAvailable: required?.available ?? false, + agent: agentMap.get(issue.assigneeAgentId ?? ""), project: projectMap.get(issue.projectId ?? ""), + company: companyRows[0], paused: pauses.has(issue.id), approval: approvalsSet.has(issue.id), + })); + } return result; } diff --git a/server/src/services/task-workspace-access.ts b/server/src/services/task-workspace-access.ts index b49ba9d9432..0c4599982e1 100644 --- a/server/src/services/task-workspace-access.ts +++ b/server/src/services/task-workspace-access.ts @@ -8,10 +8,61 @@ import { canonicalExecutionUuid } from "./execution-identity.js"; import { assertSelectedWorkspaceTrust, runRetainsIssueIdentity, taskRepositoryRunColumns, taskRepositoryResponsibleUserMatches } from "./task-repository-acquisition.js"; import { resolveCoreTrustPreset } from "./trust-preset-resolver.js"; import { isDeepStrictEqual } from "node:util"; +import { ProjectAccessDeniedError } from "./project-access-denial.js"; import { taskWorkspaceSelectionSchema, type TaskWorkspaceSelection } from "@paperclipai/shared"; +import { taskWorkspaceSelectableCondition } from "./task-workspace-selection.js"; type DbTransaction = Parameters[0]>[0]; +/** Read the project requirements of the next selected workspace, without + * admitting execution or moving files. Batch only the tasks needing a denial + * presentation; this adds at most three bounded reads, not queries per task. */ +export async function taskWorkspaceProjectRequirements(db: Db, companyId: string, issueIds: string[]) { + const result = new Map(); + if (!issueIds.length) return result; + const tasks = await db.select({ id: issues.id, projectId: issues.projectId, + executionWorkspaceId: issues.executionWorkspaceId, projectWorkspaceId: issues.projectWorkspaceId, + workspaceSelection: issues.workspaceSelection, workspacePendingSelection: issues.workspacePendingSelection, + workspaceBindingRevision: issues.workspaceBindingRevision, + }).from(issues).where(and(eq(issues.companyId, companyId), inArray(issues.id, issueIds))); + const selections = tasks.map(task => { + const pending = task.workspacePendingSelection; + const intent = pending?.intent ?? (!task.executionWorkspaceId ? task.workspaceSelection : null); + const parsed = intent ? taskWorkspaceSelectionSchema.safeParse(intent.selection) : null; + const selection = parsed?.success ? parsed.data : null; + const available = (!pending || pending.expectedBindingRevision === task.workspaceBindingRevision) + && (!parsed || parsed.success); + const workspaceId = selection ? selection.kind === "existing" ? selection.workspaceId : null : task.executionWorkspaceId; + return { task, selection, available, workspaceId }; + }); + const workspaceIds = [...new Set(selections.flatMap(row => row.workspaceId ? [row.workspaceId] : []))]; + const workspaces = workspaceIds.length ? await db.select({ id: executionWorkspaces.id, + projectId: executionWorkspaces.projectId, projectWorkspaceId: executionWorkspaces.projectWorkspaceId, + selectable: sql`${taskWorkspaceSelectableCondition()}`, + }).from(executionWorkspaces).where(and(eq(executionWorkspaces.companyId, companyId), inArray(executionWorkspaces.id, workspaceIds))) : []; + const workspaceMap = new Map(workspaces.map(workspace => [workspace.id, workspace])); + const sources = selections.map(row => { + const workspace = row.workspaceId ? workspaceMap.get(row.workspaceId) : undefined; + const sourceId = row.selection + ? row.selection.kind === "configured_source" ? row.selection.projectWorkspaceId : workspace?.projectWorkspaceId + : workspace?.projectWorkspaceId ?? row.task.projectWorkspaceId; + return { ...row, workspace, sourceId }; + }); + const sourceIds = [...new Set(sources.flatMap(row => row.sourceId ? [row.sourceId] : []))]; + const sourceRows = sourceIds.length ? await db.select({ id: projectWorkspaces.id, projectId: projectWorkspaces.projectId }) + .from(projectWorkspaces).where(and(eq(projectWorkspaces.companyId, companyId), inArray(projectWorkspaces.id, sourceIds))) : []; + const sourceMap = new Map(sourceRows.map(source => [source.id, source])); + for (const row of sources) { + const source = row.sourceId ? sourceMap.get(row.sourceId) : undefined; + result.set(row.task.id, { + projectIds: [...new Set([row.task.projectId, row.workspace?.projectId, source?.projectId].filter((id): id is string => Boolean(id)))], + available: row.available && (!row.workspaceId || Boolean(row.workspace)) && (!row.sourceId || Boolean(source)) + && (row.selection?.kind !== "existing" || row.workspace?.selectable === true), + }); + } + return result; +} + export async function assertTaskWorkspaceSourceProjectAccess( db: Db | DbTransaction, actor: AuthorizationActor, companyId: string, projectId: string | null, ) { @@ -19,7 +70,12 @@ export async function assertTaskWorkspaceSourceProjectAccess( const decision = await measureRunPhase("workspace.access.authorization", () => accessService(db as Db).decide({ actor, action: "project:read", resource: { type: "project", companyId, projectId }, })); - if (!decision.allowed) throw forbidden(decision.explanation); + if (!decision.allowed) { + if (actor.type === "agent" && decision.reason === "deny_low_trust_boundary") { + throw new ProjectAccessDeniedError(projectId); + } + throw forbidden(decision.explanation); + } } export async function assertExecutionWorkspaceRecordAccess( diff --git a/server/src/services/tool-access/profiles/projection.ts b/server/src/services/tool-access/profiles/projection.ts index 50b77ae3a79..53b92ef182f 100644 --- a/server/src/services/tool-access/profiles/projection.ts +++ b/server/src/services/tool-access/profiles/projection.ts @@ -80,9 +80,15 @@ export type ProfileApplication = Pick; +export type ProfileEntryMatch = Pick; + +export type ProfileCatalogMatch = Pick; + export function profileEntryMatchesCatalog( - entry: typeof toolProfileEntries.$inferSelect, - catalogEntry: ProfileCatalogEntry, + entry: ProfileEntryMatch, + catalogEntry: ProfileCatalogMatch, ): boolean { if (entry.selectorType === "application") return entry.applicationId === catalogEntry.applicationId; diff --git a/server/src/services/tool-access/profiles/resolution.ts b/server/src/services/tool-access/profiles/resolution.ts index 80aa6870681..c237636eb2b 100644 --- a/server/src/services/tool-access/profiles/resolution.ts +++ b/server/src/services/tool-access/profiles/resolution.ts @@ -1,56 +1,93 @@ -import { and, asc, eq, inArray } from "drizzle-orm"; +import { and, asc, eq, inArray, or, sql, type AnyColumn } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; import { agents, toolCatalogEntries, toolProfileBindings, toolProfileEntries, toolProfiles } from "@paperclipai/db"; import type { ToolCatalogEntry, ToolConnection, ToolProfileEffectiveSummary, ToolProfileWithDetails } from "@paperclipai/shared"; import { effectiveToolProfileBindings, profileIdsInBindingOrder } from "../../tool-profile-binding-precedence.js"; -import { buildProfileDetails, profileCatalogColumns, profileEntryMatchesCatalog, toProfileBinding, toProfileEntry, type ProfileCatalogEntry } from "./projection.js"; +import { buildProfileDetails, profileCatalogColumns, profileEntryMatchesCatalog, toProfileBinding, toProfileEntry, type ProfileCatalogEntry, type ProfileEntryMatch, type ProfileCatalogMatch } from "./projection.js"; type ProfileAssignmentDependencies = { assertOptionalAgent(companyId: string, agentId: string | null | undefined, label: string): Promise; resolveInstalledConnectionsForAgent(companyId: string, agentId: string): Promise; toCatalogEntry(row: typeof toolCatalogEntries.$inferSelect): ToolCatalogEntry; }; +function catalogValuesContain(column: AnyColumn, values: Array) { + const distinct = [...new Set(values)]; + return sql`(${JSON.stringify(distinct.filter(value => value !== null))}::jsonb ? ${column}::text + or (${distinct.includes(null)} and ${column} is null))`; +} + +function resolveCatalogAccess( + profiles: Array>, + entries: Array, + catalog: ProfileCatalogMatch[], +) { + const allowedCatalogIds = new Set(); + const allowedToolNames = new Set(); + const entriesByProfile = new Map(); + for (const entry of entries) { + const group = entriesByProfile.get(entry.profileId) ?? []; + group.push(entry); + entriesByProfile.set(entry.profileId, group); + } + for (const profile of profiles) { + const profileEntries = entriesByProfile.get(profile.id) ?? []; + const includes = profileEntries.filter( + (entry) => entry.effect === "include", + ); + const excludes = profileEntries.filter( + (entry) => entry.effect === "exclude", + ); + for (const catalogEntry of catalog) { + if ( + excludes.some((entry) => + profileEntryMatchesCatalog(entry, catalogEntry), + ) + ) + continue; + if ( + profile.defaultAction === "allow" || + includes.some((entry) => + profileEntryMatchesCatalog(entry, catalogEntry), + ) + ) { + allowedCatalogIds.add(catalogEntry.id); + allowedToolNames.add(catalogEntry.toolName); + } + } + for (const entry of includes.filter( + (item) => item.selectorType === "tool_name" && item.toolName, + )) { + const matchingExclude = excludes.some( + (item) => + item.selectorType === "tool_name" && + item.toolName === entry.toolName, + ); + if (!matchingExclude) allowedToolNames.add(entry.toolName!); + } + } + return { allowedCatalogIds, allowedToolNames }; +} + export function createToolProfileResolver(db: Db, { assertOptionalAgent, resolveInstalledConnectionsForAgent, toCatalogEntry }: ProfileAssignmentDependencies) { - async function resolveEffectiveProfiles( - companyId: string, - agentId: string, - loadCatalog: () => PromiseLike, - toAllowedTool: (entry: TEntry) => TAllowedTool, - ): Promise & { allowedTools: TAllowedTool[] }> { + async function readEffectiveProfileState(companyId: string, agentId: string) { await assertOptionalAgent( companyId, agentId, "Tool profile effective agent", ); - const allBindings = await db + const matchingBindings = await db .select() .from(toolProfileBindings) - .where(eq(toolProfileBindings.companyId, companyId)) + .where(and(eq(toolProfileBindings.companyId, companyId), or( + and(eq(toolProfileBindings.targetType, "company"), eq(toolProfileBindings.targetId, companyId)), + and(eq(toolProfileBindings.targetType, "agent"), eq(toolProfileBindings.targetId, agentId)), + ))) .orderBy( asc(toolProfileBindings.priority), asc(toolProfileBindings.createdAt), ); - const matchingBindings = allBindings.filter( - (binding) => - (binding.targetType === "company" && - binding.targetId === companyId) || - (binding.targetType === "agent" && binding.targetId === agentId), - ); - if (matchingBindings.length === 0) { - return { - agentId, - profiles: [], - entries: [], - bindings: [], - allowedTools: [], - allowedToolNames: [], - installedConnections: await resolveInstalledConnectionsForAgent( - companyId, - agentId, - ), - }; - } const candidateProfileIds = profileIdsInBindingOrder(matchingBindings); + if (candidateProfileIds.length === 0) return { bindings: [], activeProfiles: [] }; const candidateProfiles = await db .select() .from(toolProfiles) @@ -73,6 +110,16 @@ export function createToolProfileResolver(db: Db, { assertOptionalAgent, resolve .filter((profile): profile is typeof toolProfiles.$inferSelect => Boolean(profile && profile.status === "active"), ); + return { bindings, activeProfiles }; + } + + async function resolveEffectiveProfiles( + companyId: string, + agentId: string, + loadCatalog: () => PromiseLike, + toAllowedTool: (entry: TEntry) => TAllowedTool, + ): Promise & { allowedTools: TAllowedTool[] }> { + const { bindings, activeProfiles } = await readEffectiveProfileState(companyId, agentId); if (activeProfiles.length === 0) { return { agentId, @@ -114,44 +161,7 @@ export function createToolProfileResolver(db: Db, { assertOptionalAgent, resolve list.push(entry); entriesByProfile.set(entry.profileId, list); } - const allowedCatalogIds = new Set(); - const allowedToolNames = new Set(); - for (const profile of activeProfiles) { - const profileEntries = entriesByProfile.get(profile.id) ?? []; - const includes = profileEntries.filter( - (entry) => entry.effect === "include", - ); - const excludes = profileEntries.filter( - (entry) => entry.effect === "exclude", - ); - for (const catalogEntry of catalog) { - if ( - excludes.some((entry) => - profileEntryMatchesCatalog(entry, catalogEntry), - ) - ) - continue; - if ( - profile.defaultAction === "allow" || - includes.some((entry) => - profileEntryMatchesCatalog(entry, catalogEntry), - ) - ) { - allowedCatalogIds.add(catalogEntry.id); - allowedToolNames.add(catalogEntry.toolName); - } - } - for (const entry of includes.filter( - (item) => item.selectorType === "tool_name" && item.toolName, - )) { - const matchingExclude = excludes.some( - (item) => - item.selectorType === "tool_name" && - item.toolName === entry.toolName, - ); - if (!matchingExclude) allowedToolNames.add(entry.toolName!); - } - } + const { allowedCatalogIds, allowedToolNames } = resolveCatalogAccess(activeProfiles, entries, catalog); const agentIds = companyAgents.map((agent) => agent.id); const details: ToolProfileWithDetails[] = activeProfiles.map((profile) => buildProfileDetails({ @@ -182,7 +192,48 @@ export function createToolProfileResolver(db: Db, { assertOptionalAgent, resolve ), }; } + async function getConnectionAssignmentsForAgent(companyId: string, agentId: string, connectionIds: string[]) { + const { activeProfiles } = await readEffectiveProfileState(companyId, agentId); + if (activeProfiles.length === 0 || connectionIds.length === 0) { + return { allowedConnectionIds: [], installedConnections: await resolveInstalledConnectionsForAgent(companyId, agentId) }; + } + const catalog = await db.select({ + id: toolCatalogEntries.id, + applicationId: toolCatalogEntries.applicationId, + connectionId: toolCatalogEntries.connectionId, + toolName: toolCatalogEntries.toolName, + riskLevel: toolCatalogEntries.riskLevel, + }).from(toolCatalogEntries).where(and( + eq(toolCatalogEntries.companyId, companyId), eq(toolCatalogEntries.status, "active"), inArray(toolCatalogEntries.connectionId, connectionIds), + )); + const entries = catalog.length === 0 ? [] : await db.select({ + profileId: toolProfileEntries.profileId, + effect: toolProfileEntries.effect, + selectorType: toolProfileEntries.selectorType, + applicationId: toolProfileEntries.applicationId, + connectionId: toolProfileEntries.connectionId, + catalogEntryId: toolProfileEntries.catalogEntryId, + toolName: toolProfileEntries.toolName, + riskLevel: toolProfileEntries.riskLevel, + }).from(toolProfileEntries).where(and( + eq(toolProfileEntries.companyId, companyId), inArray(toolProfileEntries.profileId, activeProfiles.map(profile => profile.id)), + or( + and(eq(toolProfileEntries.selectorType, "application"), catalogValuesContain(toolProfileEntries.applicationId, catalog.map(entry => entry.applicationId))), + and(eq(toolProfileEntries.selectorType, "connection"), catalogValuesContain(toolProfileEntries.connectionId, catalog.map(entry => entry.connectionId))), + and(eq(toolProfileEntries.selectorType, "catalog_entry"), catalogValuesContain(toolProfileEntries.catalogEntryId, catalog.map(entry => entry.id))), + and(eq(toolProfileEntries.selectorType, "tool_name"), catalogValuesContain(toolProfileEntries.toolName, catalog.map(entry => entry.toolName))), + and(eq(toolProfileEntries.selectorType, "risk_level"), catalogValuesContain(toolProfileEntries.riskLevel, catalog.map(entry => entry.riskLevel))), + ), + )); + const { allowedCatalogIds } = resolveCatalogAccess(activeProfiles, entries, catalog); + return { + allowedConnectionIds: [...new Set(catalog.filter(entry => allowedCatalogIds.has(entry.id)).map(entry => entry.connectionId))], + installedConnections: await resolveInstalledConnectionsForAgent(companyId, agentId), + }; + } + return { + getConnectionAssignmentsForAgent, getEffectiveProfilesForAgent: (companyId: string, agentId: string) => resolveEffectiveProfiles( companyId, diff --git a/server/src/services/usage-receipts-read.test.ts b/server/src/services/usage-receipts-read.test.ts new file mode 100644 index 00000000000..83ddbed9364 --- /dev/null +++ b/server/src/services/usage-receipts-read.test.ts @@ -0,0 +1,81 @@ +import { randomBytes, randomUUID } from "node:crypto"; +import { eq } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { agents, companies, createDb, heartbeatRuns, runUsageReceipts, type Db } from "@paperclipai/db"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js"; +import { persistUsageReceipt, type UsageReceiptEnvelope } from "./usage-receipts.js"; + +const support = await getEmbeddedPostgresTestSupport(); +describe.skipIf(!support.supported)("usage receipt PostgreSQL reads", () => { + let database: Awaited>; + let db: Db; + beforeAll(async () => { + database = await startEmbeddedPostgresTestDatabase("usage-receipt-read-"); + db = createDb(database.connectionString); + }, 60_000); + afterAll(async () => { await db?.$client.end(); await database?.cleanup(); }); + + async function fixture() { + const companyId = randomUUID(), agentId = randomUUID(), runId = randomUUID(), sourceId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Receipt fixture", issuePrefix: companyId.slice(0, 8) }); + await db.insert(agents).values({ id: agentId, companyId, name: "Receipt agent", adapterType: "process", status: "active" }); + const payload = randomBytes(128 * 1024).toString("base64"); + const evidence = { contextSnapshot: { prompt: payload }, runnerProfileJson: { instructions: payload }, + resultJson: { stdout: payload }, stdoutExcerpt: payload, stderrExcerpt: payload }; + await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running", ...evidence, + usageJson: { accountingReceiptSourceId: sourceId, accountingReceiptSequence: 0, preserved: "receipt metadata" } }); + const receipt = (sequence: number): UsageReceiptEnvelope => ({ schema: "paperclip/accounting-receipt/v1", + id: randomUUID(), companyId, runId, sourceId, sequence, receivedAt: new Date().toISOString(), adapterType: "process", + receipt: { complete: false, usageBasis: "per_run", costUsd: sequence / 100, billingType: "metered_api" } }); + return { companyId, runId, evidence, receipt }; + } + + it("updates live usage with a small locked read and preserves all execution evidence", async () => { + const f = await fixture(), reads: unknown[][] = []; + const observed = new Proxy(db, { get(target, key) { + if (key !== "transaction") return Reflect.get(target, key); + return (callback: Parameters[0]) => target.transaction(tx => callback(new Proxy(tx, { + get(transaction, property) { + if (property !== "select") return Reflect.get(transaction, property); + return (...fields: Parameters) => { + const builder = transaction.select(...fields), from = builder.from.bind(builder); + builder.from = ((...args: Parameters) => { + const query = from(...args); + if (args[0] === heartbeatRuns) { + const execute = query.execute.bind(query); + query.execute = async (...executeArgs: Parameters) => { + expect(query.toSQL().sql).toContain("for update"); + const rows = await execute(...executeArgs); reads.push(rows); return rows; + }; + } + return query; + }) as typeof builder.from; + return builder; + }; + }, + }))); + } }); + await persistUsageReceipt(observed, f.receipt(1)); + expect(reads).toHaveLength(1); + expect(reads[0]).toHaveLength(1); + expect(Buffer.byteLength(JSON.stringify(reads))).toBeLessThan(2048); + const [stored] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)); + expect(stored).toMatchObject(f.evidence); + expect(stored.usageJson).toMatchObject({ preserved: "receipt metadata", accountingReceiptSequence: 1, + budgetExposure: { costCents: "1.0000000" } }); + expect(Buffer.byteLength(JSON.stringify(stored))).toBeGreaterThan(512 * 1024); + }); + + it("serializes competing checkpoints and retains replay conflicts and company isolation", async () => { + const f = await fixture(), first = f.receipt(1), second = f.receipt(2); + await Promise.all([persistUsageReceipt(db, first), persistUsageReceipt(db, second)]); + await persistUsageReceipt(db, first); + const [stored] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)); + expect(stored.usageJson).toMatchObject({ accountingReceiptSequence: 2, budgetExposure: { costCents: "2.0000000" } }); + expect(await db.select().from(runUsageReceipts).where(eq(runUsageReceipts.runId, f.runId))).toHaveLength(2); + await expect(persistUsageReceipt(db, { ...first, receipt: { ...first.receipt, costUsd: 10 } })).rejects.toThrow("Conflicting"); + const foreign = await fixture(); + await expect(persistUsageReceipt(db, { ...f.receipt(3), companyId: foreign.companyId })).rejects.toThrow("not found"); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0]).toEqual(stored); + }); +}); diff --git a/server/src/services/usage-receipts.ts b/server/src/services/usage-receipts.ts index 52bae89f8fb..0f99401d9da 100644 --- a/server/src/services/usage-receipts.ts +++ b/server/src/services/usage-receipts.ts @@ -68,7 +68,10 @@ export async function persistUsageReceipt(db: Db, raw: UsageReceiptEnvelope) { const envelope = envelopeSchema.parse(raw); const hash = receiptFingerprint(envelope); return withAccountingTransaction(db, envelope.companyId, async (tx, publications) => { - const [run] = await tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.id, envelope.runId), eq(heartbeatRuns.companyId, envelope.companyId))).for("update"); + const [run] = await tx.select({ + id: heartbeatRuns.id, agentId: heartbeatRuns.agentId, status: heartbeatRuns.status, + usageJson: heartbeatRuns.usageJson, costAccountedAt: heartbeatRuns.costAccountedAt, + }).from(heartbeatRuns).where(and(eq(heartbeatRuns.id, envelope.runId), eq(heartbeatRuns.companyId, envelope.companyId))).for("update"); if (!run) throw notFound("Accounting run not found"); const [existing] = await tx.select().from(runUsageReceipts).where(and(eq(runUsageReceipts.companyId, envelope.companyId), eq(runUsageReceipts.runId, envelope.runId), eq(runUsageReceipts.sourceId, envelope.sourceId), eq(runUsageReceipts.sequence, envelope.sequence))); if (existing && existing.receiptHash !== hash) throw conflict("Conflicting usage checkpoint replay"); diff --git a/server/src/vendor/paperclip-runner/index.ts b/server/src/vendor/paperclip-runner/index.ts index 2252b5daf6b..571daf6a929 100644 --- a/server/src/vendor/paperclip-runner/index.ts +++ b/server/src/vendor/paperclip-runner/index.ts @@ -107,6 +107,10 @@ export const PAPERCLIP_EXECUTION_PROMPT_REVISION = runner.PAPERCLIP_EXECUTION_PROMPT_REVISION; export const acpxRuntimeSessionDirectoryName = runner.acpxRuntimeSessionDirectoryName; +export const OPENCODE_ISOLATED_HOMES_DIRECTORY = + runner.OPENCODE_ISOLATED_HOMES_DIRECTORY; +export const opencodeRuntimeSessionDirectoryName = + runner.opencodeRuntimeSessionDirectoryName; export const canonicalNativeRuntimeContextDigest = runner.canonicalNativeRuntimeContextDigest; export const createNativeSessionBackend = runner.createNativeSessionBackend; diff --git a/tests/e2e/agent-chat-projects.spec.ts b/tests/e2e/agent-chat-projects.spec.ts index c8fcdc2345e..a631d3ea928 100644 --- a/tests/e2e/agent-chat-projects.spec.ts +++ b/tests/e2e/agent-chat-projects.spec.ts @@ -1,10 +1,48 @@ -import { expect, test } from "@playwright/test"; +import { expect, test, type APIRequestContext, type APIResponse } from "@playwright/test"; import { idle, json, send, setup } from "./agent-chat.shared"; test.use({ trace: "retain-on-failure" }); test.setTimeout(120_000); +test("cleanup restores agent chat experiments when cancellation fails", async ({ request }) => { + const original = await json(await request.get("/api/instance/settings/experimental")); + const response = (status: number, body: unknown) => ({ + ok: () => status >= 200 && status < 300, + status: () => status, + text: async () => JSON.stringify(body), + json: async () => body, + }) as APIResponse; + const failingRequest = new Proxy(request, { + get(target, property) { + if (property === "get") return async (...args: Parameters) => + args[0].endsWith("/live-runs") + ? response(200, [{ id: "cleanup-cancellation-fixture" }]) + : target.get(...args); + if (property === "post") return async (...args: Parameters) => + args[0].endsWith("/heartbeat-runs/cleanup-cancellation-fixture/cancel") + ? response(500, { error: "simulated cancellation failure" }) + : target.post(...args); + const value = Reflect.get(target, property, target); + return typeof value === "function" ? value.bind(target) : value; + }, + }); + try { + const f = await setup(failingRequest); + await expect(f.restore()).rejects.toThrow("500"); + const restored = await json(await request.get("/api/instance/settings/experimental")); + expect(restored).toMatchObject({ + enableAgentChat: original.enableAgentChat, + enableClassicTaskInterface: original.enableClassicTaskInterface, + }); + } finally { + await json(await request.patch("/api/instance/settings/experimental", { data: { + enableAgentChat: original.enableAgentChat, + enableClassicTaskInterface: original.enableClassicTaskInterface, + } })); + } +}); + /** * Agent chat project and content coverage: project cards, split handoff, * uploads, Ask/Plan modes, repository selection, plan approval, and shared diff --git a/tests/e2e/agent-chat.shared.ts b/tests/e2e/agent-chat.shared.ts index ab235494eb5..592af7641de 100644 --- a/tests/e2e/agent-chat.shared.ts +++ b/tests/e2e/agent-chat.shared.ts @@ -113,21 +113,24 @@ export async function setup(request: APIRequestContext) { ), }); } finally { - const runs = await json( - await request.get(`/api/companies/${company.id}/live-runs`), - ); - for (const run of runs) - await json( - await request.post(`/api/heartbeat-runs/${run.id}/cancel`), + try { + const runs = await json( + await request.get(`/api/companies/${company.id}/live-runs`), ); - await json( - await request.patch("/api/instance/settings/experimental", { - data: { - enableAgentChat: original.enableAgentChat, - enableClassicTaskInterface: original.enableClassicTaskInterface, - }, - }), - ); + for (const run of runs) + await json( + await request.post(`/api/heartbeat-runs/${run.id}/cancel`), + ); + } finally { + await json( + await request.patch("/api/instance/settings/experimental", { + data: { + enableAgentChat: original.enableAgentChat, + enableClassicTaskInterface: original.enableClassicTaskInterface, + }, + }), + ); + } } }, }; diff --git a/tests/e2e/chat-adapters-ui-messaging.spec.ts b/tests/e2e/chat-adapters-ui-messaging.spec.ts index 286f8f24f83..17208596a08 100644 --- a/tests/e2e/chat-adapters-ui-messaging.spec.ts +++ b/tests/e2e/chat-adapters-ui-messaging.spec.ts @@ -1042,7 +1042,7 @@ test.describe("Board send delivery refresh", () => { ); await page.route( new RegExp( - `/api/issues/(${issue.id}|${issue.identifier})/comments(?:\\?|$)`, + `/api/issues/(${issue.id}|${issue.identifier})/(?:recent-comments|comments)(?:\\?|$)`, ), async (route) => { if ( @@ -1050,9 +1050,7 @@ test.describe("Board send delivery refresh", () => { route.request().url().includes(`/issues/${issue.identifier}/`) ) canonicalCommentReadsAfterSend += 1; - await fulfill( - route, - sends + const comments = sends ? [ { id: publicationId, @@ -1066,8 +1064,10 @@ test.describe("Board send delivery refresh", () => { updatedAt: new Date().toISOString(), }, ] - : [], - ); + : []; + await fulfill(route, route.request().url().includes("/recent-comments") + ? { comments, latestCommentId: comments.at(-1)?.id ?? null, hasMore: false } + : comments); }, ); await page.route( diff --git a/tests/e2e/composer-stop.spec.ts b/tests/e2e/composer-stop.spec.ts index a9c747f9bf5..30c31a33a8b 100644 --- a/tests/e2e/composer-stop.spec.ts +++ b/tests/e2e/composer-stop.spec.ts @@ -380,7 +380,16 @@ for (const adapter of ["process", "paperclip_runner"] as const) { ).toEqual([]); } await menu(page, "Resume subtree"); - await page.getByRole("dialog").getByRole("checkbox").check(); + const resumeWake = page.getByRole("dialog").getByRole("checkbox"); + if (adapter === "process") { + // Stopped legacy work is blocked pending reconciliation. Removing the + // pause must not promise or request a wake through that recovery gate. + await expect(resumeWake).toHaveAccessibleName("Wake affected agents (0)"); + await expect(resumeWake).toBeDisabled(); + await expect(resumeWake).not.toBeChecked(); + } else { + await resumeWake.check(); + } await page .getByRole("dialog") .getByRole("button", { name: "Resume subtree", exact: true }) diff --git a/tests/e2e/legacy-failure-continuation.spec.ts b/tests/e2e/legacy-failure-continuation.spec.ts index 4eaa73092f9..b04ca8e1e22 100644 --- a/tests/e2e/legacy-failure-continuation.spec.ts +++ b/tests/e2e/legacy-failure-continuation.spec.ts @@ -92,8 +92,18 @@ for (const action of ["task_retry", "retry_after_reload", "inbox_retry", "messag await expect(page.getByRole("button", { name: "Steer", exact: true })).toHaveCount(0); const retry = page.getByRole("button", { name: "Retry", exact: true }); await expect(retry).toBeEnabled(); - await db.update(heartbeatRuns).set({ processPid: 999999999 }).where(eq(heartbeatRuns.id, sourceRunId)); - await retry.click(); + await page.route((url) => url.pathname === `/api/issues/${issue.id}/retry`, async route => { + if (route.request().method() !== "POST") return route.continue(); + await db.update(heartbeatRuns).set({ processPid: 999999999 }).where(eq(heartbeatRuns.id, sourceRunId)); + await route.continue(); + }, { times: 1 }); + const [response] = await Promise.all([ + page.waitForResponse(response => response.request().method() === "POST" && + new URL(response.url()).pathname === `/api/issues/${issue.id}/retry` && + response.request().postDataJSON()?.observedRunId === sourceRunId), + retry.click(), + ]); + expect(response.ok(), `${response.status()} ${await response.text()}`).toBe(true); } else if (action === "automatic_message") { // No Retry, duplicate message, or status change: the saved input runs. } else if (action === "message") { @@ -131,9 +141,15 @@ for (const action of ["task_retry", "retry_after_reload", "inbox_retry", "messag await page.reload(); await expect(page.getByText("Answered the pending follow-up once.", { exact: false })).toBeVisible(); } finally { - await request.patch(`/api/companies/${company.id}`, { data: { status: "archived" } }); - await closeRegisteredClients(url); - await rm(root, { recursive: true, force: true }); + try { + await request.patch(`/api/companies/${company.id}`, { data: { status: "archived" } }); + } finally { + try { + await closeRegisteredClients(url); + } finally { + await rm(root, { recursive: true, force: true }); + } + } } }); } diff --git a/tests/e2e/paused-composer.spec.ts b/tests/e2e/paused-composer.spec.ts index c01d083632a..f3c41525648 100644 --- a/tests/e2e/paused-composer.spec.ts +++ b/tests/e2e/paused-composer.spec.ts @@ -7,6 +7,51 @@ async function json(response: APIResponse) { } for (const classic of [false, true]) { + test(`resume preview: ${classic ? "classic" : "task chat"} does not promise to wake blocked work`, async ({ page, request }) => { + test.setTimeout(120_000); + const company = await json(await request.post("/api/companies", { data: { name: `Resume preview ${Date.now()}` } })); + const settings = await json(await request.get("/api/instance/settings/experimental")); + try { + await json(await request.patch("/api/instance/settings/experimental", { data: { enableClassicTaskInterface: classic } })); + const agent = await json(await request.post(`/api/companies/${company.id}/agents`, { data: { + name: "Resume preview fixture", role: "engineer", adapterType: "process", + adapterConfig: { command: "/usr/bin/true" }, + runtimeConfig: { heartbeat: { enabled: false, wakeOnDemand: false } }, + } })); + const task = await json(await request.post(`/api/companies/${company.id}/issues`, { data: { + title: "Blocked work stays blocked", status: "blocked", assigneeAgentId: agent.id, + } })); + await page.goto(`/${company.issuePrefix}/issues/${task.identifier}`); + await page.getByRole("button", { name: "More task actions", exact: true }).click(); + await page.getByRole("button", { name: "Pause work", exact: true }).click(); + const takeover = page.getByTestId("paused-composer-takeover"); + await expect(takeover).toBeVisible(); + // Enable dispatch only after the hold exists; assignment itself otherwise starts a run. + expect((await json(await request.get(`/api/issues/${task.id}`))).status).toBe("blocked"); + await json(await request.patch(`/api/agents/${agent.id}`, { data: { + runtimeConfig: { heartbeat: { enabled: false, wakeOnDemand: true } }, + } })); + await takeover.getByRole("button", { name: "Resume task" }).click(); + const dialog = page.getByRole("dialog"); + await expect(dialog).toContainText("The pause will be removed from 1 task."); + const wake = dialog.getByRole("checkbox", { name: "Wake affected agents (0)" }); + await expect(wake).toBeDisabled(); + await expect(wake).not.toBeChecked(); + await expect(dialog).toContainText("Blocked, backlog, and unassigned tasks will be unpaused but will not start."); + await dialog.getByRole("button", { name: "Resume work", exact: true }).click(); + await expect(dialog).toHaveCount(0); + await expect(takeover).toHaveCount(0); + await page.reload(); + const editor = (classic ? page.getByTestId("issue-chat-composer") : page).getByRole("textbox", { name: "editable markdown" }); + await expect(editor).toBeVisible(); + expect((await json(await request.get(`/api/issues/${task.id}`))).status).toBe("blocked"); + expect(await json(await request.get(`/api/companies/${company.id}/heartbeat-runs?agentId=${agent.id}`))).toEqual([]); + } finally { + await request.patch(`/api/companies/${company.id}`, { data: { status: "archived" } }); + await request.patch("/api/instance/settings/experimental", { data: { enableClassicTaskInterface: settings.enableClassicTaskInterface } }); + } + }); + test(`paused composer: ${classic ? "classic" : "task chat"} preserves drafts and requires resume`, async ({ page, request }) => { test.setTimeout(120_000); const company = await json(await request.post("/api/companies", { data: { name: `Paused composer ${Date.now()}` } })); @@ -63,8 +108,11 @@ for (const classic of [false, true]) { await takeover.getByRole("link", { name: "Resume subtree" }).click(); await expect(page).toHaveURL(new RegExp(task.identifier)); await takeover.getByRole("button", { name: "Resume subtree" }).click(); - await page.getByRole("dialog").getByRole("checkbox").uncheck(); + const subtreeWake = page.getByRole("dialog").getByRole("checkbox"); + if (await subtreeWake.isEnabled()) await subtreeWake.uncheck(); + else await expect(subtreeWake).not.toBeChecked(); await page.getByRole("dialog").getByRole("button", { name: "Resume subtree", exact: true }).click(); + await expect(page.getByRole("dialog")).toHaveCount(0); await page.goto(`/${company.issuePrefix}/issues/${child.identifier}`); await expect(takeover).toHaveCount(0); await expect(editor).toBeVisible(); diff --git a/tests/runner-e2e/EVERYDAY-WORKFLOWS.md b/tests/runner-e2e/EVERYDAY-WORKFLOWS.md index 567f365adeb..a9e395f53bb 100644 --- a/tests/runner-e2e/EVERYDAY-WORKFLOWS.md +++ b/tests/runner-e2e/EVERYDAY-WORKFLOWS.md @@ -16,7 +16,7 @@ exact execution ID explicitly. | Decide on an installed service action | `service-approve`, `service-decline` | Assign an authenticated local MCP fixture with Ask first; match its tool action and connection ID; no provider call before approval; exactly one after approval and a verified document; none after decline. | | Decline a new connection | `connection-decline` | Start without service connections; match a Notion connection intent; click Not now; verify the saved rejection, no new connection or repeated request, and an explanation followed by Done. | | Request an email address | `agentmail-setup` | Enable Chat connectors, ask for an email address, and require a durable AgentMail card for the requesting agent and user. Reload and verify one password field, the direct API-key link, and no access selectors or modal. Decline and verify no connection or repeated request. | -| Continue work after a controller restart | `recover-controller` | Observe saved source, persist a user message, restart the isolated controller, and independently test the delivered result. | +| Continue work after a controller restart | `recover-controller` | Observe saved source, persist a user message, restart the isolated controller, and independently test the delivered result. Native Daytona cases keep the controller offline for 125 seconds before restarting. | | Stop work and change direction | `stop-redirect` | Click Stop, send one new request, reload, observe exactly one stored user message and the new answer, and reach Done. | | Create and edit a company skill | `create-skill-studio` | Create one skill through the runner, verify its persisted library entry and activity-feed card, open Skill Studio, save an edit, and verify the edit after returning. Local Codex, local ACPX Claude, and warm Daytona cells are explicit. | @@ -47,8 +47,10 @@ restart, a verified first download supplies the persistence checkpoint; the controller is interrupted during a subsequent revision with another queued requirement. -The native Codex Daytona restart cell now requires a completed native tool -receipt while that revision is still running. Before and after restart it pins +The native Codex and ACPX Daytona restart cells require a completed native tool +receipt while that revision is still running. They keep the controller offline +for 125 seconds after its health listener stops, then start its replacement. +Before and after restart each pins the same run, normalized session, Runner identity, PID/start time, and active sandbox lease. The original receipt prefix must remain unchanged. A run that ends before either active observation leaves the fault boundary unexercised; @@ -56,7 +58,10 @@ it cannot pass from the final ZIP alone. The cell no longer exempts an original Codex remote run that fails with `process_lost` and is replaced. Completion requires one original provider turn and control-plane terminal, -the same native receipt prefix, no duplicate completed tool execution IDs, +the same native receipt prefix, no failed native tool receipts, and consistent +results for every stable completed tool identity. Reattached activity can +redeliver an identical start or tool receipt; changed content or a second +submission fails. It also requires a committed run and successful `workspace_finalize` receipt, and no pending workspace operation or recovery. Existing independent ZIP and preserved-download checks still apply. These receipt checks do not prove that arbitrary external diff --git a/tests/runner-e2e/GITHUB-REVIEW-JOURNEY.md b/tests/runner-e2e/GITHUB-REVIEW-JOURNEY.md index ceeb56a95e0..bc765f21390 100644 --- a/tests/runner-e2e/GITHUB-REVIEW-JOURNEY.md +++ b/tests/runner-e2e/GITHUB-REVIEW-JOURNEY.md @@ -79,6 +79,14 @@ completed state alone is also insufficient: the actual run must succeed and the associated task must reach Done. A native failure after publication fails the journey and retains the publication evidence already observed. +The rendered check must match its exact GitHub run URL, score, and reviewed +commit. Its failure status comes from that check's header. Hidden re-run menu +items, a failure on another check, and words in the model's narrative do not +prove the conclusion. `github-review-check.spec.ts` covers this boundary in the +credential-free `test:e2e:runner:browser-support` suite. Task completion reads +the visible Status property's picker, before and after reload. Hidden controls, +SVG titles, and narrative text saying Done cannot satisfy that UI checkpoint. + Each attempt records endpoint, App, PR/branch, task, and run ownership. Cleanup removes the owned endpoint, closes the owned PR/deletes its branch, and asks for the owned App's removal. App identity must match the created endpoint, company, @@ -89,6 +97,15 @@ connections, and completed task evidence. An unconfirmed cleanup fails the attempt; retain the resource receipt for recovery. Read the receipt before any manual cleanup. Never delete by a guessed name or broad account search. +Registration render waits are bounded separately from attended confirmation. +The driver retains only the GitHub origin/path and control-presence flags when +the creation control is missing. On a known registration error page without auth +fields, it also retains headings, button labels, and alerts. It excludes query +parameters and authentication content. Reauthentication is a setup boundary, +not evidence that an App exists. +Cleanup also stops failed native runs that still own a retry, and requires the +normal run-only Stop acknowledgement. It never clears pending accounting debt. + The configured deadline and observed run limit bound the attempt. They are not a hard provider-dollar cap or proof that no request outlives cancellation. Use the existing budgeted Cloud operations adapter before automated paid campaigns; @@ -113,7 +130,10 @@ cleanup. Missing usage remains unknown. Authentication screens are excluded. `github-review-report.ts` uses the existing evidence sanitizer and manifest, stages only declared screenshots, and verifies screenshot hashes and attempt path ownership. The standard report inputs are `evidence/result.json`, -`evidence/evidence-manifest.json`, and `evidence/snapshots/`. Raw private operator +`evidence/evidence-manifest.json`, and `evidence/snapshots/`. A passing result +also stages the hash-verified persisted checkpoint as `evidence/final-state.png` +for the standard report's final-state requirement. A missing persisted screenshot +rejects that passing report. Raw private operator receipts are outside the manifest. Existing report tooling imports these files; no second runner or report service is required. A presentation repair must use a new directory, preserve the original result/hash and failure, and dispatch no diff --git a/tests/runner-e2e/README.md b/tests/runner-e2e/README.md index 47853d3ba74..c6ed181c3e5 100644 --- a/tests/runner-e2e/README.md +++ b/tests/runner-e2e/README.md @@ -1,5 +1,15 @@ # Paid runner full-stack E2E +The explicit-only `daytona-restart-continuity` suite runs the existing +`recover-controller` journey with the recorded Codex and Claude staging models +on fresh Daytona fixtures. After a completed task tool in an active revision +turn, it keeps the owned development controller offline for 125 seconds, beyond +the controller's two-minute reconnect budget. The existing independent oracle +requires the original process, native session, sandbox lease and receipt prefix, +one completion and successful workspace finalization, plus verified downloaded +work. It permits one attempt per cell and retains failed evidence. This is +development API-key acceptance, distinct from staging subscription recovery. + The explicit-only [GitHub review bot journey](GITHUB-REVIEW-JOURNEY.md) tests attended bot setup and an automatic PR review against an attached deployment. Private account bindings and setup receipts live in the private eval repository. @@ -237,6 +247,11 @@ The explicit [deployment-targeted Product journeys](PRODUCT-JOURNEYS.md) reuse browser actions across managed self-hosted and Cloud targets. They include a model-free task-properties story, fresh Cloud onboarding, credential-free previews, private evidence, and prerequisite reuse; `--all` is unchanged. +Cloud onboarding may retain a versioned `onboardingConnect` receipt for the +Connect action: its last checkpoint, validation/hire response state and HTTP +status, known adapters, and allowlisted outcome/check/server codes. The case +report renders these fields explicitly; older results without the receipt remain +valid. This diagnostic does not change the journey grade or qualify a live run. ## Live provider connection journeys diff --git a/tests/runner-e2e/SECURITY.md b/tests/runner-e2e/SECURITY.md index 26a326b9cca..555a6673b20 100644 --- a/tests/runner-e2e/SECURITY.md +++ b/tests/runner-e2e/SECURITY.md @@ -14,6 +14,13 @@ their expected four-field result, exact base64 download bytes, checksums, size, and attachment/run IDs. Extra fields and arbitrary agent output are rejected. It contains no credentials, browser state, request headers, or provider text. +The optional `onboardingConnect` diagnostic is also a closed, versioned receipt. +It retains only a known checkpoint, response states, HTTP statuses, known adapter +names, and allowlisted outcome/check/server codes (at most 32 checks per response). +Unknown codes become `unrecognized`; raw bodies, messages, URLs, headers, and +browser errors are excluded. Reports validate the receipt before rendering its +explicit fields, and historical results may omit it. + ## Attended connection authentication See [PROVIDER-CONNECTIONS.md](PROVIDER-CONNECTIONS.md) for the explicit-only @@ -46,13 +53,17 @@ changes. ## GitHub authorization -The Everyday remote Codex restart oracle reads only the existing public run, +The Everyday remote native restart oracle reads only the existing public run, event, environment-lease, workspace-operation and recovery APIs. It does not execute commands in a provider sandbox or signal a provider process. Its new evidence projects closed receipt metadata and hashes the provider sandbox ID and full native event content. Provider text, tool arguments and command output do not enter these snapshots. Existing image admission, credentials, billing, retention and cleanup rules continue to apply. +The isolated server's private restart request may delay its replacement by at +most five minutes. This fault injection never stops a hosted instance or a +provider sandbox. The explicit Daytona continuity suite has one attempt and +1,000-cent company and agent hard stops. Set `RUNNER_E2E_ALLOWED_ACTOR_IDS` to a non-empty JSON array of numeric GitHub user IDs. Keep the list equal to the owners of `.github/**` in diff --git a/tests/runner-e2e/billing.test.ts b/tests/runner-e2e/billing.test.ts index 48352815c0e..037ccae91ec 100644 --- a/tests/runner-e2e/billing.test.ts +++ b/tests/runner-e2e/billing.test.ts @@ -31,6 +31,25 @@ function result(overrides: Partial = {}): RunnerE2EResult { } describe("runner E2E billing summaries", () => { + it("preserves captured token totals when a retained result has only a billing summary", () => { + const billing = summarizeExecutionBilling(result({ usage: { + inputTokens: 59_989, outputTokens: 4_974, cachedInputTokens: 488_798, costStatus: "unpriced", + } })); + const retained = result({ billing }); + expect(summarizeExecutionBilling(retained)).toEqual(billing); + expect(aggregateCampaignBilling([retained])).toMatchObject({ llm: { + inputTokens: 59_989, outputTokens: 4_974, cachedInputTokens: 488_798, + runsWithTokenUsage: 1, runsWithReportedCost: 0, + }, testsWithCompleteBilling: 0 }); + }); + + it("uses newly captured usage instead of a stale retained billing summary", () => { + const billing = summarizeExecutionBilling(result({ usage: { inputTokens: 100, outputTokens: 10, costStatus: "unpriced" } })); + const fresh = summarizeExecutionBilling(result({ billing, usage: { inputTokens: 200, outputTokens: 20, costStatus: "unpriced" } })); + expect(fresh.llm).toMatchObject({ inputTokens: 200, outputTokens: 20, runsWithReportedCost: 0 }); + expect(fresh.complete).toBe(false); + }); + it("keeps missing agent time separate from attempt duration and measured zero", () => { const interrupted = result({ status: "failed", durationMs: 108_850 }); expect(summarizeExecutionBilling(interrupted).runtime.agentRunDurationMs).toBeNull(); diff --git a/tests/runner-e2e/billing.ts b/tests/runner-e2e/billing.ts index 35f70e83569..766167c0bfb 100644 --- a/tests/runner-e2e/billing.ts +++ b/tests/runner-e2e/billing.ts @@ -278,8 +278,11 @@ export function summarizeExecutionBilling( result: Pick< RunnerE2EResult, "runIds" | "usage" | "environmentId" | "durationMs" | "runtimeUsage" | "firstTaskQuality" | "completionQuality" | "publicMcp" | "productJourney" - > & Partial>, + > & Partial>, ): RunnerE2EBillingSummary { + // Some retained journeys store their measured summary without raw run usage. + // Regeneration must preserve those measurements, including unknown prices. + if (result.usage === undefined && result.billing) return structuredClone(result.billing); const browserOnly = Boolean(result.productJourney && result.runtimeMode === "none" && !result.runIds?.length); const requestedRunCount = browserOnly ? 0 : Math.max(result.runIds?.length ?? 0, 1); const entries = browserOnly ? [] : usageEntries(result.usage, requestedRunCount); diff --git a/tests/runner-e2e/case-outcome.ts b/tests/runner-e2e/case-outcome.ts index 03316c81995..489f16f8384 100644 --- a/tests/runner-e2e/case-outcome.ts +++ b/tests/runner-e2e/case-outcome.ts @@ -1,5 +1,6 @@ import type { RunnerE2EResult } from "./types.js"; import { isBlockedConnectionResult } from "./connection-evidence.js"; +import { validateOnboardingConnectEvidence } from "./product-journeys/onboarding-connect.js"; const html = (value: unknown) => String(value ?? "").replace( /[&<>"']/g, @@ -8,6 +9,28 @@ const html = (value: unknown) => c ]!, ); + +function renderOnboardingConnect(result: RunnerE2EResult): string { + const receipt = result.productJourney?.onboardingConnect; + if (receipt === undefined) return ""; + try { + validateOnboardingConnectEvidence(receipt); + } catch { + return "

Onboarding Connect diagnostics unavailable: invalid receipt.

"; + } + return `
Onboarding Connect diagnostics +

Last checkpoint: ${html(receipt.checkpoint)}

+ ${([["validation", "Native validation"], ["hire", "Agent hire"]] as const).map(([key, label]) => { + const response = receipt[key]; + return `

${label}

${[ + ["Response state", response.state], ["HTTP status", response.httpStatus], + ["Test endpoint adapter", response.requestedAdapter], ["Returned adapter", response.adapter], + ["Validation outcome", response.outcome], ["Server code", response.serverCode], + ].map(([name, value]) => `
${name}
${html(value ?? "Unavailable")}
`).join("")}
+ ${response.checks.length ? `

Validation checks:

    ${response.checks.map(check => `
  • ${html(check.code)} · ${html(check.level)}
    • `).join("")}
` : ""} + ${response.truncatedChecks ? "

Only the first 32 validation checks were retained.

" : ""}`; + }).join("")}
`; +} /** An interrupted first-task journey is not evidence of failed downstream behavior. */ export function isIncompleteFirstTaskResult( result: RunnerE2EResult, @@ -69,6 +92,7 @@ export function renderCaseOutcome( ${result.productJourney.deployment?.effectiveFeatures ? `

Effective features:

${Object.entries(result.productJourney.deployment.effectiveFeatures).map(([key, value]) => `
${html(key)}
${html(value)}
`).join("")}
` : ""}` : ""} ${result.productJourney?.operations ? `

Eval operations: ${html(result.productJourney.operations.campaignId)} · ${result.productJourney.operations.requestCount} requests · model cost $${html(result.productJourney.operations.modelCostUsd ?? "unknown")} (setup $${html(result.productJourney.operations.setupCostUsd ?? "unknown")}, counted once). Feedback: ${html(result.productJourney.operations.feedback.collection)} · ${result.productJourney.operations.feedback.complaints} complaints · ${result.productJourney.operations.feedback.suggestions} suggestions.

` : ""} ${result.productJourney?.operations?.traceIds.length ? `
Trace correlation IDs
    ${result.productJourney.operations.traceIds.map(id => `
  • ${html(id)}
    • `).join("")}
` : ""} + ${renderOnboardingConnect(result)} ${result.providerConnection ? `

Target: ${html(result.providerConnection.target.origin)} · ${html(result.providerConnection.target.commit ?? "revision unavailable")} · ${result.providerConnection.assisted ? "Assisted login" : "Unassisted"} · ${html(result.providerConnection.authFreshness)} browser · ${html(result.providerConnection.entry)} / ${html(result.providerConnection.method)} · stopped at ${html(result.providerConnection.phase)}

` : ""} ${failures.length ? `

Failed checks:

    ${failures.map((c) => `
  • ${html(c.id)} — ${html(c.detail)}
    • `).join("")}
` : checks.length ? `

No behavioral matcher failed.${notReached.length ? " The journey is incomplete; unexercised checks are not passes." : failed ? " The overall failure came from a separate run, cleanup, or evidence check." : ""}

` : ""} ${notReached.length ? `

Not reached:

    ${notReached.map((c) => `
  • ${html(c.id)} — ${html(c.notReached)}
    • `).join("")}
` : ""} diff --git a/tests/runner-e2e/catalog.test.ts b/tests/runner-e2e/catalog.test.ts index 4f36708a091..4b87cac677d 100644 --- a/tests/runner-e2e/catalog.test.ts +++ b/tests/runner-e2e/catalog.test.ts @@ -219,10 +219,10 @@ describe("runner E2E catalog", () => { expect(localIntegrityTasks).toHaveLength(2); expect(openRouterBreadthTasks).toHaveLength(3); expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([ - 1, 3, 2, 2, 4, 4, 4, 4, 4, 4, 8, 10, 6, 63, 12, 6, 4, 2, 4, 2, 30, 3, 16, 16, 2, 6, 8, 46, 2, 23, 15, 52, 6, 6, 20, 26, 52, 28, 18, 2, 6, 6, 13, 10, 2, 48, 16, 10, 2, 1, 1, 4, 116, 2, 2, + 2, 1, 3, 2, 2, 4, 4, 4, 4, 4, 4, 8, 10, 6, 63, 12, 6, 4, 2, 4, 2, 30, 3, 16, 16, 2, 6, 8, 46, 2, 23, 15, 52, 6, 6, 20, 26, 52, 28, 18, 2, 6, 6, 13, 10, 2, 48, 16, 10, 2, 1, 1, 4, 116, 2, 2, ]); - expect(validateRunnerCatalog()).toHaveLength(765); - expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(765); + expect(validateRunnerCatalog()).toHaveLength(767); + expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(767); expect( runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"), ).toHaveLength(48); diff --git a/tests/runner-e2e/catalog.ts b/tests/runner-e2e/catalog.ts index 84c733fe035..a824e9b9dd4 100644 --- a/tests/runner-e2e/catalog.ts +++ b/tests/runner-e2e/catalog.ts @@ -1155,6 +1155,17 @@ const repositoryModelProfiles = [ ]; export const runnerSuites: readonly RunnerSuiteFixture[] = [ + { + id: "daytona-restart-continuity", label: "Daytona controller outage continuity", manualOnly: true, + description: "Keep the original active native runner, task tools and saved files through more than two minutes without a controller, then commit the result without replay.", + groups: [], profiles: repositoryModelProfiles.map(({ native }) => productionStoryProfile(native)), + environments: runnerEnvironments.filter(environment => environment.id === "daytona"), + tasks: everydayTasks.filter(task => task.id === "recover-controller") + .map(task => ({ ...task, automaticRetryPolicy: "single_attempt" as const })), expectedMatrixSize: 2, + definitionMetadata: { version: 2, scheduling: "explicit-only", controllerDowntimeMs: 125_000, + maximumAttemptsPerCell: 1, budgetMonthlyCents: 1000, instructions: "production", grading: "same-active-process-lease-native-prefix-and-workspace-receipt", + authentication: "api-key", stagingAuthentication: "subscription", stagingQualification: false }, + }, { id: "codie-latency-diagnostic", label: "Native Codex latency diagnostic", @@ -1514,7 +1525,8 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [ ...(["hire-reuse", "delegate-feedback"].includes(task.id) ? [] : [`everyday-workflows.runner-opencode.local.${task.id}`]), ]), ], - definitionMetadata: { version: 7, instructions: "production", grading: "outcome-and-invariants", scheduling: "explicit-only", + definitionMetadata: { version: 9, instructions: "production", grading: "outcome-and-invariants", scheduling: "explicit-only", + remoteControllerDowntimeMs: 125_000, remoteCodexRestart: "same-active-process-lease-native-prefix-and-workspace-receipt", procedureCases: { ids: ["hire-reuse", "delegate-feedback"], maximumAttemptsPerCell: 1, companyAndLeadBudgetCents: 1_000 } }, }, diff --git a/tests/runner-e2e/eval-gateway.test.ts b/tests/runner-e2e/eval-gateway.test.ts index aefca15a9e3..c4c9e785b8d 100644 --- a/tests/runner-e2e/eval-gateway.test.ts +++ b/tests/runner-e2e/eval-gateway.test.ts @@ -139,6 +139,9 @@ it.each([ { type: "namespace", name: "local", tools: [{ type: "web_search" }] }, { type: "namespace", name: "local", tools: [{ type: "namespace", name: "nested", tools: [{ type: "web_search" }] }] }, { type: "namespace", name: "local" }, + { type: "namespace", name: "local", tools: [{ type: "custom", name: "x", tools: [{ type: "web_search" }] }] }, + { type: "namespace", name: "local", tools: [{ type: "function", name: "x", tools: [{ type: "web_search" }] }] }, + { type: "custom", name: "x", tools: [{ type: "web_search" }] }, ])("rejects unqualified tools inside namespaces before reservation", async tool => { const f = await fixture(), transport = vi.fn(); await expect(forwardEvalRequest({ ...f, payload: { ...f.payload, tools: [tool] }, transport })).rejects.toThrow(); diff --git a/tests/runner-e2e/eval-gateway.ts b/tests/runner-e2e/eval-gateway.ts index d7cc20ab0db..6d42ce680e5 100644 --- a/tests/runner-e2e/eval-gateway.ts +++ b/tests/runner-e2e/eval-gateway.ts @@ -72,11 +72,11 @@ function boundedPayload(value: unknown, route: EvalGatewayRoute) { if (body.speed !== undefined && body.speed !== "standard" || body.inference_geo !== undefined) throw new EvalRequestError("unqualified_billing_mode"); if (body.tools !== undefined && (!Array.isArray(body.tools) || body.tools.some(tool => { const definition = row(tool), type = definition.type; - if (route.provider !== "openai") return type !== undefined && type !== "custom"; + if (route.provider !== "openai") return type !== undefined && type !== "custom" || Object.hasOwn(definition, "tools"); if (type === "namespace") return typeof definition.name !== "string" || !definition.name || !Array.isArray(definition.tools) || !definition.tools.length - || definition.tools.some(nested => !["function", "custom"].includes(String(row(nested).type))); - return !["function", "custom"].includes(String(type)); + || definition.tools.some(nested => !["function", "custom"].includes(String(row(nested).type)) || Object.hasOwn(row(nested), "tools")); + return !["function", "custom"].includes(String(type)) || Object.hasOwn(definition, "tools"); }))) throw new EvalRequestError("unqualified_hosted_tool"); const inspect = (item: unknown): void => { if (Array.isArray(item)) { item.forEach(inspect); return; } diff --git a/tests/runner-e2e/everyday-flow.ts b/tests/runner-e2e/everyday-flow.ts index 3dc84a29769..bebc81eacff 100644 --- a/tests/runner-e2e/everyday-flow.ts +++ b/tests/runner-e2e/everyday-flow.ts @@ -89,7 +89,7 @@ interface Input { workspacePath: string; privateDir: string; deadlineAt: number; - restart(): Promise; + restart(controllerDowntimeMs?: number): Promise; observe(issue: StoryIssue, runs: StoryRun[]): void; capture(id: string, label: string, file: string): Promise; evidence(name: string, value: unknown): Promise; @@ -169,7 +169,8 @@ export async function runEverydayFlow(input: Input) { let project = fixtures.project; const caseId = execution.task.id; const requireRemoteRestart = caseId === "recover-controller" && execution.environment.id === "daytona" - && execution.profile.generation === "native" && execution.profile.provider === "codex"; + && execution.profile.generation === "native" && (execution.profile.provider === "codex" + || execution.profile.provider === "acpx"); const providerChoice = caseId === "provider-decline" || caseId === "provider-second"; const nativeProviderCase = caseId === "provider-native"; let aggregatorFixture: Awaited> | undefined; @@ -949,7 +950,7 @@ export async function runEverydayFlow(input: Input) { ev.remoteRestart = { before }; await input.evidence("remote-restart-before.json", before); } else ev.allowedInterruptedRuns.push(active.id); - await input.restart(); + await input.restart(requireRemoteRestart ? 125_000 : undefined); note("controller-restarted"); await openParent(); if (requireRemoteRestart) { @@ -1643,7 +1644,9 @@ export async function runEverydayFlow(input: Input) { ev.remoteRestart!.final = final; return final; }, accept: final => gradeRemoteRestart({ ...ev.remoteRestart, final }).every(check => check.passed), - reject: final => ["failed", "cancelled", "timed_out"].includes(final.run.status) ? "Original remote run failed after restart" : undefined }); + reject: final => ["failed", "cancelled", "timed_out"].includes(final.run.status) ? "Original remote run failed after restart" + : final.run.status === "succeeded" && final.run.nativePhase === "committed" + && !gradeRemoteRestart({ ...ev.remoteRestart, final })[1]!.passed ? "Original remote turn completed with failed tools or conflicting/repeated execution receipts" : undefined }); await input.evidence("remote-restart-final.json", final); } ev.checks.push( diff --git a/tests/runner-e2e/everyday-remote-restart.test.ts b/tests/runner-e2e/everyday-remote-restart.test.ts index 938c9b30bf6..c492e4f9e72 100644 --- a/tests/runner-e2e/everyday-remote-restart.test.ts +++ b/tests/runner-e2e/everyday-remote-restart.test.ts @@ -111,11 +111,27 @@ describe("remote restart lifecycle evidence", () => { expect(gradeRemoteRestart(missingPrefix)[1]!.passed).toBe(false); }); - it.each(["replacement", "second submission", "duplicate tool receipt", "missing terminal"])("rejects %s at completion", fault => { + it("rejects failed native tool calls even when the original turn eventually succeeds", () => { + const { evidence, event } = fixture(); + const rows = [...fixture().afterRows, event(6, "tool.execution.completed", { executionId: "failed-write", status: "failed", output: "private timeout" }), + event(7, "turn.completed"), event(8, "run.result.accepted", {}, "control_plane", 1), event(9, "run.terminal", {}, "control_plane", 2)]; + evidence.final.events = projectRemoteRestartEvents(evidence.final.run, rows); + expect(gradeRemoteRestart(evidence).every(check => check.passed)).toBe(false); + }); + + it("accepts identical activity receipt redelivery without treating it as another submission", () => { + const { evidence } = fixture(); + evidence.final.events.push({ ...evidence.final.events[1]!, seq: 10, sourceSeq: 10, sourceEventId: "runner:run:10" }, + { ...evidence.final.events[3]!, seq: 11, sourceSeq: 11, sourceEventId: "runner:run:11" }); + expect(gradeRemoteRestart(evidence).every(check => check.passed)).toBe(true); + }); + + it.each(["replacement", "second submission", "changed tool receipt", "changed start receipt", "missing terminal"])("rejects %s at completion", fault => { const { evidence } = fixture(); if (fault === "replacement") evidence.final.run.id = "replacement"; if (fault === "second submission") evidence.final.events.push({ ...evidence.final.events[0]!, seq: 10, sourceSeq: 10, sourceEventId: "runner:run:10" }); - if (fault === "duplicate tool receipt") evidence.final.events.push({ ...evidence.final.events[3]!, seq: 10, sourceSeq: 10, sourceEventId: "runner:run:10" }); + if (fault === "changed tool receipt") evidence.final.events.push({ ...evidence.final.events[3]!, seq: 10, sourceSeq: 10, sourceEventId: "runner:run:10", payloadSha256: "0".repeat(64) }); + if (fault === "changed start receipt") evidence.final.events.push({ ...evidence.final.events[1]!, seq: 10, sourceSeq: 10, sourceEventId: "runner:run:10", payloadSha256: "0".repeat(64) }); if (fault === "missing terminal") evidence.final.events.pop(); expect(gradeRemoteRestart(evidence)[1]!.passed).toBe(false); }); diff --git a/tests/runner-e2e/everyday-remote-restart.ts b/tests/runner-e2e/everyday-remote-restart.ts index 59c1db429e8..9b50934eec6 100644 --- a/tests/runner-e2e/everyday-remote-restart.ts +++ b/tests/runner-e2e/everyday-remote-restart.ts @@ -12,7 +12,8 @@ const digest = (value: unknown) => createHash("sha256").update(canonicalJson(val export interface RemoteRestartEvent { seq: number; sourceSeq: number; sourceEventId: string; eventType: string; sourceKind: "runner" | "control_plane"; turnId: string | null; - toolExecutionId: string | null; toolCompleted: boolean; sha256: string; + toolExecutionId: string | null; toolCompleted: boolean; toolFailed: boolean; + payloadSha256: string; sha256: string; } export interface RemoteRestartBoundary { identity: { runId: string; companyId: string; agentId: string; issueId: string; @@ -78,6 +79,8 @@ export function projectRemoteRestartEvents(run: StoryRun, rows: readonly Row[]): turnId: id(event.turnId) ? event.turnId : null, toolExecutionId: id(payload.executionId) ? payload.executionId : null, toolCompleted: event.eventType === "tool.execution.completed" && payload.status === "completed", + toolFailed: event.eventType === "tool.execution.completed" && payload.status !== "completed", + payloadSha256: digest(payload), sha256: digest(event) }; }); } @@ -132,12 +135,23 @@ export function gradeRemoteRestart(evidence: RemoteRestartEvidence | undefined): const events = final?.events ?? []; const turnId = before?.events.find(event => event.eventType === "turn.started")?.turnId; const once = (kind: string, source: RemoteRestartEvent["sourceKind"]) => events.filter(event => event.eventType === kind && event.sourceKind === source).length === 1; - const completedTools = events.filter(event => event.toolCompleted).map(event => event.toolExecutionId); + // Reattachment can redeliver an already-observed activity notification. Its + // stable identity and exact payload must agree; a new submission, changed + // result or failed tool is not receipt replay and must never pass this cell. + const starts = events.filter(event => event.eventType === "turn.started" && event.sourceKind === "runner"); + const tools = events.filter(event => event.toolCompleted); + const toolResults = new Map(); + let consistentToolResults = true; + for (const event of tools) { + if (!event.toolExecutionId || (toolResults.has(event.toolExecutionId) && toolResults.get(event.toolExecutionId) !== event.payloadSha256)) consistentToolResults = false; + else toolResults.set(event.toolExecutionId, event.payloadSha256); + } const nativeCompletion = finalIdentity && Boolean(before && after && retainsPrefix(after.events, events)) - && once("turn.submitted", "runner") && once("turn.started", "runner") && once("turn.completed", "runner") + && once("turn.submitted", "runner") && starts.length > 0 && new Set(starts.map(event => event.payloadSha256)).size === 1 + && once("turn.completed", "runner") && once("run.result.accepted", "control_plane") && once("run.terminal", "control_plane") && events.filter(event => event.eventType.startsWith("turn.")).every(event => event.turnId === turnId || (event.eventType === "turn.submitted" && event.turnId === null)) - && new Set(completedTools).size === completedTools.length; + && consistentToolResults && !events.some(event => event.toolFailed); const operations = final?.operations ?? []; const saved = finalIdentity && run?.status === "succeeded" && run.nativePhase === "committed" && run.resultJson?.finalizationPhase === "committed" && run.resultJson.workspaceFinalizeStatus === "succeeded" diff --git a/tests/runner-e2e/github-review-browser.ts b/tests/runner-e2e/github-review-browser.ts index dffb516bcdd..f4749931dc2 100644 --- a/tests/runner-e2e/github-review-browser.ts +++ b/tests/runner-e2e/github-review-browser.ts @@ -10,6 +10,83 @@ export async function enterGitHubJourneyCompany(page: Page, origin: string, comp await page.waitForURL(url => url.origin === origin && url.pathname === pathname); } +export async function assertGitHubJourneyCheck(page: Page, expected: { repository: string; url: string; headSha: string; score: number }, timeout = 30_000) { + let target: URL; + try { target = new URL(expected.url); } + catch { throw new ConnectionFailure("github_check_reference_mismatch"); } + const prefix = `/${expected.repository}/runs/`; + const id = target.pathname.startsWith(prefix) ? target.pathname.slice(prefix.length) : ""; + if (target.origin !== "https://github.com" || target.username || target.password || target.search || target.hash + || !/^[1-9][0-9]*$/.test(id) || !/^[a-f0-9]{40,64}$/.test(expected.headSha) + || !Number.isInteger(expected.score) || expected.score < 1 || expected.score > 5) { + throw new ConnectionFailure("github_check_reference_mismatch"); + } + await expect(page).toHaveURL(target.href, { timeout }); + const check = page.locator(`#check_run_${id}`); + await expect(check.getByRole("heading", { name: `${expected.score}/5`, exact: true })).toBeVisible({ timeout }); + // GitHub keeps a hidden "Re-run failed checks" menu item before the visible + // result. Only the selected check's status header can prove its conclusion. + await expect(check.locator(":scope > .border-bottom").getByText(/failed/i)).toBeVisible({ timeout }); + await expect(check.getByText(expected.headSha, { exact: true })).toBeVisible({ timeout }); +} + +export async function assertGitHubJourneyTaskDone(page: Page, timeout = 30_000) { + const status = page.locator('[data-property-row="true"]:visible') + .filter({ has: page.locator('[data-property-label="Status"]') }) + .locator('[data-property-value="true"]'); + // Status glyphs contain SVG titles, and responsive layouts can retain hidden + // controls. Read the visible status picker rather than the first Done text. + await expect(status.getByRole("button", { name: "Change status (current: Done)", exact: true })).toBeVisible({ timeout }); +} + +export interface GitHubRegistrationBoundary { + origin: string; pathname: string; passwordRequired: boolean; oneTimeCodeRequired: boolean; + reauthenticationRequired: boolean; createControlVisible: boolean; + controls?: { headings: string[]; buttons: string[]; alerts: string[] }; +} + +/** Keep only navigation and control-presence evidence. Authentication fields, + * page content, query parameters, and grant payloads must stay out of reports. */ +export async function waitForGitHubRegistrationControl(page: Page, owner: string, timeoutMs: number, + record: (boundary: GitHubRegistrationBoundary) => Promise = async () => {}, reauthenticate?: () => Promise) { + const control = page.getByRole("button", { name: new RegExp(`^Create GitHub App(?: for ${owner})?$`) }); + const observe = async (includeErrorControls = false) => { + const url = new URL(page.url()); + if (url.origin !== "https://github.com") throw new ConnectionFailure("github_registration_origin_mismatch"); + const boundary: GitHubRegistrationBoundary = { origin: url.origin, pathname: url.pathname, + passwordRequired: await page.locator('input[type="password"]:visible').count() > 0, + oneTimeCodeRequired: await page.locator('input[autocomplete="one-time-code"]:visible').count() > 0, + reauthenticationRequired: await page.getByRole("heading", { name: /^Confirm access$/i }).isVisible() + || await page.getByRole("button", { name: /^Use (?:your password|passkey)$/i }).isVisible(), + createControlVisible: await control.isVisible() }; + boundary.reauthenticationRequired ||= boundary.passwordRequired || boundary.oneTimeCodeRequired + || /\/(?:login|sessions)(?:\/|$)/.test(boundary.pathname); + if (includeErrorControls && !boundary.reauthenticationRequired + && ["/settings/apps/new", "/settings/apps/manifest"].includes(url.pathname)) { + boundary.controls = { headings: await page.getByRole("heading").allTextContents(), + buttons: await page.getByRole("button").allTextContents(), alerts: await page.getByRole("alert").allTextContents() }; + } + await record(boundary); + return boundary; + }; + await observe(); + let authenticationAttempted = false; + for (;;) { + try { await control.waitFor({ timeout: Math.min(timeoutMs, 30_000) }); break; } + catch (error) { + if (page.isClosed()) throw error; + const boundary = await observe(true); + if (boundary.reauthenticationRequired) { + if (reauthenticate && !authenticationAttempted) { authenticationAttempted = true; await reauthenticate(); continue; } + throw new ConnectionBlock("awaiting_user", "github_registration_reauthentication_required"); + } + throw new ConnectionFailure("github_registration_create_control_missing"); + } + } + await observe(); + return control; +} + /** Capture measured pages only, excluding authentication and unrelated GitHub data. */ export function isGitHubJourneyEvidencePage(value: string, targetOrigin: string, repository: string, appSlug?: string) { const url = new URL(value); diff --git a/tests/runner-e2e/github-review-check.spec.ts b/tests/runner-e2e/github-review-check.spec.ts new file mode 100644 index 00000000000..705bc5246a9 --- /dev/null +++ b/tests/runner-e2e/github-review-check.spec.ts @@ -0,0 +1,94 @@ +import { expect, test, type Page } from "@playwright/test"; +import { assertGitHubJourneyCheck, assertGitHubJourneyTaskDone } from "./github-review-browser.js"; + +const expected = { repository: "fixture/repo", url: "https://github.com/fixture/repo/runs/123", headSha: "a".repeat(40), score: 2 }; + +async function openCheck(page: Page, input: { status?: string; score?: number; headSha?: string; unrelatedFailure?: boolean; url?: string } = {}) { + const body = ` +
+
Fixture bot / Paperclip Review ${input.status ?? "failed"}
+

${input.score ?? expected.score}/5

+

Reviewed commit: ${input.headSha ?? expected.headSha}

+

Review rationale: failed tests are discussed in the summary.

+
+ ${input.unrelatedFailure ? '
Another check failed

2/5

' : ""}`; + await page.route("https://github.com/**", route => route.fulfill({ contentType: "text/html", body })); + await page.goto(input.url ?? expected.url); +} + +test("accepts the visible failed check despite an earlier hidden re-run menu item", async ({ page }) => { + await openCheck(page); + await expect(page.getByText(/failed/i).first()).toBeHidden(); + await assertGitHubJourneyCheck(page, expected, 150); +}); + +test("rejects a successful check even when its narrative and another check say failed", async ({ page }) => { + await openCheck(page, { status: "succeeded", unrelatedFailure: true }); + await expect(assertGitHubJourneyCheck(page, expected, 150)).rejects.toThrow(); +}); + +test("rejects the wrong reviewed commit", async ({ page }) => { + await openCheck(page, { headSha: "b".repeat(40) }); + await expect(assertGitHubJourneyCheck(page, expected, 150)).rejects.toThrow(); +}); + +test("rejects a different score", async ({ page }) => { + await openCheck(page, { score: 3 }); + await expect(assertGitHubJourneyCheck(page, expected, 150)).rejects.toThrow(); +}); + +test("rejects a page for another repository", async ({ page }) => { + await openCheck(page, { url: "https://github.com/foreign/repo/runs/123" }); + await expect(assertGitHubJourneyCheck(page, expected, 150)).rejects.toThrow(); +}); + +test("rejects a check reference outside the configured repository", async ({ page }) => { + const url = "https://github.com/foreign/repo/runs/123"; + await openCheck(page, { url }); + await expect(assertGitHubJourneyCheck(page, { ...expected, url }, 150)).rejects.toThrow(); +}); + +for (const [name, override] of [ + ["foreign origin", { url: "https://example.com/fixture/repo/runs/123" }], + ["embedded credentials", { url: "https://fixture-user:fixture-password@github.com/fixture/repo/runs/123" }], + ["query suffix", { url: `${expected.url}?x=1` }], + ["hash suffix", { url: `${expected.url}#fragment` }], + ["non-numeric run ID", { url: "https://github.com/fixture/repo/runs/abc" }], + ["short commit", { headSha: "a".repeat(7) }], + ["score below range", { score: 0 }], + ["score above range", { score: 6 }], +] as const) { + test(`rejects a malformed check reference: ${name}`, async ({ page }) => { + // These must reject before navigation or DOM inspection. + await expect(assertGitHubJourneyCheck(page, { ...expected, ...override }, 150)) + .rejects.toThrow("github_check_reference_mismatch"); + }); +} + +function taskStatus(status: string, hidden = false) { + return `
+ Status
+ +
`; +} + +test("accepts visible Done status despite hidden controls and SVG title text", async ({ page }) => { + await page.setContent(`${taskStatus("Done", true)}${taskStatus("Done")}`); + await expect(page.getByText("Done", { exact: true }).first()).toBeHidden(); + await assertGitHubJourneyTaskDone(page, 150); +}); + +test("rejects Done in the narrative when the task is still in progress", async ({ page }) => { + await page.setContent(`

Done

${taskStatus("In Progress")}`); + await expect(assertGitHubJourneyTaskDone(page, 150)).rejects.toThrow(); +}); + +test("rejects hidden Done status when the visible status is in progress", async ({ page }) => { + await page.setContent(`${taskStatus("Done", true)}${taskStatus("In Progress")}`); + await expect(assertGitHubJourneyTaskDone(page, 150)).rejects.toThrow(); +}); + +test("rejects a Done picker in an unrelated property", async ({ page }) => { + await page.setContent(taskStatus("Done").replace('data-property-label="Status">Status', 'data-property-label="Assignee">Assignee')); + await expect(assertGitHubJourneyTaskDone(page, 150)).rejects.toThrow(); +}); diff --git a/tests/runner-e2e/github-review-flow.ts b/tests/runner-e2e/github-review-flow.ts index c580ed767fa..d43e919ddd8 100644 --- a/tests/runner-e2e/github-review-flow.ts +++ b/tests/runner-e2e/github-review-flow.ts @@ -3,7 +3,7 @@ import type { GitHubTaskReview } from "../../packages/shared/src/types/chat-gith import type { GitHubRepositoryPage } from "../../packages/shared/src/types/chat-channels.js"; import { ConnectionBlock, ConnectionFailure } from "./connection-config.js"; import { ConnectionApi } from "./connection-target.js"; -import { awaitJourneyCondition, createGitHubJourneyPullRequest, enterGitHubJourneyCompany, observeGitHubHead, selectGitHubJourneyRepository } from "./github-review-browser.js"; +import { assertGitHubJourneyCheck, assertGitHubJourneyTaskDone, awaitJourneyCondition, createGitHubJourneyPullRequest, enterGitHubJourneyCompany, observeGitHubHead, selectGitHubJourneyRepository, waitForGitHubRegistrationControl, type GitHubRegistrationBoundary } from "./github-review-browser.js"; import { githubCartFixture, hasOnlyOpenedAutomaticReview, matchesGitHubJourneyEnvironment, retainGitHubReview, verifyGitHubReview, verifyGitHubReviewRun, type GitHubReviewJourneyEvidence } from "./github-review-evidence.js"; import type { GitHubReviewJourneyConfig } from "./github-review-config.js"; import { captureOwnedGitHubApp, readOwnedGitHubReviewRuns, type GitHubJourneyResources } from "./github-review-ownership.js"; @@ -15,6 +15,8 @@ export async function runGitHubReviewFlow(input: { checkpoint(id: keyof GitHubReviewJourneyEvidence["checkpoints"], page: Page): Promise; phase(id: string): Promise; persistResources(): Promise; + recordRegistrationBoundary?(boundary: GitHubRegistrationBoundary): Promise; + authenticateRegistration?(page: Page): Promise; }) { const { page, github, api, config, resources, evidence } = input; const capabilities = await api.get('/api/instance/settings/experimental'); @@ -69,8 +71,8 @@ export async function runGitHubReviewFlow(input: { // Manifest flow navigates the same tab. External App grants remain attended. evidence.assisted = true; await awaitJourneyCondition(async () => new URL(page.url()).hostname === "github.com", 30_000, "github_manifest_handoff_missing", input.signal); - const createApp = page.getByRole("button", { name: new RegExp(`^Create GitHub App(?: for ${config.github.appOwner})?$`) }); - await createApp.waitFor({ timeout: config.browser.loginTimeoutMs }); + const createApp = await waitForGitHubRegistrationControl(page, config.github.appOwner, config.browser.loginTimeoutMs, + input.recordRegistrationBoundary, input.authenticateRegistration ? () => input.authenticateRegistration!(page) : undefined); await page.bringToFront(); await input.checkpoint("app_creation_consent", page); console.log(`[github-review-journey] Authorize creation of only ${resources.appName} in the dedicated QA browser. Install it only on ${config.github.repository}.`); @@ -220,17 +222,20 @@ export async function runGitHubReviewFlow(input: { text: await inline.locator(".review-comment .comment-body").filter({ hasText: /precision|decimal|floating|cents/i }).first().innerText() }; await inline.scrollIntoViewIfNeeded(); await input.checkpoint("github_inline_delivery", github); + await input.phase("github_check_projection"); + if (!review!.checkUrl) throw new ConnectionFailure("github_check_reference_missing"); const checkPopup = page.context().waitForEvent("page"); await page.getByRole("link", { name: "Check", exact: true }).click(); const check = await checkPopup; - await expect(check.getByText(`${evidence.score}/5`, { exact: false }).first()).toBeVisible(); - await expect(check.getByText(/failed/i).first()).toBeVisible(); + await assertGitHubJourneyCheck(check, { repository: config.github.repository, url: review!.checkUrl, headSha, score: evidence.score }); await input.checkpoint("check_projection", check); + await input.phase("task_done"); await page.getByRole("link", { name: "Paperclip task", exact: true }).click(); - await expect(page.getByText("Done", { exact: true }).first()).toBeVisible(); + await assertGitHubJourneyTaskDone(page); await input.checkpoint("task_done", page); + await input.phase("persisted"); await page.reload(); await github.reload(); - await expect(page.getByText("Done", { exact: true }).first()).toBeVisible(); + await assertGitHubJourneyTaskDone(page); await expect(github.getByText(`${evidence.score}/5`, { exact: false }).first()).toBeVisible(); await input.checkpoint("persisted", page); } diff --git a/tests/runner-e2e/github-review-journey.test.ts b/tests/runner-e2e/github-review-journey.test.ts index 8489b4596b0..09ce3e869dd 100644 --- a/tests/runner-e2e/github-review-journey.test.ts +++ b/tests/runner-e2e/github-review-journey.test.ts @@ -8,9 +8,9 @@ import type { GitHubTaskReview } from "../../packages/shared/src/types/chat-gith import { verifyConnectionTarget } from "./connection-target.js"; import { parseConnectionConfig } from "./connection-config.js"; import type { APIRequestContext } from "@playwright/test"; -import { captureOwnedGitHubApp, readOwnedGitHubReviewRuns, type GitHubJourneyResources } from "./github-review-ownership.js"; +import { captureOwnedGitHubApp, readOwnedGitHubReviewRuns, stopOwnedGitHubReviewRun, type GitHubJourneyResources } from "./github-review-ownership.js"; import type { ChatEndpoint } from "../../packages/shared/src/types/chat-channels.js"; -import { enterGitHubJourneyCompany, isGitHubJourneyEvidencePage } from "./github-review-browser.js"; +import { enterGitHubJourneyCompany, isGitHubJourneyEvidencePage, waitForGitHubRegistrationControl } from "./github-review-browser.js"; import { attendGitHubJourneyAction } from "./github-review-attendance.js"; import { PassThrough } from "node:stream"; import type { Locator, Page } from "@playwright/test"; @@ -19,7 +19,104 @@ import { runGitHubReviewCampaign } from "./github-review-launch.js"; import type { ConnectionApi } from "./connection-target.js"; import { runGitHubReviewFlow } from "./github-review-flow.js"; +describe("GitHub registration handoff", () => { + const page = (pathname: string, password: boolean, available: boolean) => ({ + url: () => `https://github.com${pathname}?return_to=private-grant&token=never-retain`, isClosed: () => false, + locator: (selector: string) => ({ count: async () => selector.includes('password') && password ? 1 : 0 }), + getByRole: (_role?: string, options?: { name?: string | RegExp }) => ({ isVisible: async () => String(options?.name).includes('Create GitHub App') && available, allTextContents: async () => ['Registration error'], + waitFor: vi.fn(async () => { if (!available) throw new Error("control_missing"); }) }), + }); + it("classifies a reauthentication boundary and retains no query or auth content", async () => { + const record = vi.fn(); + await expect(waitForGitHubRegistrationControl(page('/sessions/sudo', true, false) as unknown as Page, 'fixture', 600000, record)) + .rejects.toThrow('github_registration_reauthentication_required'); + expect(record.mock.calls.at(-1)?.[0]).toEqual({ origin: 'https://github.com', pathname: '/sessions/sudo', passwordRequired: true, oneTimeCodeRequired: false, reauthenticationRequired: true, createControlVisible: false }); + expect(JSON.stringify(record.mock.calls)).not.toMatch(/private-grant|never-retain/); + }); + it("fails a missing creation control on an ordinary registration page", async () => { + const record = vi.fn(); + await expect(waitForGitHubRegistrationControl(page('/settings/apps/manifest', false, false) as unknown as Page, 'fixture', 600000, record)) + .rejects.toThrow('github_registration_create_control_missing'); + expect(record.mock.calls.at(-1)?.[0].controls.alerts).toEqual(['Registration error']); + }); + it("admits the ordinary creation control with a bounded render wait", async () => { + const target = page('/settings/apps/new', false, true); + const control = target.getByRole('button', { name: /^Create GitHub App/ }); + const get = target.getByRole; + target.getByRole = (role, options) => String(options?.name).includes('Create GitHub App') ? control : get(role, options); + await expect(waitForGitHubRegistrationControl(target as unknown as Page, 'fixture', 600000)).resolves.toBe(control); + expect(control.waitFor).toHaveBeenCalledWith({ timeout: 30000 }); + }); + it("keeps passkey-first sudo at the manifest path out of authentication content diagnostics", async () => { + const target = page('/settings/apps/manifest', false, false), get = target.getByRole; + target.getByRole = (role, options) => String(options?.name).includes('Confirm access') + ? { ...get(role, options), isVisible: async () => true } : get(role, options); + const record = vi.fn(); + await expect(waitForGitHubRegistrationControl(target as unknown as Page, 'fixture', 600000, record)) + .rejects.toThrow('github_registration_reauthentication_required'); + expect(record.mock.calls.at(-1)?.[0].reauthenticationRequired).toBe(true); + expect(record.mock.calls.at(-1)?.[0].controls).toBeUndefined(); + }); + it("uses a supplied ordinary reauthentication adapter once before retrying creation readiness", async () => { + const target = page('/settings/apps/manifest', true, false), get = target.getByRole; + let authenticated = false; + const control = { ...get(), isVisible: async () => authenticated, + waitFor: vi.fn(async () => { if (!authenticated) throw new Error('sudo_required'); }) }; + target.getByRole = (role, options) => String(options?.name).includes('Create GitHub App') ? control : get(role, options); + const authenticate = vi.fn(async () => { authenticated = true; }); + await expect(waitForGitHubRegistrationControl(target as unknown as Page, 'fixture', 600000, async () => {}, authenticate)).resolves.toBe(control); + expect(authenticate).toHaveBeenCalledOnce(); + }); +}); + describe("reused target ownership", () => { + it("stops a scheduled legacy retry and waits until its cancellation finishes", async () => { + const scope = { companyId: 'company', agentId: 'agent', issueId: 'issue' }; + const scheduled = { ...scope, id: 'run', status: 'scheduled_retry', runtimeMode: 'legacy' }; + const get = vi.fn().mockResolvedValueOnce(scheduled).mockResolvedValueOnce(scheduled) + .mockResolvedValue({ ...scheduled, status: 'cancelled' }); + const post = vi.fn().mockResolvedValue({}); + await stopOwnedGitHubReviewRun({ get, post } as unknown as ConnectionApi, 'run', scope, new AbortController().signal); + expect(post).toHaveBeenCalledExactlyOnceWith('/api/heartbeat-runs/run/cancel', {}); + expect(get).toHaveBeenCalledTimes(3); + }); + it.each(['legacy', undefined])("stops a queued %s run without a native-only cancellation request", async runtimeMode => { + const scope = { companyId: 'company', agentId: 'agent', issueId: 'issue' }; + const queued = { ...scope, id: 'run', status: 'queued', runtimeMode }; + const get = vi.fn().mockResolvedValueOnce(queued).mockResolvedValue({ ...queued, status: 'cancelled' }); + const post = vi.fn(async (_route: string, body: Record) => { + if (body.cancellationRequestId) throw new Error('Correlated Stop requires a native run'); + return {}; + }); + await stopOwnedGitHubReviewRun({ get, post } as unknown as ConnectionApi, 'run', scope, new AbortController().signal); + expect(post).toHaveBeenCalledExactlyOnceWith('/api/heartbeat-runs/run/cancel', {}); + expect(get).toHaveBeenCalledTimes(2); + }); + it.each([true, false])("stops a failed native retry and verifies its audited acknowledgement (dispatched=%s)", async dispatched => { + const scope = { companyId: 'company', agentId: 'agent', issueId: 'issue' }; + const failed = { ...scope, id: 'run', status: 'failed', runtimeMode: 'native', nativePhase: 'retryable_failure' }; + const get = vi.fn().mockResolvedValueOnce(failed).mockResolvedValue({ ...failed, status: 'cancelled', costAccountingPending: true, + resultJson: { nativeCancellation: { schema: 'paperclip.native-cancellation.v1', runId: 'run', companyId: 'company', issueId: 'issue', scope: 'run', reasonCode: 'cancellation_run_only', dispatched, dispatchState: 'acknowledged', intentAuditId: 'intent', acknowledgementAuditId: 'ack' } } }); + const post = vi.fn().mockResolvedValue({}); + await stopOwnedGitHubReviewRun({ get, post } as unknown as ConnectionApi, 'run', scope, new AbortController().signal); + expect(post).toHaveBeenCalledExactlyOnceWith('/api/heartbeat-runs/run/cancel', { cancellationRequestId: expect.any(String) }); + expect(get).toHaveBeenCalledTimes(2); + }); + it("refuses to stop a related run belonging to a different task", async () => { + const post = vi.fn(); + const api = { get: async () => ({ id: 'run', companyId: 'company', agentId: 'agent', issueId: 'other', status: 'running' }), post }; + await expect(stopOwnedGitHubReviewRun(api as unknown as ConnectionApi, 'run', { companyId: 'company', agentId: 'agent', issueId: 'issue' }, new AbortController().signal)) + .rejects.toThrow('cleanup_run_owner_mismatch'); + expect(post).not.toHaveBeenCalled(); + }); + it("does not accept cancelled status as native Stop acknowledgement", async () => { + const scope = { companyId: 'company', agentId: 'agent', issueId: 'issue' }; + const get = vi.fn().mockResolvedValueOnce({ ...scope, id: 'run', status: 'running', runtimeMode: 'native' }) + .mockResolvedValue({ ...scope, id: 'run', status: 'cancelled', runtimeMode: 'native' }); + await expect(stopOwnedGitHubReviewRun({ get, post: async () => ({}) } as unknown as ConnectionApi, 'run', scope, new AbortController().signal, 1)) + .rejects.toThrow('owned_run_cancellation_unconfirmed'); + expect(get.mock.calls.length).toBeGreaterThan(1); + }); it("admits an inherited environment before setup and still leaves actual-run verification mandatory", async () => { const setup = parseGitHubReviewJourneyConfig(config()); const company = { id: setup.target.companyId, issuePrefix: "QA" }; diff --git a/tests/runner-e2e/github-review-launch.ts b/tests/runner-e2e/github-review-launch.ts index a78365f6add..83f4667d90c 100644 --- a/tests/runner-e2e/github-review-launch.ts +++ b/tests/runner-e2e/github-review-launch.ts @@ -12,7 +12,7 @@ import { createPrivateJourneyRepository, ensureGitHubJourneyLogin, closeOwnedGit import { loadGitHubReviewJourneyConfig } from "./github-review-config.js"; import { gradeGitHubReviewJourney, type GitHubReviewJourneyEvidence } from "./github-review-evidence.js"; import { runGitHubReviewFlow } from "./github-review-flow.js"; -import { captureOwnedGitHubApp, readOwnedGitHubReviewRuns, type GitHubJourneyResources } from "./github-review-ownership.js"; +import { captureOwnedGitHubApp, readOwnedGitHubReviewRuns, stopOwnedGitHubReviewRun, type GitHubJourneyResources } from "./github-review-ownership.js"; import { JourneyDependencyError, resolveJourneyCredential } from "./journey-dependencies.js"; import { sanitizeJson, assertSecretFree } from "./redaction.js"; import { writeGitHubReviewReport } from "./github-review-report.js"; @@ -101,7 +101,14 @@ export async function runGitHubReviewCampaign(input: { executions: readonly Matr const identity = await verifyConnectionTarget(config.target.baseURL, common, browser.context.request); evidence.targetCommit = identity.commit; evidence.checkpoints.target = true; await runGitHubReviewFlow({ page, github, githubUsername: config.github.account, api, config, resources, evidence, signal, - phase: checkpoint, persistResources: () => save("owned-resources.json", resources), checkpoint: async (id, tab) => { + phase: checkpoint, persistResources: () => save("owned-resources.json", resources), + ...(typeof authentication?.authenticateGitHubJourneyRegistration === "function" ? { + authenticateRegistration: (tab: Page) => authentication.authenticateGitHubJourneyRegistration({ page: tab, + configFile: config!.authentication!.configFile, expectedAccount: config!.github.account, signal, + deadlineAt: Date.now() + Math.min(config!.browser.loginTimeoutMs, 120_000), + registerSecret: (value: string) => secrets.push(value), checkpoint: (id: string) => checkpoint(id) }), + } : {}), + recordRegistrationBoundary: boundary => save("github-registration-boundary.json", boundary), checkpoint: async (id, tab) => { evidence.checkpoints[id] = true; // Authentication pages and forms are deliberately never captured. const consentUrl = new URL(tab.url()); @@ -180,12 +187,8 @@ export async function runGitHubReviewCampaign(input: { executions: readonly Matr await save("owned-resources.json", resources); await api.post(`/api/chat-endpoints/${resources.endpointId}/setup`, { action: "remove" }); for (const runId of resources.runIds) { - const run = await api.get(`/api/heartbeat-runs/${runId}`); - if (run.companyId !== config!.target.companyId || run.agentId !== config!.agent.id) throw new ConnectionFailure("cleanup_run_owner_mismatch"); - if (["queued", "running"].includes(run.status)) { - await api.post(`/api/heartbeat-runs/${runId}/cancel`); - await awaitJourneyCondition(async () => !["queued", "running"].includes((await api!.get(`/api/heartbeat-runs/${runId}`)).status), 30_000, "owned_run_cancellation_unconfirmed", cleanupSignal, "failed"); - } + if (!resources.issueId) throw new ConnectionFailure("cleanup_run_issue_missing"); + await stopOwnedGitHubReviewRun(api, runId, { companyId: config!.target.companyId, agentId: config!.agent.id, issueId: resources.issueId }, cleanupSignal); } if (resources.issueId && result.status !== "passed") { const issue = await api.get(`/api/issues/${resources.issueId}`); diff --git a/tests/runner-e2e/github-review-ownership.ts b/tests/runner-e2e/github-review-ownership.ts index f1d15ee4855..867ff16e84c 100644 --- a/tests/runner-e2e/github-review-ownership.ts +++ b/tests/runner-e2e/github-review-ownership.ts @@ -2,6 +2,10 @@ import type { ChatEndpoint } from "../../packages/shared/src/types/chat-channels import { ConnectionFailure } from "./connection-config.js"; import type { GitHubReviewJourneyConfig } from "./github-review-config.js"; import type { ConnectionApi } from "./connection-target.js"; +import { randomUUID } from "node:crypto"; +import { awaitJourneyCondition } from "./github-review-browser.js"; + +const liveRunStatuses = new Set(["queued", "running", "scheduled_retry"]); /** The issue run source includes history beyond company list windows. Related * activity can name a run for another task, so hydrate and check direct ownership. */ @@ -13,6 +17,33 @@ export async function readOwnedGitHubReviewRuns(api: Pick, return runs.filter(run => run.issueId === scope.issueId && run.agentId === scope.agentId); } +/** A failed native run can still own a durable retry. Stop it through the + * normal audited route and require acknowledgement, not just terminal status. */ +export async function stopOwnedGitHubReviewRun(api: Pick, runId: string, + scope: { companyId: string; agentId: string; issueId: string }, signal: AbortSignal, timeoutMs = 30_000) { + const read = async () => { + const run = await api.get>(`/api/heartbeat-runs/${runId}`); + if (run.id !== runId || run.companyId !== scope.companyId || run.agentId !== scope.agentId || run.issueId !== scope.issueId) { + throw new ConnectionFailure("cleanup_run_owner_mismatch"); + } + return run; + }; + const run = await read(); + const retry = run.runtimeMode === "native" && run.status === "failed" && run.nativePhase === "retryable_failure"; + if (!liveRunStatuses.has(run.status) && !retry) return; + await api.post(`/api/heartbeat-runs/${runId}/cancel`, run.runtimeMode === "native" ? { cancellationRequestId: randomUUID() } : {}); + await awaitJourneyCondition(async () => { + const stopped = await read(); + if (run.runtimeMode !== "native") return !liveRunStatuses.has(stopped.status); + const intent = stopped.resultJson?.nativeCancellation; + return stopped.status === "cancelled" && intent?.schema === "paperclip.native-cancellation.v1" + && intent.runId === runId && intent.companyId === scope.companyId && intent.issueId === scope.issueId + && intent.scope === "run" && intent.reasonCode === "cancellation_run_only" && typeof intent.dispatched === "boolean" + && intent.dispatchState === "acknowledged" && typeof intent.intentAuditId === "string" && intent.intentAuditId.length > 0 + && typeof intent.acknowledgementAuditId === "string" && intent.acknowledgementAuditId.length > 0; + }, timeoutMs, "owned_run_cancellation_unconfirmed", signal, "failed"); +} + export interface GitHubJourneyResources { endpointId?: string; pullNumber?: number; branch: string; appName: string; appId?: string; appSlug?: string; appSettingsUrl?: string; installationSettingsUrl?: string; diff --git a/tests/runner-e2e/github-review-report.test.ts b/tests/runner-e2e/github-review-report.test.ts index 8840fcf171d..9ad23acbcd5 100644 --- a/tests/runner-e2e/github-review-report.test.ts +++ b/tests/runner-e2e/github-review-report.test.ts @@ -5,6 +5,7 @@ import path from "node:path"; import { afterEach, expect, it } from "vitest"; import { runnerMatrix } from "./catalog.js"; import { writeGitHubReviewReport } from "./github-review-report.js"; +import { githubJourneyCheckpoints } from "./github-review-evidence.js"; import { validateRetainedRunnerResult } from "./result-validation.js"; import type { RunnerE2EResult } from "./types.js"; @@ -49,3 +50,30 @@ it("refuses an altered screenshot rather than retaining its claimed digest", asy await expect(writeGitHubReviewReport({ output, campaignId: "altered", result: fixture(), resources: { branch: "owned", appName: "owned", runIds: [] }, catalog: runnerMatrix, secrets: [] })) .rejects.toThrow("journey_screenshot_digest_mismatch"); }); + +it("stages the verified persisted checkpoint as the standard report's final state", async () => { + const output = await mkdtemp(path.join(os.tmpdir(), "github-report-test-")); temporary.push(output); + const input = fixture(); + input.status = "passed"; delete input.error; + input.githubReviewJourney!.outcome = "passed"; + input.githubReviewJourney!.checkpoints = Object.fromEntries(githubJourneyCheckpoints.map(id => [id, true])); + input.screenshots = [{ id: "persisted", label: "Persisted completion", file: "persisted.png", sha256: createHash("sha256").update(pixel).digest("hex") }]; + await writeFile(path.join(output, "persisted.png"), pixel); + const result = await writeGitHubReviewReport({ output, campaignId: "passing-journey", result: input, resources: { branch: "owned", appName: "owned", runIds: ["run"] }, catalog: runnerMatrix, secrets: [] }); + expect(result.status).toBe("passed"); + const manifest = JSON.parse(await readFile(path.join(output, "evidence/evidence-manifest.json"), "utf8")); + expect(manifest.files).toContain("final-state.png"); + expect(await readFile(path.join(output, "evidence/final-state.png"))).toEqual(pixel); + expect(await readFile(path.join(output, "evidence/snapshots/persisted.png"))).toEqual(pixel); +}); + +it("refuses a passing report without its persisted screenshot", async () => { + const output = await mkdtemp(path.join(os.tmpdir(), "github-report-test-")); temporary.push(output); + const input = fixture(); + input.status = "passed"; delete input.error; + input.githubReviewJourney!.outcome = "passed"; + input.githubReviewJourney!.checkpoints = Object.fromEntries(githubJourneyCheckpoints.map(id => [id, true])); + await writeFile(path.join(output, "failure.png"), pixel); + await expect(writeGitHubReviewReport({ output, campaignId: "missing-final-state", result: input, resources: { branch: "owned", appName: "owned", runIds: ["run"] }, catalog: runnerMatrix, secrets: [] })) + .rejects.toThrow("journey_final_state_missing"); +}); diff --git a/tests/runner-e2e/github-review-report.ts b/tests/runner-e2e/github-review-report.ts index 6ad2cc05149..7d0e3332619 100644 --- a/tests/runner-e2e/github-review-report.ts +++ b/tests/runner-e2e/github-review-report.ts @@ -29,6 +29,7 @@ export async function writeGitHubReviewReport(input: { try { await mkdir(path.join(stage, "snapshots"), { mode: 0o700 }); const result = structuredClone(input.result); + let finalStateStaged = false; for (const shot of result.screenshots ?? []) { const source = path.resolve(output, shot.file); if (!source.startsWith(`${output}${path.sep}`) || !/^[a-z0-9_-]+$/.test(shot.id) || path.extname(source) !== ".png") throw new Error("unsafe_journey_screenshot_reference"); @@ -37,8 +38,13 @@ export async function writeGitHubReviewReport(input: { if (shot.sha256 && createHash("sha256").update(bytes).digest("hex") !== shot.sha256) throw new Error("journey_screenshot_digest_mismatch"); shot.file = `snapshots/${shot.id}.png`; await writeFile(path.join(stage, shot.file), bytes, { mode: 0o600 }); + if (result.status === "passed" && shot.id === "persisted" && result.githubReviewJourney?.checkpoints.persisted === true) { + await writeFile(path.join(stage, "final-state.png"), bytes, { mode: 0o600 }); + finalStateStaged = true; + } } validateRetainedRunnerResult(result); + if (result.status === "passed" && !finalStateStaged) throw new Error("journey_final_state_missing"); await save(stage, "result.json", result); await save(stage, "snapshots/journey-resources.json", input.resources); await save(stage, "snapshots/journey-evidence.json", result.githubReviewJourney); diff --git a/tests/runner-e2e/history.test.ts b/tests/runner-e2e/history.test.ts index c717441e321..b6216af3745 100644 --- a/tests/runner-e2e/history.test.ts +++ b/tests/runner-e2e/history.test.ts @@ -32,6 +32,7 @@ import { renderRunnerHistoryIndex } from "./history-index.js"; import { renderPublicCampaignSummary } from "./public-summary-image.js"; import { isPublicRunnerScreenshotRoute } from "./screenshot-policy.js"; import { connectionCheckpoints, type ConnectionEvidence } from "./connection-evidence.js"; +import { githubJourneyCheckpoints, type GitHubReviewJourneyEvidence } from "./github-review-evidence.js"; import type { MatrixExecution, RunnerE2EResult } from "./types.js"; const temporaryDirectories: string[] = []; @@ -65,6 +66,12 @@ function result(execution: MatrixExecution, status: "passed" | "failed") { durationMs: 1_000, cleanup: "passed", ...(execution.task.flow === "provider_connection" ? { providerConnection: { version: 1, outcome: status, phase: "finished", assisted: false, authFreshness: "signed-in", target: { mode: "attach", origin: "http://127.0.0.1:3100", commit: "a".repeat(40), deploymentMode: "local_trusted" }, entry: "agent", method: "api-key", checkpoints: Object.fromEntries(connectionCheckpoints.map(key => [key, true])), waits: [] } satisfies ConnectionEvidence } : {}), + ...(execution.task.flow === "github_review_journey" ? { githubReviewJourney: { + version: 1, outcome: status, phase: "finished", setupId: "fixture", hosting: "self-hosted", + targetCommit: "a".repeat(40), assisted: false, dependencies: [], + checkpoints: Object.fromEntries(githubJourneyCheckpoints.map(key => [key, true])), + cleanup: { endpoint: true, pullRequest: true, branch: true, app: true }, + } satisfies GitHubReviewJourneyEvidence } : {}), } satisfies RunnerE2EResult; } diff --git a/tests/runner-e2e/live-fixtures.ts b/tests/runner-e2e/live-fixtures.ts index cc861c6232c..c5d27531b43 100644 --- a/tests/runner-e2e/live-fixtures.ts +++ b/tests/runner-e2e/live-fixtures.ts @@ -171,7 +171,7 @@ export async function setupLiveFixtures(input: { : ["native-completion", "native-instruction-consolidation", "native-connection-guidance"].includes(execution.suite.id) || (execution.suite.id === "everyday-workflows" && ["hire-reuse", "delegate-feedback"].includes(execution.task.id)) ? NATIVE_COMPLETION_BUDGET_CENTS : execution.suite.id === "task-titles" ? TASK_TITLE_BUDGET_CENTS - : ["stock-harness", "task-workspaces", "case-handoffs", "feedback-persistence"].includes(execution.suite.id) ? 1_000 : 0, + : ["stock-harness", "task-workspaces", "case-handoffs", "feedback-persistence", "daytona-restart-continuity"].includes(execution.suite.id) ? 1_000 : 0, }); await input.onCompanyCreated?.(company.id); return company; @@ -365,7 +365,7 @@ export async function setupLiveFixtures(input: { setupProbe.defaultRunner = selected.defaultRunner; } } - if (["stock-harness", "native-connection-guidance", "feedback-persistence"].includes(execution.suite.id) + if (["stock-harness", "native-connection-guidance", "feedback-persistence", "daytona-restart-continuity"].includes(execution.suite.id) || (execution.suite.id === "everyday-workflows" && ["hire-reuse", "delegate-feedback"].includes(execution.task.id))) { agent.budgetMonthlyCents = 1_000; } diff --git a/tests/runner-e2e/playwright-support.config.ts b/tests/runner-e2e/playwright-support.config.ts index bdaefb27292..0256f9ef884 100644 --- a/tests/runner-e2e/playwright-support.config.ts +++ b/tests/runner-e2e/playwright-support.config.ts @@ -3,7 +3,7 @@ import { defineConfig } from "@playwright/test"; /** Browser-only harness regressions: no Paperclip instance or provider credentials. */ export default defineConfig({ testDir: ".", - testMatch: ["screenshot-readiness.spec.ts", "service-worker-reload.spec.ts", "lost-send.spec.ts", "chat-restart.spec.ts", "settings-toggle.spec.ts", "browser-bootstrap-diagnostics.spec.ts", "service-worker-module-loading.spec.ts", "browser-bootstrap-recovery.spec.ts", "blocker-input.spec.ts", "native-blocker-visible.spec.ts", "copilot-permission-card.spec.ts"], + testMatch: ["screenshot-readiness.spec.ts", "service-worker-reload.spec.ts", "lost-send.spec.ts", "chat-restart.spec.ts", "settings-toggle.spec.ts", "browser-bootstrap-diagnostics.spec.ts", "service-worker-module-loading.spec.ts", "browser-bootstrap-recovery.spec.ts", "blocker-input.spec.ts", "native-blocker-visible.spec.ts", "copilot-permission-card.spec.ts", "github-review-check.spec.ts"], workers: 1, retries: 0, timeout: 10_000, diff --git a/tests/runner-e2e/product-journeys/cloud-onboarding.test.ts b/tests/runner-e2e/product-journeys/cloud-onboarding.test.ts index 94589ca5069..0878ade8065 100644 --- a/tests/runner-e2e/product-journeys/cloud-onboarding.test.ts +++ b/tests/runner-e2e/product-journeys/cloud-onboarding.test.ts @@ -86,6 +86,8 @@ function fixture(entry: Entry, options: { intentFailure?: boolean; qualification }, }); const page = { + on() { return this; }, + off() { return this; }, getByLabel(label: string) { return locator(() => controls().filter(control => control.name === label && control.role === "textbox")); }, getByRole(role: string, { name: label }: { name: string | RegExp }) { return locator(() => controls().filter(control => control.role === role && match(control.name, label))); @@ -93,6 +95,7 @@ function fixture(entry: Entry, options: { intentFailure?: boolean; qualification locator(selector: string) { expect(selector).toBe("summary"); return locator(() => controls().filter(control => control.role === "summary")); }, }; const context = { page, company: { id: "company" }, identity: { kind: "cloud" }, startingState: "recipe", api: { + origin: "https://tenant.example.test", async get(route: string) { if (route.endsWith("/ai-connections")) return { connections: saved ? [connection] : [] }; if (route.endsWith("/agents")) return hired ? [{ id: "agent", name, adapterType: "paperclip_runner", adapterConfig: { provider: "codex", model: gateway.model } }] : []; diff --git a/tests/runner-e2e/product-journeys/cloud-onboarding.ts b/tests/runner-e2e/product-journeys/cloud-onboarding.ts index 3173676c3ad..6f9a75634a7 100644 --- a/tests/runner-e2e/product-journeys/cloud-onboarding.ts +++ b/tests/runner-e2e/product-journeys/cloud-onboarding.ts @@ -1,5 +1,6 @@ import { expect } from "@playwright/test"; import { createConnectionProof } from "../connection-evidence.js"; +import { onboardingConnect } from "./onboarding-connect.js"; import { onboardingFirstTask } from "./onboarding-first-task.js"; import { JourneyError, type JourneyContext, type JourneyEvidence, type TargetSession } from "./types.js"; @@ -69,11 +70,12 @@ export async function cloudOnboarding(input: CloudOnboardingInput): Promise { evidence.onboardingConnect = value; } }); const agents = await api.get(`/api/companies/${company.id}/agents`); const agent = agents.find(a => a.name === name); if (!agent || agents.length !== 1 || agent.adapterType !== "paperclip_runner" || agent.adapterConfig?.provider !== "codex" || agent.adapterConfig?.model !== input.gateway.model) throw new JourneyError("failed", "onboarding_agent_configuration_mismatch"); + connect.checkpoint = "agent-verification"; await input.own({ kind: "agent", name, id: agent.id }); evidence.checks.push({ id: "ui-connection-and-native-agent", passed: true }); await input.own({ kind: "issue", name: "Paperclip onboarding" }); diff --git a/tests/runner-e2e/product-journeys/contracts.test.ts b/tests/runner-e2e/product-journeys/contracts.test.ts index 754d6c7bab9..b02057281a3 100644 --- a/tests/runner-e2e/product-journeys/contracts.test.ts +++ b/tests/runner-e2e/product-journeys/contracts.test.ts @@ -13,6 +13,7 @@ import type { RunnerE2EResult } from "../types.js"; import type { JourneyContext, JourneyEvidence, TargetSession } from "./types.js"; import { createConnectionProof } from "../connection-evidence.js"; import { retainOnboardingArtifact } from "./artifact-evidence.js"; +import type { OnboardingConnectEvidence } from "./onboarding-connect.js"; vi.mock("@playwright/test", () => ({ expect: Object.assign( (locator: { textContent(): Promise }) => ({ async toContainText(value: string) { expect(await locator.textContent()).toContain(value); } }), @@ -29,7 +30,60 @@ function result(): RunnerE2EResult { return { schema: "paperclip.runner-e2e.result/v2", suiteId: "product-journeys", suiteDefinitionHash: "test", executionId: "product-journeys.local-qa.local.task-properties", profileId: "local-qa", environmentId: "local", caseId: "task-properties", provider: null, model: null, runtimeMode: "none", status: "passed", cleanup: "passed", attempt: 1, startedAt: "2026-01-01T00:00:00Z", finishedAt: "2026-01-01T00:00:01Z", durationMs: 1000, productJourney: { outcome: "passed", startingState: "recipe", phases: [{phase: "journey", outcome: "passed"}], checks: [{id:"assignment-persisted", passed:true},{id:"no-agent-work",passed:true}], privacy: "private", resourceDisposition: "released", deployment: { kind: "self-hosted", targetId: "local-qa", origin: "http://127.0.0.1:3100", commit: "a".repeat(40), cloudRelease: null, runtimeImage: null, configurationHash: "test" } } }; } +function connectReceipt(): OnboardingConnectEvidence { + return { + version: 1, checkpoint: "hire", + validation: { state: "json", httpStatus: 200, requestedAdapter: "paperclip_runner", adapter: "paperclip_runner", outcome: "fail", serverCode: null, checks: [{ code: "adapter_auth_missing", level: "error" }], truncatedChecks: false }, + hire: { state: "json", httpStatus: 422, requestedAdapter: null, adapter: null, outcome: null, serverCode: "agent_runner_unavailable", checks: [], truncatedChecks: false }, + }; +} describe("deployment journey contracts", () => { + it("accepts historical results without Connect diagnostics and optional closed receipts", () => { + const retained = result(); + retained.caseId = "cloud-onboarding"; + retained.status = "failed"; + expect(() => validateRetainedRunnerResult(retained)).not.toThrow(); + expect(renderCaseOutcome(retained, true, [])).not.toContain("Onboarding Connect diagnostics"); + retained.productJourney!.onboardingConnect = connectReceipt(); + expect(() => validateRetainedRunnerResult(JSON.parse(JSON.stringify(retained)))).not.toThrow(); + retained.productJourney!.onboardingConnect.validation = { state: "pending", httpStatus: 200, requestedAdapter: "codex_local", adapter: null, outcome: null, serverCode: null, checks: [], truncatedChecks: false }; + expect(() => validateRetainedRunnerResult(retained)).not.toThrow(); + }); + it.each([ + ["request headers", () => ({ ...connectReceipt(), requestHeaders: { authorization: "must-not-retain" } })], + ["response bodies", () => ({ ...connectReceipt(), validation: { ...connectReceipt().validation, body: "must-not-retain" } })], + ["check messages", () => ({ ...connectReceipt(), validation: { ...connectReceipt().validation, checks: [{ code: "adapter_auth_missing", level: "error", message: "must-not-retain" }] } })], + ["unknown server codes", () => ({ ...connectReceipt(), hire: { ...connectReceipt().hire, serverCode: "raw-provider-error" } })], + ["unknown check codes", () => ({ ...connectReceipt(), validation: { ...connectReceipt().validation, checks: [{ code: "raw-provider-error", level: "error" }] } })], + ["unknown adapters", () => ({ ...connectReceipt(), validation: { ...connectReceipt().validation, adapter: "raw-provider-name" } })], + ["unknown requested adapters", () => ({ ...connectReceipt(), validation: { ...connectReceipt().validation, requestedAdapter: "raw-provider-name" } })], + ["unknown outcomes", () => ({ ...connectReceipt(), validation: { ...connectReceipt().validation, outcome: "raw-provider-message" } })], + ["unknown response states", () => ({ ...connectReceipt(), validation: { ...connectReceipt().validation, state: "raw-provider-message" } })], + ["unknown check levels", () => ({ ...connectReceipt(), validation: { ...connectReceipt().validation, checks: [{ code: "adapter_auth_missing", level: "fatal" }] } })], + ["oversized check arrays", () => ({ ...connectReceipt(), validation: { ...connectReceipt().validation, checks: Array.from({ length: 33 }, () => ({ code: "adapter_auth_missing", level: "error" })) } })], + ["invalid HTTP statuses", () => ({ ...connectReceipt(), validation: { ...connectReceipt().validation, httpStatus: 600 } })], + ["non-boolean truncation", () => ({ ...connectReceipt(), validation: { ...connectReceipt().validation, truncatedChecks: "true" } })], + ["missing response fields", () => ({ ...connectReceipt(), hire: { state: "missing" } })], + ["unsupported receipt versions", () => ({ ...connectReceipt(), version: 2 })], + ["unknown checkpoints", () => ({ ...connectReceipt(), checkpoint: "raw-browser-text" })], + ] as const)("rejects Connect receipts containing %s", (_label, makeReceipt) => { + const retained = result(); + retained.productJourney!.onboardingConnect = makeReceipt() as OnboardingConnectEvidence; + expect(() => validateRetainedRunnerResult(retained)).toThrow("result.productJourney.onboardingConnect"); + }); + it("renders only validated Connect fields and marks missing or truncated evidence", () => { + const retained = result(); + retained.productJourney!.onboardingConnect = connectReceipt(); + const output = renderCaseOutcome(retained, true, []); + for (const text of ["Onboarding Connect diagnostics", "Last checkpoint", "Native validation", "Agent hire", "HTTP status", "Test endpoint adapter", "Returned adapter", "200", "422", "paperclip_runner", "adapter_auth_missing", "agent_runner_unavailable", "error", "Unavailable"]) expect(output).toContain(text); + retained.productJourney!.onboardingConnect.validation.truncatedChecks = true; + expect(renderCaseOutcome(retained, true, [])).toContain("Only the first 32 validation checks were retained"); + Object.assign(retained.productJourney!.onboardingConnect.validation, { body: "" }); + const invalid = renderCaseOutcome(retained, false, []); + expect(invalid).toContain("Onboarding Connect diagnostics unavailable: invalid receipt"); + expect(invalid).not.toContain("must-not-render"); + expect(invalid).not.toContain("adapter_auth_missing"); + }); it("validates optional historical artifact evidence without trusting extra fields or a different run", () => { const retained = result(); retained.caseId = "cloud-onboarding"; diff --git a/tests/runner-e2e/product-journeys/launch.ts b/tests/runner-e2e/product-journeys/launch.ts index 2d0233bd2e9..410bb7725bf 100644 --- a/tests/runner-e2e/product-journeys/launch.ts +++ b/tests/runner-e2e/product-journeys/launch.ts @@ -99,7 +99,7 @@ export async function launchProductJourneys(args: string[], repositoryRoot: stri try { sha = execFileSync("git", ["rev-parse", "HEAD"], { cwd: repositoryRoot, encoding: "utf8", stdio: ["ignore", "pipe", "ignore"] }).trim(); } catch { /* Packaged harnesses may not include Git metadata. */ } const source = resolveRunnerE2ESource({ sha, ref: null, workflowRunUrl: null }); const definitionHash = createHash("sha256").update(JSON.stringify(productJourneys)); - for (const name of ["catalog.ts", "task-properties.ts", "cloud-onboarding.ts", "onboarding-first-task.ts", "artifact-evidence.ts", "fixtures.ts", "launch.ts", "types.ts", "collection.ts", "local-target.ts"]) definitionHash.update(await readFile(path.join(import.meta.dirname, name))); + for (const name of ["catalog.ts", "task-properties.ts", "cloud-onboarding.ts", "onboarding-connect.ts", "onboarding-first-task.ts", "artifact-evidence.ts", "fixtures.ts", "launch.ts", "types.ts", "collection.ts", "local-target.ts"]) definitionHash.update(await readFile(path.join(import.meta.dirname, name))); const suiteDefinitionHash = definitionHash.digest("hex"); const expected = config.journeys.map(id => `product-journeys.${config.id}.${id === "cloud-onboarding" ? config.modelProfileId : "browser"}.${config.executionEnvironment}.${id}`); let reportPersisted = false; diff --git a/tests/runner-e2e/product-journeys/onboarding-connect.test.ts b/tests/runner-e2e/product-journeys/onboarding-connect.test.ts new file mode 100644 index 00000000000..be9535cf0c7 --- /dev/null +++ b/tests/runner-e2e/product-journeys/onboarding-connect.test.ts @@ -0,0 +1,182 @@ +import { EventEmitter } from "node:events"; +import { afterEach, describe, expect, it, vi } from "vitest"; +import { onboardingConnect, validateOnboardingConnectEvidence, type OnboardingConnectEvidence } from "./onboarding-connect.js"; + +const origin = "https://tenant.example.test", companyId = "company-1"; +const validation = `/api/companies/${companyId}/adapters/paperclip_runner/test-environment`; +const hire = `/api/companies/${companyId}/agent-hires`; +const secret = "PRIVATE_SENTINEL_do_not_retain"; +function fixture(options: { ready?: boolean; durationMs?: number; click?: () => Promise | void; wait?: () => Promise } = {}) { + const events = new EventEmitter(); + let retained!: OnboardingConnectEvidence; + const page = Object.assign(events, { getByRole(_role: string, { name }: { name: string }) { + return { click: vi.fn(async () => { expect(name).toBe("Connect"); await options.click?.(); }), + waitFor: vi.fn(({ timeout }: { timeout: number }) => { + expect(name).toBe("Get started"); + if (options.wait) return options.wait(); + return options.ready ? Promise.resolve() : new Promise((_resolve, reject) => { + setTimeout(() => reject(Object.assign(new Error(secret), { name: "TimeoutError" })), timeout); + }); + }) }; + } }); + const request = (path = validation, method = "POST", requestOrigin = origin) => ({ url: () => requestOrigin + path, method: () => method }); + const respond = (body: unknown, status = 200, path = validation, method = "POST", requestOrigin = origin) => { + const req = request(path, method, requestOrigin); + const response = { request: () => req, status: () => status, body: () => Promise.resolve(Buffer.isBuffer(body) ? body : Buffer.from(JSON.stringify(body))) }; + events.emit("request", req); events.emit("response", response); + return response; + }; + const run = () => onboardingConnect({ page: page as any, origin, companyId, deadlineAt: Date.now() + (options.durationMs ?? 100), retain: value => { retained = value; } }); + return { page, events, request, respond, run, receipt: () => retained }; +} +afterEach(() => { vi.useRealTimers(); }); + +describe("one onboarding Connect diagnostic boundary", () => { + it("does not press Connect after the action deadline already expired", async () => { + const click = vi.fn(); + const f = fixture({ durationMs: 0, click }); + await expect(f.run()).rejects.toMatchObject({ code: "onboarding_connect_timeout" }); + expect(click).not.toHaveBeenCalled(); + expect(f.receipt().checkpoint).toBe("connect"); + expect(f.events.eventNames()).toEqual([]); + }); + + it("rejects a delayed failure body even after Get started becomes visible", async () => { + let resolveBody!: (bytes: Buffer) => void; + const body = new Promise(resolve => { resolveBody = resolve; }); + let reviewSeen!: () => void; + const review = new Promise(resolve => { reviewSeen = resolve; }); + const f = fixture({ wait: async () => { reviewSeen(); }, click: () => { + const req = f.request(); f.events.emit("request", req); + f.events.emit("response", { request: () => req, status: () => 200, body: () => body }); + } }); + const result = expect(f.run()).rejects.toMatchObject({ code: "onboarding_native_validation_failed" }); + await review; + for (let i = 0; i < 10 && f.events.listenerCount("response"); i++) await Promise.resolve(); + expect(f.events.listenerCount("response")).toBe(0); + resolveBody(Buffer.from(JSON.stringify({ status: "fail", checks: [] }))); + await result; + expect(f.receipt().validation.outcome).toBe("fail"); + expect(f.receipt().checkpoint).toBe("validation"); + expect(f.events.eventNames()).toEqual([]); + }); + + it.each(["details", "top-level"])("retains a known %s server rejection without its free text", async location => { + const f = fixture({ click: () => { f.respond({ error: secret, ...(location === "details" ? { details: { code: "agent_runner_config_incompatible", fields: [secret] } } : { code: "agent_runner_unavailable" }) }, 422); } }); + await expect(f.run()).rejects.toMatchObject({ code: "onboarding_native_validation_failed" }); + expect(f.receipt()).toMatchObject({ checkpoint: "validation", validation: { state: "json", httpStatus: 422, serverCode: location === "details" ? "agent_runner_config_incompatible" : "agent_runner_unavailable" }, hire: { state: "missing" } }); + expect(JSON.stringify(f.receipt())).not.toContain(secret); + validateOnboardingConnectEvidence(f.receipt()); + expect(f.events.listenerCount("response")).toBe(0); + }); + + it.each([{ status: "fail", checks: [] }, { status: "pass", checks: [{ code: "adapter_auth_missing", level: "error" }] }])("fails on a 200 validation rejection: %j", async body => { + const f = fixture({ click: () => { f.respond(body); } }); + await expect(f.run()).rejects.toMatchObject({ code: "onboarding_native_validation_failed" }); + expect(f.receipt().validation.httpStatus).toBe(200); + }); + + it("retains the hire boundary rejection and does not wait for Get started", async () => { + const f = fixture({ click: () => { f.respond({ details: { code: "agent_runner_platform_probe_failed" }, error: secret }, 503, hire); } }); + await expect(f.run()).rejects.toMatchObject({ code: "onboarding_agent_hire_failed" }); + expect(f.receipt()).toMatchObject({ checkpoint: "hire", hire: { httpStatus: 503, serverCode: "agent_runner_platform_probe_failed" } }); + }); + + it("projects only enums from malicious nested fields, including unknown codes and bounded checks", async () => { + const f = fixture({ ready: true, click: () => { f.respond({ adapterType: secret, status: secret, code: { credential: secret }, details: { code: secret }, + message: secret, hint: secret, headers: { authorization: secret }, checks: Array.from({ length: 40 }, () => ({ code: secret, level: secret, message: secret, nested: { apiKey: secret } })) }); } }); + const result = await f.run(); + expect(result.validation).toMatchObject({ adapter: "unrecognized", outcome: "unrecognized", serverCode: "unrecognized", truncatedChecks: true }); + expect(result.validation.checks).toHaveLength(32); + expect(result.validation.checks[0]).toEqual({ code: "unrecognized", level: "unrecognized" }); + expect(JSON.stringify(result)).not.toContain(secret); + validateOnboardingConnectEvidence(result); + expect(() => validateOnboardingConnectEvidence({ ...result, validation: { ...result.validation, requestedAdapter: ["codex_local"] } })).toThrow(); + }); + + it("ignores wrong origin, company, method, query, and responses without an in-action request", async () => { + vi.useFakeTimers(); + const f = fixture({ click: () => { + f.respond({}, 500, validation, "POST", "https://foreign.example.test"); + f.respond({}, 500, validation.replace(companyId, "company-2")); + f.respond({}, 500, validation, "GET"); + f.respond({}, 500, validation + "?private=" + secret); + f.events.emit("response", { request: () => f.request(), status: () => 500 }); + } }); + const result = expect(f.run()).rejects.toMatchObject({ code: "onboarding_get_started_timeout" }); + await vi.advanceTimersByTimeAsync(101); await result; + expect(f.receipt()).toMatchObject({ checkpoint: "connect", validation: { state: "missing" }, hire: { state: "missing" } }); + expect(f.events.eventNames()).toEqual([]); + }); + + it.each([Buffer.from("{not-json"), Buffer.alloc(65_537, "x")])("keeps malformed or oversized bodies unknown and still requires Get started", async bytes => { + vi.useFakeTimers(); + const f = fixture({ click: () => { f.respond(bytes); } }); + const result = expect(f.run()).rejects.toMatchObject({ code: "onboarding_get_started_timeout" }); + await vi.advanceTimersByTimeAsync(101); await result; + expect(f.receipt().validation.state).toBe(bytes.byteLength > 65_536 ? "oversized" : "unreadable"); + expect(f.receipt().checkpoint).toBe("validation"); + }); + + it("records missing responses and ignores late responses after timeout", async () => { + vi.useFakeTimers(); + const f = fixture({ click: () => { f.events.emit("request", f.request()); } }); + const result = expect(f.run()).rejects.toMatchObject({ code: "onboarding_get_started_timeout" }); + await vi.advanceTimersByTimeAsync(101); await result; + const sealed = JSON.stringify(f.receipt()); + f.respond({ status: "pass", adapterType: "paperclip_runner", checks: [] }); + expect(JSON.stringify(f.receipt())).toBe(sealed); + expect(f.receipt().validation.state).toBe("missing"); + }); + + it("bounds stalled JSON reads and cannot mutate retained evidence after return or reject unhandled", async () => { + vi.useFakeTimers(); + let rejectBody!: (reason: Error) => void; + const body = new Promise((_resolve, reject) => { rejectBody = reject; }); + const f = fixture({ ready: true, durationMs: 10_000, click: () => { + const req = f.request(); f.events.emit("request", req); + f.events.emit("response", { request: () => req, status: () => 200, body: () => body }); + } }); + const result = f.run(); + await vi.advanceTimersByTimeAsync(5001); + expect((await result).validation.state).toBe("unreadable"); + const sealed = JSON.stringify(f.receipt()); + rejectBody(new Error(secret)); await Promise.resolve(); + expect(JSON.stringify(f.receipt())).toBe(sealed); + expect(f.events.eventNames()).toEqual([]); + }); + + it("preserves the Get started timeout when a previously received body arrives later", async () => { + vi.useFakeTimers(); + let resolveBody!: (bytes: Buffer) => void; + const body = new Promise(resolve => { resolveBody = resolve; }); + const f = fixture(); + const result = expect(f.run()).rejects.toMatchObject({ code: "onboarding_get_started_timeout" }); + await vi.advanceTimersByTimeAsync(10); + const req = f.request(); f.events.emit("request", req); + f.events.emit("response", { request: () => req, status: () => 200, body: () => body }); + await vi.advanceTimersByTimeAsync(90); + resolveBody(Buffer.from(JSON.stringify({ status: "fail" }))); + await vi.advanceTimersByTimeAsync(1); await result; + expect(f.receipt().checkpoint).toBe("validation"); + expect(f.events.eventNames()).toEqual([]); + }); + + it("keeps a browser error at its own boundary without retaining the browser text", async () => { + const f = fixture({ click: () => { throw new Error(secret); } }); + await expect(f.run()).rejects.toMatchObject({ code: "onboarding_connect_failed" }); + expect(f.receipt().checkpoint).toBe("connect"); + expect(JSON.stringify(f.receipt())).not.toContain(secret); + expect(f.events.eventNames()).toEqual([]); + }); + + it("observes both successful responses and requires the visible review step", async () => { + const f = fixture({ ready: true, click: () => { + f.respond({ adapterType: "paperclip_runner", status: "pass", checks: [{ code: "codex_hello_probe_passed", level: "info", message: secret }] }, 200, validation.replace("paperclip_runner", "codex_local")); + f.respond({ agent: { id: secret }, approval: null }, 201, hire); + } }); + expect(await f.run()).toMatchObject({ checkpoint: "review", validation: { requestedAdapter: "codex_local", adapter: "paperclip_runner", outcome: "pass", checks: [{ code: "codex_hello_probe_passed", level: "info" }] }, hire: { state: "json", httpStatus: 201 } }); + expect(JSON.stringify(f.receipt())).not.toContain(secret); + expect(f.events.eventNames()).toEqual([]); + }); +}); diff --git a/tests/runner-e2e/product-journeys/onboarding-connect.ts b/tests/runner-e2e/product-journeys/onboarding-connect.ts new file mode 100644 index 00000000000..4aa6e43becf --- /dev/null +++ b/tests/runner-e2e/product-journeys/onboarding-connect.ts @@ -0,0 +1,131 @@ +import type { Page, Response, Request } from "@playwright/test"; +import { JourneyError } from "./types.js"; + +export const onboardingServerCodes = ["agent_runner_config_incompatible", "agent_runner_unavailable", "agent_runner_platform_probe_failed", "managed_sandbox_unavailable", "environment_company_mismatch"] as const; +export const onboardingCheckCodes = ["adapter_auth_missing", "codex_hello_probe_failed", "codex_hello_probe_passed", "codex_hello_probe_timeout", "codex_hello_probe_auth_required", "environment_lease_acquire_failed", "environment_workspace_realize_failed", "environment_target_failed", "environment_target_unsupported", "environment_not_found", "environment_target_unavailable", "managed_sandbox_unavailable", "environment_env_binding_missing", "environment_env_resolution_failed", "ai_connection_api_key_rejected", "ai_connection_verification_failed", "sandbox_test_identity"] as const; +const checkpoints = ["connect", "validation", "hire", "review", "agent-verification"] as const; +const states = ["missing", "pending", "json", "unreadable", "oversized"] as const; +const requestedAdapters = ["codex_local", "paperclip_runner"] as const; +const adapters = ["codex_local", "paperclip_runner", "unrecognized"] as const; +const outcomes = ["pass", "warn", "fail", "unrecognized"] as const; +const levels = ["info", "warn", "error", "unrecognized"] as const; +type Known = T[number] | "unrecognized"; +export interface OnboardingResponseEvidence { + state: typeof states[number]; httpStatus: number | null; + requestedAdapter: "codex_local" | "paperclip_runner" | null; + adapter: typeof adapters[number] | null; outcome: typeof outcomes[number] | null; + serverCode: Known | null; + checks: Array<{ code: Known; level: typeof levels[number] }>; + truncatedChecks: boolean; +} +export interface OnboardingConnectEvidence { + version: 1; checkpoint: typeof checkpoints[number]; validation: OnboardingResponseEvidence; hire: OnboardingResponseEvidence; +} +const blank = (): OnboardingResponseEvidence => ({ state: "missing", httpStatus: null, requestedAdapter: null, adapter: null, outcome: null, serverCode: null, checks: [], truncatedChecks: false }); +const record = (v: unknown): v is Record => Boolean(v && typeof v === "object" && !Array.isArray(v)); +const member = (values: T, v: unknown): v is T[number] => typeof v === "string" && values.includes(v); +const closed = (v: unknown, keys: string[]) => record(v) && Object.keys(v).length === keys.length && keys.every(key => Object.hasOwn(v, key)); +export function validateOnboardingConnectEvidence(value: unknown): void { + if (!closed(value, ["version", "checkpoint", "validation", "hire"]) || !record(value) || value.version !== 1 || !member(checkpoints, value.checkpoint)) throw new Error("invalid_onboarding_connect_evidence"); + for (const response of [value.validation, value.hire]) { + if (!closed(response, ["state", "httpStatus", "requestedAdapter", "adapter", "outcome", "serverCode", "checks", "truncatedChecks"]) || !record(response) || + !member(states, response.state) || !(response.httpStatus === null || Number.isSafeInteger(response.httpStatus) && Number(response.httpStatus) >= 100 && Number(response.httpStatus) <= 599) || + !(response.requestedAdapter === null || member(requestedAdapters, response.requestedAdapter)) || + !(response.adapter === null || member(adapters, response.adapter)) || !(response.outcome === null || member(outcomes, response.outcome)) || + !(response.serverCode === null || response.serverCode === "unrecognized" || member(onboardingServerCodes, response.serverCode)) || + typeof response.truncatedChecks !== "boolean" || !Array.isArray(response.checks) || response.checks.length > 32 || + response.checks.some(check => !closed(check, ["code", "level"]) || !record(check) || !(check.code === "unrecognized" || member(onboardingCheckCodes, check.code)) || !member(levels, check.level))) throw new Error("invalid_onboarding_connect_evidence"); + } +} + +/** Observe only responses to this Connect action; never retain bodies, text, headers or browser errors. */ +export async function onboardingConnect(input: { page: Page; origin: string; companyId: string; deadlineAt: number; retain(value: OnboardingConnectEvidence): void }): Promise { + const { page } = input, receipt: OnboardingConnectEvidence = { version: 1, checkpoint: "connect", validation: blank(), hire: blank() }; + input.retain(receipt); + let active = true, failure: JourneyError | undefined; + let signalFailure!: (error: JourneyError) => void; + const failed = new Promise(resolve => { signalFailure = resolve; }); + const requests = new Set(), pending = new Set>(); + const deadlineAt = Math.min(input.deadlineAt, Date.now() + 180_000); + const remaining = () => Math.max(1, deadlineAt - Date.now()); + const advance = (checkpoint: OnboardingConnectEvidence["checkpoint"]) => { + if (checkpoints.indexOf(checkpoint) > checkpoints.indexOf(receipt.checkpoint)) receipt.checkpoint = checkpoint; + }; + const reject = (code: string) => { failure ??= new JourneyError("failed", code); signalFailure(failure); }; + const route = (request: Request): { kind: "validation" | "hire"; adapter: "codex_local" | "paperclip_runner" | null } | null => { + try { + const url = new URL(request.url()); + if (request.method() !== "POST" || url.origin !== input.origin || url.search || url.hash) return null; + const prefix = `/api/companies/${encodeURIComponent(input.companyId)}`; + if (url.pathname === `${prefix}/agent-hires`) return { kind: "hire", adapter: null }; + for (const adapter of ["codex_local", "paperclip_runner"] as const) if (url.pathname === `${prefix}/adapters/${adapter}/test-environment`) return { kind: "validation", adapter }; + } catch { /* An unrelated or malformed URL is not this action's evidence. */ } + return null; + }; + const onRequest = (request: Request) => { if (active && route(request)) requests.add(request); }; + const onResponse = (response: Response) => { + const request = response.request(), matched = route(request); + if (!active || !matched || !requests.delete(request)) return; + const observed = receipt[matched.kind]; + // One Connect action can report each boundary once. Later traffic cannot replace its failure. + if (observed.state !== "missing") return; + advance(matched.kind); observed.state = "pending"; + const status = response.status(); + observed.httpStatus = Number.isSafeInteger(status) && status >= 100 && status <= 599 ? status : null; + observed.requestedAdapter = matched.adapter; + if (response.status() >= 400) reject(matched.kind === "validation" ? "onboarding_native_validation_failed" : "onboarding_agent_hire_failed"); + const classify = async () => { + let timer: ReturnType | undefined; + try { + const bytes = await Promise.race([response.body(), new Promise(resolve => { timer = setTimeout(() => resolve(null), Math.min(5000, remaining())); })]); + if (!active) return; + if (!bytes) { observed.state = "unreadable"; return; } + if (bytes.byteLength > 65_536) { observed.state = "oversized"; return; } + let body: unknown; try { body = JSON.parse(bytes.toString("utf8")); } catch { observed.state = "unreadable"; return; } + if (!record(body)) { observed.state = "unreadable"; return; } + observed.state = "json"; + const detailCode = record(body.details) ? body.details.code : undefined; + const code = member(onboardingServerCodes, detailCode) ? detailCode : member(onboardingServerCodes, body.code) ? body.code : detailCode ?? body.code; + if (code !== undefined) observed.serverCode = member(onboardingServerCodes, code) ? code : "unrecognized"; + if (matched.kind === "validation") { + observed.adapter = member(adapters, body.adapterType) ? body.adapterType : "unrecognized"; + observed.outcome = member(outcomes, body.status) ? body.status : "unrecognized"; + if (Array.isArray(body.checks)) { + observed.truncatedChecks = body.checks.length > 32; + observed.checks = body.checks.slice(0, 32).map(check => ({ code: record(check) && member(onboardingCheckCodes, check.code) ? check.code : "unrecognized", level: record(check) && member(levels, check.level) ? check.level : "unrecognized" })); + if (body.status === "fail" || body.checks.some(check => record(check) && check.code === "adapter_auth_missing")) reject("onboarding_native_validation_failed"); + } else if (body.status === "fail") reject("onboarding_native_validation_failed"); + } + } catch { if (active) observed.state = "unreadable"; } + finally { clearTimeout(timer); } + }; + const work = classify(); pending.add(work); void work.then(() => pending.delete(work), () => pending.delete(work)); + }; + page.on("request", onRequest); page.on("response", onResponse); + try { + if (Date.now() >= deadlineAt) throw new JourneyError("failed", "onboarding_connect_timeout"); + const clicked = page.getByRole("button", { name: "Connect", exact: true }).click({ timeout: remaining() }) + .then(() => null, error => new JourneyError("failed", error?.name === "TimeoutError" ? "onboarding_connect_timeout" : "onboarding_connect_failed")); + const clickResult = await Promise.race([failed, clicked]); + if (clickResult) throw clickResult; + if (failure) throw failure; + const result = await Promise.race([failed, page.getByRole("button", { name: "Get started", exact: true }).waitFor({ timeout: remaining() }) + .then(() => null, error => new JourneyError("failed", error?.name === "TimeoutError" ? "onboarding_get_started_timeout" : "onboarding_get_started_failed"))]); + const boundaryFailure = failure; + // Seal the observed boundary before awaiting body reads. A response arriving + // after Get started or its timeout cannot change this action's outcome. + page.off("request", onRequest); page.off("response", onResponse); + if (pending.size) await Promise.all([...pending]); + if (boundaryFailure) throw boundaryFailure; + if (result) throw result; + if (failure) throw failure; + if (Date.now() >= deadlineAt) throw new JourneyError("failed", "onboarding_get_started_timeout"); + advance("review"); + return receipt; + } finally { + // Settle only already observed responses within their bounded read; detach before later traffic. + page.off("request", onRequest); page.off("response", onResponse); + if (pending.size) await Promise.all([...pending]); + active = false; requests.clear(); + } +} diff --git a/tests/runner-e2e/product-journeys/types.ts b/tests/runner-e2e/product-journeys/types.ts index 65941f30d86..da0b054a606 100644 --- a/tests/runner-e2e/product-journeys/types.ts +++ b/tests/runner-e2e/product-journeys/types.ts @@ -28,6 +28,7 @@ export interface JourneyEvidence { attemptId?: string; operations?: JourneyOperations; artifact?: OnboardingArtifactEvidence; + onboardingConnect?: import("./onboarding-connect.js").OnboardingConnectEvidence; } export interface OnboardingArtifactEvidence { schema: "paperclip.product-journey.artifact/v1"; diff --git a/tests/runner-e2e/result-validation.ts b/tests/runner-e2e/result-validation.ts index 883d928c510..56f38ed0bec 100644 --- a/tests/runner-e2e/result-validation.ts +++ b/tests/runner-e2e/result-validation.ts @@ -4,6 +4,7 @@ import { githubJourneyCheckpoints, gradeGitHubReviewJourney } from "./github-rev import { STARTUP_REUSE_LAYERS, STARTUP_REUSE_REASONS } from "../../packages/adapter-utils/src/startup-reuse.js"; import type { RunnerE2ECampaignAccounting, RunnerE2EResult } from "./types.js"; import { DEPENDABLE_CELLS } from "./dependable-qualification.js"; +import { validateOnboardingConnectEvidence } from "./product-journeys/onboarding-connect.js"; import { validateOnboardingArtifact } from "./product-journeys/artifact-evidence.js"; type Rule = (value: unknown, at: string) => void; @@ -344,6 +345,7 @@ const fields = { productJourney: optional(shape({ targetId: optional(string), campaignId: optional(string), attemptId: optional(string), + onboardingConnect: optional((value, at) => { try { validateOnboardingConnectEvidence(value); } catch { invalid(at); } }), artifact: optional((value, at) => { try { validateOnboardingArtifact(value); } catch { invalid(at); } }), operations: optional(closedShape({ campaignId: string, attemptId: string, modelCostUsd: nullable(number), setupCostUsd: nullable(number), requestCount: integer, feedback: closedShape({ collection: oneOf("complete", "partial", "unavailable"), complaints: integer, suggestions: integer }), traceIds: array(string) })), diff --git a/tests/runner-e2e/runner.spec.ts b/tests/runner-e2e/runner.spec.ts index 420d8c97cf5..8836e27dfe8 100644 --- a/tests/runner-e2e/runner.spec.ts +++ b/tests/runner-e2e/runner.spec.ts @@ -264,6 +264,7 @@ async function restartIsolatedPaperclipServer(input: { deadlineAt: number; preserveRunner?: RestartRunnerIdentity; crashAfterResult?: FinalizationCrashBinding; + controllerDowntimeMs?: number; }): Promise { const { controlDirectory, @@ -274,7 +275,8 @@ async function restartIsolatedPaperclipServer(input: { const temporaryRequestPath = `${requestPath}.${process.pid}.${input.requestId}.tmp`; await writeFile( temporaryRequestPath, - JSON.stringify({ requestId: input.requestId, ...(input.preserveRunner ? { preserveRunner: input.preserveRunner } : {}), + JSON.stringify({ requestId: input.requestId, ...(input.controllerDowntimeMs !== undefined ? { controllerDowntimeMs: input.controllerDowntimeMs } : {}), + ...(input.preserveRunner ? { preserveRunner: input.preserveRunner } : {}), ...(input.crashAfterResult ? { crashAfterResult: input.crashAfterResult } : {}) }), { encoding: "utf8", mode: 0o600 }, ); @@ -1240,7 +1242,7 @@ for (const execution of executions) { const story = await runEverydayFlow({ page, api, fixtures, execution, nonce, workspacePath, privateDir, deadlineAt: startedAtMs + deadlineMs, - restart: () => restartIsolatedPaperclipServer({ api, requestId: `story-${nonce}`, deadlineAt: startedAtMs + deadlineMs }), + restart: controllerDowntimeMs => restartIsolatedPaperclipServer({ api, requestId: `story-${nonce}`, deadlineAt: startedAtMs + deadlineMs, controllerDowntimeMs }), observe: (storyIssue, storyRuns) => { issue = storyIssue; selectedRuns = storyRuns; }, capture: captureScreenshot, evidence: (name, data) => writeSanitizedJson(snapshotsDir, name, data, secrets), diff --git a/tests/runner-e2e/server.ts b/tests/runner-e2e/server.ts index 8956ac3f5aa..451ed91e034 100644 --- a/tests/runner-e2e/server.ts +++ b/tests/runner-e2e/server.ts @@ -293,6 +293,7 @@ async function waitForHealthToStop() { interface RestartRequest { requestId: string; + controllerDowntimeMs?: number; preserveRunner?: RestartRunnerIdentity; crashAfterResult?: FinalizationCrashBinding; } @@ -321,9 +322,15 @@ async function readRestartRequest(): Promise { return null; } const preserveRunner = (value as { preserveRunner?: unknown }).preserveRunner; + const controllerDowntimeMs = (value as { controllerDowntimeMs?: unknown }).controllerDowntimeMs; + if (controllerDowntimeMs !== undefined && (!Number.isSafeInteger(controllerDowntimeMs) + || Number(controllerDowntimeMs) < 0 || Number(controllerDowntimeMs) > 300_000)) { + throw new Error("Controller downtime must be an integer between zero and five minutes"); + } const crashAfterResult = (value as { crashAfterResult?: unknown }).crashAfterResult; if (crashAfterResult !== undefined && preserveRunner !== undefined) throw new Error("A finalization crash cannot preserve a running provider"); - return { requestId, ...(preserveRunner === undefined ? {} : { preserveRunner: parseRestartRunnerIdentity(preserveRunner) }), + return { requestId, ...(controllerDowntimeMs === undefined ? {} : { controllerDowntimeMs: Number(controllerDowntimeMs) }), + ...(preserveRunner === undefined ? {} : { preserveRunner: parseRestartRunnerIdentity(preserveRunner) }), ...(crashAfterResult === undefined ? {} : { crashAfterResult: parseFinalizationCrashBinding(crashAfterResult) }) }; } @@ -348,7 +355,7 @@ async function writeRestartAck( await rename(temporaryAckPath, restartAckPath); } -async function restartServer({ requestId, preserveRunner, crashAfterResult }: RestartRequest) { +async function restartServer({ requestId, preserveRunner, crashAfterResult, controllerDowntimeMs = 0 }: RestartRequest) { activeRestartRequestId = requestId; appendLog(`\nRestart request ${requestId}: stopping Paperclip\n`); const previous = child; @@ -365,6 +372,11 @@ async function restartServer({ requestId, preserveRunner, crashAfterResult }: Re // Do not mistake an orphaned old server for a healthy replacement. The port // must stop answering before the next launcher is allowed to start. await waitForHealthToStop(); + const downtimeDeadline = Date.now() + controllerDowntimeMs; + while (Date.now() < downtimeDeadline) { + if (shutdownRequested()) throw new Error("Wrapper shutdown interrupted controller downtime"); + await delay(Math.min(250, downtimeDeadline - Date.now())); + } if (shutdownRequested()) { throw new Error("Wrapper shutdown interrupted the Paperclip restart"); } diff --git a/ui/src/adapters/codex-local/config-fields.tsx b/ui/src/adapters/codex-local/config-fields.tsx index d691b30c5c2..3ccf6abacf4 100644 --- a/ui/src/adapters/codex-local/config-fields.tsx +++ b/ui/src/adapters/codex-local/config-fields.tsx @@ -309,7 +309,7 @@ export function CodexLocalConfigFields({ updateRunnerSchemaValue("allowUnmeteredProvider", value)} /> } - {runnerManaged && runnerProvider === "acpx" && runnerSchemaValue("acpxAgent", "claude") !== "grok" && ( + {runnerManaged && !hideRunnerHarness && runnerProvider === "acpx" && runnerSchemaValue("acpxAgent", "claude") !== "grok" && ( setQuery(event.target.value)} /> +
{matches.map(project => )}
+ {!matches.length &&

No projects found.

} + + } + + {visibleIds.length ?
    + {visibleIds.map(id =>
  • + {name(id)} + {projectIds.includes(id) ? Allowed + : <>To add} +
    • )} +
:

No projects allowed yet.

} + {!canManage &&

A person with agent-management permission can add projects here.

} + {pending.length > 0 &&
+

Add access to {pending.map(name).join(", ")}

+

{agentName} will be able to access work and repositories in {pending.length === 1 ? "this project" : "these projects"}. Existing project access and low-trust restrictions stay in place.

+
} + {error &&

Couldn’t save project access. Your selection is kept. {error}

} + {saved &&

Project access saved. Return to your task to retry.

} + {canManage &&
+ + +
} + ; +} diff --git a/ui/src/components/ExecutionBlockerNotice.tsx b/ui/src/components/ExecutionBlockerNotice.tsx index 9c34d72748f..17bd169750c 100644 --- a/ui/src/components/ExecutionBlockerNotice.tsx +++ b/ui/src/components/ExecutionBlockerNotice.tsx @@ -2,9 +2,10 @@ import type { TaskInterruptionPresentation } from "@paperclipai/shared"; import { Button } from "./ui/button"; import { cn } from "../lib/utils"; import { useSecondTick } from "../hooks/useSecondTick"; +import { TaskProjectAccessNotice } from "./TaskProjectAccessNotice"; /** One ordinary interruption surface. Technical records remain in diagnostics. */ -export function ExecutionBlockerNotice({ presentation, canRetry, onRetry, optimistic, takingLonger, error, className }: { +export function ExecutionBlockerNotice({ presentation, canRetry, onRetry, optimistic, takingLonger, error, className, issueId, agentName, projectName }: { presentation: TaskInterruptionPresentation | null | undefined; canRetry: boolean; onRetry: () => void; @@ -12,8 +13,14 @@ export function ExecutionBlockerNotice({ presentation, canRetry, onRetry, optimi takingLonger?: boolean; error?: Error | null; className?: string; + issueId?: string; + agentName?: string; + projectName?: string; }) { useSecondTick(presentation?.state === "retry_scheduled"); + if (presentation?.projectAccess && !optimistic && !takingLonger) return ; if (error && !optimistic && !takingLonger) return
{error.message}{canRetry && }
; if (takingLonger) return
Taking longer than usual…{canRetry && }
; if (optimistic || !presentation || presentation.responsibility !== "system" || diff --git a/ui/src/components/ProjectAccessNotice.tsx b/ui/src/components/ProjectAccessNotice.tsx new file mode 100644 index 00000000000..07386a68de5 --- /dev/null +++ b/ui/src/components/ProjectAccessNotice.tsx @@ -0,0 +1,38 @@ +import { ShieldCheck, ShieldX, ArrowRight } from "lucide-react"; +import { Button } from "./ui/button"; +import { cn } from "../lib/utils"; + +export interface ProjectAccessNoticeProps { + agentName?: string; + projectName?: string; + ready: boolean; + canManage: boolean; + canRetry: boolean; + onManage: () => void; + onRetry: () => void; + error?: Error | null; + className?: string; +} + +export function ProjectAccessNotice({ agentName = "This agent", projectName, ready, canManage, canRetry, onManage, onRetry, error, className }: ProjectAccessNoticeProps) { + return
+
+ {ready ? : } +
+

{ready ? "Project access is ready" : `${agentName} can’t access this project`}

+

{ready + ? "The agent can now access the required project. Retry to continue with your saved message." + : `This task needs ${projectName ?? "a project"} outside the agent’s allowed projects. The agent hasn’t started. Your message is saved.`}

+ {!ready && !canManage &&

A person with agent-management permission can add this project in the agent’s Permissions / Trust settings.

} +
+
+
+ {ready ? canRetry && + : } +
+ {error &&

{error.message}

} + {!ready &&
Run details +
Stopped during
Workspace setup
Reason
Project outside allowed scope
Provider started
No
+
} +
; +} diff --git a/ui/src/components/TaskChatThread.test.tsx b/ui/src/components/TaskChatThread.test.tsx index 4f7b6c85308..3884bf88197 100644 --- a/ui/src/components/TaskChatThread.test.tsx +++ b/ui/src/components/TaskChatThread.test.tsx @@ -361,6 +361,16 @@ it.each([true, false])("defers hidden content until history is ready and keeps i expect(container.querySelector('.task-chat-history-pending')).toBeNull(); }); +it.each([true, false])("keeps cold-open messages readable while history is pending (mobile=%s)", (mobile) => { + sidebarState.isMobile = mobile; + render(More messages} onAdd={async () => {}} />); + expect(container.querySelector('[data-testid="task-chat-history-loading"]')).toBeNull(); + expect(container.textContent).toContain("Thread message 1"); + expect(container.textContent).toContain("More messages"); + expect(container.querySelector('[data-testid="mock-editor"]')).not.toBeNull(); +}); + it("defers initial history for a saved conversation with its own draft key", () => { render( {}} issueId="saved-chat" conversationMode draftKey="agent-chat:company:saved-agent" initialHistoryPending />); diff --git a/ui/src/components/TaskChatThread.tsx b/ui/src/components/TaskChatThread.tsx index 8d77a9b03ee..24717cbb567 100644 --- a/ui/src/components/TaskChatThread.tsx +++ b/ui/src/components/TaskChatThread.tsx @@ -23,6 +23,7 @@ import { useRef, useState, type ComponentProps, + type ReactNode, } from "react"; import { IssueAssigneePausedNotice, @@ -422,6 +423,8 @@ function resolvedWithoutUserFacingResponse(value: unknown): boolean { export type TaskChatThreadProps = ComponentProps & { conversationMode?: boolean; creationActivity?: ActivityEvent[]; + compactHistory?: boolean; + historyControl?: ReactNode; initialHistoryPending?: boolean; initialHistoryError?: boolean; onRetryInitialHistory?: () => void; @@ -898,6 +901,7 @@ export function TaskChatThread(props: TaskChatThreadProps) { for (const r of linkedRuns ?? []) { map.set(r.runId, { id: r.runId, + createdAt: r.createdAt, status: r.status, adapterType: r.adapterType ?? "", runtimeMode: r.runtimeMode, @@ -908,6 +912,7 @@ export function TaskChatThread(props: TaskChatThreadProps) { for (const r of liveRuns ?? []) { map.set(r.id, { id: r.id, + createdAt: r.createdAt, status: r.status, adapterType: r.adapterType, runtimeMode: r.runtimeMode, @@ -919,6 +924,7 @@ export function TaskChatThread(props: TaskChatThreadProps) { if (activeRun) { map.set(activeRun.id, { id: activeRun.id, + createdAt: activeRun.createdAt, status: activeRun.status, adapterType: activeRun.adapterType, runtimeMode: activeRun.runtimeMode, @@ -929,9 +935,13 @@ export function TaskChatThread(props: TaskChatThreadProps) { return [...map.values()]; }, [linkedRuns, liveRuns, activeRun]); + const openedAt = useMemo(() => Date.now(), [issueId]); const nativeRuns = useMemo( - () => runs.filter((run) => run.runtimeMode === "native"), - [runs], + () => runs.filter((run) => run.runtimeMode === "native").map((run) => ({ + ...run, + conversationOnly: Boolean(props.compactHistory && (!run.createdAt || new Date(run.createdAt).getTime() <= openedAt)), + })), + [runs, props.compactHistory, openedAt], ); const { transcriptByRun: nativeTranscriptByRun, @@ -942,6 +952,9 @@ export function TaskChatThread(props: TaskChatThreadProps) { retry: retryNativeEvents, } = useNativeRunTranscripts(nativeRuns); const logRuns = useMemo(() => runs.filter((run) => + (!nativeRuns.some((nativeRun) => nativeRun.id === run.id && nativeRun.conversationOnly) || + nativeTranscriptErrorsByRun.has(run.id) || + (hydratedNativeRunIds.has(run.id) && (nativeTranscriptByRun.get(run.id)?.length ?? 0) === 0)) && ( run.runtimeMode !== "native" || // Active runs still need the websocket log stream. Only settled native // history can skip the legacy transport when event history is available. @@ -949,14 +962,14 @@ export function TaskChatThread(props: TaskChatThreadProps) { nativeTranscriptErrorsByRun.has(run.id) || (hydratedNativeRunIds?.has(run.id) && (nativeTranscriptByRun.get(run.id)?.length ?? 0) === 0) - ), [runs, nativeTranscriptErrorsByRun, hydratedNativeRunIds, nativeTranscriptByRun]); + )), [runs, nativeRuns, nativeTranscriptErrorsByRun, hydratedNativeRunIds, nativeTranscriptByRun]); const { transcriptByRun: logTranscriptByRun, isInitialHydrating: logsAreInitiallyHydrating, hydratedRunIds: hydratedLogRunIds, errorsByRun: logErrorsByRun, retry: retryLogs, - } = useLiveRunTranscripts({ runs: logRuns, companyId }); + } = useLiveRunTranscripts({ runs: logRuns, companyId, ...(props.compactHistory ? { logReadLimitBytes: 32_000 } : {}) }); const fallbackByRunRef = useRef( new Map>>(), ); @@ -2813,8 +2826,8 @@ export function TaskChatThread(props: TaskChatThreadProps) { ); const renderBrief = useCallback( - () => issueBrief ? : null, - [issueBrief], + () => issueBrief ? <>{props.historyControl} : null, + [issueBrief, props.historyControl], ); const assignedAgentForNotice = useMemo(() => { @@ -2968,10 +2981,10 @@ export function TaskChatThread(props: TaskChatThreadProps) { : logsAreInitiallyHydrating; }); // Hydrating a run inserts activity around its saved reply and can move the - // latest viewport substantially. Wait for the initial comment window's run - // history and plan before revealing; older runs outside that window do not - // delay it, and the latch below keeps subsequent refreshes visible. - const historyPending = initialHistoryPending || planLoading || transcriptHistoryPending; + // latest viewport substantially in the full-history view. Cold opens reveal + // the brief and recent messages independently; neither plans nor transcripts + // may hold them behind the loading screen. + const historyPending = !props.compactHistory && (initialHistoryPending || planLoading || transcriptHistoryPending); const scrollNavigation = useTaskChatScrollNavigation(); useEffect(() => { recordIssueDetailReadiness({ plan: !planLoading, transcripts: !transcriptHistoryPending, history: !historyPending }); @@ -3124,7 +3137,7 @@ export function TaskChatThread(props: TaskChatThreadProps) { {threadHeaderWithBlockers}{!issueBrief ? props.historyControl : null}} renderInteraction={renderInteraction} renderBrief={renderBrief} renderMessageActions={renderMessageActions} diff --git a/ui/src/components/TaskProjectAccessNotice.tsx b/ui/src/components/TaskProjectAccessNotice.tsx new file mode 100644 index 00000000000..021d955f8ec --- /dev/null +++ b/ui/src/components/TaskProjectAccessNotice.tsx @@ -0,0 +1,23 @@ +import { useQuery } from "@tanstack/react-query"; +import type { TaskInterruptionPresentation } from "@paperclipai/shared"; +import { agentsApi } from "../api/agents"; +import { useNavigate } from "../lib/router"; +import { queryKeys } from "../lib/queryKeys"; +import { ProjectAccessNotice } from "./ProjectAccessNotice"; + +export function TaskProjectAccessNotice({ access, issueId, agentName, projectName, ...props }: { + access: NonNullable; + issueId?: string; + agentName?: string; + projectName?: string; + canRetry: boolean; + onRetry: () => void; + error?: Error | null; + className?: string; +}) { + const navigate = useNavigate(); + const permission = useQuery({ queryKey: queryKeys.agents.projectAccess(access.agentId), queryFn: () => agentsApi.projectAccess(access.agentId) }); + return navigate(`/agents/${access.agentId}/permissions${issueId ? `?fromTask=${encodeURIComponent(issueId)}` : ""}`)} />; +} diff --git a/ui/src/components/TaskTreeControls.tsx b/ui/src/components/TaskTreeControls.tsx index d2fd9137547..dfd0d1aaf1f 100644 --- a/ui/src/components/TaskTreeControls.tsx +++ b/ui/src/components/TaskTreeControls.tsx @@ -78,6 +78,7 @@ export function TaskTreeControlDialog({ scope, affectedCount, affectedAgentCount, + warnings = [], loading, error, pending, @@ -93,6 +94,7 @@ export function TaskTreeControlDialog({ scope: "leaf" | "subtree"; affectedCount: number; affectedAgentCount: number; + warnings?: string[]; loading: boolean; error?: string | null; pending: boolean; @@ -129,7 +131,9 @@ export function TaskTreeControlDialog({ ? "Loading…" : cancel ? `${tasks} will be cancelled.` - : `${tasks} will ${mode === "restore" ? "be restored" : "resume"}.`} + : mode === "restore" + ? `${tasks} will be restored.` + : `The pause will be removed from ${tasks}.`} {error ? ( @@ -151,13 +155,16 @@ export function TaskTreeControlDialog({ ) : null} + {!loading ? warnings.map((warning) => ( +

{warning}

+ )) : null} + ) : null, + } : {})} executionBlocked={executionBlocked} interruption={interruption} retryPreparing={retryPreparing} @@ -2526,7 +2575,7 @@ const IssueDetailChatTab = memo(function IssueDetailChatTab({ browsers={browsers} onOpenBrowser={onOpenBrowser} hasOlderComments={hasOlderComments} - initialHistoryPending={!!issueId && ( + initialHistoryPending={historyExpanded && !!issueId && ( initialHistoryPending || commentsInitialLoading || // These responses add rows or change the composer takeover. @@ -2540,6 +2589,7 @@ const IssueDetailChatTab = memo(function IssueDetailChatTab({ initialHistoryError={ initialHistoryError || activityError || + (!historyExpanded && inputFacts.isError) || linkedRunsError || liveRunsError || (activeRunQueryEnabled && activeRunError) @@ -2547,6 +2597,7 @@ const IssueDetailChatTab = memo(function IssueDetailChatTab({ onRetryInitialHistory={() => { onRetryInitialHistory?.(); void refetchActivity(); + if (!historyExpanded && inputCommentIds.length) void inputFacts.refetch(); void refetchLinkedRuns(); void refetchLiveRuns(); if (activeRunQueryEnabled) void refetchActiveRun(); @@ -3223,11 +3274,22 @@ export function TaskDetailSurface({ conversation, tasksTab }: { tasksTab?: TaskS [location.state, resolvedIssueDetailState], ); + const coreIssueQuery = useQuery({ + queryKey: [...queryKeys.issues.detail(issueId!), "core"], + queryFn: ({ signal }) => issuesApi.getCore(issueId!, { signal }), + enabled: !!issueId, + }); + const { historyExpanded, expandHistory, recent: recentComments } = useIssueHistoryWindow( + issueId, + Boolean(conversation || classicTaskInterfaceEnabled || location.hash || detailTab !== "chat"), + coreIssueQuery.data, + ); const { data: queriedIssue, isLoading, isPlaceholderData, error, + refetch: refetchIssueDetail, } = useQuery({ ...getIssueDetailQueryOptions(queryClient, issueId!, { placeholderIssue: issueHeaderSeed @@ -3237,12 +3299,15 @@ export function TaskDetailSurface({ conversation, tasksTab }: { tasksTab?: TaskS } : null, }), - enabled: !!issueId, + enabled: !!issueId && (coreIssueQuery.isFetched || !!queryClient.getQueryData(queryKeys.issues.detail(issueId))), }); useEffect(() => { - if (issueId) void prefetchIssueThread(queryClient, issueId); - }, [issueId, queryClient]); - const issue = queriedIssue ?? conversation?.issue ?? draftIssue; + if (issueId && historyExpanded) void prefetchIssueThread(queryClient, issueId); + }, [issueId, historyExpanded, queryClient]); + const accessError = [error, coreIssueQuery.error].find((value) => + value instanceof ApiError && [401, 403, 404].includes(value.status), + ); + const issue = accessError ? undefined : queriedIssue ?? coreIssueQuery.data ?? conversation?.issue ?? draftIssue; const taskRetry = useTaskRetry(issue ?? undefined); const resolveWritableIssueId = async () => { if (!conversation) return issueId!; @@ -3258,7 +3323,7 @@ export function TaskDetailSurface({ conversation, tasksTab }: { tasksTab?: TaskS // or upload against the previous task while the requested task is loading. const loadedIssue = !isPlaceholderData && - !error && + (!error || !!coreIssueQuery.data) && issue && issueId && (issue.id.toLowerCase() === issueId.toLowerCase() || @@ -3275,6 +3340,7 @@ export function TaskDetailSurface({ conversation, tasksTab }: { tasksTab?: TaskS !hasLegacyIssueDetailQuery(location.search) )); const resolvedCompanyId = issue?.companyId ?? selectedCompanyId; + const hasLiveConnection = useTaskRunLiveUpdates({ companyId: resolvedCompanyId, issueId: issue?.id ?? issueId, issueQueryRef: issueId ?? "pending", assigneeAgentId: issue?.assigneeAgentId }); const externalObjectsState = useIssueExternalObjects(conversation && !conversation.issue ? null : issue?.id ?? null); // A closed isolated workspace no longer blocks the composer. The server reopens // the workspace when the next comment or resume arrives, so the composer stays @@ -3304,7 +3370,7 @@ export function TaskDetailSurface({ conversation, tasksTab }: { tasksTab?: TaskS limit: ISSUE_COMMENT_PAGE_SIZE, ...(pageParam ? { after: pageParam } : {}), }), - enabled: !!issueId, + enabled: !!issueId && historyExpanded, initialPageParam: null as string | null, getNextPageParam: (lastPage) => getNextIssueCommentPageParam(lastPage, ISSUE_COMMENT_PAGE_SIZE), @@ -3313,8 +3379,8 @@ export function TaskDetailSurface({ conversation, tasksTab }: { tasksTab?: TaskS >(issueId ?? "pending"), }); const comments = useMemo( - () => flattenIssueCommentPages(commentPages?.pages), - [commentPages?.pages], + () => mergeWindowComments(recentComments.data?.comments ?? [], flattenIssueCommentPages(commentPages?.pages)), + [commentPages?.pages, recentComments.data?.comments], ); useLayoutEffect(() => { @@ -3352,7 +3418,7 @@ export function TaskDetailSurface({ conversation, tasksTab }: { tasksTab?: TaskS ); const shouldPrefetchOlderComments = useMemo( () => - shouldAutoloadOlderIssueComments({ + historyExpanded && shouldAutoloadOlderIssueComments({ activeDetailTab: detailTab, hasOlderComments: hasOlderComments ?? false, loadedCommentCount: comments.length, @@ -3364,6 +3430,7 @@ export function TaskDetailSurface({ conversation, tasksTab }: { tasksTab?: TaskS comments.length, commentsLoading, commentsLoadingOlder, + historyExpanded, detailTab, hasOlderComments, ], @@ -3414,7 +3481,7 @@ export function TaskDetailSurface({ conversation, tasksTab }: { tasksTab?: TaskS queryKey: queryKeys.issues.liveRuns(issueId!), queryFn: () => heartbeatsApi.liveRunsForIssue(issueId!), enabled: !!issueId, - refetchInterval: 3000, + refetchInterval: (query) => liveRunRefetchInterval(hasLiveConnection, query.state, 3000), select: (runs) => runs.length, placeholderData: keepPreviousDataForSameQueryTail( issueId ?? "pending", @@ -3431,7 +3498,7 @@ export function TaskDetailSurface({ conversation, tasksTab }: { tasksTab?: TaskS queryFn: () => heartbeatsApi.activeRunForIssue(issueId!), enabled: !!issueId && (!!issue?.executionRunId || issue?.status === "in_progress"), - refetchInterval: liveRunCount > 0 ? false : 3000, + refetchInterval: (query) => liveRunCount > 0 ? false : liveRunRefetchInterval(hasLiveConnection, query.state, 3000), select: (run) => !!run, placeholderData: keepPreviousDataForSameQueryTail( issueId ?? "pending", @@ -4084,18 +4151,33 @@ export function TaskDetailSurface({ conversation, tasksTab }: { tasksTab?: TaskS }, [issueId, queryClient]); const removeCommentFromCache = useCallback( - (commentId: string) => { - queryClient.setQueryData< - InfiniteData | undefined - >(queryKeys.issues.comments(issueId!), (current) => { - if (!current) return current; - return { - ...current, - pages: removeIssueCommentFromPages(current.pages, commentId), - }; - }); + async (commentId: string) => { + // Cancel reads started before deletion so they cannot restore stale rows. + await Promise.all(issueCacheRefs.map((ref) => + queryClient.cancelQueries({ queryKey: queryKeys.issues.comments(ref) }), + )); + for (const ref of issueCacheRefs) { + queryClient.setQueryData>( + queryKeys.issues.comments(ref), + (current) => current && { ...current, pages: removeIssueCommentFromPages(current.pages, commentId) }, + ); + queryClient.setQueryData( + [...queryKeys.issues.comments(ref), "recent"], + (current) => { + if (!current) return current; + const comments = mergeWindowComments(current.comments.filter((comment) => comment.id !== commentId)); + return { + ...current, + comments, + latestCommentId: current.latestCommentId === commentId + ? comments.at(-1)?.id ?? null + : current.latestCommentId, + }; + }, + ); + } }, - [issueId, queryClient], + [issueCacheRefs, queryClient], ); const clearCommentHashIfCurrent = useCallback( @@ -4112,7 +4194,10 @@ export function TaskDetailSurface({ conversation, tasksTab }: { tasksTab?: TaskS ); const upsertCommentInCache = useCallback( - (comment: IssueComment) => { + async (comment: IssueComment) => { + await Promise.all(issueCacheRefs.map((ref) => + queryClient.cancelQueries({ queryKey: queryKeys.issues.comments(ref) }), + )); for (const ref of issueCacheRefs) { queryClient.setQueryData< InfiniteData | undefined @@ -4124,6 +4209,13 @@ export function TaskDetailSurface({ conversation, tasksTab }: { tasksTab?: TaskS } : current, ); + queryClient.setQueryData( + [...queryKeys.issues.comments(ref), "recent"], + (current) => current && { + ...current, + comments: current.comments.map((existing) => existing.id === comment.id ? comment : existing), + }, + ); } }, [issueCacheRefs, queryClient], @@ -4877,6 +4969,7 @@ export function TaskDetailSurface({ conversation, tasksTab }: { tasksTab?: TaskS ); try { await issuesApi.cancelComment(comment.issueId, comment.id); + await removeCommentFromCache(comment.id); invalidateIssueDetail(); invalidateIssueThreadLazily(); invalidateIssueCollections(); @@ -4918,6 +5011,11 @@ export function TaskDetailSurface({ conversation, tasksTab }: { tasksTab?: TaskS pages: upsertIssueCommentInPages(undefined, comment), }, ); + // A locally seeded page is not a complete history page. Expansion must + // fetch the real first page even within the normal cache freshness window. + if (!historyExpanded) { + await queryClient.invalidateQueries({ queryKey: queryKeys.issues.comments(issueId ?? comment.issueId), exact: true, refetchType: "none" }); + } if (context?.optimisticCommentId) { setOptimisticComments((current) => current.filter( @@ -5295,6 +5393,7 @@ export function TaskDetailSurface({ conversation, tasksTab }: { tasksTab?: TaskS ); try { await issuesApi.cancelComment(issueId!, comment.id); + await removeCommentFromCache(comment.id); invalidateIssueDetail(); invalidateIssueThreadLazily(); invalidateIssueCollections(); @@ -5339,6 +5438,9 @@ export function TaskDetailSurface({ conversation, tasksTab }: { tasksTab?: TaskS pages: upsertIssueCommentInPages(undefined, comment), }, ); + if (!historyExpanded) { + await queryClient.invalidateQueries({ queryKey: queryKeys.issues.comments(issueId!), exact: true, refetchType: "none" }); + } } if (context?.optimisticCommentId) { setOptimisticComments((current) => @@ -5403,14 +5505,14 @@ export function TaskDetailSurface({ conversation, tasksTab }: { tasksTab?: TaskS const cancelQueuedComment = useMutation({ mutationFn: async ({ commentId }: { commentId: string }) => issuesApi.cancelComment(issueId!, commentId), - onSuccess: (comment) => { + onSuccess: async (comment) => { setLocallyQueuedCommentRunIds((current) => { if (!current.has(comment.id)) return current; const next = new Map(current); next.delete(comment.id); return next; }); - removeCommentFromCache(comment.id); + await removeCommentFromCache(comment.id); restoreQueuedCommentDraft(comment.body); invalidateIssueDetail(); invalidateIssueThreadLazily(); @@ -5436,8 +5538,8 @@ export function TaskDetailSurface({ conversation, tasksTab }: { tasksTab?: TaskS const deleteComment = useMutation({ mutationFn: async ({ commentId }: { commentId: string }) => issuesApi.deleteComment(issueId!, commentId), - onSuccess: (comment) => { - upsertCommentInCache(comment); + onSuccess: async (comment) => { + await upsertCommentInCache(comment); clearCommentHashIfCurrent(comment.id); invalidateIssueDetail(); invalidateIssueThreadLazily(); @@ -7044,9 +7146,9 @@ export function TaskDetailSurface({ conversation, tasksTab }: { tasksTab?: TaskS ); }, [activePauseHold, issue]); - if (isLoading) + if (!issue && !accessError && (isLoading || coreIssueQuery.isPending)) return ; - if (error) return

{error.message}

; + if (!issue && (error || coreIssueQuery.error)) return

{(error || coreIssueQuery.error)?.message}

; if (!issue) return null; // Do not expose a file chooser on the outgoing UUID/company/interface // branch: its input can be detached before the chosen file is returned. @@ -7793,6 +7895,12 @@ export function TaskDetailSurface({ conversation, tasksTab }: { tasksTab?: TaskS : "max-w-3xl space-y-6" } > + {error && coreIssueQuery.data ? ( +
+ Some task details could not be loaded. + +
+ ) : null} {/* Parent chain breadcrumb (redesign: rendered inside the thread viewport) */} {taskChatShellEnabled ? null : ancestorsNav} {taskChatShellEnabled ? null : createdFromNote} @@ -7814,6 +7922,7 @@ export function TaskDetailSurface({ conversation, tasksTab }: { tasksTab?: TaskS )} {resolvedDetailTab === "chat" && ( )} @@ -8149,6 +8258,7 @@ export function TaskDetailSurface({ conversation, tasksTab }: { tasksTab?: TaskS interruption={issue.interruption} retryPreparing={taskRetry.optimistic} queryIssueId={issueId ?? issue.id} + hasLiveConnection={hasLiveConnection} onOpenSkill={handleOpenSkill} browsers={browserQuery.data} onOpenBrowser={(id) => { setOpenBrowserId(id); if (isMobile) setMobilePropsOpen(true); else openTaskSidePanel(); }} @@ -8228,7 +8338,10 @@ export function TaskDetailSurface({ conversation, tasksTab }: { tasksTab?: TaskS canFalsePositiveRecoveryAction={canResolveBoardRecoveryAction} legacyRecoverySourceIssue={legacyRecoverySourceIssue} comments={threadComments} - commentsInitialLoading={commentsLoading} + commentsInitialLoading={historyExpanded ? commentsLoading : recentComments.isLoading} + historyExpanded={historyExpanded} + onExpandHistory={expandHistory} + hasEarlierMessages={recentComments.data?.hasMore ?? false} initialHistoryPending={linkedCommentPending} initialMetadataPending={ interactionsLoading || @@ -8236,13 +8349,13 @@ export function TaskDetailSurface({ conversation, tasksTab }: { tasksTab?: TaskS workProductsLoading } initialHistoryError={ - commentsError || + (historyExpanded ? commentsError : recentComments.isError) || interactionsError || attachmentsError || workProductsError } onRetryInitialHistory={() => { - void refetchComments(); + void (historyExpanded ? refetchComments() : recentComments.refetch()); void refetchInteractions(); void refetchAttachments(); void refetchWorkProducts(); @@ -8317,6 +8430,7 @@ export function TaskDetailSurface({ conversation, tasksTab }: { tasksTab?: TaskS onImageUpload={handleCommentImageUpload} onAttachImage={handleCommentAttachImage} onInterruptQueued={handleInterruptQueuedRun} + onRemoveCommentFromCache={removeCommentFromCache} onDeleteComment={(commentId) => deleteComment .mutateAsync({ commentId }) @@ -8490,6 +8604,7 @@ export function TaskDetailSurface({ conversation, tasksTab }: { tasksTab?: TaskS scope={treeControlScope} affectedCount={previewAffectedIssueCount} affectedAgentCount={previewAffectedAgentCount} + warnings={treeControlPreview?.warnings.map((warning) => warning.message)} loading={treeControlPreviewLoading} error={ treeControlPreviewError @@ -8511,7 +8626,7 @@ export function TaskDetailSurface({ conversation, tasksTab }: { tasksTab?: TaskS executeTreeControl.mutate({ mode: treeControlMode, scope: treeControlScope, - wakeAgents: treeControlWakeAgentsOnResume, + wakeAgents: treeControlWakeAgentsOnResume && previewAffectedAgentCount > 0, }) } /> diff --git a/ui/storybook/stories/design-proposals/project-access.stories.tsx b/ui/storybook/stories/design-proposals/project-access.stories.tsx new file mode 100644 index 00000000000..8efb0bc3a7e --- /dev/null +++ b/ui/storybook/stories/design-proposals/project-access.stories.tsx @@ -0,0 +1,121 @@ +import type { Meta, StoryObj } from "@storybook/react-vite"; +import { expect, userEvent, waitFor, within } from "storybook/test"; +import { ProjectAccessProposal } from "../../support/design-proposals/ProjectAccessProposal"; + +const meta = { + id: "design-project-access", + title: "Design proposals/Patterns/Project access recovery", + component: ProjectAccessProposal, + parameters: { + layout: "fullscreen", + initialEntries: ["/PAP/issues/DEMO-42"], + docs: { description: { component: "Start at Blocked task. Manage project access opens the proposed agent settings. Add Agent Extensions, review the additive access change, save, return to the task and retry. Low-trust review stays active. Settings supports multiple allowed projects without replacing the existing one. Read-only, pending save, failed save and failed retry states remain actionable. The reply and every state transition are local fixtures; no project permissions change and no provider runs. Reset the story to restore the starting state." } }, + paperclip: { + capabilities: ["recovery/diagnosis", "recovery/source-action", "agents/instructions"], + home: "design-proposals", status: "proposed", + exception: "Requested design for a visible project denial and additive project access repair before production implementation.", + implementation: "prototype", boundary: "fixtures", + notes: "Reuses DesignPageShell, DesignSidebar, AgentAvatar, IssueStatusBadge, TaskChatBubble, TaskChatLiveRunPill and shared form primitives. Notice, access editor and navigation are a local prototype. Synthetic names and repositories only. Save retains existing projects; failed saves preserve draft but grant nothing. Retry is unavailable before a saved grant and does not run on save. Read-only viewers cannot edit. No production API, persistence, permission, provider or recovery proof. The implemented notice and editor are covered by the task-project-access stories; this retains the original design snapshot. Server regression tests and application verification cover authorization, persistence and explicit retry separately.", + }, + }, +} satisfies Meta; +export default meta; +type Story = StoryObj; + +export const BlockedTask: Story = {}; +export const AgentSettings: Story = { args: { initialView: "permissions" } }; +export const AccessSaved: Story = { args: { initialState: "access_saved" } }; +export const ReadOnly: Story = { args: { canManage: false }, play: async ({ canvasElement }) => { + const canvas = within(canvasElement); + await expect(await canvas.findByText("Reviewer can’t access this project")).toBeVisible(); + await expect(canvas.queryByRole("button", { name: "Manage project access" })).not.toBeInTheDocument(); + await userEvent.click(canvas.getByRole("button", { name: "View agent access" })); + await expect(canvas.getByText("Only people with agent configuration permission can change this access.")).toBeVisible(); + await expect(canvas.queryByRole("button", { name: "Add project" })).not.toBeInTheDocument(); +} }; +export const Light: Story = { globals: { theme: "light" } }; +export const Mobile: Story = { globals: { viewport: { value: "mobile1", isRotated: false } } }; +export const MobileSettings: Story = { args: { initialView: "permissions" }, globals: { viewport: { value: "mobile1", isRotated: false } } }; + +async function addProject(canvasElement: HTMLElement, name = "Agent Extensions") { + const canvas = within(canvasElement), screen = within(canvasElement.ownerDocument.body); + await userEvent.click(await canvas.findByRole("button", { name: "Add project" })); + await userEvent.click(await screen.findByRole("checkbox", { name })); + await userEvent.click(screen.getByRole("button", { name: "Done" })); +} + +export const RepairJourney: Story = { play: async ({ canvasElement, step }) => { + const canvas = within(canvasElement); + await step("Explain the denied project before any provider work", async () => { + await expect(await canvas.findByText("Reviewer can’t access this project")).toBeVisible(); + await expect(canvas.getByRole("alert")).toHaveTextContent("The agent hasn’t started. Your message is saved."); + await expect(canvas.queryByRole("button", { name: "Retry task" })).not.toBeInTheDocument(); + await userEvent.click(canvas.getByRole("button", { name: "Manage project access" })); + }); + await step("Add a project while preserving existing containment", async () => { + await expect(canvas.getByRole("button", { name: "Save project access" })).toBeDisabled(); + await addProject(canvasElement); + await expect(canvas.getByTestId("project-product")).toHaveTextContent("Allowed"); + await expect(canvas.getByTestId("project-extensions")).toHaveTextContent("To add"); + await expect(canvas.getByText("Low-trust review", { exact: true })).toBeVisible(); + await userEvent.click(canvas.getByRole("button", { name: "Save project access" })); + await expect(await canvas.findByText("Project access saved.")).toBeVisible(); + await expect(canvas.getByTestId("project-product")).toHaveTextContent("Allowed"); + await expect(canvas.getByTestId("project-extensions")).toHaveTextContent("Allowed"); + }); + await step("Explicitly retry the saved message after saving access", async () => { + await userEvent.click(canvas.getByRole("button", { name: "Return to task" })); + await expect(canvas.getByText("Project access is ready")).toBeVisible(); + await waitFor(() => expect(canvas.getByText("Are you there?")).toBeVisible()); + await userEvent.click(canvas.getByRole("button", { name: "Retry task" })); + await waitFor(() => expect(canvas.getByText("Yes, I’m here. What would you like me to look at?")).toBeVisible()); + await expect(canvas.queryByRole("alert")).not.toBeInTheDocument(); + }); +} }; + +export const CancelJourney: Story = { args: { initialView: "permissions" }, play: async ({ canvasElement }) => { + const canvas = within(canvasElement); + await addProject(canvasElement); + await userEvent.click(canvas.getByRole("button", { name: "Cancel" })); + await expect(canvas.getByText("Reviewer can’t access this project")).toBeVisible(); + await userEvent.click(canvas.getByRole("button", { name: "Manage project access" })); + await expect(canvas.queryByTestId("project-extensions")).not.toBeInTheDocument(); + await expect(canvas.getByTestId("project-product")).toHaveTextContent("Allowed"); +} }; + +export const SaveFailure: Story = { args: { initialView: "permissions", failSave: true }, play: async ({ canvasElement }) => { + const canvas = within(canvasElement); + await addProject(canvasElement); + await userEvent.click(canvas.getByRole("button", { name: "Save project access" })); + await expect(await canvas.findByRole("alert")).toHaveTextContent("No permissions changed. Your selection is kept."); + await expect(canvas.getByTestId("project-extensions")).toHaveTextContent("To add"); + await expect(canvas.getByTestId("project-product")).toHaveTextContent("Allowed"); +} }; +export const SaveFailureJourney: Story = { args: { initialView: "permissions", failSave: true }, play: async ({ canvasElement }) => { + const canvas = within(canvasElement); + await addProject(canvasElement); + await userEvent.click(canvas.getByRole("button", { name: "Save project access" })); + await userEvent.click(await canvas.findByRole("button", { name: "Try saving again" })); + await expect(await canvas.findByText("Project access saved.")).toBeVisible(); + await expect(canvas.getByTestId("project-extensions")).toHaveTextContent("Allowed"); +} }; +export const Saving: Story = { args: { initialView: "permissions", holdSaving: true }, play: async ({ canvasElement }) => { + const canvas = within(canvasElement); + await addProject(canvasElement); + await userEvent.click(canvas.getByRole("button", { name: "Save project access" })); + await expect(canvas.getByRole("button", { name: "Saving access…" })).toBeDisabled(); + await expect(canvas.getByRole("button", { name: "Cancel" })).toBeDisabled(); +} }; +export const RetryFailure: Story = { args: { initialState: "access_saved", failRetry: true }, play: async ({ canvasElement }) => { + const canvas = within(canvasElement); + await userEvent.click(await canvas.findByRole("button", { name: "Retry task" })); + await expect(await canvas.findByRole("alert")).toHaveTextContent("Couldn’t start the retry. Project access is saved."); +} }; +export const MultipleProjects: Story = { args: { initialView: "permissions" }, play: async ({ canvasElement }) => { + const canvas = within(canvasElement); + await addProject(canvasElement); + await addProject(canvasElement, "Website"); + await userEvent.click(canvas.getByRole("button", { name: "Save project access" })); + await expect(await canvas.findByText("Project access saved.")).toBeVisible(); + for (const id of ["product", "extensions", "website"]) await expect(canvas.getByTestId(`project-${id}`)).toHaveTextContent("Allowed"); +} }; diff --git a/ui/storybook/stories/recovery/project-access.stories.tsx b/ui/storybook/stories/recovery/project-access.stories.tsx new file mode 100644 index 00000000000..3349e6605b3 --- /dev/null +++ b/ui/storybook/stories/recovery/project-access.stories.tsx @@ -0,0 +1,77 @@ +import type { Meta, StoryObj } from "@storybook/react-vite"; +import { useState } from "react"; +import { expect, fn, userEvent, within } from "storybook/test"; +import { AllowedProjectsEditor } from "@/components/AllowedProjectsEditor"; +import { ProjectAccessNotice } from "@/components/ProjectAccessNotice"; + +const meta = { + id: "task-project-access", + title: "Projects and execution/Recovery/Task chat/Project access", + component: ProjectAccessNotice, + parameters: { paperclip: { + capabilities: ["recovery/diagnosis", "agents/instructions"], implementation: "production", boundary: "fixtures", + notes: "Production notice and additive project editor. Local save callbacks cover selection, read-only, cancellation and failed-save states. API authorization, persistence and explicit retry admission are separate server regressions and local application verification.", + } }, + decorators: [(Story) =>
], + args: { agentName: "Reviewer", projectName: "Agent Extensions", ready: false, canManage: true, canRetry: false, onManage: fn(), onRetry: fn() }, +} satisfies Meta; +export default meta; +type Story = StoryObj; + +export const BlockedTask: Story = { play: async ({ canvasElement }) => { + const canvas = within(canvasElement); + await expect(canvas.getByRole("alert")).toHaveTextContent("Your message is saved."); + await expect(canvas.queryByRole("button", { name: "Retry task" })).not.toBeInTheDocument(); + await expect(canvas.getByRole("button", { name: "Manage project access" })).toBeVisible(); +} }; +export const AccessSaved: Story = { args: { ready: true, canRetry: true } }; +export const ReadOnly: Story = { args: { canManage: false } }; +export const Light: Story = { globals: { theme: "light" } }; +export const Mobile: Story = { globals: { viewport: { value: "mobile1", isRotated: false } } }; + +const projects = [{ id: "product", name: "Product" }, { id: "extensions", name: "Agent Extensions" }, { id: "website", name: "Website" }]; +function Editor({ fail = false, canManage = true }: { fail?: boolean; canManage?: boolean }) { + const [ids, setIds] = useState(["product"]); + const [failed, setFailed] = useState(false); + return { + if (fail && !failed) { setFailed(true); throw new Error("Connection interrupted."); } + setIds(current => [...new Set([...current, ...added])]); + }} />; +} +async function selectProject(canvasElement: HTMLElement) { + const canvas = within(canvasElement), screen = within(canvasElement.ownerDocument.body); + await userEvent.click(canvas.getByRole("button", { name: "Add project" })); + await expect(await screen.findByRole("checkbox", { name: "Product" })).toBeDisabled(); + await userEvent.click(screen.getByRole("checkbox", { name: "Agent Extensions" })); + await userEvent.click(screen.getByRole("button", { name: "Done" })); +} +export const AgentSettings: Story = { render: () => }; +export const SaveAccess: Story = { ...AgentSettings, play: async ({ canvasElement }) => { + const canvas = within(canvasElement); + await selectProject(canvasElement); + await userEvent.click(canvas.getByRole("button", { name: "Save project access" })); + await expect(await canvas.findByRole("status")).toHaveTextContent("Project access saved"); + await expect(canvas.getAllByText("Allowed", { exact: true })).toHaveLength(2); + await expect(canvas.queryByRole("button", { name: "Retry task" })).not.toBeInTheDocument(); +} }; +export const CancelAddition: Story = { ...AgentSettings, play: async ({ canvasElement }) => { + const canvas = within(canvasElement); + await selectProject(canvasElement); + await userEvent.click(canvas.getByRole("button", { name: "Cancel" })); + await expect(canvas.queryByText("Agent Extensions", { exact: true })).not.toBeInTheDocument(); + await expect(canvas.getByText("Product")).toBeVisible(); +} }; +export const SaveFailure: Story = { render: () => , play: async ({ canvasElement }) => { + const canvas = within(canvasElement); + await selectProject(canvasElement); + await userEvent.click(canvas.getByRole("button", { name: "Save project access" })); + await expect(await canvas.findByRole("alert")).toHaveTextContent("Your selection is kept"); + await expect(canvas.getByText("To add")).toBeVisible(); + await userEvent.click(canvas.getByRole("button", { name: "Try saving again" })); + await expect(await canvas.findByRole("status")).toHaveTextContent("Project access saved"); +} }; +export const ReadOnlySettings: Story = { render: () => , play: async ({ canvasElement }) => { + await expect(within(canvasElement).queryByRole("button", { name: "Add project" })).not.toBeInTheDocument(); +} }; +export const MobileSettings: Story = { ...AgentSettings, globals: { viewport: { value: "mobile1", isRotated: false } } }; diff --git a/ui/storybook/support/design-proposals/ProjectAccessProposal.tsx b/ui/storybook/support/design-proposals/ProjectAccessProposal.tsx new file mode 100644 index 00000000000..a8894e19af5 --- /dev/null +++ b/ui/storybook/support/design-proposals/ProjectAccessProposal.tsx @@ -0,0 +1,180 @@ +import { useState } from "react"; +import { ArrowLeft, ArrowRight, Check, FolderOpen, Loader2, Plus, ShieldCheck, ShieldX, X } from "lucide-react"; +import { AgentAvatar } from "@/components/AgentAvatar"; +import { DesignPageShell } from "@/components/DesignPageShell"; +import { DesignSidebar } from "@/components/DesignSidebar"; +import { IssueStatusBadge } from "@/components/StatusBadge"; +import { TaskChatBubble } from "@/components/task-chat/TaskChatBubble"; +import { TaskChatLiveRunPill } from "@/components/task-chat/TaskChatLiveRunPill"; +import { Button } from "@/components/ui/button"; +import { Checkbox } from "@/components/ui/checkbox"; +import { Input } from "@/components/ui/input"; +import { Popover, PopoverContent, PopoverTrigger } from "@/components/ui/popover"; +import { DesignProposalAccess } from "./DesignProposalAccess"; + +const projects = [ + { id: "product", name: "Product", repository: "acme/product" }, + { id: "extensions", name: "Agent Extensions", repository: "acme/agent-extensions" }, + { id: "website", name: "Website", repository: "acme/website" }, +]; +const agent = { id: "reviewer-preview", name: "Reviewer" }; +const wait = () => new Promise(resolve => setTimeout(resolve, 650)); + +export interface ProjectAccessProposalProps { + initialView?: "task" | "permissions"; + initialState?: "blocked" | "access_saved"; + canManage?: boolean; + failSave?: boolean; + failRetry?: boolean; + holdSaving?: boolean; +} + +/** Design-only state machine. It never calls a permissions or execution API. */ +function Proposal({ initialView = "task", initialState = "blocked", canManage = true, + failSave = false, failRetry = false, holdSaving = false }: ProjectAccessProposalProps) { + const [view, setView] = useState(initialView); + const [allowed, setAllowed] = useState(initialState === "access_saved" ? ["product", "extensions"] : ["product"]); + const [draft, setDraft] = useState(allowed); + const [pickerOpen, setPickerOpen] = useState(false); + const [query, setQuery] = useState(""); + const [saving, setSaving] = useState(false); + const [saveError, setSaveError] = useState(false); + const [saveAttempted, setSaveAttempted] = useState(false); + const [savedNotice, setSavedNotice] = useState(false); + const [run, setRun] = useState<"stopped" | "starting" | "replied">("stopped"); + const [retryError, setRetryError] = useState(false); + const [retryAttempted, setRetryAttempted] = useState(false); + const [shellNotice, setShellNotice] = useState(""); + const added = projects.filter(project => draft.includes(project.id) && !allowed.includes(project.id)); + const accessReady = allowed.includes("extensions"); + + function openPermissions() { + setDraft(allowed); + setSavedNotice(false); + setSaveError(false); + setView("permissions"); + } + + async function save() { + if (!canManage || saving || !added.length) return; + setSaving(true); + setSaveError(false); + if (holdSaving) return; + await wait(); + setSaving(false); + setSaveAttempted(true); + if (failSave && !saveAttempted) { setSaveError(true); return; } + setAllowed([...draft]); + setSavedNotice(true); + } + + async function retry() { + if (!accessReady || run === "starting") return; + setRun("starting"); + setRetryError(false); + await wait(); + setRetryAttempted(true); + if (failRetry && !retryAttempted) { setRun("stopped"); setRetryError(true); return; } + setRun("replied"); + } + + const task =
+
+
+ DEMO-42·Agent Extensions +
+

Quick check-in

+ +
+ Reviewer +
+ + {run === "starting" ?
+ : run === "replied" ? + :
+
+ {accessReady ? : } +
+

{accessReady ? "Project access is ready" : "Reviewer can’t access this project"}

+

{accessReady + ? "Agent Extensions is now in Reviewer’s allowed projects. Retry to continue with your saved message." + : "This task uses Agent Extensions, which isn’t in Reviewer’s allowed projects. The agent hasn’t started. Your message is saved."}

+ {!accessReady && !canManage &&

An organization admin can add this project in Reviewer’s Permissions / Trust settings.

} +
+
+
+ {accessReady ? + : canManage ? + : } +
+ {retryError &&

Couldn’t start the retry. Project access is saved. Try again.

} + {!accessReady &&
Run details +
Stopped during
Workspace setup
Reason
Project outside allowed scope
Provider started
No
+
} +
} + {run === "replied" && } +
; + + const permissions =
+ +

Reviewer

Permissions / Trust

+
+ +

Low-trust review

Runs in an isolated sandbox. Untrusted output stays quarantined.

+
+
+
+

Allowed projects

+

Projects whose tasks and repositories this agent may access.

+
+ {canManage && + + + setQuery(event.target.value)} /> +
{projects.filter(project => `${project.name} ${project.repository}`.toLowerCase().includes(query.toLowerCase())).map(project => )}
+ {!projects.some(project => `${project.name} ${project.repository}`.toLowerCase().includes(query.toLowerCase())) &&

No projects found.

} + + } +
+
    + {projects.filter(project => draft.includes(project.id)).map(project =>
  • +

    {project.name}

    {project.repository}

    + {allowed.includes(project.id) ? Allowed + : <>To add} +
    • )} +
+ {!accessReady &&

DEMO-42 is waiting for access to Agent Extensions.

} + {!canManage &&

Only people with agent configuration permission can change this access.

} + {added.length > 0 &&
+

Add access to {added.map(project => project.name).join(" and ")}

+

Reviewer will be able to access work and repositories in {added.length === 1 ? "this project" : "these projects"}. Existing project access and low-trust restrictions stay in place.

+
} + {saveError &&

Couldn’t save project access. No permissions changed. Your selection is kept.

} + {savedNotice &&
+ Project access saved. + +
} + {canManage &&
+ + +
} +
+
; + + return
+ setShellNotice("This preview focuses on project access. Use Manage project access to try the repair.")} />} + headerActions={Design preview}> + {shellNotice &&

{shellNotice}

} + {view === "task" ? task : permissions} + +
; +} + +export function ProjectAccessProposal(props: ProjectAccessProposalProps) { + return ; +}