diff --git a/doc/DATABASE.md b/doc/DATABASE.md index b3f7070103c..46a104d61a6 100644 --- a/doc/DATABASE.md +++ b/doc/DATABASE.md @@ -1,5 +1,54 @@ # Database +### Durable native provider backups + +`native_harness_backups` journals encrypted provider-session archives in object +storage. One current row per company and session scope selects the published +backup. Pending uploads remain recorded after failure; a controller must still +own the native run when it publishes. Superseded controllers cannot replace the +selected backup. Archive bytes never enter the database. + +The dedicated volume migration imports verified surviving backups without +removing their source files or changing task-session pointers. Replacement +readiness rejects retained session pointers whose backups remain local. The +existing bounded startup retention pass retires current backups only after its +task, run, lease and age checks pass. Old superseded objects have a separate +retention path that preserves the current backup and exact backups referenced +by live leases. Abandoned pending uploads also require a settled run without an +owner. Object deletion is acknowledged before removing the ledger row. This +adds no idle polling. + +Archives stream up to 4 GiB of file data and 100,000 entries, with a separate +4.75 GiB encoded-object bound. Contained relative symbolic links are preserved +verbatim; absolute or escaping targets, link chains, writes beneath links, and +special files are rejected. Offline verification streams file hashes and honors +cancellation. A changed local manifest cannot be silently skipped just because +a previous migration published an object. Backups whose source run was deleted +remain untouched on the original volume and do not block enrollment. + +The same offline maintenance process imports surviving legacy controller state +into `native_runner_checkpoints` and binds its existing session snapshot through +`nativeControllerRecovery` metadata. Runner validates the original protocol +state and recorded artifact; Core requires a finished run with no live owner. +Blocked or pending recovery records remain unchanged while their surviving +state is preserved; migration does not mark their work completed. +The normal bounded continuation selector handles inherited, unstarted follow-ups. +Never-started queued turns keep their task pointers and remain queued. Original +files stay intact, canonical database checkpoints take precedence, and missing +source files are reported separately rather than fabricated. This migration +does not recover provider state that was already lost before enrollment. + +For an already-failed run, a valid legacy controller can lack proof of the +original Runner artifact after its old bootstrap tickets were pruned. The +migration preserves those exact controller and supervisor bytes as an encrypted +`paperclip.legacy-controller-archive.v1` object through the same backup ledger. +Its scope is the hash of its manifest and is separate from the provider session +scope. Publication requires a terminal-failure coordinator with no live owner. +The archive does not become an executable Runner checkpoint. Task pointers, +failed results and recovery blockers stay unchanged. A successful retained +session without artifact proof still blocks migration. The migration receipt +reports failed-session archives separately from imported recovery authorities. + ### Proactive agent chat messages Migration `0419_slippery_impossible_man.sql` adds an agent-scoped unique constraint diff --git a/doc/RUNNER-INTEGRATION-PLAYBOOK.md b/doc/RUNNER-INTEGRATION-PLAYBOOK.md index 1480ebc49c9..d3f39a9418a 100644 --- a/doc/RUNNER-INTEGRATION-PLAYBOOK.md +++ b/doc/RUNNER-INTEGRATION-PLAYBOOK.md @@ -781,6 +781,13 @@ case/flow, matcher, result validator, and report test. Follow this sequence: 2. Select existing environments and relevant journeys. Add a case only when a promised behavior lacks an independent oracle; keep one durable purpose per suite. Record expected turns, deadline, spending bound, and cleanup obligations. + Native sandbox integrations must pass `rolling-web-deployment` and + `rolling-web-deployment-warm` before cloud qualification. + The shared suite includes registered native Daytona profiles automatically. + It replaces the web controller during real work, deletes its entire home, + checks continuous reads and writes, and requires a later turn to retain the + native and provider session identity after another empty-home replacement. + A profile's presence in the matrix is not a passing qualification receipt. 3. Make setup create an isolated company, workspace, agent, and secrets through supported APIs. Execute the graded action through ordinary browser/API surfaces. Avoid direct database writes and test-only provider shortcuts. diff --git a/packages/db/src/migrations/0429_high_freak.sql b/packages/db/src/migrations/0429_high_freak.sql new file mode 100644 index 00000000000..7571d5f5566 --- /dev/null +++ b/packages/db/src/migrations/0429_high_freak.sql @@ -0,0 +1,15 @@ +CREATE TABLE IF NOT EXISTS "native_harness_backups" ( + "id" uuid PRIMARY KEY DEFAULT gen_random_uuid() NOT NULL, + "company_id" uuid NOT NULL, + "issue_id" uuid NOT NULL, + "run_id" uuid NOT NULL, + "scope_digest" text NOT NULL, + "manifest_sha256" text NOT NULL, + "status" text DEFAULT 'pending' NOT NULL, + "reference" jsonb NOT NULL, + "created_at" timestamp with time zone DEFAULT now() NOT NULL, + "updated_at" timestamp with time zone DEFAULT now() NOT NULL +); +--> statement-breakpoint +CREATE UNIQUE INDEX IF NOT EXISTS "native_harness_backups_current_idx" ON "native_harness_backups" USING btree ("company_id","scope_digest") WHERE "native_harness_backups"."status" = 'current';--> statement-breakpoint +CREATE INDEX IF NOT EXISTS "native_harness_backups_scope_idx" ON "native_harness_backups" USING btree ("company_id","scope_digest","manifest_sha256"); diff --git a/packages/db/src/migrations/meta/0424_snapshot.json b/packages/db/src/migrations/meta/0429_snapshot.json similarity index 99% rename from packages/db/src/migrations/meta/0424_snapshot.json rename to packages/db/src/migrations/meta/0429_snapshot.json index d36bfdfe18c..ce1d9aceaee 100644 --- a/packages/db/src/migrations/meta/0424_snapshot.json +++ b/packages/db/src/migrations/meta/0429_snapshot.json @@ -1,6 +1,6 @@ { - "id": "cb093c1d-3e54-4ab2-8009-9c243097224c", - "prevId": "fe0b877c-d63e-46d9-8798-c7729fd13eb7", + "id": "809726a1-762c-4c9d-a7fd-55670f1c856f", + "prevId": "fb4383f6-cbe6-4aff-9fd2-6eeb69a6927a", "version": "7", "dialect": "postgresql", "tables": { @@ -28825,6 +28825,337 @@ "checkConstraints": {}, "isRLSEnabled": false }, + "public.execution_admission_project_charges": { + "name": "execution_admission_project_charges", + "schema": "", + "columns": { + "admission_id": { + "name": "admission_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "project_id": { + "name": "project_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + } + }, + "indexes": { + "execution_admission_project_charges_project_idx": { + "name": "execution_admission_project_charges_project_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "project_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "execution_admission_project_charges_company_admission_fk": { + "name": "execution_admission_project_charges_company_admission_fk", + "tableFrom": "execution_admission_project_charges", + "tableTo": "execution_admissions", + "columnsFrom": [ + "company_id", + "admission_id" + ], + "columnsTo": [ + "company_id", + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "execution_admission_project_charges_company_project_fk": { + "name": "execution_admission_project_charges_company_project_fk", + "tableFrom": "execution_admission_project_charges", + "tableTo": "projects", + "columnsFrom": [ + "company_id", + "project_id" + ], + "columnsTo": [ + "company_id", + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": { + "execution_admission_project_charges_admission_id_project_id_pk": { + "name": "execution_admission_project_charges_admission_id_project_id_pk", + "columns": [ + "admission_id", + "project_id" + ] + } + }, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.execution_admissions": { + "name": "execution_admissions", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true + }, + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "run_id": { + "name": "run_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "attempt": { + "name": "attempt", + "type": "integer", + "primaryKey": false, + "notNull": true + }, + "primary_project_id": { + "name": "primary_project_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "context_revision": { + "name": "context_revision", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "admitted_project_ids": { + "name": "admitted_project_ids", + "type": "uuid[]", + "primaryKey": false, + "notNull": false + }, + "settlement_id": { + "name": "settlement_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "settled_at": { + "name": "settled_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "execution_admissions_company_id_uq": { + "name": "execution_admissions_company_id_uq", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + }, + "execution_admissions_run_attempt_uq": { + "name": "execution_admissions_run_attempt_uq", + "columns": [ + { + "expression": "run_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "attempt", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + }, + "execution_admissions_live_run_uq": { + "name": "execution_admissions_live_run_uq", + "columns": [ + { + "expression": "run_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "where": "\"execution_admissions\".\"settled_at\" is null", + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "execution_admissions_company_id_companies_id_fk": { + "name": "execution_admissions_company_id_companies_id_fk", + "tableFrom": "execution_admissions", + "tableTo": "companies", + "columnsFrom": [ + "company_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + }, + "execution_admissions_run_id_heartbeat_runs_id_fk": { + "name": "execution_admissions_run_id_heartbeat_runs_id_fk", + "tableFrom": "execution_admissions", + "tableTo": "heartbeat_runs", + "columnsFrom": [ + "run_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": { + "execution_admissions_attempt_check": { + "name": "execution_admissions_attempt_check", + "value": "\"execution_admissions\".\"attempt\" > 0" + }, + "execution_admissions_settlement_check": { + "name": "execution_admissions_settlement_check", + "value": "(\"execution_admissions\".\"settlement_id\" is null) = (\"execution_admissions\".\"settled_at\" is null)" + }, + "execution_admissions_project_set_check": { + "name": "execution_admissions_project_set_check", + "value": "case when \"execution_admissions\".\"admitted_project_ids\" is null\n then \"execution_admissions\".\"settled_at\" is not null else array_position(\"execution_admissions\".\"admitted_project_ids\", null) is null and cardinality(\"execution_admissions\".\"admitted_project_ids\") <= 100 end" + } + }, + "isRLSEnabled": false + }, + "public.project_task_execution_policies": { + "name": "project_task_execution_policies", + "schema": "", + "columns": { + "project_id": { + "name": "project_id", + "type": "uuid", + "primaryKey": true, + "notNull": true + }, + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "max_concurrent_tasks": { + "name": "max_concurrent_tasks", + "type": "integer", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": {}, + "foreignKeys": { + "project_task_execution_policies_company_project_fk": { + "name": "project_task_execution_policies_company_project_fk", + "tableFrom": "project_task_execution_policies", + "tableTo": "projects", + "columnsFrom": [ + "company_id", + "project_id" + ], + "columnsTo": [ + "company_id", + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": { + "project_task_execution_policies_maximum_check": { + "name": "project_task_execution_policies_maximum_check", + "value": "\"project_task_execution_policies\".\"max_concurrent_tasks\" is null or \"project_task_execution_policies\".\"max_concurrent_tasks\" > 0" + } + }, + "isRLSEnabled": false + }, "public.execution_workspace_repositories": { "name": "execution_workspace_repositories", "schema": "", @@ -29999,6 +30330,22 @@ "concurrently": false, "method": "btree", "with": {} + }, + "execution_workspace_snapshots_imported_children_idx": { + "name": "execution_workspace_snapshots_imported_children_idx", + "columns": [ + { + "expression": "(\"metadata\"->'importedWorkspaceChildren') jsonb_path_ops", + "asc": true, + "isExpression": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"execution_workspace_snapshots\".\"metadata\" ? 'importedWorkspaceChildren'", + "concurrently": false, + "method": "gin", + "with": {} } }, "foreignKeys": {}, @@ -43679,6 +44026,133 @@ }, "isRLSEnabled": false }, + "public.native_harness_backups": { + "name": "native_harness_backups", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "issue_id": { + "name": "issue_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "run_id": { + "name": "run_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "scope_digest": { + "name": "scope_digest", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "manifest_sha256": { + "name": "manifest_sha256", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'pending'" + }, + "reference": { + "name": "reference", + "type": "jsonb", + "primaryKey": false, + "notNull": true + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "native_harness_backups_current_idx": { + "name": "native_harness_backups_current_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "scope_digest", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "where": "\"native_harness_backups\".\"status\" = 'current'", + "concurrently": false, + "method": "btree", + "with": {} + }, + "native_harness_backups_scope_idx": { + "name": "native_harness_backups_scope_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "scope_digest", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "manifest_sha256", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, "public.native_run_finalizations": { "name": "native_run_finalizations", "schema": "", @@ -62448,6 +62922,12 @@ "primaryKey": false, "notNull": false }, + "cleanup_verified_at": { + "name": "cleanup_verified_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, "created_at": { "name": "created_at", "type": "timestamp with time zone", diff --git a/packages/db/src/migrations/meta/_journal.json b/packages/db/src/migrations/meta/_journal.json index a2f1da3ce13..1d66ff4fa0c 100644 --- a/packages/db/src/migrations/meta/_journal.json +++ b/packages/db/src/migrations/meta/_journal.json @@ -2990,6 +2990,13 @@ "when": 1793743828770, "tag": "0428_retained_admitted_project_sets", "breakpoints": true + }, + { + "idx": 429, + "version": "7", + "when": 1793760539457, + "tag": "0429_high_freak", + "breakpoints": true } ] } \ No newline at end of file diff --git a/packages/db/src/schema/index.ts b/packages/db/src/schema/index.ts index f12096d60cb..91c9770c6b1 100644 --- a/packages/db/src/schema/index.ts +++ b/packages/db/src/schema/index.ts @@ -146,6 +146,7 @@ export { completionContracts } from "./completion_contracts.js"; export { nativeRunResults } from "./native_run_results.js"; export { nativeRunFinalizations } from "./native_run_finalizations.js"; export { nativeRunnerCheckpoints } from "./native_runner_checkpoints.js"; +export { nativeHarnessBackups } from "./native_harness_backups.js"; export { workAssessments } from "./work_assessments.js"; export { statusDecisions } from "./status_decisions.js"; export { statusDecisionEffects } from "./status_decision_effects.js"; diff --git a/packages/db/src/schema/native_harness_backups.ts b/packages/db/src/schema/native_harness_backups.ts new file mode 100644 index 00000000000..9f39797534b --- /dev/null +++ b/packages/db/src/schema/native_harness_backups.ts @@ -0,0 +1,19 @@ +import { sql } from "drizzle-orm"; +import { index, jsonb, pgTable, text, timestamp, uniqueIndex, uuid } from "drizzle-orm/pg-core"; + +export const nativeHarnessBackups = pgTable("native_harness_backups", { + id: uuid("id").primaryKey().defaultRandom(), + companyId: uuid("company_id").notNull(), + issueId: uuid("issue_id").notNull(), + runId: uuid("run_id").notNull(), + scopeDigest: text("scope_digest").notNull(), + manifestSha256: text("manifest_sha256").notNull(), + status: text("status").notNull().default("pending"), + reference: jsonb("reference").$type>().notNull(), + createdAt: timestamp("created_at", { withTimezone: true }).notNull().defaultNow(), + updatedAt: timestamp("updated_at", { withTimezone: true }).notNull().defaultNow(), +}, table => ({ + current: uniqueIndex("native_harness_backups_current_idx").on(table.companyId, table.scopeDigest) + .where(sql`${table.status} = 'current'`), + scope: index("native_harness_backups_scope_idx").on(table.companyId, table.scopeDigest, table.manifestSha256), +})); diff --git a/packages/paperclip-runner/docs/runner-protocol-live-evals.md b/packages/paperclip-runner/docs/runner-protocol-live-evals.md index 1ef927f5825..3760e60f40c 100644 --- a/packages/paperclip-runner/docs/runner-protocol-live-evals.md +++ b/packages/paperclip-runner/docs/runner-protocol-live-evals.md @@ -659,6 +659,19 @@ billable usage. These receipt corrections also apply to non-eval ACPX sessions. The durable redactor preserves these known numeric token-counter fields; string/object values under the same names remain redacted as sensitive data. +Grok 1.0.13 reports whole-prompt usage in `session/prompt` response +`_meta.usage`. The sibling `_meta` counters describe only the last model call. +Runner binds the nested receipt to the outbound JSON-RPC request and session, +then emits it before the terminal event. Replayed, conflicting, malformed, or +incomplete receipts cannot supply complete accounting. ACP input includes cache +reads and cache creation; Runner separates those buckets and does not add the +reasoning subset to output again. A complete `costUsdTicks` uses 10 billion ticks +per USD. Missing or partial cost remains unknown. The sidecar's `cost.basis` +distinguishes a `turn` total from the default session-cumulative cost, so later +turns do not inherit or subtract an earlier turn's bill. +These semantics follow the upstream [prompt usage contract](https://github.com/xai-org/grok-build/blob/2bdd1d6a6369de0e8c68132ea4539e9abd9e14a8/crates/codegen/xai-grok-shell/src/extensions/notification.rs) +and [prompt response tests](https://github.com/xai-org/grok-build/blob/2bdd1d6a6369de0e8c68132ea4539e9abd9e14a8/crates/codegen/xai-grok-shell/src/agent/mvp_agent/prompt_response_meta_tests.rs). + `all` means the maintained enabled campaign, not every matching file in the roster directory. A missing campaign file fails closed. Within an enabled lane, a missing remote profile or unavailable provider is retained as an diff --git a/packages/paperclip-runner/protocol/provider-schemas/acpx-sidecar.schema.json b/packages/paperclip-runner/protocol/provider-schemas/acpx-sidecar.schema.json index bd1717d102f..7578065f002 100644 --- a/packages/paperclip-runner/protocol/provider-schemas/acpx-sidecar.schema.json +++ b/packages/paperclip-runner/protocol/provider-schemas/acpx-sidecar.schema.json @@ -225,6 +225,14 @@ }, "additionalProperties": false, "allOf": [ + { + "if": { "properties": { "eventType": { "const": "runtime.event" }, "payload": { + "type": "object", + "properties": { "type": { "const": "status" }, "tag": { "const": "usage_update" } }, + "required": ["type", "tag"] + } } }, + "then": { "properties": { "payload": { "type": "object", "properties": { "cost": { "$ref": "#/$defs/usageCost" } } } } } + }, { "if": { "properties": { "eventType": { "const": "runtime.rich_event" } } }, "then": { @@ -237,6 +245,16 @@ } ] }, + "usageCost": { + "description": "Provider cost is session-cumulative unless basis explicitly identifies a complete turn total. A turn total is never a session counter.", + "type": ["object", "null"], + "properties": { + "amount": { "type": ["number", "null"], "minimum": 0 }, + "currency": { "type": ["string", "null"], "maxLength": 16 }, + "basis": { "enum": ["turn", "session_cumulative"] } + }, + "additionalProperties": false + }, "richEvent": { "type": "object", "required": [ diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/provider_events.rs b/packages/paperclip-runner/runner/crates/runner-core/src/provider_events.rs index b1d76fef47a..390b8a69ca3 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/provider_events.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/provider_events.rs @@ -1208,9 +1208,16 @@ fn normalize_acpx_status( Some(Value::String(currency)) => currency.eq_ignore_ascii_case("USD"), Some(_) => false, }; - // ACPX 0.13.1 documents breakdown as per-turn usage while cost is - // session-cumulative. Keep those authorities separate so consumers do - // not add the same tokens twice or treat cumulative cost as a delta. + let turn_cost = payload.pointer("/cost/basis").and_then(Value::as_str) == Some("turn"); + let provider_cost = if cost_is_usd { + payload + .pointer("/cost/amount") + .and_then(Value::as_f64) + .filter(|value| value.is_finite() && *value >= 0.0) + .unwrap_or(0.0) + } else { + 0.0 + }; let cumulative = json!({ "inputTokens": 0, "outputTokens": 0, @@ -1218,15 +1225,7 @@ fn normalize_acpx_status( "cacheWriteTokens": 0, "activeSeconds": 0.0, "requests": provider_requests, - "providerCostUsd": if cost_is_usd { - payload - .pointer("/cost/amount") - .and_then(Value::as_f64) - .filter(|value| value.is_finite() && *value >= 0.0) - .unwrap_or(0.0) - } else { - 0.0 - }, + "providerCostUsd": if turn_cost { 0.0 } else { provider_cost }, }); let run_delta = json!({ "inputTokens": nonnegative_u64(breakdown.get("inputTokens")), @@ -1243,7 +1242,7 @@ fn normalize_acpx_status( ), "activeSeconds": 0.0, "requests": 1, - "providerCostUsd": 0.0, + "providerCostUsd": if turn_cost { provider_cost } else { 0.0 }, }); return vec![NormalizedProviderEvent { event_type: "usage.reported".to_owned(), @@ -1499,6 +1498,31 @@ fn has_rfc_uri_scheme_prefix(value: &str) -> bool { mod tests { use super::*; + #[test] + fn acpx_turn_cost_is_not_a_cumulative_session_price() { + let base = json!({"tag": "usage_update", "breakdown": { + "inputTokens": 30787, "outputTokens": 5763, "cachedReadTokens": 475008, + "cachedWriteTokens": 0, "thoughtTokens": 0 + }, "cost": {"amount": 0.336842, "currency": "USD", "basis": "turn"}}); + let first = normalize_acpx_status(&base, "item", "turn-1", 1); + assert_eq!(first[0].payload["runDeltaAvailable"], true); + assert_eq!(first[0].payload["runDelta"]["providerCostUsd"], 0.336842); + assert_eq!(first[0].payload["cumulative"]["providerCostUsd"], 0.0); + assert_eq!(first[0].payload["runDelta"]["inputTokens"], 30787); + assert_eq!(first[0].payload["runDelta"]["outputTokens"], 5763); + let mut second = base.clone(); + second["cost"]["amount"] = json!(0.001); + let next = normalize_acpx_status(&second, "next", "turn-2", 2); + assert_eq!(next[0].payload["runDelta"]["providerCostUsd"], 0.001); + second["cost"].as_object_mut().unwrap().remove("basis"); + let cumulative = normalize_acpx_status(&second, "legacy", "turn-3", 3); + assert_eq!( + cumulative[0].payload["cumulative"]["providerCostUsd"], + 0.001 + ); + assert_eq!(cumulative[0].payload["runDelta"]["providerCostUsd"], 0.0); + } + #[test] fn performance_diagnostics_keep_only_bounded_content_free_measurements() { let mut spans = vec![json!({"name":"private prompt","startedAtMs":1,"endedAtMs":2})]; diff --git a/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.test.ts b/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.test.ts index 5a96c9aebea..9741e14641b 100644 --- a/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.test.ts +++ b/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.test.ts @@ -8,7 +8,7 @@ import { appendSemanticToolReceipt, readNativeSemanticReceipt, semanticInputSha2 import { startRunnerToolBridge, type RunnerToolCall } from "../drivers/runner-tool-bridge.js"; import { validatePrpStructuredRunResult } from "../protocol/replay-contract.js"; import { resolveCompletionContinuation } from "../drivers/completion-continuation.js"; -import { acpxUsageEstimateNotice, persistedAcpxTurnUsage, persistedCursorUsageNotice } from "../drivers/acpx/usage-accounting.js"; +import { acpxUsageEstimateNotice, persistedAcpxTurnUsage, persistedCursorUsageNotice, qualifiedAcpxUsageBreakdown } from "../drivers/acpx/usage-accounting.js"; import { stripTypeScriptTypes } from "node:module"; import { cursorPlanToolIdentity, cursorToolIdentity } from "../drivers/acpx/cursor-plan-tool-identity.js"; import { createHash, randomUUID } from "node:crypto"; @@ -21,6 +21,22 @@ import { createKimiMessageProjection } from "../drivers/acpx/kimi-message-bounda import { afterEach, describe, expect, it, vi } from "vitest"; describe("live sidecar pending request snapshots", () => { + it("preserves explicit turn cost through the sidecar's closed usage projection", () => { + const source = readFileSync(new URL("./acpx-runtime-sidecar.ts", import.meta.url), "utf8"); + const start = source.indexOf("function safeUsage("); + const end = source.indexOf("function safeOutput(", start); + const project = new Function("record", "qualifiedAcpxUsageBreakdown", "initializedAgent", "safeText", "text", + `${stripTypeScriptTypes(source.slice(start, end))}; return safeUsage;`)( + (value: unknown) => value && typeof value === "object" ? value : {}, qualifiedAcpxUsageBreakdown, "grok", + (value: string) => value, (value: unknown) => typeof value === "string" ? value : "", + ); + expect(project({ amount: 0.336842, currency: "USD", basis: "turn", rawSecret: "excluded" }, null)).toEqual({ + cost: { amount: 0.336842, currency: "USD", basis: "turn" }, breakdown: null, + }); + expect(project({ amount: 0.7, currency: "USD" }, null)).toEqual({ cost: { amount: 0.7, currency: "USD" }, breakdown: null }); + expect(project(undefined, null)).toEqual({ cost: null, breakdown: null }); + }); + it("attests current callback identities after the live status barrier without exposing forms", async () => { const source = readFileSync(new URL("./acpx-runtime-sidecar.ts", import.meta.url), "utf8"); const start = source.indexOf(' if (request.command === "session.snapshot") {'); diff --git a/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.ts b/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.ts index 155f963fd99..8510323ef77 100644 --- a/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.ts +++ b/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.ts @@ -1199,6 +1199,7 @@ function safeUsage(cost: unknown, breakdown: unknown): Record { ? null : { amount: safeNonNegativeNumber(nativeCost.amount), + ...(nativeCost.basis === "turn" ? { basis: "turn" } : {}), currency: safeText(text(nativeCost.currency)).slice(0, 16) || null, }, breakdown: diff --git a/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts b/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts index 5c4bb9d17ad..3f9736ba32d 100644 --- a/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts +++ b/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts @@ -31,6 +31,7 @@ import { digestPaperclipSemanticContent } from "../semantic-tools/receipts.js"; import { DurablePrpControlPlane, decodeDurablePrpCheckpoint, + captureLegacyDurablePrpCheckpoint, inspectDurablePrpCheckpointHeader, inspectStagedRunAttachment, inspectRetainedRunInstructions, @@ -220,6 +221,40 @@ it("restores authentication and commands from opaque checkpoint records", async } }); +it("imports the authenticated legacy authority without changing its source and restores it in an empty home", async () => { + const root = mkdtempSync(resolve(tmpdir(), "runner-legacy-import-")); + const options = { stateDirectory: resolve(root, "old/control-plane"), identity, expectedRunnerVersion, expectedRunnerDigest }; + const core = new DurablePrpControlPlane(options); + let restored: DurablePrpControlPlane | undefined; + try { + await core.start(); + const client = (await authenticate(core, core.issueBootstrapTicket()))!; + core.queueCommand("session.snapshot", {}, "retained-maintenance"); + client.socket.destroy(); + await core.stop(); + const file = resolve(options.stateDirectory, "control-plane-state.json"); + const source = readFileSync(file, "utf8"); + const checkpoint = captureLegacyDurablePrpCheckpoint(source, identity); + expect(readFileSync(file, "utf8")).toBe(source); + expect(checkpoint.runnerDigest).toBe(expectedRunnerDigest); + expect(checkpoint.recoveryEvidence.authenticatedConnectionCount).toBe(1); + rmSync(resolve(root, "old"), { recursive: true }); + checkpoint.restoreSessionDirectory(resolve(root, "new")); + restored = new DurablePrpControlPlane({ ...options, stateDirectory: resolve(root, "new/control-plane") }); + expect(restored.getCommand("retained-maintenance")).toMatchObject({ type: "session.snapshot" }); + expect(() => captureLegacyDurablePrpCheckpoint(source, { ...identity, runId: "another-run" })).toThrow("binding_mismatch"); + const state = JSON.parse(source); + expect(() => captureLegacyDurablePrpCheckpoint(JSON.stringify({ ...state, tickets: {} }), identity)).toThrow("artifact_unproven"); + expect(captureLegacyDurablePrpCheckpoint(JSON.stringify({ ...state, tickets: {} }), identity, + { version: expectedRunnerVersion, digest: expectedRunnerDigest }).runnerDigest).toBe(expectedRunnerDigest); + expect(() => captureLegacyDurablePrpCheckpoint(source, identity, + { version: expectedRunnerVersion, digest: `sha256:${"0".repeat(64)}` })).toThrow("artifact_unproven"); + expect(() => captureLegacyDurablePrpCheckpoint(JSON.stringify({ schema: "paperclip.runner.external-control-plane-state.v1", identity }), identity)).toThrow("binding_mismatch"); + } finally { + await restored?.stop(); await core.stop(); rmSync(root, { recursive: true, force: true }); + } +}); + it.each(["missing record", "invalid state", "wrong sequence", "invalid preparation identity", "duplicate preparation", "malformed header", "malformed cursor"])("rejects %s at checkpoint decoding", async fault => { const root = mkdtempSync(resolve(tmpdir(), "runner-checkpoint-invalid-")); const core = new DurablePrpControlPlane({ stateDirectory: root, identity, expectedRunnerVersion, expectedRunnerDigest }); diff --git a/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts b/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts index f0c727bac5b..87e7eae5042 100644 --- a/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts +++ b/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts @@ -392,6 +392,33 @@ function captureCheckpoint(state: StoredCoreState, runnerVersion: string, runner }; } +export function captureLegacyDurablePrpCheckpoint(contents: string, expected: { + runId: string; runnerInstanceId: string; normalizedSessionId: string; +}, verifiedArtifact?: { version: string; digest: string }): DurablePrpCheckpoint { + if (Buffer.byteLength(contents) > maxStateBytes) throw new Error("native_runner_checkpoint_invalid"); + const value: unknown = JSON.parse(contents); + if (!isRecord(value) || !exactIdentity(value.identity) || !isStoredCoreState(value, value.identity) + || value.identity.runId !== expected.runId || value.identity.runnerInstanceId !== expected.runnerInstanceId + || value.identity.normalizedSessionId !== expected.normalizedSessionId + || checkpointRunId(value) !== expected.runId || value.warmTransition + || value.stagedRunAttachment && !stagedTargetActivated(value)) { + throw new Error("native_runner_checkpoint_binding_mismatch"); + } + const artifacts: Array<{ runnerVersion: string; runnerDigest: string }> = value.completedWarmTransition + ? [value.completedWarmTransition.receipt] + : Object.values(value.tickets).filter(ticket => ticket.usedAt !== null + && Number.isFinite(Date.parse(ticket.usedAt)) + && canonicalJson(ticket.identity) === canonicalJson(value.identity)); + if (verifiedArtifact) artifacts.push({ runnerVersion: verifiedArtifact.version, runnerDigest: verifiedArtifact.digest }); + const artifact = artifacts[0]; + if (!artifact || typeof artifact.runnerVersion !== "string" || !artifact.runnerVersion + || !runnerDigestPattern.test(artifact.runnerDigest) + || artifacts.some(candidate => candidate.runnerVersion !== artifact.runnerVersion || candidate.runnerDigest !== artifact.runnerDigest)) { + throw new Error("native_runner_checkpoint_artifact_unproven"); + } + return decodeDurablePrpCheckpoint(captureCheckpoint(value, artifact.runnerVersion, artifact.runnerDigest).records); +} + function parseCheckpointHeader(contents: string): Record & { runnerVersion: string; runnerDigest: string; state: Record & { identity: DurableRecoveryIdentity }; diff --git a/packages/paperclip-runner/src/drivers/acpx/codex-runtime-adapter.test.ts b/packages/paperclip-runner/src/drivers/acpx/codex-runtime-adapter.test.ts index 0df913dbe3d..6c8f9295f73 100644 --- a/packages/paperclip-runner/src/drivers/acpx/codex-runtime-adapter.test.ts +++ b/packages/paperclip-runner/src/drivers/acpx/codex-runtime-adapter.test.ts @@ -19,6 +19,7 @@ import { describe, expect, it, vi } from "vitest"; import type { VerifiedAcpxCommandLease } from "./installation-integrity.js"; import { openCodexAcpxRuntime } from "./codex-runtime-adapter.js"; import { cursorInstructionBinding } from "./cursor-instructions.js"; +import { persistedAcpxTurnUsage } from "./usage-accounting.js"; import { createAcpxCommandLeaseOwner } from "./command-lease-owner.js"; import { resolveQualifiedAcpxProfile } from "./qualified-profiles.js"; import type { AcpxRuntimePortOpenOptions } from "./runtime-host.js"; @@ -913,6 +914,83 @@ describe("Codex ACPX runtime adapter", () => { expect(assertWorkspaceHeld).toHaveBeenCalledOnce(); expect(command.spawn).not.toHaveBeenCalled(); }); + it("exposes only the active Grok prompt response as terminal usage across consecutive turns", async () => { + const runtime = fakeRuntime(); + const durableRecord = { acpxRecordId: "record-1", acpSessionId: "backend-1", agentSessionId: "agent-1", lastRequestId: "previous" }; + const durableStore: AcpSessionStore = { load: vi.fn(async () => structuredClone(durableRecord) as never), save: vi.fn() }; + let created!: AcpRuntimeOptions; + const options = openOptions(fakeCommand()); + options.profile = { ...options.profile, agent: "grok" }; + const port = await openCodexAcpxRuntime(options, { createRegistry: () => registry(), createStore: () => durableStore, + createRuntime: value => { created = value; return runtime; } }); + let sequence = 0; + vi.mocked(runtime.startTurn).mockImplementation(input => { + const wireId = ++sequence; + durableRecord.lastRequestId = input.requestId; + created.onAcpMessage?.("outbound", { id: wireId, method: "session/prompt", params: { sessionId: "backend-1" } }); + created.onAcpMessage?.("inbound", { id: wireId, result: { _meta: { + sessionId: "backend-1", promptId: `prompt-${wireId}`, requestId: `prompt-${wireId}`, + usage: { inputTokens: 50, outputTokens: 5, totalTokens: 55, cachedReadTokens: 20, cacheCreationTokens: 0, + reasoningTokens: 2, modelCalls: 1, costUsdTicks: wireId === 1 ? 10000000 : 5000000 }, + } } }); + return { requestId: input.requestId, promptStarted: Promise.resolve(), events: (async function* () {})(), + result: Promise.resolve({ status: "completed" as const }), cancel: async () => undefined, closeStream: async () => undefined }; + }); + try { + for (const [requestId, amount] of [["runner-1", 0.001], ["runner-2", 0.0005]] as const) { + const before = await port.getStatus(); + const turn = port.startTurn({ requestId, text: "fixture" }); + await turn.result; + const after = await port.getStatus(); + expect(persistedAcpxTurnUsage(before, after, requestId)).toMatchObject({ + cost: { amount, currency: "USD", basis: "turn" }, + breakdown: { inputTokens: 30, outputTokens: 5, cachedReadTokens: 20, thoughtTokens: 0 }, + }); + created.onAcpMessage?.("inbound", { id: sequence, error: { message: "late conflicting response" } }); + expect(await port.getStatus()).toEqual(after); + } + } finally { await port.close({ reason: "fixture done" }); } + }); + + it.each(["cancel", "closeStream"] as const)("retains the bound Grok usage response during %s until the prompt settles", async (stop) => { + const runtime = fakeRuntime(); + const durableRecord = { acpxRecordId: "record-1", acpSessionId: "backend-1", agentSessionId: "agent-1", lastRequestId: "previous" }; + const durableStore: AcpSessionStore = { load: vi.fn(async () => structuredClone(durableRecord) as never), save: vi.fn() }; + let created!: AcpRuntimeOptions; + let settle!: (value: { status: "completed" }) => void; + const result = new Promise<{ status: "completed" }>(resolve => { settle = resolve; }); + const options = openOptions(fakeCommand()); + options.profile = { ...options.profile, agent: "grok" }; + const port = await openCodexAcpxRuntime(options, { createRegistry: () => registry(), createStore: () => durableStore, + createRuntime: value => { created = value; return runtime; } }); + vi.mocked(runtime.startTurn).mockImplementation(input => { + durableRecord.lastRequestId = input.requestId; + created.onAcpMessage?.("outbound", { id: 1, method: "session/prompt", params: { sessionId: "backend-1" } }); + return { requestId: input.requestId, promptStarted: Promise.resolve(), events: (async function* () {})(), result, + cancel: async () => undefined, closeStream: async () => undefined }; + }); + try { + const before = await port.getStatus(); + const turn = port.startTurn({ requestId: "runner-cancelled", text: "fixture" }); + await turn[stop]({ reason: "fixture stopped" }); + created.onAcpMessage?.("outbound", { id: 2, method: "session/prompt", params: { sessionId: "backend-1" } }); + created.onAcpMessage?.("inbound", { id: 1, result: { _meta: { + sessionId: "backend-1", promptId: "prompt-1", requestId: "prompt-1", + usage: { inputTokens: 50, outputTokens: 5, totalTokens: 55, cachedReadTokens: 20, cacheCreationTokens: 0, + reasoningTokens: 2, modelCalls: 1, costUsdTicks: 10000000 }, + } } }); + settle({ status: "completed" }); + await turn.result; + const after = await port.getStatus(); + expect(persistedAcpxTurnUsage(before, after, "runner-cancelled")).toMatchObject({ + cost: { amount: 0.001, currency: "USD", basis: "turn" }, + breakdown: { inputTokens: 30, outputTokens: 5, cachedReadTokens: 20, thoughtTokens: 0 }, + }); + created.onAcpMessage?.("inbound", { id: 1, error: { message: "late conflicting response" } }); + expect(await port.getStatus()).toEqual(after); + } finally { settle({ status: "completed" }); await port.close({ reason: "fixture done" }); } + }); + it("reads verified status from durable state without draining live updates", async () => { const runtime = fakeRuntime(); vi.mocked(runtime.getStatus!).mockReturnValue(new Promise(() => {})); diff --git a/packages/paperclip-runner/src/drivers/acpx/codex-runtime-adapter.ts b/packages/paperclip-runner/src/drivers/acpx/codex-runtime-adapter.ts index 580bf58453d..48d3eea115c 100644 --- a/packages/paperclip-runner/src/drivers/acpx/codex-runtime-adapter.ts +++ b/packages/paperclip-runner/src/drivers/acpx/codex-runtime-adapter.ts @@ -1,6 +1,7 @@ import { admitAcpxSessionContext } from "./session-context-admission.js"; import { createKimiInstructionAdmission } from "./kimi-instructions.js"; import { kimiReadOnlyPermissionAllowed } from "./kimi-config.js"; +import { createGrokPromptUsageCollector } from "./grok-usage.js"; import { resolveGrokDisableWebSearch } from "./grok-web-search.js"; import { resolveGrokEffort, createGrokEffortAdmission } from "./grok-effort.js"; import { resolveClaudeEffort, createClaudeEffortAdmission } from "./claude-effort.js"; @@ -295,6 +296,7 @@ export async function openQualifiedAcpxRuntime( const extensionProfile = ACPX_CAPABILITY_PROFILES[options.profile.agent]; const extensionRequests = new Set(extensionProfile.extensionRequests); const extensionNotifications = new Set(extensionProfile.extensionNotifications); + const promptUsage = options.profile.agent === "grok" ? createGrokPromptUsageCollector() : undefined; const extensionBoundary: AcpxRuntimeExtensionBoundary = { active: null, sessionIds: new Set(), steering: null, }; @@ -466,6 +468,10 @@ export async function openQualifiedAcpxRuntime( : null; }, onSessionNotification: acceptGoalNotification, + onAcpMessage: (direction, message) => { + const active = extensionBoundary.active; + if (active && (direction === "inbound" || !active.signal.aborted)) promptUsage?.observe(direction, message); + }, spawnEnvironment: () => ({ ...definedEnvironment(options.launchEnvironment), ...(options.profile.agent === "claude" @@ -685,6 +691,7 @@ export async function openQualifiedAcpxRuntime( extensionBoundary, options.profile.agent === "copilot", options.profile.agent === "copilot" ? assertCopilotPromptPolicy : undefined, + promptUsage, ); } catch (error) { const cleanupReason = "ACPX runtime identity validation failed"; @@ -1117,6 +1124,7 @@ function runtimePort( extensionBoundary: AcpxRuntimeExtensionBoundary, monitorProviderDeath: boolean, assertPromptPolicy?: (text: string) => void, + promptUsage?: ReturnType, ): AcpxRuntimePort { extensionBoundary.sessionIds = new Set([identity.backendSessionId]); let extensionControls: Promise = Promise.resolve(); @@ -1375,7 +1383,8 @@ function runtimePort( return structuredClone(identity); }, async getStatus() { - return await persistedRuntimeStatus(sessionStore, handle, identity); + const status = await persistedRuntimeStatus(sessionStore, handle, identity); + return promptUsage ? { ...status, terminalPromptUsage: promptUsage.snapshot() } : status; }, steeringCapability() { return extensionBoundary.steering === null ? null : structuredClone(extensionBoundary.steering); @@ -1488,6 +1497,7 @@ function runtimePort( onRequest: input.onExtensionRequest, onNotification: input.onExtensionNotification, }; extensionBoundary.active = extensionTurn; + promptUsage?.begin(input.requestId, identity.backendSessionId); // A guardian can outlive the provider while ACP awaits a callback. // Native exit or a typed ACP agent disconnect retires this turn; losing // the controller connection must preserve deliverable input instead. diff --git a/packages/paperclip-runner/src/drivers/acpx/grok-usage.test.ts b/packages/paperclip-runner/src/drivers/acpx/grok-usage.test.ts new file mode 100644 index 00000000000..5ce3acd199b --- /dev/null +++ b/packages/paperclip-runner/src/drivers/acpx/grok-usage.test.ts @@ -0,0 +1,94 @@ +import { describe, expect, it } from "vitest"; +import { createGrokPromptUsageCollector, grokPromptUsage } from "./grok-usage.js"; +import { persistedAcpxTurnUsage } from "./usage-accounting.js"; + +const usage = { inputTokens: 505795, outputTokens: 5763, totalTokens: 511558, cachedReadTokens: 475008, + cacheCreationTokens: 0, reasoningTokens: 531, modelCalls: 20, costUsdTicks: 3368420000 }; +const meta = { sessionId: "session-1", promptId: "prompt-1", requestId: "prompt-1", usage, + inputTokens: 999, outputTokens: 999, totalTokens: 999 }; +const request = { jsonrpc: "2.0", id: 7, method: "session/prompt", params: { sessionId: "session-1" } }; +const response = { jsonrpc: "2.0", id: 7, result: { stopReason: "end_turn", _meta: meta } }; + +describe("Grok terminal prompt usage", () => { + it("uses the complete prompt ledger with disjoint cache buckets and exact USD tick conversion", () => { + expect(grokPromptUsage({ ...meta, usage: { ...usage, cacheCreationTokens: 100 } }, "session-1")).toEqual({ + promptId: "prompt-1", event: { type: "status", tag: "usage_update", text: "terminal prompt usage", + cost: { amount: 0.336842, currency: "USD", basis: "turn" }, + breakdown: { inputTokens: 30687, outputTokens: 5763, totalTokens: 511558, cachedReadTokens: 475008, + cachedWriteTokens: 100, thoughtTokens: 0 } }, + }); + }); + + it.each([ + { usageIsIncomplete: true }, { usageIsIncomplete: "false" }, { cachedReadTokens: 505796 }, + { inputTokens: -1 }, { outputTokens: 1.5 }, { reasoningTokens: -1 }, { reasoningTokens: 1.5 }, { totalTokens: 1 }, + { modelCalls: undefined }, { inputTokens: Number.MAX_SAFE_INTEGER + 1 }, + ])("refuses incomplete or invalid token authority: %j", patch => { + expect(grokPromptUsage({ ...meta, usage: { ...usage, ...patch } }, "session-1")).toBeNull(); + }); + + it("settles a real Grok prompt whose reasoning counter exceeds its output counter", () => { + const collector = createGrokPromptUsageCollector(); + collector.begin("runner-1", "session-1"); + collector.observe("outbound", request); + collector.observe("inbound", { ...response, result: { _meta: { ...meta, usage: { + inputTokens: 517350, outputTokens: 7969, totalTokens: 525319, cachedReadTokens: 458624, + cacheCreationTokens: 0, reasoningTokens: 9859, modelCalls: 15, costUsdTicks: 4537320000, + } } } }); + const state = { lastRequestId: "runner-1", terminalPromptUsage: collector.snapshot() }; + expect(persistedAcpxTurnUsage({}, state, "runner-1")).toEqual({ + type: "status", tag: "usage_update", text: "terminal prompt usage", + cost: { amount: 0.453732, currency: "USD", basis: "turn" }, + breakdown: { inputTokens: 58726, outputTokens: 7969, totalTokens: 525319, + cachedReadTokens: 458624, cachedWriteTokens: 0, thoughtTokens: 0 }, + }); + }); + + it.each([{ costUsdTicks: undefined }, { costUsdTicks: -1 }, { costIsPartial: true }, { costIsPartial: "false" }])( + "preserves complete tokens without inventing cost: %j", patch => { + const result = grokPromptUsage({ ...meta, usage: { ...usage, ...patch } }, "session-1"); + expect(result?.event.breakdown).toMatchObject({ inputTokens: 30787, outputTokens: 5763 }); + expect(result?.event).not.toHaveProperty("cost"); + }); + + it("binds only the matching prompt response, rejects replay and leaves the next turn's price independent", () => { + const collector = createGrokPromptUsageCollector(); + collector.begin("runner-1", "session-1"); + collector.observe("inbound", response); + expect(collector.snapshot()).toBeNull(); + collector.observe("outbound", request); + collector.observe("inbound", { ...response, id: 6 }); + collector.observe("inbound", { ...response, method: "_x.ai/session/update" }); + expect(collector.snapshot()).toBeNull(); + collector.observe("inbound", response); + const first = collector.snapshot(); + collector.observe("inbound", response); + expect(collector.snapshot()).toEqual(first); + const before = { lastRequestId: "runner-1", terminalPromptUsage: first }; + expect(persistedAcpxTurnUsage({}, before, "runner-1")?.cost).toEqual({ amount: 0.336842, currency: "USD", basis: "turn" }); + expect(persistedAcpxTurnUsage(before, before, "runner-1")).toBeNull(); + expect(persistedAcpxTurnUsage({}, before, "foreign")).toBeNull(); + collector.begin("runner-2", "session-1"); + collector.observe("inbound", response); + expect(collector.snapshot()).toBeNull(); + collector.observe("outbound", { ...request, id: 8 }); + collector.observe("inbound", { ...response, id: 8, result: { _meta: { ...meta, promptId: "prompt-2", requestId: "prompt-2", + usage: { ...usage, costUsdTicks: 10000000 } } } }); + const after = { lastRequestId: "runner-2", terminalPromptUsage: collector.snapshot() }; + expect(persistedAcpxTurnUsage(before, after, "runner-2")?.cost).toEqual({ amount: 0.001, currency: "USD", basis: "turn" }); + }); + + it.each(["foreign-session", "conflicting-receipt", "second-prompt", "rpc-error", "reused-prompt"])("rejects %s", kind => { + const collector = createGrokPromptUsageCollector(); + collector.begin("runner-1", "session-1"); + collector.observe("outbound", request); + collector.observe("inbound", response); + if (kind === "reused-prompt") { collector.begin("runner-2", "session-1"); collector.observe("outbound", request); } + if (kind === "second-prompt") collector.observe("outbound", request); + else collector.observe("inbound", kind === "rpc-error" ? { ...response, error: {} } : { ...response, result: { _meta: { + ...meta, ...(kind === "foreign-session" ? { sessionId: "session-2" } : {}), + ...(kind === "conflicting-receipt" ? { usage: { ...usage, costUsdTicks: 1 } } : {}), + } } }); + expect(collector.snapshot()).toBeNull(); + }); +}); diff --git a/packages/paperclip-runner/src/drivers/acpx/grok-usage.ts b/packages/paperclip-runner/src/drivers/acpx/grok-usage.ts new file mode 100644 index 00000000000..12e337b8fce --- /dev/null +++ b/packages/paperclip-runner/src/drivers/acpx/grok-usage.ts @@ -0,0 +1,62 @@ +type RecordValue = Record; +const record = (value: unknown): RecordValue => value !== null && typeof value === "object" && !Array.isArray(value) ? value as RecordValue : {}; +const integer = (value: unknown): value is number => Number.isSafeInteger(value) && Number(value) >= 0; +const id = (value: unknown): value is string => typeof value === "string" && /^[A-Za-z0-9][A-Za-z0-9._:-]{0,239}$/.test(value); + +export interface TerminalPromptUsage { + requestId: string; + promptId: string; + event: RecordValue; +} + +export function grokPromptUsage(metaValue: unknown, sessionId: string): Omit | null { + const meta = record(metaValue), usage = record(meta.usage); + if (meta.sessionId !== sessionId || !id(meta.promptId) || meta.requestId !== meta.promptId + || (usage.usageIsIncomplete !== undefined && usage.usageIsIncomplete !== false)) return null; + const { inputTokens, outputTokens, totalTokens, cachedReadTokens, cacheCreationTokens, reasoningTokens, modelCalls } = usage; + if (![inputTokens, outputTokens, totalTokens, cachedReadTokens, cacheCreationTokens, reasoningTokens, modelCalls].every(integer) + || Number(inputTokens) + Number(outputTokens) !== totalTokens + || Number(cachedReadTokens) + Number(cacheCreationTokens) > Number(inputTokens)) return null; + const costKnown = integer(usage.costUsdTicks) && (usage.costIsPartial === undefined || usage.costIsPartial === false); + return { + promptId: meta.promptId, + event: { + type: "status", tag: "usage_update", text: "terminal prompt usage", + ...(costKnown ? { cost: { amount: Number(usage.costUsdTicks) / 10_000_000_000, currency: "USD", basis: "turn" } } : {}), + breakdown: { + inputTokens: Number(inputTokens) - Number(cachedReadTokens) - Number(cacheCreationTokens), + outputTokens, totalTokens, cachedReadTokens, cachedWriteTokens: cacheCreationTokens, thoughtTokens: 0, + }, + }, + }; +} + +export function createGrokPromptUsageCollector() { + let active: { requestId: string; sessionId: string; wireId?: string | number; invalid: boolean; receipt: TerminalPromptUsage | null } | null = null; + let previousPromptId: string | undefined; + return { + begin(requestId: string, sessionId: string) { + previousPromptId = active?.receipt?.promptId ?? previousPromptId; + active = { requestId, sessionId, invalid: false, receipt: null }; + }, + observe(direction: "inbound" | "outbound", value: unknown) { + if (!active || active.invalid) return; + const message = record(value); + if (direction === "outbound") { + if (message.method !== "session/prompt" || record(message.params).sessionId !== active.sessionId) return; + if (active.wireId !== undefined || !(id(message.id) || integer(message.id))) { active.invalid = true; return; } + active.wireId = message.id; + return; + } + if (active.wireId === undefined || message.id !== active.wireId || message.method !== undefined) return; + const receipt = message.error === undefined ? grokPromptUsage(record(message.result)._meta, active.sessionId) : null; + if (!receipt || receipt.promptId === previousPromptId) { active.invalid = true; return; } + const bound = { ...receipt, requestId: active.requestId }; + if (active.receipt && JSON.stringify(active.receipt) !== JSON.stringify(bound)) { active.invalid = true; return; } + active.receipt = bound; + }, + snapshot(): TerminalPromptUsage | null { + return active && !active.invalid ? structuredClone(active.receipt) : null; + }, + }; +} diff --git a/packages/paperclip-runner/src/drivers/acpx/usage-accounting.ts b/packages/paperclip-runner/src/drivers/acpx/usage-accounting.ts index 4143f655a68..6640ce97efa 100644 --- a/packages/paperclip-runner/src/drivers/acpx/usage-accounting.ts +++ b/packages/paperclip-runner/src/drivers/acpx/usage-accounting.ts @@ -43,6 +43,10 @@ export function persistedAcpxTurnUsage( ): Record | null { const current = record(after); if (current.lastRequestId !== requestId) return null; + const terminal = record(current.terminalPromptUsage); + const priorTerminal = record(record(before).terminalPromptUsage); + if (terminal.requestId === requestId && terminal.promptId !== priorTerminal.promptId + && priorTerminal.requestId !== requestId) return record(terminal.event); const previousReceipts = record(record(before).requestTokenUsage); const receipts = record(current.requestTokenUsage); const added = Object.keys(receipts).filter( diff --git a/packages/paperclip-runner/src/index.ts b/packages/paperclip-runner/src/index.ts index 8a1b75988e5..30e064085a0 100644 --- a/packages/paperclip-runner/src/index.ts +++ b/packages/paperclip-runner/src/index.ts @@ -46,6 +46,7 @@ export { type DurablePrpControlPlaneOptions, type DurablePrpCheckpoint, decodeDurablePrpCheckpoint, + captureLegacyDurablePrpCheckpoint, inspectDurablePrpCheckpointHeader, type PrpWireConnection, type PrpWireAttachment, diff --git a/packages/paperclip-runner/test/acpx-sidecar-contract.test.mjs b/packages/paperclip-runner/test/acpx-sidecar-contract.test.mjs index 2b262d59641..7de94511896 100644 --- a/packages/paperclip-runner/test/acpx-sidecar-contract.test.mjs +++ b/packages/paperclip-runner/test/acpx-sidecar-contract.test.mjs @@ -53,6 +53,18 @@ test("the ACPX sidecar schema accepts each versioned message family", () => { } }); +test("usage prices declare turn or session scope without changing legacy cumulative prices", () => { + const event = cost => ({ protocolVersion, sequence: 1, eventType: "runtime.event", runId: "run-1", turnId: "turn-1", + payload: { type: "status", tag: "usage_update", cost } }); + for (const cost of [null, { amount: 0.1, currency: "USD" }, { amount: 0.1, currency: "USD", basis: "turn" }, + { amount: 0.1, currency: "USD", basis: "session_cumulative" }]) { + assert.equal(validate(event(cost)), true, JSON.stringify(validate.errors)); + } + for (const cost of [{ amount: -1 }, { amount: 0.1, basis: "estimated" }, { amount: 0.1, rawProviderReceipt: {} }]) { + assert.equal(validate(event(cost)), false); + } +}); + test("failed responses accept the bounded provider cause and reject extra diagnostic fields", () => { const failure = { schema: "paperclip.provider-failure.v1", provider: "anthropic", phase: "startup", category: "rate_limit", message: "Account rate limit reached.", httpStatus: 429, retryability: "unknown" }; diff --git a/server/src/__tests__/ephemeral-server-contract.test.ts b/server/src/__tests__/ephemeral-server-contract.test.ts index bfa5e4acae8..e677036a7e2 100644 --- a/server/src/__tests__/ephemeral-server-contract.test.ts +++ b/server/src/__tests__/ephemeral-server-contract.test.ts @@ -1,6 +1,6 @@ import { afterEach, expect, it, vi } from "vitest"; import { assertEphemeralServerConfig } from "../services/ephemeral-server-contract.js"; -import { assertEphemeralWorkspaceExecution, assertEphemeralNativeLifecycle } from "../services/remote-workspaces.js"; +import { assertEphemeralWorkspaceExecution, assertEphemeralNativeRuntime } from "../services/remote-workspaces.js"; import { assertEphemeralAgentInstructions } from "../services/agent-instructions.js"; const config = { databaseUrl: "postgresql://runtime:fixture@ep-example-pooler.neon.test/tenant", storageProvider: "s3" as const }; @@ -24,20 +24,20 @@ it("never connects the listener with migration privileges, another tenant, or th expect(() => assertEphemeralServerConfig(config, { ...env, PAPERCLIP_DATABASE_LISTEN_URL: url })).toThrow("runtime role"); } }); -it("rejects disk-dependent execution and instructions while retaining remote warm execution", () => { +it("requires remote workspace authority and native execution independently of provider lifecycle", () => { vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "true"); const remote = { externalConversation: false, adapterType: "paperclip_runner", pluginTaskWorkspace: false, remoteAuthority: true }; expect(() => assertEphemeralWorkspaceExecution(remote)).not.toThrow(); expect(() => assertEphemeralWorkspaceExecution({ ...remote, remoteAuthority: false, pluginTaskWorkspace: true })).not.toThrow(); for (const patch of [{ remoteAuthority: false }, { adapterType: "process" }]) { - expect(() => assertEphemeralWorkspaceExecution({ ...remote, ...patch })).toThrow("warm Paperclip Runner"); + expect(() => assertEphemeralWorkspaceExecution({ ...remote, ...patch })).toThrow("native Paperclip Runner"); } - expect(() => assertEphemeralNativeLifecycle("warm")).not.toThrow(); - expect(() => assertEphemeralNativeLifecycle("per_turn")).toThrow("warm Paperclip Runner"); - expect(() => assertEphemeralNativeLifecycle(undefined)).toThrow("warm Paperclip Runner"); + expect(() => assertEphemeralNativeRuntime(true)).not.toThrow(); + expect(() => assertEphemeralNativeRuntime(false)).toThrow("native Paperclip Runner"); const agent = { id: "agent", companyId: "company", name: "Agent", adapterConfig: {} }; expect(() => assertEphemeralAgentInstructions(agent)).not.toThrow(); expect(() => assertEphemeralAgentInstructions({ ...agent, adapterConfig: { instructionsBundleMode: "external", instructionsRootPath: "/old-volume" } })).toThrow("managed agent instructions"); vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "false"); expect(() => assertEphemeralWorkspaceExecution({ ...remote, remoteAuthority: false })).not.toThrow(); + expect(() => assertEphemeralNativeRuntime(false)).not.toThrow(); }); diff --git a/server/src/adapters/registry.test.ts b/server/src/adapters/registry.test.ts index a1cea55ab57..4c9210bafa5 100644 --- a/server/src/adapters/registry.test.ts +++ b/server/src/adapters/registry.test.ts @@ -4,6 +4,7 @@ import { listServerAdapters, requireServerAdapter } from "./registry.js"; import * as executionTarget from "@paperclipai/adapter-utils/execution-target"; import { BUILTIN_ADAPTER_TYPES } from "./builtin-adapter-types.js"; import { QUALIFIED_ACPX_PROFILES, KIMI_DISTRIBUTION_CLOSURE_SHA256 } from "../vendor/paperclip-runner/index.js"; +import { assertHarnessTestPassed } from "../services/agent-harness-verification.js"; const { probeInstallation, probeGrokInstallation, probePiInstallation, probeKimiInstallation, probeKimiModels, probeKimiRemote, verifyApiKey } = vi.hoisted(() => ({ verifyApiKey: vi.fn(), @@ -222,6 +223,24 @@ describe("native ACPX environment checks", () => { expect(result.checks.map(check => check.code)).toEqual(["acpx_runtime_ready", "ai_connection_api_key_reverified"]); }); + it.each(["verified", "rejected", "unavailable", "missing", "managed"])("keeps remote Grok adoption tied to account verification: %s", async state => { + if (state === "rejected") verifyApiKey.mockRejectedValue({ details: { code: "ai_connection_api_key_rejected" } }); + if (state === "unavailable") verifyApiKey.mockRejectedValue(new Error("Provider unavailable")); + const result = await requireServerAdapter("paperclip_runner").testEnvironment!({ ...context, + config: { provider: "acpx", acpxAgent: "grok", model: "grok-4.7", + env: state === "missing" ? {} : { XAI_API_KEY: "fixture-xai-key" }, + ...(state === "managed" ? { managedAiConnection: { method: "api_key" } } : {}) }, + executionTarget: { kind: "remote", transport: "sandbox", remoteCwd: "/workspace", providerKey: "test", + runner: { execute: vi.fn().mockResolvedValue({ exitCode: 0, timedOut: false, stdout: "Linux\nx86_64\n" }) } }, + }); + if (["missing", "managed"].includes(state)) expect(verifyApiKey).not.toHaveBeenCalled(); + else expect(verifyApiKey).toHaveBeenCalledExactlyOnceWith("xai", "fixture-xai-key"); + expect(result.checks[0]).toMatchObject({ code: "acpx_remote_runtime_unverified", level: "warn" }); + if (state === "verified") expect(() => assertHarnessTestPassed(result)).not.toThrow(); + else expect(() => assertHarnessTestPassed(result)).toThrow("The harness test failed"); + expect(probeGrokInstallation).not.toHaveBeenCalled(); + }); + it.each(["ai_connection_api_key_rejected", "ai_connection_verification_failed"])("fails setup on %s", async code => { verifyApiKey.mockRejectedValue({ details: { code } }); const result = await requireServerAdapter("paperclip_runner").testEnvironment!({ ...context, diff --git a/server/src/adapters/registry.ts b/server/src/adapters/registry.ts index 7540a1e672a..a96f57763b1 100644 --- a/server/src/adapters/registry.ts +++ b/server/src/adapters/registry.ts @@ -150,29 +150,28 @@ import { resolvePaperclipRunnerProviderProfile, } from "../services/native-runtime/provider-profile.js"; -/** Reuse the existing provider key check without requiring a global Claude CLI. - * Readiness and account access are separate checks; model access is proven by a task. - */ -async function verifyNativeClaudeApiKey( +async function verifyNativeAcpxApiKey( config: Record, result: AdapterEnvironmentTestResult, ): Promise { const env = config.env as Record | undefined; - if (config.acpxAgent !== "claude" || !env || typeof env.ANTHROPIC_API_KEY !== "string" - || !env.ANTHROPIC_API_KEY || config.managedAiRouting || config.managedAiConnection - || ["ANTHROPIC_BASE_URL", "ANTHROPIC_AUTH_TOKEN", "CLAUDE_CODE_USE_BEDROCK", "CLAUDE_CODE_USE_VERTEX", "CLAUDE_CODE_USE_FOUNDRY"].some(key => env[key])) return result; + const account = config.acpxAgent === "claude" ? { provider: "anthropic" as const, name: "Anthropic", key: "ANTHROPIC_API_KEY", + routing: ["ANTHROPIC_BASE_URL", "ANTHROPIC_AUTH_TOKEN", "CLAUDE_CODE_USE_BEDROCK", "CLAUDE_CODE_USE_VERTEX", "CLAUDE_CODE_USE_FOUNDRY"] } + : config.acpxAgent === "grok" ? { provider: "xai" as const, name: "xAI", key: "XAI_API_KEY", routing: ["XAI_BASE_URL", "GROK_BASE_URL"] } : null; + if (!account || !env || typeof env[account.key] !== "string" || !env[account.key] + || config.managedAiRouting || config.managedAiConnection || account.routing.some(key => env[key])) return result; try { - await validateAiApiKey("anthropic", env.ANTHROPIC_API_KEY); + await validateAiApiKey(account.provider, env[account.key] as string); return { ...result, checks: [...result.checks, { code: "ai_connection_api_key_reverified", level: "info", - message: "Anthropic verified this API key. Model access is checked when the agent runs.", + message: `${account.name} verified this API key. Model access is checked when the agent runs.`, }] }; } catch (error) { const details = error && typeof error === "object" && "details" in error ? error.details as { code?: string } | undefined : undefined; return { ...result, status: "fail", checks: [...result.checks, { code: details?.code === "ai_connection_api_key_rejected" ? details.code : "ai_connection_verification_failed", - level: "error", message: "Could not verify the Anthropic API key. Check the connection and try again.", + level: "error", message: `Could not verify the ${account.name} API key. Check the connection and try again.`, }] }; } } @@ -527,7 +526,7 @@ const paperclipRunnerAdapter: ServerAdapterModule = { if (!((os === "Linux" && (arch === "x86_64" || (profile.acpxAgent === "claude" && ["aarch64", "arm64"].includes(arch)))) || (os === "Darwin" && (arch === "arm64" || (!["grok"].includes(profile.acpxAgent) && arch === "x86_64"))))) { throw new Error(`ACPX ${profile.acpxAgent} requires a qualified Linux ${profile.acpxAgent === "claude" ? "x64/ARM64" : "x64"} or macOS architecture.`); } - return verifyNativeClaudeApiKey(context.config, { + return verifyNativeAcpxApiKey(context.config, { adapterType: "paperclip_runner", status: "warn" as const, testedAt: new Date().toISOString(), checks: [{ code: "acpx_remote_runtime_unverified", level: "warn" as const, message: "The remote platform is supported. Runtime package integrity and readiness must still be verified by the remote runner before launch." }], @@ -535,7 +534,7 @@ const paperclipRunnerAdapter: ServerAdapterModule = { } const { probeAcpxClaudeInstallation, probeAcpxGrokInstallation, probeAcpxCursorInstallation, probeAcpxPiInstallation } = await import("../vendor/paperclip-runner/live/index.js"); await (profile.acpxAgent === "pi" ? probeAcpxPiInstallation : profile.acpxAgent === "grok" ? probeAcpxGrokInstallation : profile.acpxAgent === "cursor" ? probeAcpxCursorInstallation : probeAcpxClaudeInstallation)(profile.model); - return verifyNativeClaudeApiKey(context.config, { + return verifyNativeAcpxApiKey(context.config, { adapterType: "paperclip_runner", status: "pass" as const, testedAt: new Date().toISOString(), checks: [{ code: "acpx_runtime_ready", level: "info" as const, message: `ACPX ${profile.acpxAgent} runtime is installed and verified. Model access is checked when it runs.` }], }); diff --git a/server/src/maintenance/ephemeral-migration-data.ts b/server/src/maintenance/ephemeral-migration-data.ts index a815a51db65..44acf0a3635 100644 --- a/server/src/maintenance/ephemeral-migration-data.ts +++ b/server/src/maintenance/ephemeral-migration-data.ts @@ -9,6 +9,8 @@ import { skillPackageMaintenance } from "../services/skill-package-maintenance.j import { hasUnsettledLocalInstructionCopies } from "../services/agent-instruction-working-copies.js"; import { providerTraceStore } from "../services/provider-trace-store.js"; import { migrateWorkspaceTrees } from "../services/remote-workspace-migration-batch.js"; +import { migrateNativeHarnessBackups } from "../services/native-runtime/native-harness-backup-migration.js"; +import { migrateNativeControllers } from "../services/native-runtime/native-controller-migration.js"; /** Runs only in the maintenance process after Cloud has confirmed all legacy * deployments stopped. Each owner retains its own migration/publication rules; @@ -25,12 +27,22 @@ export async function migrateEphemeralServerData(db: Db, input: { const before = await readEphemeralServerDataEligibility(db); const unsupported = before.blockers.filter(blocker => ![ "agent_files_require_migration", "run_logs_require_migration", "provider_traces_require_migration", "skills_require_migration", "workspaces_require_migration", + "native_harness_backups_require_migration", + "native_controllers_require_migration", ].includes(blocker)); if (unsupported.length) throw new Error(unsupported[0]); - phase("workspaces"); const storage = getStorageService(); if (storage.provider !== "s3") throw new Error("shared_object_storage_required"); + phase("native_controllers"); + const controllers = await migrateNativeControllers(db, { writersStopped: true, signal, storage }); + counts.nativeControllers = controllers.imported; + counts.nativeControllersUnchanged = controllers.unchanged; + counts.nativeControllerSourcesAlreadyAbsent = controllers.sourceAlreadyAbsent; + counts.nativeControllersArchivedFailed = controllers.archivedFailed; + phase("native_harness_backups"); + counts.nativeHarnessBackups = await migrateNativeHarnessBackups(db, { storage, writersStopped: true, signal }); + phase("workspaces"); counts.workspaces = await migrateWorkspaceTrees(db, { storage, signal, phase }); phase("agent_files"); diff --git a/server/src/services/environment-runtime.ts b/server/src/services/environment-runtime.ts index bb3c56d268f..6f53cbe45e9 100644 --- a/server/src/services/environment-runtime.ts +++ b/server/src/services/environment-runtime.ts @@ -56,7 +56,7 @@ import { import { environmentService } from "./environments.js"; import { instanceSettingsService } from "./instance-settings.js"; import { hasSandboxSnapshotQualification, qualifiedSandboxSnapshotId, usesManagedCloudSandboxImage, type SandboxTemplateIdentity } from "./sandbox-snapshot-qualification.js"; -import { verifyNativeHarnessBackupStamp } from "./native-runtime/native-harness-backup-stamp.js"; +import { verifyRecoverableNativeHarnessBackup } from "./native-runtime/native-harness-backup-storage.js"; import { collectEnvironmentSecretRefs, parseEnvironmentDriverConfig, @@ -1847,10 +1847,8 @@ function createSandboxEnvironmentDriver( if (!providerLease) { if ( input.adapterType === "paperclip_runner" && - !verifyNativeHarnessBackupStamp( - reusableLease.metadata?.nativeHarnessBackup, - reusableLease.providerLeaseId, - ) + !await verifyRecoverableNativeHarnessBackup({ db, companyId: reusableLease.companyId, + stamp: reusableLease.metadata?.nativeHarnessBackup, providerLeaseId: reusableLease.providerLeaseId }) ) { throw new RunnerHarnessBackupUnavailableError( reusableLease.providerLeaseId, @@ -3986,10 +3984,8 @@ export function environmentRuntimeService( "paperclip_runner" && leaseSnapshot.metadata?.sandboxLeaseAcquisition && (!leaseSnapshot.providerLeaseId || - !verifyNativeHarnessBackupStamp( - leaseSnapshot.metadata.nativeHarnessBackup, - leaseSnapshot.providerLeaseId, - )) + !await verifyRecoverableNativeHarnessBackup({ db, companyId: leaseSnapshot.companyId, + stamp: leaseSnapshot.metadata.nativeHarnessBackup, providerLeaseId: leaseSnapshot.providerLeaseId })) ) { throw new RunnerHarnessBackupUnavailableError( leaseSnapshot.providerLeaseId ?? leaseSnapshot.id, diff --git a/server/src/services/ephemeral-server-contract.ts b/server/src/services/ephemeral-server-contract.ts index 396885b3f57..78cc9c54502 100644 --- a/server/src/services/ephemeral-server-contract.ts +++ b/server/src/services/ephemeral-server-contract.ts @@ -4,6 +4,8 @@ import type { Config } from "../config.js"; import { isValidLocalEncryptedMasterKey } from "../secrets/local-encrypted-provider.js"; import { agentInstructionsBundleMode } from "./agent-instructions.js"; import { skillInstallationMaintenance } from "./skill-installation-maintenance.js"; +import { hasLocalNativeHarnessObligations } from "./native-runtime/native-harness-backup-migration.js"; +import { hasLocalNativeControllerObligations } from "./native-runtime/native-controller-migration.js"; export const ephemeralServerEnabled = () => process.env.PAPERCLIP_EPHEMERAL_SERVER === "true"; @@ -64,6 +66,8 @@ export async function readEphemeralServerDataEligibility(db: Db, options: { purp isNull(providerTraceRecords.deletedAt), sql`${providerTraceRecords.traceRef} not like 's3-segments-v1:%'`, )).limit(1); if (localTrace) blockers.push("provider_traces_require_migration"); + if (await hasLocalNativeHarnessObligations(db)) blockers.push("native_harness_backups_require_migration"); + if (await hasLocalNativeControllerObligations(db)) blockers.push("native_controllers_require_migration"); const skills = skillInstallationMaintenance(db); let after: string | undefined; for (;;) { diff --git a/server/src/services/heartbeat/runtime-selection.test.ts b/server/src/services/heartbeat/runtime-selection.test.ts index 032a80e73c6..3c7c3bab0d5 100644 --- a/server/src/services/heartbeat/runtime-selection.test.ts +++ b/server/src/services/heartbeat/runtime-selection.test.ts @@ -432,14 +432,14 @@ describe.skipIf(!support.supported)("heartbeat runtime selection boundary", () = const { input, run } = await fixture("process"); vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "true"); expect(input.nativeRuntimeResolution.kind).toBe("legacy"); - await expect(selectHeartbeatRuntime(db, input)).rejects.toThrow("warm Paperclip Runner"); + await expect(selectHeartbeatRuntime(db, input)).rejects.toThrow("native Paperclip Runner"); expect((await read(run.id)).runtimeMode).toBe("legacy"); expect(await coordinators(run.id)).toHaveLength(0); expect(input.onNativeLifecycleSelected).not.toHaveBeenCalled(); expect(prepareNativeWorkspaceSync).not.toHaveBeenCalled(); }); - it.each(["warm", "per_turn"] as const)("checks resolved %s policy for ephemeral admission", async mode => { + it.each(["warm", "per_turn"] as const)("preserves resolved %s policy on an ephemeral server", async mode => { const { input, run } = await fixture(); vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "true"); // Deliberately disagree with the resolved provider policy in both cases. @@ -447,14 +447,41 @@ describe.skipIf(!support.supported)("heartbeat runtime selection boundary", () = input.workspace.executionTarget = { ...sandbox, reusableLeaseConfigured: true, effectiveCapabilities: { reusableLeases: true } as never, runnerLifecyclePolicy: mode === "warm" ? { mode, idleTimeoutMs: 300_000 } : { mode, idleTimeoutMs: null } }; - if (mode === "per_turn") { - await expect(selectHeartbeatRuntime(db, input)).rejects.toThrow("warm Paperclip Runner"); - expect(await coordinators(run.id)).toHaveLength(0); - } else { - const selected = await selectHeartbeatRuntime(db, input); - expect(selected.selected && selected.nativeExecution?.session.lifecyclePolicy.mode).toBe("warm"); - expect((await read(run.id)).runnerProfileJson).toMatchObject({ nativeExecutionInput: { session: { lifecyclePolicy: { mode: "warm" } } } }); - } + const selected = await selectHeartbeatRuntime(db, input); + expect(selected.selected && selected.nativeExecution?.session.lifecyclePolicy.mode).toBe(mode); + expect((await read(run.id)).runnerProfileJson).toMatchObject({ nativeExecutionInput: { session: { lifecyclePolicy: { mode } } } }); + expect(await coordinators(run.id)).toHaveLength(1); + }); + + it.each([ + { provider: "acpx", acpxAgent: "claude", model: "claude-opus-5-5", credentialProvider: "anthropic" as const, method: "subscription" as const }, + { provider: "acpx", acpxAgent: "grok", model: "grok-4.7", credentialProvider: "xai" as const, method: "subscription" as const }, + { provider: "opencode", model: "openrouter/deepseek/deepseek-v4-flash-0731", credentialProvider: "openrouter" as const, method: "api_key" as const }, + ])("admits managed $provider/$acpxAgent on an ephemeral server without changing credential lifecycle", async profile => { + vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "true"); + const { input, run, issue } = await fixture(); + const { credentialProvider, method, ...providerConfig } = profile; + const adapterConfig = { ...providerConfig, lifecycleMode: "warm" }; + input.agent = { ...input.agent, adapterConfig }; + await db.update(agents).set({ adapterConfig }).where(eq(agents.id, input.agent.id)); + input.workspace.executionTarget = { ...sandbox, reusableLeaseConfigured: true, + effectiveCapabilities: { reusableLeases: true } as never, runnerLifecyclePolicy: { mode: "warm", idleTimeoutMs: 300_000 } }; + input.nativeRuntimeResolution = resolveHeartbeatNativeRuntimeMode({ persisted: run, enabled: true, runtimeConfig: {}, adapterConfig, + agent: input.agent, issue, target: input.workspace.executionTarget, workspaceId: null }); + expect(input.nativeRuntimeResolution.kind).toBe("native"); + const attribution = { connectionId: randomUUID(), grantId: randomUUID(), provider: credentialProvider, method, + mode: "responsible_user" as const, responsibleUserId: "task-owner" }; + input.config.managedAiRuntime = { sessionIdentity: "account:epoch-1", identity: "account:generation-1", + config: { env: {}, managedAiConnection: { ...attribution, identity: "account:generation-1", sessionIdentity: "account:epoch-1" } }, + attribution, accountName: "Saved account", accountOwnerUserId: "task-owner", home: instructionRoot, cleanup: async () => {} }; + const selected = await selectHeartbeatRuntime(db, input); + if (!selected.selected || !selected.nativeExecution) throw new Error("Expected native selection"); + expect(selected.nativeExecution.provider.kind).toBe(profile.provider); + expect(selected.nativeExecution.session.lifecyclePolicy).toEqual({ mode: "per_turn", idleTimeoutMs: null }); + expect((await read(run.id)).runnerProfileJson).toMatchObject({ + nativeExecutionInput: { session: { lifecyclePolicy: { mode: "per_turn" } } }, + }); + expect(await coordinators(run.id)).toHaveLength(1); }); it.each([ diff --git a/server/src/services/heartbeat/runtime-selection.ts b/server/src/services/heartbeat/runtime-selection.ts index b73ca32b3aa..d9613c3dc9b 100644 --- a/server/src/services/heartbeat/runtime-selection.ts +++ b/server/src/services/heartbeat/runtime-selection.ts @@ -1,7 +1,7 @@ import { resolveProviderApiPlacement } from "../native-runtime/provider-api-placement.js"; import { readClaimedNativeStopRecovery } from "../native-runtime/native-cancellation-request.js"; import { museRunnerBroker } from "../muse-runner-broker.js"; -import { assertEphemeralNativeLifecycle, readRemoteWorkspaceAuthority } from "../remote-workspaces.js"; +import { assertEphemeralNativeRuntime, readRemoteWorkspaceAuthority } from "../remote-workspaces.js"; import { measureRunTransaction } from "../execution-transaction-timing.js"; import { measureRunPhase } from "../execution-preparation-timing.js"; import { @@ -269,7 +269,7 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele const { runtimeConfig, runtimeSkillEntries, runScopedMentionedSkillKeys, instructionCopy, agentIdentity, issueAssigneeOverrides, resolvedConfig, managedAiRuntime } = input.config; const runOptions = input.recovery; const { providerTraceRequested, providerTraceCapture } = input.trace; - if (nativeRuntimeResolution.kind !== "native" && !isDotRun) assertEphemeralNativeLifecycle(undefined); + if (!isDotRun) assertEphemeralNativeRuntime(nativeRuntimeResolution.kind === "native"); let nativeExecution: NativeExecutionInput | null = null; let getNativeFreshSessionHandoff: (() => Promise) | undefined; let nativeRunnerInstanceId: string | null = null; @@ -522,9 +522,6 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele (["openai_dot_mcp", "grok_bot_mcp", "muse_external", "cursor_cloud_api"].includes(nativeRuntimeResolution.profile.backend) || managedAiRuntime && !supportsManagedWarmSession ? { mode: "per_turn" as const, idleTimeoutMs: null } : environmentLifecyclePolicy ?? agentLifecyclePolicy); - if (!["openai_dot_mcp", "grok_bot_mcp", "muse_external", "cursor_cloud_api"].includes(nativeRuntimeResolution.profile.backend)) { - assertEphemeralNativeLifecycle(effectiveLifecyclePolicy.mode); - } if ( effectiveLifecyclePolicy.mode === "warm" && executionTarget?.kind === "remote" && diff --git a/server/src/services/heartbeat/upgrade-readiness.test.ts b/server/src/services/heartbeat/upgrade-readiness.test.ts index 0dfb106e3c7..fb605c53be1 100644 --- a/server/src/services/heartbeat/upgrade-readiness.test.ts +++ b/server/src/services/heartbeat/upgrade-readiness.test.ts @@ -88,13 +88,14 @@ postgresTests("upgrade readiness persisted execution evidence", () => { finally { await core.stop(); await rm(root, { recursive: true, force: true }); } expect(await readUpgradeReadiness(db, input)).toMatchObject({ ready: true, recoverableRuns: 1 }); const cipher = createLocalEncryptedCipher("write"); - // Real staging headers contain retained launch metadata around 113 KiB. - const representativeHeader = JSON.stringify({ ...JSON.parse(header), padding: "x".repeat(100_000) }); + const parsedHeader = JSON.parse(header); + const representativeHeader = JSON.stringify({ ...parsedHeader, state: { ...parsedHeader.state, + runAttachTemplate: { provider: { instructions: "x".repeat(300_000) } } } }); await db.update(nativeRunnerCheckpoints).set({ material: cipher.encrypt(JSON.stringify({ ...binding, part: "header", contents: representativeHeader })) }) .where(eq(nativeRunnerCheckpoints.runId, runId)); expect((await readUpgradeReadiness(db, input)).ready).toBe(true); for (const contents of ["{", "{}", JSON.stringify({ ...JSON.parse(header), schema: "paperclip.runner.controller-checkpoint.v1" }), - JSON.stringify({ ...JSON.parse(header), padding: "x".repeat(400_000) }), + JSON.stringify({ ...JSON.parse(header), padding: "x".repeat(1_100_000) }), JSON.stringify({ ...JSON.parse(header), state: { ...JSON.parse(header).state, identity: { ...JSON.parse(header).state.identity, runId: randomUUID() } } })]) { await db.update(nativeRunnerCheckpoints).set({ material: cipher.encrypt(JSON.stringify({ ...binding, part: "header", contents })) }) .where(eq(nativeRunnerCheckpoints.runId, runId)); diff --git a/server/src/services/heartbeat/upgrade-readiness.ts b/server/src/services/heartbeat/upgrade-readiness.ts index 1ef14aed3b9..99da469b2ec 100644 --- a/server/src/services/heartbeat/upgrade-readiness.ts +++ b/server/src/services/heartbeat/upgrade-readiness.ts @@ -1,7 +1,8 @@ import type { Db } from "@paperclipai/db"; import { sql } from "drizzle-orm"; import { remoteWorkspaceRunReferenceSchema } from "@paperclipai/shared"; -import { inspectNativeRunnerUpgradeHeader } from "../native-runtime/native-runner-checkpoint.js"; +import { inspectNativeRunnerUpgradeHeader, NATIVE_CHECKPOINT_HEADER_INSPECTION_MAX_BYTES, + NATIVE_CHECKPOINT_INSPECTION_TOTAL_MAX_BYTES } from "../native-runtime/native-runner-checkpoint.js"; import { readNativeWorkspaceSyncReference } from "../native-runtime/native-workspace-sync.js"; import { REMOTE_RUNNER_RECONNECT_GRACE_MS } from "../native-runtime/native-runner-recovery-contract.js"; import { currentNativeControllerIdentity } from "../native-runtime/native-restart-recovery.js"; @@ -30,7 +31,8 @@ export async function readUpgradeReadiness(db: Db, input: { r.runner_profile_json->'nativeWorkspaceSync' as workspace, r.runner_profile_json->'remoteWorkspace' as "remoteWorkspace", r.runner_profile_json ? 'remoteWorkspace' as "hasRemoteWorkspace", p.id as "pluginLeaseId", - case when pg_column_size(c.material) <= 262144 and sum(pg_column_size(c.material)) over () <= 8388608 then c.material else null end as material, ( + case when pg_column_size(c.material) <= ${NATIVE_CHECKPOINT_HEADER_INSPECTION_MAX_BYTES} + and sum(pg_column_size(c.material)) over () <= ${NATIVE_CHECKPOINT_INSPECTION_TOTAL_MAX_BYTES} then c.material else null end as material, ( r.runtime_mode = 'native' and a.adapter_type = 'paperclip_runner' and r.runner_instance_id is not null and r.native_session_id is not null and r.process_pid is not null and r.process_started_at is not null diff --git a/server/src/services/heartbeat/workspace-preparation.test.ts b/server/src/services/heartbeat/workspace-preparation.test.ts index 3a218e50b8e..3134fb688d1 100644 --- a/server/src/services/heartbeat/workspace-preparation.test.ts +++ b/server/src/services/heartbeat/workspace-preparation.test.ts @@ -444,7 +444,7 @@ describe.skipIf(!support.supported && !externalDatabaseUrl)("heartbeat workspace const resolve = vi.spyOn(input.services, "resolveWorkspaceForRun").mockRejectedValue(new Error("local folder retained")); process.env.PAPERCLIP_EPHEMERAL_SERVER = String(ephemeral); await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow(ephemeral - ? "Volume-free servers require warm Paperclip Runner execution with remote workspace storage." : "local folder retained"); + ? "Volume-free servers require native Paperclip Runner execution with remote workspace storage." : "local folder retained"); if (ephemeral) expect(resolve).not.toHaveBeenCalled(); else expect(resolve).toHaveBeenCalledOnce(); expect(await db.select().from(executionWorkspaces)).toHaveLength(0); diff --git a/server/src/services/native-runtime/native-controller-archive.ts b/server/src/services/native-runtime/native-controller-archive.ts new file mode 100644 index 00000000000..d0602ce24f1 --- /dev/null +++ b/server/src/services/native-runtime/native-controller-archive.ts @@ -0,0 +1,40 @@ +import { createHash } from "node:crypto"; +import { mkdtemp, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import type { Db } from "@paperclipai/db"; +import type { StorageService } from "../../storage/types.js"; +import { removeNativeHarnessBackup } from "./native-harness-backup-cleanup.js"; +import { persistNativeHarnessBackup, verifyDurableNativeHarnessBackup } from "./native-harness-backup-storage.js"; + +type Source = { companyId: string; issueId: string; runId: string; controller: string; supervisor?: string }; +const sha256 = (value: string) => createHash("sha256").update(value).digest("hex"); + +export function legacyNativeControllerArchive(source: Source) { + const manifest = JSON.stringify({ schema: "paperclip.legacy-controller-archive.v1", companyId: source.companyId, + issueId: source.issueId, runId: source.runId, reason: "original_runner_artifact_unproven", + files: { "control-plane-state.json": sha256(source.controller), + ...(source.supervisor === undefined ? {} : { "supervisor.json": sha256(source.supervisor) }) } }); + return { manifest, scopeDigest: sha256(manifest), manifestSha256: `sha256:${sha256(manifest)}` }; +} + +export async function hasArchivedLegacyNativeController(db: Db, input: Source & { storage: StorageService; signal?: AbortSignal }) { + return verifyDurableNativeHarnessBackup({ db, storage: input.storage, companyId: input.companyId, + ...legacyNativeControllerArchive(input), signal: input.signal }); +} + +export async function archiveFailedLegacyNativeController(db: Db, input: Source & { + storage: StorageService; writersStopped: true; signal: AbortSignal; +}) { + if (await hasArchivedLegacyNativeController(db, input)) return; + const descriptor = legacyNativeControllerArchive(input); + const root = await mkdtemp(join(tmpdir(), "pc-legacy-controller-archive-")); + try { + await writeFile(join(root, "manifest.json"), descriptor.manifest, { mode: 0o600 }); + await writeFile(join(root, "control-plane-state.json"), input.controller, { mode: 0o600 }); + if (input.supervisor !== undefined) await writeFile(join(root, "supervisor.json"), input.supervisor, { mode: 0o600 }); + await persistNativeHarnessBackup({ db, storage: input.storage, root, scopeDigest: descriptor.scopeDigest, signal: input.signal, + owner: { kind: "offline_migration", writersStopped: input.writersStopped, terminalFailureOnly: true, + companyId: input.companyId, issueId: input.issueId, runId: input.runId } }); + } finally { await removeNativeHarnessBackup(root); } +} diff --git a/server/src/services/native-runtime/native-controller-migration.integration.test.ts b/server/src/services/native-runtime/native-controller-migration.integration.test.ts new file mode 100644 index 00000000000..ea5d334e0ec --- /dev/null +++ b/server/src/services/native-runtime/native-controller-migration.integration.test.ts @@ -0,0 +1,156 @@ +import { createHash, randomUUID } from "node:crypto"; +import { mkdir, readFile, rm, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { eq } from "drizzle-orm"; +import { afterEach, expect, it, vi } from "vitest"; +import { agentTaskSessions, heartbeatRuns, issues, nativeHarnessBackups, nativeRunFinalizations, nativeRunnerCheckpoints } from "@paperclipai/db"; +import { DurablePrpControlPlane, parseNativeExecutionInput } from "../../vendor/paperclip-runner/index.js"; +import { describeEmbeddedPostgres, nativeRestartFixture } from "../../__tests__/helpers/native-restart.js"; +import { nativeToolContractFingerprintForTarget } from "./native-session-resume.js"; +import { nativeControllerMigrationDescriptor } from "./native-session-executor.js"; +import { hasLocalNativeControllerObligations, migrateNativeControllers } from "./native-controller-migration.js"; +import { resolveNativeControllerRecovery } from "./native-controller-recovery.js"; +import { resolveRemoteProviderPackAuthority } from "./remote-provider-pack-authority.js"; +import { canonicalProviderPackJson } from "./remote-provider-pack-verification.js"; +import { nativeSha256 } from "./canonical.js"; +import { createLocalDiskStorageProvider } from "../../storage/local-disk-provider.js"; +import { createStorageService } from "../../storage/service.js"; +import { legacyNativeControllerArchive } from "./native-controller-archive.js"; +import { restoreDurableNativeHarnessBackup } from "./native-harness-backup-storage.js"; + +describeEmbeddedPostgres.each(["import", "archive", "success"])("offline native controller migration (%s)", scenario => { + const environment = nativeRestartFixture(); + afterEach(() => vi.unstubAllEnvs()); + async function fixture() { + vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "73".repeat(32)); + const f = await environment.seedRun(randomUUID()); + const { companyId, agentId, paperclipHome, runtimeRoot } = environment.current; + const binding = { companyId, agentId, issueId: f.issueId, runId: f.runId, executionWorkspaceId: f.native.environmentLeaseId }; + const execution = parseNativeExecutionInput({ + schema: "paperclip.native-execution-input.v1", provider: { kind: "codex", model: null }, binding, + task: { identifier: "MIG-1", title: "Retained work", description: null, prompt: "Continue the work.", workMode: "standard" }, + workspace: { cwd: "/workspace", repoUrl: null, repoRef: null, branchName: null }, + session: { normalizedSessionId: f.native.normalizedSessionId, driverKind: "codex_app_server", protocolVersion: 1, + lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } }, + completionContract: { id: randomUUID(), sha256: "fixture", schemaVersion: "paperclip.completion-contract.v1", + contract: { revision: "1", objective: "Continue", criteria: [{ id: "objective", requirement: "Save work." }] } }, + credentialBindings: [], interactionResponses: [], + }); + const fingerprint = nativeToolContractFingerprintForTarget("remote", "codex"); + const paths = nativeControllerMigrationDescriptor(execution, fingerprint); + const identity = { runId: f.runId, runnerInstanceId: f.native.runnerInstanceId, normalizedSessionId: f.native.normalizedSessionId, + environmentLeaseId: f.native.environmentLeaseId, turnId: f.native.turnId, itemId: f.native.itemId }; + const core = new DurablePrpControlPlane({ stateDirectory: join(paths.root, "control-plane"), identity, + expectedRunnerVersion: "migration-fixture", expectedRunnerDigest: `sha256:${"a".repeat(64)}` }); + core.issueBootstrapTicket(); core.queueCommand("session.snapshot", {}, "retained-command"); await core.stop(); + const file = join(paths.root, "control-plane/control-plane-state.json"); + const state = JSON.parse(await readFile(file, "utf8")); + state.tickets = {}; + const source = JSON.stringify(state); await writeFile(file, source, { mode: 0o600 }); + const packBinding = { companyId, agentId, workspaceId: binding.executionWorkspaceId, + sessionId: f.native.normalizedSessionId, runnerInstanceId: f.native.runnerInstanceId, + environmentLeaseId: f.native.environmentLeaseId, remoteCwd: "/remote/workspace", providerDigest: nativeSha256(execution.provider) }; + const payload = { + runnerSourceRevision: "a".repeat(40), target: { platform: "linux", architecture: "x64" }, + pins: { opencode: "1.18.34" }, artifacts: { + nodeCommand: { path: "node", sha256: `sha256:${"1".repeat(64)}` }, + opencodeCommand: { path: "opencode", sha256: `sha256:${"2".repeat(64)}` }, + }, + }; + const pack = resolveRemoteProviderPackAuthority({ stateDirectory: paths.root, reattach: false, binding: packBinding, + current: () => ({ schema: "paperclip-runner/remote-provider-pack/v1", payload, + digest: `sha256:${createHash("sha256").update(canonicalProviderPackJson(payload)).digest("hex")}` }), + runnerArtifact: () => ({ version: "migration-fixture", digest: `sha256:${"a".repeat(64)}` }) }); + pack.verified(); + const snapshot = { backendKind: "mock", sessionId: f.native.normalizedSessionId, + identity: { companyId, agentId, issueId: f.issueId, runId: f.runId, sessionId: f.native.normalizedSessionId } }; + const profile = { ...f.preparedProfile, nativeExecutionInput: execution, nativeToolContractFingerprint: fingerprint, sessionCheckpoint: snapshot, + nativeWorkspaceSync: { schema: "paperclip.native-workspace-sync/v1", state: "prepared", descriptorSha256: "a".repeat(64), + baselineSha256: "b".repeat(64), finalHostSha256: null, workspaceId: randomUUID(), leaseId: f.native.environmentLeaseId, + providerLeaseId: randomUUID(), remoteCwd: "/remote/workspace", resourceDisposition: "destroy" } }; + await f.db.update(heartbeatRuns).set({ status: "failed", finishedAt: new Date(), runnerProfileJson: profile }).where(eq(heartbeatRuns.id, f.runId)); + await f.db.update(nativeRunFinalizations).set({ phase: "terminal_failure", leaseOwner: null }).where(eq(nativeRunFinalizations.runId, f.runId)); + const queuedId = randomUUID(); + await f.db.insert(heartbeatRuns).values({ id: queuedId, companyId, agentId, issueId: f.issueId, status: "queued" }); + await f.db.update(issues).set({ executionRunId: queuedId }).where(eq(issues.id, f.issueId)); + const [session] = await f.db.insert(agentTaskSessions).values({ companyId, agentId, adapterType: "paperclip_runner", + taskKey: f.issueId, lastRunId: f.runId }).returning(); + return { ...f, companyId, agentId, paperclipHome, runtimeRoot, paths, file, source, profile, session, queuedId, identity, snapshot }; + } + if (scenario === "import") it("imports an exact settled authority once, preserves queued work, and resumes from an empty home", async () => { + const f = await fixture(); + const { companyId, agentId, paperclipHome, runtimeRoot, profile, file, source, session, queuedId, identity, snapshot } = f; + expect(await hasLocalNativeControllerObligations(f.db)).toBe(true); + await f.db.update(heartbeatRuns).set({ runnerProfileJson: { ...profile, + nativeWorkspaceSync: { ...profile.nativeWorkspaceSync, leaseId: randomUUID() } } }).where(eq(heartbeatRuns.id, f.runId)); + await expect(migrateNativeControllers(f.db, { writersStopped: true, signal: AbortSignal.timeout(30_000) })) + .rejects.toThrow("native_controller_migration_artifact_binding"); + await f.db.update(heartbeatRuns).set({ runnerProfileJson: profile }).where(eq(heartbeatRuns.id, f.runId)); + await f.db.update(nativeRunFinalizations).set({ leaseOwner: "unsettled-owner" }).where(eq(nativeRunFinalizations.runId, f.runId)); + await expect(migrateNativeControllers(f.db, { writersStopped: true, signal: AbortSignal.timeout(30_000) })).rejects.toThrow("normal_ownership_recovery_required"); + expect(await f.db.select().from(nativeRunnerCheckpoints).where(eq(nativeRunnerCheckpoints.runId, f.runId))).toHaveLength(0); + await f.db.update(nativeRunFinalizations).set({ leaseOwner: "expired-owner", leaseExpiresAt: new Date(0), + phase: "retryable_failure", recoveryState: "resuming_session" }).where(eq(nativeRunFinalizations.runId, f.runId)); + const [preservedOwner] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)); + expect(await migrateNativeControllers(f.db, { writersStopped: true, signal: AbortSignal.timeout(30_000) })) + .toMatchObject({ imported: 1, sourceAlreadyAbsent: 0 }); + expect(await migrateNativeControllers(f.db, { writersStopped: true, signal: AbortSignal.timeout(30_000) })).toMatchObject({ unchanged: 1, imported: 0 }); + expect(await hasLocalNativeControllerObligations(f.db)).toBe(false); + expect((await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0]).toEqual(preservedOwner); + expect(await readFile(file, "utf8")).toBe(source); + expect((await f.db.select().from(agentTaskSessions).where(eq(agentTaskSessions.id, session.id)))[0].lastRunId).toBe(f.runId); + expect((await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, queuedId)))[0].status).toBe("queued"); + await rm(paperclipHome, { recursive: true, force: true }); + await f.db.update(heartbeatRuns).set({ nativeIssueId: f.issueId, nativeSessionId: f.native.normalizedSessionId, + status: "running", runtimeMode: "native" }).where(eq(heartbeatRuns.id, queuedId)); + const recovered = await resolveNativeControllerRecovery(f.db, { companyId, agentId, issueId: f.issueId, + normalizedSessionId: f.native.normalizedSessionId, currentRunId: queuedId }); + expect(recovered.kind).toBe("recoverable"); + if (recovered.kind !== "recoverable") throw new Error(JSON.stringify(recovered)); + const replacement = join(runtimeRoot, "replacement"); await mkdir(replacement, { mode: 0o700 }); + recovered.checkpoint.restoreSessionDirectory(replacement); + expect(JSON.parse(await readFile(join(replacement, "control-plane/control-plane-state.json"), "utf8")).identity).toEqual(identity); + expect(recovered.snapshot).toEqual(snapshot); + }); + if (scenario === "archive") it("preserves an already-failed legacy controller without inventing executable recovery authority", async () => { + const f = await fixture(); + await rm(join(f.paths.root, "provider-pack-authority.json")); + await f.db.update(nativeRunFinalizations).set({ recoveryState: "blocked", failureCode: "native_provider_terminal_failed" }) + .where(eq(nativeRunFinalizations.runId, f.runId)); + const [originalRun] = await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)); + const [originalOwner] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)); + const storage = createStorageService(createLocalDiskStorageProvider(join(f.runtimeRoot, "durable-objects"))); + const options = { storage, writersStopped: true as const, signal: AbortSignal.timeout(30_000) }; + const put = vi.spyOn(storage, "putFile").mockRejectedValue(new Error("storage unavailable")); + await expect(migrateNativeControllers(f.db, options)).rejects.toThrow("storage unavailable"); + put.mockRestore(); + expect(await hasLocalNativeControllerObligations(f.db, storage)).toBe(true); + expect(await migrateNativeControllers(f.db, options)).toMatchObject({ archivedFailed: 1, imported: 0 }); + expect(await migrateNativeControllers(f.db, options)).toMatchObject({ archivedFailed: 1, imported: 0 }); + expect(await hasLocalNativeControllerObligations(f.db, storage)).toBe(false); + expect(await f.db.select().from(nativeRunnerCheckpoints).where(eq(nativeRunnerCheckpoints.runId, f.runId))).toHaveLength(0); + expect((await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0]).toEqual(originalRun); + expect((await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0]).toEqual(originalOwner); + expect((await f.db.select().from(issues).where(eq(issues.id, f.issueId)))[0].executionRunId).toBe(f.queuedId); + expect((await f.db.select().from(agentTaskSessions).where(eq(agentTaskSessions.id, f.session.id)))[0].lastRunId).toBe(f.runId); + await rm(f.paperclipHome, { recursive: true, force: true }); + const descriptor = legacyNativeControllerArchive({ companyId: f.companyId, issueId: f.issueId, runId: f.runId, controller: f.source }); + const replacement = join(f.runtimeRoot, "archived-inspection"); + expect(await restoreDurableNativeHarnessBackup({ db: f.db, storage, companyId: f.companyId, + scopeDigest: descriptor.scopeDigest, backupRoot: replacement })).toBe(true); + expect(await readFile(join(replacement, "current", "control-plane-state.json"), "utf8")).toBe(f.source); + const recovery = await resolveNativeControllerRecovery(f.db, { companyId: f.companyId, agentId: f.agentId, + issueId: f.issueId, normalizedSessionId: f.native.normalizedSessionId, currentRunId: f.queuedId }); + expect(recovery.kind).not.toBe("recoverable"); + }); + if (scenario === "success") it("does not substitute a failure archive for a successful retained session", async () => { + const f = await fixture(); + await rm(join(f.paths.root, "provider-pack-authority.json")); + await f.db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, f.runId)); + const storage = createStorageService(createLocalDiskStorageProvider(join(f.runtimeRoot, "durable-objects"))); + await expect(migrateNativeControllers(f.db, { storage, writersStopped: true, signal: AbortSignal.timeout(30_000) })) + .rejects.toThrow("native_runner_checkpoint_artifact_unproven"); + expect(await f.db.select().from(nativeHarnessBackups)).toHaveLength(0); + expect(await hasLocalNativeControllerObligations(f.db, storage)).toBe(true); + }); +}); diff --git a/server/src/services/native-runtime/native-controller-migration.ts b/server/src/services/native-runtime/native-controller-migration.ts new file mode 100644 index 00000000000..e760f7071dd --- /dev/null +++ b/server/src/services/native-runtime/native-controller-migration.ts @@ -0,0 +1,163 @@ +import { constants } from "node:fs"; +import { lstat, open, readdir } from "node:fs/promises"; +import { join } from "node:path"; +import { and, eq, sql } from "drizzle-orm"; +import { heartbeatRuns, type Db } from "@paperclipai/db"; +import { parseNativeExecutionInput } from "../../vendor/paperclip-runner/index.js"; +import { offlineMigrationExecutionWhere } from "../offline-migration-execution.js"; +import { findNativeSessionResumeRun } from "./native-session-resume.js"; +import { nativeControllerMigrationDescriptor } from "./native-session-executor.js"; +import { hasNativeRunnerCheckpoint, loadNativeRunnerCheckpoint } from "./native-runner-checkpoint.js"; +import { importNativeControllerRecovery, readNativeControllerRecovery } from "./native-controller-recovery.js"; +import { nativeHarnessStateBase } from "./native-harness-backup-stamp.js"; +import { providerPackWorkspaceId, resolveRemoteProviderPackAuthority } from "./remote-provider-pack-authority.js"; +import { nativeSha256 } from "./canonical.js"; +import { readNativeWorkspaceSyncReference } from "./native-workspace-sync.js"; +import type { StorageService } from "../../storage/types.js"; +import { durableNativeHarnessStorage } from "./native-harness-backup-storage.js"; +import { archiveFailedLegacyNativeController, hasArchivedLegacyNativeController } from "./native-controller-archive.js"; + +type Selected = { id: string; companyId: string; agentId: string; issueId: string; sessionId: string; createdAt: Date }; + +async function* retainedSources(db: Db, signal?: AbortSignal) { + let after: string | undefined; + for (;;) { + signal?.throwIfAborted(); + const rows = await db.execute(sql`select s.id, s.company_id as "companyId", s.agent_id as "agentId", + r.native_issue_id as "issueId", r.native_session_id as "sessionId", r.created_at as "createdAt" + from agent_task_sessions s join heartbeat_runs r on r.id = s.last_run_id + and r.company_id = s.company_id and r.agent_id = s.agent_id + where s.adapter_type = 'paperclip_runner' and r.native_issue_id is not null and r.native_session_id is not null + ${after ? sql`and s.id > ${after}::uuid` : sql``} order by s.id limit 100`); + for (const selected of rows) { + signal?.throwIfAborted(); + const source = await findNativeSessionResumeRun(db, { companyId: selected.companyId, agentId: selected.agentId, + issueId: selected.issueId, normalizedSessionId: selected.sessionId, + beforeCreatedAt: new Date(selected.createdAt), includeProgressBarrier: true }); + if (source) yield { selected, source }; + } + if (rows.length < 100) return; + after = rows.at(-1)!.id; + } +} + +async function readPrivateSource(file: string, maxBytes: number): Promise { + const handle = await open(file, constants.O_RDONLY | constants.O_NOFOLLOW).catch((error: NodeJS.ErrnoException) => { + if (error.code === "ENOENT") return undefined; throw error; + }); + if (!handle) return undefined; + try { + const stat = await handle.stat(); + if (!stat.isFile() || stat.nlink !== 1 || stat.size > maxBytes) throw new Error("native_controller_migration_unsafe_source"); + return await handle.readFile("utf8"); + } finally { await handle.close(); } +} + +async function localPaths(source: NonNullable>>) { + const execution = parseNativeExecutionInput(source.runnerProfileJson.nativeExecutionInput); + const descriptor = nativeControllerMigrationDescriptor(execution, source.runnerProfileJson.nativeToolContractFingerprint); + const root = await lstat(descriptor.root).catch((error: NodeJS.ErrnoException) => { + if (error.code === "ENOENT") return undefined; throw error; + }); + if (root && (!root.isDirectory() || root.isSymbolicLink())) throw new Error("native_controller_migration_unsafe_source"); + const control = join(descriptor.root, "control-plane"); + const directory = await lstat(control).catch((error: NodeJS.ErrnoException) => { + if (error.code === "ENOENT") return undefined; throw error; + }); + if (directory && (!directory.isDirectory() || directory.isSymbolicLink())) throw new Error("native_controller_migration_unsafe_source"); + return { ...descriptor, execution, controlFile: join(control, "control-plane-state.json") }; +} + +export async function hasLocalNativeControllerObligations(db: Db, storage?: StorageService | null): Promise { + const roots = await readdir(nativeHarnessStateBase()).catch((error: NodeJS.ErrnoException) => { + if (error.code === "ENOENT") return [] as string[]; throw error; + }); + if (!roots.length) return false; + for await (const { selected, source } of retainedSources(db)) { + const binding = { companyId: selected.companyId, issueId: selected.issueId, runId: source.id }; + if (readNativeControllerRecovery(source.runnerProfileJson.nativeControllerRecovery, source.runnerProfileJson.sessionCheckpoint) + && await hasNativeRunnerCheckpoint(db, binding)) continue; + const paths = await localPaths(source); + const present = await lstat(paths.controlFile).catch((error: NodeJS.ErrnoException) => { + if (error.code === "ENOENT") return undefined; throw error; + }); + if (present && source.status === "failed") { + const archiveStorage = storage === undefined ? durableNativeHarnessStorage() : storage; + const controller = await readPrivateSource(paths.controlFile, 256 * 1024 * 1024); + const supervisor = paths.execution.session.lifecyclePolicy.mode === "warm" + ? await readPrivateSource(paths.supervisorPath, 4 * 1024 * 1024) : undefined; + if (controller !== undefined && archiveStorage && await hasArchivedLegacyNativeController(db, { ...binding, storage: archiveStorage, controller, supervisor })) continue; + } + if (present || await hasNativeRunnerCheckpoint(db, binding)) return true; + } + return false; +} + +export async function migrateNativeControllers(db: Db, input: { writersStopped: true; signal: AbortSignal; storage?: StorageService }) { + const [active] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(offlineMigrationExecutionWhere()).limit(1); + if (!input.writersStopped || active) throw new Error("native_controller_migration_requires_offline_writers"); + const counts = { inspected: 0, imported: 0, unchanged: 0, sourceAlreadyAbsent: 0, archivedFailed: 0 }; + for await (const { selected, source } of retainedSources(db, input.signal)) { + counts.inspected++; + const binding = { companyId: selected.companyId, issueId: selected.issueId, runId: source.id }; + const checkpoint = await loadNativeRunnerCheckpoint(db, binding); + const metadata = readNativeControllerRecovery(source.runnerProfileJson.nativeControllerRecovery, source.runnerProfileJson.sessionCheckpoint); + const snapshot = source.runnerProfileJson.sessionCheckpoint as { identity?: Record } | null; + if (checkpoint && metadata?.checkpoint?.runId === source.id + && metadata.checkpoint.runnerInstanceId === source.runnerInstanceId + && metadata.checkpoint.normalizedSessionId === selected.sessionId + && snapshot?.identity?.runId === source.id && snapshot.identity.companyId === selected.companyId + && snapshot.identity.agentId === selected.agentId && snapshot.identity.issueId === selected.issueId + && snapshot.identity.sessionId === selected.sessionId + && checkpoint.recoveryEvidence.runnerInstanceId === source.runnerInstanceId + && checkpoint.recoveryEvidence.normalizedSessionId === selected.sessionId) { counts.unchanged++; continue; } + const paths = await localPaths(source); + const legacyControllerState = checkpoint ? undefined : await readPrivateSource(paths.controlFile, 256 * 1024 * 1024); + if (!checkpoint && legacyControllerState === undefined) { counts.sourceAlreadyAbsent++; continue; } + let legacyRunnerArtifact: { version: string; digest: string } | undefined; + if (legacyControllerState !== undefined && await readPrivateSource(join(paths.root, "provider-pack-authority.json"), 512 * 1024)) { + const identity = JSON.parse(legacyControllerState).identity; + const execution = paths.execution; + const [admission] = await db.select({ workspace: sql`${heartbeatRuns.runnerProfileJson}->'nativeWorkspaceSync'` }) + .from(heartbeatRuns).where(and(eq(heartbeatRuns.id, source.id), eq(heartbeatRuns.companyId, selected.companyId), + eq(heartbeatRuns.agentId, selected.agentId))); + const workspace = readNativeWorkspaceSyncReference(admission?.workspace); + const remoteCwd = workspace?.remoteCwd ?? execution.workspace.cwd; + if (typeof identity?.environmentLeaseId !== "string" || !remoteCwd + || (workspace && workspace.leaseId !== identity.environmentLeaseId)) { + throw new Error("native_controller_migration_artifact_binding"); + } + const unavailable = (): never => { throw new Error("native_controller_migration_artifact_unavailable"); }; + legacyRunnerArtifact = resolveRemoteProviderPackAuthority({ stateDirectory: paths.root, reattach: true, + binding: { companyId: selected.companyId, agentId: selected.agentId, + workspaceId: providerPackWorkspaceId(execution.binding.executionWorkspaceId, source.id, identity.environmentLeaseId), + sessionId: selected.sessionId, runnerInstanceId: source.runnerInstanceId!, + environmentLeaseId: identity.environmentLeaseId, remoteCwd, + providerDigest: nativeSha256(execution.provider) }, + current: unavailable, runnerArtifact: unavailable }).runnerArtifact; + } + const supervisor = paths.execution.session.lifecyclePolicy.mode === "warm" + ? await readPrivateSource(paths.supervisorPath, 4 * 1024 * 1024) : undefined; + input.signal.throwIfAborted(); + let result: Awaited>; + try { + result = await importNativeControllerRecovery(db, { ...binding, agentId: selected.agentId, + normalizedSessionId: selected.sessionId, expectedConfigDigest: paths.configDigest, + supervisor: supervisor === undefined ? undefined : JSON.parse(supervisor), + offline: { writersStopped: true, legacyControllerState, legacyRunnerArtifact } }); + } catch (error) { + const storage = input.storage ?? durableNativeHarnessStorage(); + if (!(error instanceof Error) || error.message !== "native_runner_checkpoint_artifact_unproven" + || source.status !== "failed" || legacyControllerState === undefined || !storage) throw error; + await archiveFailedLegacyNativeController(db, { ...binding, storage, controller: legacyControllerState, supervisor, + writersStopped: true, signal: input.signal }); + counts.archivedFailed++; + continue; + } + if (result.kind !== "imported" && result.kind !== "unchanged") { + throw new Error(`native_controller_migration_blocked:${source.id}:${"reason" in result ? result.reason : result.kind}`); + } + counts[result.kind]++; + } + return counts; +} diff --git a/server/src/services/native-runtime/native-controller-recovery-inventory.ts b/server/src/services/native-runtime/native-controller-recovery-inventory.ts index 88f0077f9bd..f965c569de8 100644 --- a/server/src/services/native-runtime/native-controller-recovery-inventory.ts +++ b/server/src/services/native-runtime/native-controller-recovery-inventory.ts @@ -1,6 +1,7 @@ import type { Db } from "@paperclipai/db"; import { sql } from "drizzle-orm"; -import { inspectNativeRunnerUpgradeHeader } from "./native-runner-checkpoint.js"; +import { inspectNativeRunnerUpgradeHeader, NATIVE_CHECKPOINT_HEADER_INSPECTION_MAX_BYTES, + NATIVE_CHECKPOINT_INSPECTION_TOTAL_MAX_BYTES } from "./native-runner-checkpoint.js"; /** An explicit, paginated diagnostic. Never called by an idle/background scan. * This inventories selected retained task-session pointers, not all history. @@ -23,7 +24,8 @@ export async function readNativeControllerRecoveryInventory(db: Db, input: { aft r.native_session_id as "sessionId", r.runner_instance_id as "runnerId", case when pg_column_size(r.runner_profile_json->'nativeControllerRecovery') <= 4096 then r.runner_profile_json->'nativeControllerRecovery' else null end as metadata, - case when pg_column_size(c.material) <= 262144 and sum(pg_column_size(c.material)) over () <= 8388608 + case when pg_column_size(c.material) <= ${NATIVE_CHECKPOINT_HEADER_INSPECTION_MAX_BYTES} + and sum(pg_column_size(c.material)) over () <= ${NATIVE_CHECKPOINT_INSPECTION_TOTAL_MAX_BYTES} then c.material else null end as material, (r.finished_at is null or r.status not in ('succeeded', 'failed', 'cancelled', 'interrupted', 'timed_out') or f.run_id is null or f.lease_owner is not null or f.recovery_state is not null diff --git a/server/src/services/native-runtime/native-controller-recovery.integration.test.ts b/server/src/services/native-runtime/native-controller-recovery.integration.test.ts index c4315273e81..d1e5a5a475f 100644 --- a/server/src/services/native-runtime/native-controller-recovery.integration.test.ts +++ b/server/src/services/native-runtime/native-controller-recovery.integration.test.ts @@ -3,7 +3,7 @@ import { mkdtemp, rm } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { eq } from "drizzle-orm"; -import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; import { agentTaskSessions, agents, companies, createDb, environmentLeases, environments, heartbeatRuns, issues, nativeRunFinalizations, nativeRunResults } from "@paperclipai/db"; import { DurablePrpControlPlane, parseNativeExecutionInput, type PersistedNativeSession } from "../../vendor/paperclip-runner/index.js"; import { startRecoveryPostgres, recoveryPostgresSupport } from "../../__tests__/helpers/recovery-postgres.js"; @@ -92,6 +92,18 @@ describe.skipIf(!recoveryPostgresSupport.supported)("native controller durable d expect(await resolveNativeControllerRecovery(db, { ...fixture.scope, currentRunId })).toEqual({ kind: "waiting_for_owner", deadline }); }); + it("uses the database clock when an import finds an expired owner", async () => { + const fixture = await seed(); + const now = Date.now(); + await db.update(nativeRunFinalizations).set({ leaseOwner: "expired-owner", leaseExpiresAt: new Date(now - 60_000) }) + .where(eq(nativeRunFinalizations.runId, fixture.scope.runId)); + vi.useFakeTimers({ toFake: ["Date"] }); vi.setSystemTime(now - 3_600_000); + try { + expect(await importNativeControllerRecovery(db, { ...fixture.scope, expectedConfigDigest: compatibility.configDigest, + supervisor: fixture.supervisor })).toEqual({ kind: "missing_evidence", reason: "normal_ownership_recovery_required" }); + } finally { vi.useRealTimers(); } + }); + it("holds mismatched snapshots and runner bindings instead of falling back to fresh execution", async () => { const fixture = await seed(), currentRunId = randomUUID(); const metadata = createNativeControllerRecovery(fixture.snapshot, compatibility, { runId: fixture.scope.runId, diff --git a/server/src/services/native-runtime/native-controller-recovery.ts b/server/src/services/native-runtime/native-controller-recovery.ts index 223e5f69bdc..64e9f6d1d23 100644 --- a/server/src/services/native-runtime/native-controller-recovery.ts +++ b/server/src/services/native-runtime/native-controller-recovery.ts @@ -1,12 +1,13 @@ import { and, eq, ne, sql } from "drizzle-orm"; import { environmentLeases, heartbeatRuns, issues, nativeRunFinalizations, nativeRunnerCheckpoints, type Db } from "@paperclipai/db"; import type { AdapterExecutionTarget } from "@paperclipai/adapter-utils/execution-target"; -import type { DurablePrpCheckpoint, PersistedNativeSession } from "../../vendor/paperclip-runner/index.js"; +import { captureLegacyDurablePrpCheckpoint, parseNativeExecutionInput, type DurablePrpCheckpoint, type PersistedNativeSession } from "../../vendor/paperclip-runner/index.js"; import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js"; import { readRunCancellation } from "../run-cancellation.js"; import { nativeSha256 } from "./canonical.js"; import { findNativeSessionResumeRun, isNativeSessionId, isUnusedNativeSessionBootstrap, nativeSessionProviderEvidence } from "./native-session-resume.js"; -import { loadNativeRunnerCheckpoint } from "./native-runner-checkpoint.js"; +import { insertMigratedNativeRunnerCheckpoint, loadNativeRunnerCheckpoint } from "./native-runner-checkpoint.js"; +import { offlineMigrationExecutionWhere } from "../offline-migration-execution.js"; import { redactRunnerApiPayloads } from "./runner-api-log-redaction.js"; export const NATIVE_CONTROLLER_RECOVERY_MAX_BYTES = 4 * 1024; @@ -223,10 +224,11 @@ export async function resolveNativeControllerRecovery(db: Db, input: { export async function importNativeControllerRecovery(db: Db, input: { companyId: string; agentId: string; issueId: string; runId: string; normalizedSessionId: string; expectedConfigDigest: string; - supervisor: unknown; + supervisor?: unknown; + offline?: { writersStopped: true; legacyControllerState?: string; legacyRunnerArtifact?: { version: string; digest: string } }; }): Promise<{ kind: "imported" | "unchanged" } | Exclude> { const supervisor = record(input.supervisor); - if (supervisor.schema !== "paperclip.native-session-supervisor.v1" || supervisor.configDigest !== input.expectedConfigDigest) { + if (input.supervisor !== undefined && (supervisor.schema !== "paperclip.native-session-supervisor.v1" || supervisor.configDigest !== input.expectedConfigDigest)) { return { kind: "unsupported", reason: "supervisor_compatibility" }; } return db.transaction(async tx => { @@ -236,25 +238,48 @@ export async function importNativeControllerRecovery(db: Db, input: { eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.issueId, input.issueId), eq(nativeRunFinalizations.runId, input.runId))).for("update"); if (!task || !owner) return { kind: "missing_evidence" as const, reason: "owner_binding" }; - if (owner.leaseOwner || task.executionRunId) { - return owner.leaseExpiresAt && owner.leaseExpiresAt > new Date() + if (input.offline) { + const [active] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(offlineMigrationExecutionWhere()).limit(1); + if (!input.offline.writersStopped || active) return { kind: "missing_evidence" as const, reason: "offline_writers_required" }; + } + const [clock] = await tx.execute<{ now: Date }>(sql`select clock_timestamp() as now`); + const liveOwner = owner.leaseOwner && (!owner.leaseExpiresAt || owner.leaseExpiresAt > new Date(clock.now)); + if (input.offline ? liveOwner : owner.leaseOwner || task.executionRunId) { + return owner.leaseExpiresAt && owner.leaseExpiresAt > new Date(clock.now) ? { kind: "waiting_for_owner" as const, deadline: owner.leaseExpiresAt } : { kind: "missing_evidence" as const, reason: "normal_ownership_recovery_required" }; } - if (owner.phase !== "committed" || !owner.resultId || owner.recoveryState) { + if (!input.offline && (!(owner.phase === "committed" && owner.resultId) || owner.recoveryState)) { return { kind: "missing_evidence" as const, reason: "pending_finalization" }; } const [run] = await tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.agentId, input.agentId), eq(heartbeatRuns.nativeIssueId, input.issueId), eq(heartbeatRuns.nativeSessionId, input.normalizedSessionId))).for("update"); const profile = record(run?.runnerProfileJson), snapshot = record(profile.sessionCheckpoint), identity = record(snapshot.identity); - if (run?.runtimeMode !== "native" || run.status !== "succeeded" || !run.finishedAt + if (run?.runtimeMode !== "native" || !(run.status === "succeeded" || input.offline && ["failed", "cancelled", "interrupted", "timed_out"].includes(run.status)) || !run.finishedAt || identity.runId !== input.runId || identity.companyId !== input.companyId || identity.issueId !== input.issueId || identity.agentId !== input.agentId || identity.sessionId !== input.normalizedSessionId - || nativeSha256(snapshot) !== nativeSha256(redactRunnerApiPayloads(supervisor.snapshot))) { + || input.supervisor !== undefined && nativeSha256(snapshot) !== nativeSha256(redactRunnerApiPayloads(supervisor.snapshot))) { return { kind: "missing_evidence" as const, reason: "stale_supervisor_snapshot" }; } - const checkpoint = await loadNativeRunnerCheckpoint(tx, input); + if (input.supervisor === undefined) { + try { + const admitted = parseNativeExecutionInput(profile.nativeExecutionInput); + if (!input.offline || admitted.session.lifecyclePolicy.mode !== "per_turn" + || admitted.binding.runId !== input.runId || admitted.binding.companyId !== input.companyId + || admitted.binding.agentId !== input.agentId || admitted.binding.issueId !== input.issueId) { + return { kind: "missing_evidence" as const, reason: "admitted_per_turn_binding" }; + } + } catch { return { kind: "unsupported" as const, reason: "admitted_input_format" }; } + } + let checkpoint = await loadNativeRunnerCheckpoint(tx, input); + const importing = !checkpoint && input.offline?.legacyControllerState !== undefined; + if (importing) { + if (!run.runnerInstanceId) return { kind: "missing_evidence" as const, reason: "checkpoint_binding" }; + checkpoint = captureLegacyDurablePrpCheckpoint(input.offline!.legacyControllerState!, { + runId: input.runId, runnerInstanceId: run.runnerInstanceId, normalizedSessionId: input.normalizedSessionId, + }, input.offline!.legacyRunnerArtifact); + } if (!checkpoint || checkpoint.recoveryEvidence.normalizedSessionId !== input.normalizedSessionId || checkpoint.recoveryEvidence.runnerInstanceId !== run.runnerInstanceId) { return { kind: "missing_evidence" as const, reason: "checkpoint_binding" }; @@ -265,10 +290,13 @@ export async function importNativeControllerRecovery(db: Db, input: { }, { runId: input.runId, runnerInstanceId: checkpoint.recoveryEvidence.runnerInstanceId, normalizedSessionId: input.normalizedSessionId }); if (!readNativeControllerRecovery(metadata, snapshot)) return { kind: "unsupported" as const, reason: "metadata_format" }; if (profile.nativeControllerRecovery) { - return nativeSha256(profile.nativeControllerRecovery) === nativeSha256(metadata) - ? { kind: "unchanged" as const } - : { kind: "missing_evidence" as const, reason: "existing_metadata_conflict" }; + if (nativeSha256(profile.nativeControllerRecovery) !== nativeSha256(metadata)) { + return { kind: "missing_evidence" as const, reason: "existing_metadata_conflict" }; + } + if (importing) await insertMigratedNativeRunnerCheckpoint(tx, input, checkpoint); + return { kind: importing ? "imported" as const : "unchanged" as const }; } + if (importing) await insertMigratedNativeRunnerCheckpoint(tx, input, checkpoint); await tx.update(heartbeatRuns).set({ runnerProfileJson: { ...profile, nativeControllerRecovery: metadata }, updatedAt: new Date() }) .where(eq(heartbeatRuns.id, input.runId)); return { kind: "imported" as const }; diff --git a/server/src/services/native-runtime/native-harness-backup-archive.test.ts b/server/src/services/native-runtime/native-harness-backup-archive.test.ts new file mode 100644 index 00000000000..a08ca880f2c --- /dev/null +++ b/server/src/services/native-runtime/native-harness-backup-archive.test.ts @@ -0,0 +1,80 @@ +import { mkdtemp, mkdir, open, readFile, readlink, rm, stat, symlink, writeFile } from "node:fs/promises"; +import { createWriteStream } from "node:fs"; +import { join } from "node:path"; +import { tmpdir } from "node:os"; +import { pipeline } from "node:stream/promises"; +import tar from "tar-stream"; +import { afterEach, expect, it } from "vitest"; +import { captureNativeHarnessArchive, restoreNativeHarnessArchive, MAX_NATIVE_HARNESS_BYTES } from "./native-harness-backup-archive.js"; + +const roots: string[] = []; +afterEach(async () => { for (const root of roots.splice(0)) await rm(root, { recursive: true, force: true }); }); +async function fixture() { const root = await mkdtemp(join(tmpdir(), "native-backup-archive-test-")); roots.push(root); return root; } +async function maliciousArchive(root: string, headers: Array) { + const archive = tar.pack(), file = join(root, "malicious.tar"); + const written = pipeline(archive, createWriteStream(file)); + for (const header of headers) archive.entry(header, Buffer.alloc(0)); + archive.finalize(); await written; return file; +} +it.each([ + { name: "../escaped", type: "file" as const }, + { name: "/absolute", type: "file" as const }, + { name: "link", type: "symlink" as const, linkname: "../outside" }, + { name: "device", type: "character-device" as const }, + { name: "parent/child", type: "file" as const }, +])("rejects unsafe archive entry $name", async header => { + const root = await fixture(), file = await maliciousArchive(root, [header]); + await expect(restoreNativeHarnessArchive(file, join(root, "destination"))).rejects.toThrow("native_harness_archive_invalid"); + await expect(stat(join(root, "escaped"))).rejects.toMatchObject({ code: "ENOENT" }); +}); +it("rejects duplicate paths instead of overwriting a restored file", async () => { + const root = await fixture(), file = await maliciousArchive(root, [{ name: "manifest.json", type: "file" }, { name: "manifest.json", type: "file" }]); + await expect(restoreNativeHarnessArchive(file, join(root, "destination"))).rejects.toThrow("native_harness_archive_invalid"); +}); +it("refuses a source symlink without reading its target", async () => { + const root = await fixture(), outside = join(root, "outside"); + await writeFile(outside, "private external file"); + const { mkdir } = await import("node:fs/promises"); const source = join(root, "source"); await mkdir(source); + await symlink(outside, join(source, "linked")); + await expect(captureNativeHarnessArchive(source, join(root, "archive"))).rejects.toThrow("native_harness_archive_invalid"); + expect(await readFile(outside, "utf8")).toBe("private external file"); +}); +it("honors cancellation before walking the checkpoint tree", async () => { + const root = await fixture(); + await expect(captureNativeHarnessArchive(root, join(root, "archive"), AbortSignal.abort(new Error("test cancelled")))) + .rejects.toThrow(); +}); + +it("preserves contained provider links", async () => { + const root = await fixture(), source = join(root, "source"), target = join(root, "restored"); + await mkdir(join(source, "provider", ".bin"), { recursive: true }); + await writeFile(join(source, "manifest.json"), "{}"); + await writeFile(join(source, "provider", "tool.js"), "provider state"); + await symlink("../tool.js", join(source, "provider", ".bin", "tool")); + const archive = join(root, "archive"); + await captureNativeHarnessArchive(source, archive); + await restoreNativeHarnessArchive(archive, target); + expect(await readlink(join(target, "provider", ".bin", "tool"))).toBe("../tool.js"); + expect(await readFile(join(target, "provider", ".bin", "tool"), "utf8")).toBe("provider state"); +}); +it("rejects writes beneath a symlink even when its target is contained", async () => { + const root = await fixture(), file = await maliciousArchive(root, [ + { name: "manifest.json", type: "file" }, { name: "directory", type: "directory" }, + { name: "link", type: "symlink", linkname: "directory" }, { name: "link/child", type: "file" }, + ]); + await expect(restoreNativeHarnessArchive(file, join(root, "destination"))).rejects.toThrow("native_harness_archive_invalid"); +}); +it("rejects indirect symlink targets instead of allowing chained traversal", async () => { + const root = await fixture(), file = await maliciousArchive(root, [ + { name: "manifest.json", type: "file" }, { name: "first", type: "symlink", linkname: "second/child" }, + { name: "second", type: "symlink", linkname: "elsewhere" }, + ]); + await expect(restoreNativeHarnessArchive(file, join(root, "destination"))).rejects.toThrow("native_harness_archive_invalid"); +}); +it("rejects oversized source files before streaming their payload", async () => { + const root = await fixture(), source = join(root, "source"); await mkdir(source); + const handle = await open(join(source, "oversized"), "wx"); + await handle.truncate(MAX_NATIVE_HARNESS_BYTES + 1); await handle.close(); + await expect(captureNativeHarnessArchive(source, join(root, "archive"))).rejects.toThrow("native_harness_archive_invalid"); + expect((await stat(join(root, "archive"))).size).toBeLessThan(1024); +}); diff --git a/server/src/services/native-runtime/native-harness-backup-archive.ts b/server/src/services/native-runtime/native-harness-backup-archive.ts new file mode 100644 index 00000000000..f6249d5f118 --- /dev/null +++ b/server/src/services/native-runtime/native-harness-backup-archive.ts @@ -0,0 +1,114 @@ +import { constants, createReadStream, createWriteStream } from "node:fs"; +import { chmod, lstat, mkdir, open, readdir, readlink, symlink } from "node:fs/promises"; +import { join, posix } from "node:path"; +import { Transform } from "node:stream"; +import { pipeline } from "node:stream/promises"; +import tar from "tar-stream"; + +export const MAX_NATIVE_HARNESS_BYTES = 4 * 1024 * 1024 * 1024; +export const MAX_NATIVE_HARNESS_OBJECT_BYTES = MAX_NATIVE_HARNESS_BYTES + 768 * 1024 * 1024; +export const MAX_NATIVE_HARNESS_ENTRIES = 100_000; +const fail = () => new Error("native_harness_archive_invalid"); +const safeName = (name: string) => name.length > 0 && Buffer.byteLength(name) <= 4096 && !name.includes("\\") + && !/[\u0000-\u001f\u007f]/u.test(name) && !posix.isAbsolute(name) + && name.split("/").every(part => part !== ".." && part !== "." && part !== ""); + +const safeLink = (name: string, target: string) => target.length > 0 && Buffer.byteLength(target) <= 4096 + && !target.includes("\\") && !/[\u0000-\u001f\u007f]/u.test(target) && !posix.isAbsolute(target) + && !["..", "."].includes(posix.normalize(posix.join(posix.dirname(name), target))) + && !posix.normalize(posix.join(posix.dirname(name), target)).startsWith("../"); +const validateLinks = (links: Map) => { + for (const [name, target] of links) { + const resolved = posix.normalize(posix.join(posix.dirname(name), target)).split("/"); + for (let index = 1; index <= resolved.length; index++) { + if (links.has(resolved.slice(0, index).join("/"))) throw fail(); + } + } +}; +const boundedArchive = () => { + let bytes = 0; + return new Transform({ transform(chunk, _encoding, callback) { + bytes += chunk.length; + callback(bytes > MAX_NATIVE_HARNESS_OBJECT_BYTES ? fail() : null, chunk); + } }); +}; + +export async function captureNativeHarnessArchive(root: string, target: string, signal?: AbortSignal) { + if (!(await lstat(root)).isDirectory()) throw fail(); + const archive = tar.pack(); + const output = pipeline(archive, boundedArchive(), createWriteStream(target, { flags: "wx", mode: 0o600 }), { signal }); + void output.catch(() => undefined); + let count = 0, bytes = 0; + const links = new Map(); + const visit = async (relative: string): Promise => { + for (const name of (await readdir(join(root, relative))).sort()) { + signal?.throwIfAborted(); + const entry = relative ? `${relative}/${name}` : name; + if (!safeName(entry) || ++count > MAX_NATIVE_HARNESS_ENTRIES) throw fail(); + const file = join(root, entry), stat = await lstat(file); + if (stat.isDirectory()) { + await new Promise((resolve, reject) => archive.entry({ name: entry, type: "directory", mode: stat.mode & 0o777 }, + error => error ? reject(error) : resolve())); + await visit(entry); + } else if (stat.isFile()) { + bytes += stat.size; + if (bytes > MAX_NATIVE_HARNESS_BYTES) throw fail(); + const handle = await open(file, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK); + try { + const before = await handle.stat(); + if (!before.isFile() || before.ino !== stat.ino || before.dev !== stat.dev || before.size !== stat.size) throw fail(); + const stream = archive.entry({ name: entry, size: stat.size, type: "file", mode: stat.mode & 0o777 }); + await pipeline(handle.createReadStream({ autoClose: false }), stream, { signal }); + const after = await handle.stat(); + if (before.size !== after.size || before.mtimeMs !== after.mtimeMs || before.ctimeMs !== after.ctimeMs) throw fail(); + } finally { await handle.close(); } + } else if (stat.isSymbolicLink()) { + const linkname = await readlink(file); + if (!safeLink(entry, linkname)) throw fail(); + links.set(entry, linkname); + await new Promise((resolve, reject) => archive.entry({ name: entry, type: "symlink", linkname }, + error => error ? reject(error) : resolve())); + } else throw fail(); + } + }; + try { await visit(""); validateLinks(links); archive.finalize(); await output; } + catch (error) { archive.destroy(error as Error); await output.catch(() => undefined); throw error; } +} + +export async function restoreNativeHarnessArchive(source: string, target: string, signal?: AbortSignal) { + await mkdir(target, { mode: 0o700 }); + const archive = tar.extract(), names = new Set(), links = new Map(); + const directories: Array<{ path: string; mode: number }> = [], directoryNames = new Set(); + let bytes = 0; + archive.on("entry", (header, body, next) => { + body.on("error", () => undefined); + void (async () => { + signal?.throwIfAborted(); + const name = header.type === "directory" ? header.name.replace(/\/$/u, "") : header.name; + if (!safeName(name) || names.has(name) || names.size >= MAX_NATIVE_HARNESS_ENTRIES + || !["directory", "file", "symlink"].includes(header.type ?? "") || !Number.isSafeInteger(header.size ?? 0) + || (header.size ?? 0) < 0 || (bytes += header.size ?? 0) > MAX_NATIVE_HARNESS_BYTES) throw fail(); + names.add(name); + const file = join(target, name), mode = (header.mode ?? 0o600) & 0o777; + const parent = posix.dirname(name); + if (parent !== "." && !directoryNames.has(parent)) throw fail(); + if (header.type === "directory") { + if (header.size) throw fail(); + await mkdir(file, { mode: 0o700 }); directories.push({ path: file, mode }); directoryNames.add(name); + for await (const _ of body) throw fail(); + } else if (header.type === "symlink") { + if (header.size || !header.linkname || !safeLink(name, header.linkname)) throw fail(); + links.set(name, header.linkname); + await symlink(header.linkname, file); + for await (const _ of body) throw fail(); + } else { + await pipeline(body, createWriteStream(file, { flags: "wx", mode }), { signal }); + await chmod(file, mode); + } + })().then(() => next(), error => archive.destroy(error as Error)); + }); + await pipeline(createReadStream(source), boundedArchive(), archive, { signal }); + if (!names.has("manifest.json")) throw fail(); + validateLinks(links); + for (const directory of directories.reverse()) await chmod(directory.path, directory.mode); +} diff --git a/server/src/services/native-runtime/native-harness-backup-expiration.ts b/server/src/services/native-runtime/native-harness-backup-expiration.ts index 6e5ab7c424d..c326d232098 100644 --- a/server/src/services/native-runtime/native-harness-backup-expiration.ts +++ b/server/src/services/native-runtime/native-harness-backup-expiration.ts @@ -2,7 +2,7 @@ import { notifyExternalConversationSettlement } from "../external-conversation-s import { randomUUID } from "node:crypto"; import { promises as fs } from "node:fs"; import path from "node:path"; -import { and, asc, eq, gt, inArray, isNull, lt, or, sql } from "drizzle-orm"; +import { and, asc, eq, gt, gte, inArray, isNull, lt, or, sql } from "drizzle-orm"; import { agentTaskSessions, environmentLeases, @@ -10,6 +10,7 @@ import { issues, nativeRunFinalizations, nativeRunnerCheckpoints, + nativeHarnessBackups, type Db, } from "@paperclipai/db"; import { logger } from "../../middleware/logger.js"; @@ -18,6 +19,9 @@ import { nativeSessionScopeDigestIsBusy } from "./native-session-executor.js"; import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js"; import { DELIVERY_QUEUES } from "../delivery-work-notifications.js"; import type { NativeHarnessBackupExpirationScope } from "../native-harness-backup-expiration-notifications.js"; +import { deleteRetiredNativeHarnessBackup, durableNativeHarnessStorage } from "./native-harness-backup-storage.js"; +import { retireObsoleteNativeHarnessBackup } from "./native-harness-backup-retention.js"; +import type { StorageService } from "../../storage/types.js"; export const NATIVE_HARNESS_BACKUP_EXPIRATION_INTERVAL_MS = 60 * 60 * 1000; export const NATIVE_HARNESS_BACKUP_EXPIRATION_STARTUP_DELAY_MS = 2 * 60 * 1000; @@ -51,6 +55,7 @@ export type NativeHarnessBackupExpirationOptions = { stateBase?: string; shouldContinue?: () => boolean; isSessionBusy?: (scopeDigest: string) => boolean; + storage?: StorageService; }; export type NativeHarnessBackupExpirationResult = { @@ -291,6 +296,73 @@ export function nativeHarnessBackupExpirationService( let evidenceDueAt = 0; let passDueAt = Number.POSITIVE_INFINITY; let checkpointCursor: string | null = null; + let durableBackupCursor: string | null = null; + let checkpointsEnumerated = false; + + async function expireDurableBackups(options: { + closedBefore: Date; dryRun: boolean; maxEntries: number; maxRemovals: number; shouldContinue: () => boolean; + }, result: NativeHarnessBackupExpirationResult) { + const storage = defaults.storage ?? durableNativeHarnessStorage(); + if (!storage) return; + const candidates = await db.select({ id: nativeHarnessBackups.id, scopeDigest: nativeHarnessBackups.scopeDigest, + companyId: nativeHarnessBackups.companyId, runId: nativeHarnessBackups.runId, survivingRunId: heartbeatRuns.id, + sessionId: heartbeatRuns.nativeSessionId, status: nativeHarnessBackups.status }) + .from(nativeHarnessBackups).leftJoin(heartbeatRuns, eq(heartbeatRuns.id, nativeHarnessBackups.runId)) + .where(and(durableBackupCursor ? gt(nativeHarnessBackups.id, durableBackupCursor) : undefined, + or(eq(nativeHarnessBackups.status, "retired"), lt(nativeHarnessBackups.createdAt, options.closedBefore)))) + .orderBy(asc(nativeHarnessBackups.id)).limit(options.maxEntries + 1); + let processed = 0; + for (const candidate of candidates) { + if (!options.shouldContinue() || processed++ >= options.maxEntries || result.expired >= options.maxRemovals) { + result.truncated = true; return; + } + durableBackupCursor = candidate.id; + if (candidate.survivingRunId && !candidate.sessionId) continue; + try { + const retired = candidate.status === "retired" || await db.transaction(async tx => { + if (["pending", "superseded"].includes(candidate.status)) { + const obsolete = await retireObsoleteNativeHarnessBackup(tx, { ...candidate, + closedBefore: options.closedBefore, dryRun: options.dryRun }); + if (obsolete) result.candidates++; + return obsolete && !options.dryRun; + } + if (!candidate.survivingRunId) { + result.candidates++; + if (options.dryRun) return false; + await tx.update(nativeHarnessBackups).set({ status: "retired", updatedAt: new Date() }) + .where(eq(nativeHarnessBackups.id, candidate.id)); + return true; + } + const blocker = await sessionRetentionBlocker(tx, { scopeDigest: candidate.scopeDigest, checkpoint: true, + closedBefore: options.closedBefore, + identity: { runId: candidate.runId, normalizedSessionId: candidate.sessionId!, environmentLeaseId: null } }); + if ("reason" in blocker) return false; + const [recent] = await tx.select({ id: nativeHarnessBackups.id }).from(nativeHarnessBackups).where(and( + eq(nativeHarnessBackups.companyId, blocker.companyId), eq(nativeHarnessBackups.scopeDigest, candidate.scopeDigest), + gte(nativeHarnessBackups.createdAt, options.closedBefore))).limit(1); + if (recent) return false; + result.candidates++; + if (options.dryRun) return false; + await notifyExternalConversationSettlement(tx, { companyId: blocker.companyId }); + await tx.update(nativeRunFinalizations).set({ controllerGeneration: sql`${nativeRunFinalizations.controllerGeneration} + 1` }) + .where(inArray(nativeRunFinalizations.runId, blocker.runIds)); + await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${`${blocker.companyId}:${candidate.scopeDigest}`}, 0))`); + const cleared = await tx.delete(agentTaskSessions).where(and(eq(agentTaskSessions.companyId, blocker.companyId), + eq(agentTaskSessions.agentId, blocker.agentId), or(inArray(agentTaskSessions.lastRunId, blocker.runIds), + sql`${agentTaskSessions.sessionParamsJson}->>'sessionId' = ${candidate.sessionId}`))).returning({ id: agentTaskSessions.id }); + result.clearedTaskSessions += cleared.length; + await tx.update(nativeHarnessBackups).set({ status: "retired", updatedAt: new Date() }).where(and( + eq(nativeHarnessBackups.companyId, blocker.companyId), eq(nativeHarnessBackups.scopeDigest, candidate.scopeDigest))); + return true; + }); + if (retired && !options.dryRun && await deleteRetiredNativeHarnessBackup(db, storage, candidate.id, AbortSignal.timeout(60_000))) result.expired++; + } catch (err) { + result.failed++; + logger.warn({ err, backupId: candidate.id }, "durable native harness backup expiration skipped an object"); + } + } + durableBackupCursor = null; + } async function expireCheckpoints(options: { closedBefore: Date; dryRun: boolean; maxEntries: number; maxRemovals: number; shouldContinue: () => boolean; @@ -704,13 +776,20 @@ export function nativeHarnessBackupExpirationService( const result = emptyResult(); const shouldContinue = options.shouldContinue ?? defaults.shouldContinue ?? (() => true); if (!shouldContinue()) return { ...result, truncated: true }; - await expireCheckpoints({ + const pass = { closedBefore: new Date((options.now ?? defaults.now ?? new Date()).getTime() - closedForDays * DAY_MS), dryRun: options.dryRun ?? defaults.dryRun ?? false, maxEntries: Math.max(1, options.maxEntries ?? defaults.maxEntries ?? 200), maxRemovals: Math.max(1, options.maxRemovals ?? defaults.maxRemovals ?? 50), shouldContinue, - }, result); + }; + if (!checkpointsEnumerated) { + await expireCheckpoints(pass, result); + if (result.truncated) return result; + checkpointsEnumerated = true; + } + await expireDurableBackups(pass, result); + if (!result.truncated) checkpointsEnumerated = false; return result; } diff --git a/server/src/services/native-runtime/native-harness-backup-migration.ts b/server/src/services/native-runtime/native-harness-backup-migration.ts new file mode 100644 index 00000000000..d2d7aa0d6c3 --- /dev/null +++ b/server/src/services/native-runtime/native-harness-backup-migration.ts @@ -0,0 +1,79 @@ +import { createHash } from "node:crypto"; +import { lstat, readFile, readdir } from "node:fs/promises"; +import { join } from "node:path"; +import { and, eq, sql } from "drizzle-orm"; +import { environmentLeases, heartbeatRuns, nativeHarnessBackups, type Db } from "@paperclipai/db"; +import type { StorageService } from "../../storage/types.js"; +import { nativeHarnessStateBase, readNativeHarnessBackupStamp, verifyNativeHarnessBackupStampAsync } from "./native-harness-backup-stamp.js"; +import { persistNativeHarnessBackup, verifyDurableNativeHarnessBackup } from "./native-harness-backup-storage.js"; +import { offlineMigrationExecutionWhere } from "../offline-migration-execution.js"; + +export async function hasLocalNativeHarnessObligations(db: Db): Promise { + const rows = await db.execute(sql`select 1 from agent_task_sessions s + join heartbeat_runs r on r.id = s.last_run_id and r.company_id = s.company_id and r.agent_id = s.agent_id + join environment_leases e on e.company_id = r.company_id and e.issue_id = r.native_issue_id + and e.metadata->'nativeHarnessBackup'->>'normalizedSessionId' = r.native_session_id::text + where s.adapter_type = 'paperclip_runner' + and e.metadata->'nativeHarnessBackup'->>'schema' = 'paperclip.native-harness-backup-stamp.v2' + and not exists (select 1 from native_harness_backups b where b.company_id = r.company_id + and b.scope_digest = e.metadata->'nativeHarnessBackup'->>'sessionScopeSha256' and b.status = 'current') + limit 1`); + return rows.length > 0; +} + +export async function migrateNativeHarnessBackups(db: Db, input: { + storage: StorageService; writersStopped: true; signal: AbortSignal; +}): Promise { + if (!input.writersStopped) throw new Error("native_harness_migration_requires_offline_writers"); + const [active] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(offlineMigrationExecutionWhere()).limit(1); + if (active) throw new Error("native_harness_migration_requires_offline_writers"); + const base = nativeHarnessStateBase(); + const directories = await readdir(base).catch((error: NodeJS.ErrnoException) => { + if (error.code === "ENOENT") return [] as string[]; throw error; + }); + let count = 0; + for (const scopeDigest of directories.filter(name => /^[a-f0-9]{64}$/.test(name)).sort()) { + input.signal.throwIfAborted(); + const root = join(base, scopeDigest, "failover-backups", "current"); + const stat = await lstat(root).catch((error: NodeJS.ErrnoException) => { + if (error.code === "ENOENT") return null; throw error; + }); + if (!stat) continue; + if (!stat.isDirectory() || stat.isSymbolicLink()) throw new Error("native_harness_migration_unsafe_source"); + const manifestBytes = await readFile(join(root, "manifest.json")); + const manifest = JSON.parse(manifestBytes.toString("utf8")); + const state = JSON.parse(await readFile(join(root, "runner", "runner-state.json"), "utf8")); + if (state.lifecycle !== "suspended" || typeof state.runId !== "string" + || manifest.schema !== "paperclip.native-harness-backup.v1") throw new Error("native_harness_migration_source_unsettled"); + const [run] = await db.select({ companyId: heartbeatRuns.companyId, issueId: heartbeatRuns.nativeIssueId, + sessionId: heartbeatRuns.nativeSessionId, runnerId: heartbeatRuns.runnerInstanceId }).from(heartbeatRuns) + .where(eq(heartbeatRuns.id, state.runId)); + if (!run) continue; + if (!run.issueId || run.sessionId !== manifest.normalizedSessionId || run.runnerId !== manifest.runnerInstanceId) { + throw new Error("native_harness_migration_binding_mismatch"); + } + const [current] = await db.select().from(nativeHarnessBackups).where(and(eq(nativeHarnessBackups.companyId, run.companyId), + eq(nativeHarnessBackups.scopeDigest, scopeDigest), eq(nativeHarnessBackups.status, "current"))); + const manifestSha256 = `sha256:${createHash("sha256").update(manifestBytes).digest("hex")}`; + if (current) { + if (current.manifestSha256 !== manifestSha256) throw new Error("native_harness_migration_durable_source_conflict"); + await verifyDurableNativeHarnessBackup({ db, storage: input.storage, companyId: run.companyId, + scopeDigest, manifestSha256: current.manifestSha256, signal: input.signal }); + count++; continue; + } + const leases = await db.select({ stamp: sql`${environmentLeases.metadata}->'nativeHarnessBackup'`, + providerLeaseId: environmentLeases.providerLeaseId }).from(environmentLeases).where(and( + eq(environmentLeases.companyId, run.companyId), eq(environmentLeases.issueId, run.issueId), + sql`${environmentLeases.metadata}->'nativeHarnessBackup'->>'manifestSha256' = ${manifestSha256}`)); + let valid = false; + for (const lease of leases) { + const stamp = readNativeHarnessBackupStamp(lease.stamp, lease.providerLeaseId ?? ""); + if (stamp?.sessionScopeSha256 === scopeDigest && await verifyNativeHarnessBackupStampAsync(stamp, lease.providerLeaseId!, input.signal)) { valid = true; break; } + } + if (!valid) throw new Error("native_harness_migration_backup_unverified"); + await persistNativeHarnessBackup({ db, storage: input.storage, root, scopeDigest, signal: input.signal, + owner: { kind: "offline_migration", writersStopped: true, companyId: run.companyId, issueId: run.issueId, runId: state.runId } }); + count++; + } + return count; +} diff --git a/server/src/services/native-runtime/native-harness-backup-publication.test.ts b/server/src/services/native-runtime/native-harness-backup-publication.test.ts index df1029f55f9..66dcc70cf00 100644 --- a/server/src/services/native-runtime/native-harness-backup-publication.test.ts +++ b/server/src/services/native-runtime/native-harness-backup-publication.test.ts @@ -53,6 +53,15 @@ describe("native harness backup publication", () => { expect(await fs.readdir(backupRoot)).toEqual(["current"]); expect(changes).toEqual([{ companyId: input.companyId, scopeDigest: input.scopeDigest }]); }); + it("restores the old current directory when the pending rename fails", async () => { + const input = await fixture(); const changes: unknown[] = []; + unsubscribe = onNativeHarnessBackupChange(input.db, change => changes.push(change)); + await fs.rm(input.pendingRoot, { recursive: true }); + await expect(publishNativeHarnessBackup(input)).rejects.toMatchObject({ code: "ENOENT" }); + expect(await fs.readFile(path.join(input.current, "manifest.json"), "utf8")).toBe("saved"); + expect(changes).toEqual([]); + }); + it("serves an unrelated request while a retired backup is being removed", async () => { const input = await fixture(); for (let index = 0; index < 40; index++) { diff --git a/server/src/services/native-runtime/native-harness-backup-retention.ts b/server/src/services/native-runtime/native-harness-backup-retention.ts new file mode 100644 index 00000000000..86e0a25311f --- /dev/null +++ b/server/src/services/native-runtime/native-harness-backup-retention.ts @@ -0,0 +1,34 @@ +import { and, eq, inArray, isNotNull, sql } from "drizzle-orm"; +import { environmentLeases, heartbeatRuns, nativeHarnessBackups, nativeRunFinalizations, type Db } from "@paperclipai/db"; + +type Transaction = Parameters[0]>[0]; + +export async function retireObsoleteNativeHarnessBackup(tx: Transaction, input: { + id: string; companyId: string; scopeDigest: string; closedBefore: Date; dryRun: boolean; +}): Promise { + await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${`${input.companyId}:${input.scopeDigest}`}, 0))`); + const [candidate] = await tx.select().from(nativeHarnessBackups).where(and( + eq(nativeHarnessBackups.id, input.id), eq(nativeHarnessBackups.companyId, input.companyId))).for("update"); + if (!candidate || !["pending", "superseded"].includes(candidate.status) || candidate.updatedAt >= input.closedBefore) return false; + const [current] = await tx.select({ id: nativeHarnessBackups.id }).from(nativeHarnessBackups).where(and( + eq(nativeHarnessBackups.companyId, input.companyId), eq(nativeHarnessBackups.scopeDigest, input.scopeDigest), + eq(nativeHarnessBackups.status, "current"))).limit(1); + if (!current && candidate.status === "superseded") return false; + const [lease] = await tx.select({ id: environmentLeases.id }).from(environmentLeases).where(and( + eq(environmentLeases.companyId, input.companyId), + inArray(environmentLeases.status, ["active", "released", "retained", "pending_cleanup"]), + sql`${environmentLeases.metadata}->'nativeHarnessBackup'->>'sessionScopeSha256' = ${input.scopeDigest}`, + sql`${environmentLeases.metadata}->'nativeHarnessBackup'->>'manifestSha256' = ${candidate.manifestSha256}`)).limit(1); + if (lease) return false; + if (candidate.status === "pending") { + const [run] = await tx.select({ status: heartbeatRuns.status, finishedAt: heartbeatRuns.finishedAt }) + .from(heartbeatRuns).where(eq(heartbeatRuns.id, candidate.runId)); + if (run && (!run.finishedAt || !["succeeded", "failed", "cancelled", "interrupted", "timed_out"].includes(run.status))) return false; + const [owner] = await tx.select({ id: nativeRunFinalizations.runId }).from(nativeRunFinalizations).where(and( + eq(nativeRunFinalizations.runId, candidate.runId), isNotNull(nativeRunFinalizations.leaseOwner))).limit(1); + if (owner) return false; + } + if (!input.dryRun) await tx.update(nativeHarnessBackups).set({ status: "retired", updatedAt: new Date() }) + .where(eq(nativeHarnessBackups.id, candidate.id)); + return true; +} diff --git a/server/src/services/native-runtime/native-harness-backup-stamp.ts b/server/src/services/native-runtime/native-harness-backup-stamp.ts index 865a2b853cb..3d377cc08c5 100644 --- a/server/src/services/native-runtime/native-harness-backup-stamp.ts +++ b/server/src/services/native-runtime/native-harness-backup-stamp.ts @@ -1,5 +1,7 @@ import { createHash } from "node:crypto"; -import { lstatSync, readFileSync, readdirSync, readlinkSync } from "node:fs"; +import { constants, lstatSync, readFileSync, readdirSync, readlinkSync } from "node:fs"; +import { lstat, open, readdir, readlink } from "node:fs/promises"; +import { MAX_NATIVE_HARNESS_BYTES, MAX_NATIVE_HARNESS_ENTRIES } from "./native-harness-backup-archive.js"; import { dirname, resolve } from "node:path"; import { resolvePaperclipInstanceRoot } from "../../home-paths.js"; @@ -142,12 +144,12 @@ export function createNativeHarnessBackupStamp(input: { }; } -export function verifyNativeHarnessBackupStamp( +export function readNativeHarnessBackupStamp( value: unknown, expectedProviderLeaseId: string, -): boolean { - if (!expectedProviderLeaseId) return false; - if (!value || typeof value !== "object" || Array.isArray(value)) return false; +): NativeHarnessBackupStamp | null { + if (!expectedProviderLeaseId) return null; + if (!value || typeof value !== "object" || Array.isArray(value)) return null; const stamp = value as Record; if ( stamp.schema !== "paperclip.native-harness-backup-stamp.v2" || @@ -160,69 +162,96 @@ export function verifyNativeHarnessBackupStamp( !stamp.sourceProviderLeaseId || stamp.authorizedProviderLeaseId !== expectedProviderLeaseId || typeof stamp.manifestSha256 !== "string" || - !stamp.manifestSha256.startsWith("sha256:") + !/^sha256:[a-f0-9]{64}$/.test(stamp.manifestSha256) || + !/^[a-f0-9]{64}$/.test(stamp.sessionScopeSha256) || + typeof stamp.completedAt !== "string" || !Number.isFinite(Date.parse(stamp.completedAt)) ) - return false; + return null; + return stamp as unknown as NativeHarnessBackupStamp; +} + +function matchingBackupDirectories(candidate: string, stamp: NativeHarnessBackupStamp): Array<{ path: string; sha256: string; bytes: number }> | null { + const manifestPath = resolve(candidate, "manifest.json"); + if (!isRealDirectory(candidate) || !isRealFile(manifestPath) || lstatSync(manifestPath).size > 4 * 1024 * 1024) return null; + const bytes = readFileSync(manifestPath); + if (`sha256:${createHash("sha256").update(bytes).digest("hex")}` !== stamp.manifestSha256) return null; + const manifest = JSON.parse(bytes.toString("utf8")); + if (manifest.schema !== "paperclip.native-harness-backup.v1" || manifest.normalizedSessionId !== stamp.normalizedSessionId + || manifest.runnerInstanceId !== stamp.runnerInstanceId || manifest.sourceProviderLeaseId !== stamp.sourceProviderLeaseId + || !Array.isArray(manifest.directories) || manifest.directories.length === 0) return null; + const directories: Array<{ path: string; sha256: string; bytes: number }> = []; + for (const entry of manifest.directories) { + if (!entry || typeof entry !== "object" || Array.isArray(entry) || typeof entry.name !== "string" + || !/^[A-Za-z0-9._-]+$/.test(entry.name) || [".", ".."].includes(entry.name) + || typeof entry.sha256 !== "string" || !Number.isSafeInteger(entry.bytes) || entry.bytes < 0) return null; + const path = resolve(candidate, entry.name); + if (!isRealDirectory(path)) return null; + directories.push({ path, sha256: entry.sha256, bytes: entry.bytes }); + } + return directories; +} + +function backupCandidates(stamp: NativeHarnessBackupStamp) { const root = stateRootFromDigest(stamp.sessionScopeSha256); - if (!root || !isRealDirectory(root)) return false; - const backupRoot = resolve(root, "failover-backups"); - for (const candidate of [ - resolve(backupRoot, "current"), - resolve(backupRoot, "previous"), - ]) { - const manifestPath = resolve(candidate, "manifest.json"); - if (!isRealDirectory(candidate) || !isRealFile(manifestPath)) continue; + return !root || !isRealDirectory(root) ? [] : ["current", "previous"].map(name => resolve(root, "failover-backups", name)); +} + +export function verifyNativeHarnessBackupStamp(value: unknown, expectedProviderLeaseId: string): boolean { + const stamp = readNativeHarnessBackupStamp(value, expectedProviderLeaseId); + if (!stamp) return false; + for (const candidate of backupCandidates(stamp)) { try { - const bytes = readFileSync(manifestPath); - const manifestSha256 = `sha256:${createHash("sha256").update(bytes).digest("hex")}`; - if (manifestSha256 !== stamp.manifestSha256) continue; - const manifest = JSON.parse(bytes.toString("utf8")) as Record< - string, - unknown - >; - if ( - manifest.schema !== "paperclip.native-harness-backup.v1" || - manifest.normalizedSessionId !== stamp.normalizedSessionId || - manifest.runnerInstanceId !== stamp.runnerInstanceId || - manifest.sourceProviderLeaseId !== stamp.sourceProviderLeaseId || - !Array.isArray(manifest.directories) || - manifest.directories.length === 0 - ) - continue; - let valid = true; - for (const entry of manifest.directories) { - if (!entry || typeof entry !== "object" || Array.isArray(entry)) { - valid = false; - break; - } - const declared = entry as Record; - if ( - typeof declared.name !== "string" || - !/^[A-Za-z0-9._-]+$/.test(declared.name) || - typeof declared.sha256 !== "string" || - typeof declared.bytes !== "number" - ) { - valid = false; - break; - } - const directory = resolve(candidate, declared.name); - if (!isRealDirectory(directory)) { - valid = false; - break; - } - const actual = digestDirectory(directory); - if ( - actual.sha256 !== declared.sha256 || - actual.bytes !== declared.bytes - ) { - valid = false; - break; - } + const directories = matchingBackupDirectories(candidate, stamp); + if (directories?.every(directory => { + const actual = digestDirectory(directory.path); + return actual.sha256 === directory.sha256 && actual.bytes === directory.bytes; + })) return true; + } catch {} + } + return false; +} + +export async function verifyNativeHarnessBackupStampAsync(value: unknown, expectedProviderLeaseId: string, signal: AbortSignal): Promise { + const stamp = readNativeHarnessBackupStamp(value, expectedProviderLeaseId); + if (!stamp) return false; + for (const candidate of backupCandidates(stamp)) { + signal.throwIfAborted(); + try { + const directories = matchingBackupDirectories(candidate, stamp); + if (!directories || directories.reduce((total, entry) => total + entry.bytes, 0) > MAX_NATIVE_HARNESS_BYTES) continue; + let valid = true, count = 0; + for (const directory of directories) { + const hash = createHash("sha256"); let bytes = 0; + const visit = async (current: string, relative: string): Promise => { + signal.throwIfAborted(); + const entries = (await readdir(current, { withFileTypes: true })).sort((a, b) => a.name.localeCompare(b.name)); + if (!entries.length) hash.update(`directory:${relative}\0`); + for (const entry of entries) { + if (++count > MAX_NATIVE_HARNESS_ENTRIES) throw new Error("native_harness_backup_entry_limit"); + const path = resolve(current, entry.name), name = relative ? `${relative}/${entry.name}` : entry.name; + const stats = await lstat(path); + if (stats.isDirectory()) { hash.update(`directory:${name}:${stats.mode & 0o777}\0`); await visit(path, name); } + else if (stats.isSymbolicLink()) hash.update(`symlink:${name}:${await readlink(path)}\0`); + else if (stats.isFile()) { + hash.update(`file:${name}:${stats.mode & 0o777}:${stats.size}\0`); + const handle = await open(path, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK); + try { + const before = await handle.stat(); + if (!before.isFile() || stats.ino !== before.ino || stats.dev !== before.dev || stats.size !== before.size) throw new Error("native_harness_backup_source_changed"); + for await (const chunk of handle.createReadStream({ signal, autoClose: false })) { + bytes += chunk.length; if (bytes > directory.bytes) throw new Error("native_harness_backup_size_mismatch"); hash.update(chunk); + } + const after = await handle.stat(); + if (before.size !== after.size || before.mtimeMs !== after.mtimeMs || before.ctimeMs !== after.ctimeMs) throw new Error("native_harness_backup_source_changed"); + } finally { await handle.close(); } + } else throw new Error("native_harness_backup_unsupported_entry"); + } + }; + await visit(directory.path, ""); + if (`sha256:${hash.digest("hex")}` !== directory.sha256 || bytes !== directory.bytes) { valid = false; break; } } if (valid) return true; - } catch { - // Try the previous atomically-published backup. - } + } catch { signal.throwIfAborted(); } } return false; } diff --git a/server/src/services/native-runtime/native-harness-backup-storage.integration.test.ts b/server/src/services/native-runtime/native-harness-backup-storage.integration.test.ts new file mode 100644 index 00000000000..c7c2b5a7f0d --- /dev/null +++ b/server/src/services/native-runtime/native-harness-backup-storage.integration.test.ts @@ -0,0 +1,257 @@ +import { createHash, randomUUID } from "node:crypto"; +import { chmod, mkdir, readFile, rename, stat, symlink, readlink, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { and, eq } from "drizzle-orm"; +import { agentTaskSessions, environmentLeases, heartbeatRuns, issues, nativeHarnessBackups, nativeRunFinalizations } from "@paperclipai/db"; +import { afterEach, expect, it, vi } from "vitest"; +import { describeEmbeddedPostgres, nativeRestartFixture } from "../../__tests__/helpers/native-restart.js"; +import { createLocalDiskStorageProvider } from "../../storage/local-disk-provider.js"; +import { createStorageService } from "../../storage/service.js"; +import { nativeHarnessBackupOwner, persistNativeHarnessBackup, restoreDurableNativeHarnessBackup, + verifyDurableNativeHarnessBackup, deleteRetiredNativeHarnessBackup } from "./native-harness-backup-storage.js"; +import { removeNativeHarnessBackup } from "./native-harness-backup-cleanup.js"; +import { buildNativeHarnessBackupManifest } from "./native-session-executor.js"; +import { createNativeHarnessBackupStamp, verifyNativeHarnessBackupStamp, verifyNativeHarnessBackupStampAsync } from "./native-harness-backup-stamp.js"; +import { hasLocalNativeHarnessObligations, migrateNativeHarnessBackups } from "./native-harness-backup-migration.js"; +import { nativeHarnessBackupExpirationService } from "./native-harness-backup-expiration.js"; + +describeEmbeddedPostgres("durable native provider backups", () => { + const environment = nativeRestartFixture(); + const ownedHomes: string[] = []; + afterEach(async () => { + for (const home of ownedHomes.splice(0)) await removeNativeHarnessBackup(home); + vi.unstubAllEnvs(); + }); + async function fixture(provider = "codex") { + vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "51".repeat(32)); + const f = await environment.seedRun(randomUUID()); + const { companyId, runtimeRoot, successor } = environment.current; + await f.db.update(nativeRunFinalizations).set({ controllerBootId: successor.bootId, controllerGeneration: 1, + leaseOwner: "backup-writer", leaseExpiresAt: new Date(Date.now() + 120_000) }).where(eq(nativeRunFinalizations.runId, f.runId)); + const owner = await nativeHarnessBackupOwner(f.db, { companyId, issueId: f.issueId, runId: f.runId, bootId: successor.bootId }); + const home = join(runtimeRoot, randomUUID()), root = join(home, "failover-backups", "current"); + ownedHomes.push(home); + await mkdir(join(root, provider, "sessions"), { recursive: true }); + await mkdir(join(root, "runner")); + const identity = { normalizedSessionId: randomUUID(), providerSessionId: randomUUID(), provider }; + await writeFile(join(root, provider, "sessions", "history.json"), JSON.stringify(identity), { mode: 0o600 }); + await writeFile(join(root, "runner", "runner-state.json"), JSON.stringify({ runId: f.runId, lifecycle: "suspended" }), { mode: 0o600 }); + const manifest = JSON.stringify({ schema: "paperclip.native-harness-backup.v1", ...identity, directories: [provider, "runner"] }); + await writeFile(join(root, "manifest.json"), manifest, { mode: 0o600 }); + await chmod(join(root, provider, "sessions"), 0o500); + const sessionScopeId = randomUUID(), scopeDigest = createHash("sha256").update(sessionScopeId).digest("hex"); + const storage = createStorageService(createLocalDiskStorageProvider(join(runtimeRoot, "durable-objects"))); + return { ...f, home, root, companyId, owner, scopeDigest, sessionScopeId, storage, identity, + manifestSha256: `sha256:${createHash("sha256").update(manifest).digest("hex")}` }; + } + it.each(["codex", "claude", "opencode", "grok", "new-provider"])("restores %s session bytes after deleting the entire controller home", async provider => { + const f = await fixture(provider); + await persistNativeHarnessBackup(f); + const [row] = await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest)); + expect(row.status).toBe("current"); + const object = await f.storage.getObject(f.companyId, row.reference.objectKey as string); + const parts: Buffer[] = []; for await (const part of object.stream) parts.push(part); + expect(Buffer.concat(parts).includes(Buffer.from(f.identity.providerSessionId))).toBe(false); + await removeNativeHarnessBackup(f.home); + const replacement = join(environment.current.runtimeRoot, randomUUID(), "failover-backups"); + ownedHomes.push(replacement); + expect(await restoreDurableNativeHarnessBackup({ ...f, backupRoot: replacement })).toBe(true); + expect(JSON.parse(await readFile(join(replacement, "current", provider, "sessions", "history.json"), "utf8"))).toEqual(f.identity); + expect((await stat(join(replacement, "current", provider, "sessions"))).mode & 0o777).toBe(0o500); + expect(await verifyDurableNativeHarnessBackup(f)).toBe(true); + }); + it("keeps the last published backup when object storage fails", async () => { + const f = await fixture(); await persistNativeHarnessBackup(f); + const [before] = await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest)); + const put = vi.spyOn(f.storage, "putFile").mockRejectedValue(new Error("storage unavailable")); + await expect(persistNativeHarnessBackup(f)).rejects.toThrow("storage unavailable"); put.mockRestore(); + const rows = await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest)); + expect(rows.filter(row => row.status === "current").map(row => row.id)).toEqual([before.id]); + expect(rows.filter(row => row.status === "pending")).toHaveLength(1); + expect(await readFile(join(f.root, "manifest.json"), "utf8")).toContain("native-harness-backup"); + }); + it("fences an old controller whose upload finishes after ownership changes", async () => { + const f = await fixture(); await persistNativeHarnessBackup(f); + const [before] = await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest)); + const put = f.storage.putFile.bind(f.storage); + f.storage.putFile = async input => { + const result = await put(input); + await f.db.update(nativeRunFinalizations).set({ controllerBootId: randomUUID(), controllerGeneration: 2 }) + .where(eq(nativeRunFinalizations.runId, f.runId)); + return result; + }; + await expect(persistNativeHarnessBackup(f)).rejects.toThrow("native_runner_checkpoint_owner_lost"); + const rows = await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest)); + expect(rows.filter(row => row.status === "current").map(row => row.id)).toEqual([before.id]); + }); + it("refuses publication after a pending upload was retired without demoting the current backup", async () => { + const f = await fixture(); await persistNativeHarnessBackup(f); + const [current] = await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest)); + const put = f.storage.putFile.bind(f.storage); + f.storage.putFile = async input => { + const result = await put(input); + await f.db.update(nativeHarnessBackups).set({ status: "retired" }).where(and(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest), eq(nativeHarnessBackups.status, "pending"))); + return result; + }; + await expect(persistNativeHarnessBackup(f)).rejects.toThrow("native_harness_storage_publication_retired"); + expect((await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.id, current.id)))[0].status).toBe("current"); + const [retired] = await f.db.select().from(nativeHarnessBackups).where(and(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest), eq(nativeHarnessBackups.status, "retired"))); + expect(await deleteRetiredNativeHarnessBackup(f.db, f.storage, retired.id)).toBe(true); + }); + it("rejects corrupt objects and cross-company lookups without publishing a local cache", async () => { + const f = await fixture(); await persistNativeHarnessBackup(f); + const replacement = join(environment.current.runtimeRoot, randomUUID(), "failover-backups"); + expect(await restoreDurableNativeHarnessBackup({ ...f, companyId: randomUUID(), backupRoot: replacement })).toBe(false); + const [row] = await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest)); + await f.storage.deleteObject(f.companyId, row.reference.objectKey as string); + await f.storage.putFile({ companyId: f.companyId, namespace: "native-harness-backups", objectKey: row.reference.objectKey as string, + originalFilename: null, contentType: "application/octet-stream", body: Buffer.from("corrupted") }); + await expect(restoreDurableNativeHarnessBackup({ ...f, backupRoot: replacement })).rejects.toThrow("hash_mismatch"); + await expect(stat(join(replacement, "current"))).rejects.toMatchObject({ code: "ENOENT" }); + }); + async function settle(f: { db: Awaited>["db"]; runId: string; issueId: string }, closedAt = new Date()) { + await f.db.update(heartbeatRuns).set({ status: "failed", finishedAt: closedAt }).where(eq(heartbeatRuns.id, f.runId)); + await f.db.update(nativeRunFinalizations).set({ phase: "terminal_failure", leaseOwner: null, leaseExpiresAt: null }) + .where(eq(nativeRunFinalizations.runId, f.runId)); + await f.db.update(issues).set({ status: "done", executionRunId: null, completedAt: closedAt }).where(eq(issues.id, f.issueId)); + } + it("migrates a lease-verified local backup once and keeps its selected session intact", async () => { + const f = await fixture("codex-home"); + const base = join(environment.current.runtimeRoot, randomUUID()); ownedHomes.push(base); await mkdir(base); + await rename(f.home, join(base, f.scopeDigest)); + const root = join(base, f.scopeDigest, "failover-backups", "current"); + vi.stubEnv("PAPERCLIP_RUNNER_STATE_DIR", base); + const runnerInstanceId = randomUUID(), providerLeaseId = randomUUID(); + const execution = { provider: { kind: "codex" }, binding: { companyId: f.companyId, issueId: f.issueId, + runId: f.runId, agentId: environment.current.agentId, executionWorkspaceId: randomUUID() }, + workspace: { cwd: "/workspace", repoUrl: null, repoRef: null, branchName: null }, + session: { normalizedSessionId: f.identity.normalizedSessionId, driverKind: "codex_app_server", + protocolVersion: 1, lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } } } as never; + await chmod(join(root, "codex-home", "sessions"), 0o700); + await symlink("history.json", join(root, "codex-home", "sessions", "history-link")); + await chmod(join(root, "codex-home", "sessions"), 0o500); + const manifest = buildNativeHarnessBackupManifest({ backupRoot: root, execution, runnerInstanceId, + providerSessionIdentity: { providerSessionId: f.identity.providerSessionId }, sourceProviderLeaseId: providerLeaseId }); + await writeFile(join(root, "manifest.json"), JSON.stringify(manifest)); + const stamp = createNativeHarnessBackupStamp({ manifestPath: join(root, "manifest.json"), sessionScopeId: f.sessionScopeId, + authorizedProviderLeaseId: providerLeaseId, runnerInstanceId, normalizedSessionId: f.identity.normalizedSessionId, completedAt: manifest.completedAt }); + expect(verifyNativeHarnessBackupStamp(stamp, providerLeaseId)).toBe(true); + expect(await verifyNativeHarnessBackupStampAsync(stamp, providerLeaseId, AbortSignal.timeout(10_000))).toBe(true); + await expect(verifyNativeHarnessBackupStampAsync(stamp, providerLeaseId, AbortSignal.abort(new Error("cancelled")))).rejects.toThrow("cancelled"); + await f.db.update(heartbeatRuns).set({ nativeSessionId: f.identity.normalizedSessionId, runnerInstanceId }).where(eq(heartbeatRuns.id, f.runId)); + await f.db.insert(environmentLeases).values({ companyId: f.companyId, issueId: f.issueId, heartbeatRunId: f.runId, + providerLeaseId, status: "expired", metadata: { nativeHarnessBackup: stamp } }); + const [selected] = await f.db.insert(agentTaskSessions).values({ companyId: f.companyId, agentId: environment.current.agentId, + adapterType: "paperclip_runner", taskKey: f.issueId, lastRunId: f.runId, + sessionParamsJson: { sessionId: f.identity.normalizedSessionId } }).returning(); + expect(await hasLocalNativeHarnessObligations(f.db)).toBe(true); + await expect(migrateNativeHarnessBackups(f.db, { storage: f.storage, writersStopped: true, signal: AbortSignal.timeout(10_000) })) + .rejects.toThrow("native_harness_migration_requires_offline_writers"); + const ownedRuns = await f.db.select({ id: heartbeatRuns.id, issueId: heartbeatRuns.nativeIssueId }).from(heartbeatRuns) + .where(eq(heartbeatRuns.companyId, f.companyId)); + for (const run of ownedRuns) await settle({ ...f, runId: run.id, issueId: run.issueId! }); + await f.db.update(nativeRunFinalizations).set({ recoveryState: "blocked" }).where(eq(nativeRunFinalizations.runId, f.runId)); + const [preservedOwner] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)); + const [queued] = await f.db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: environment.current.agentId, + status: "queued", invocationSource: "on_demand", issueId: f.issueId }).returning(); + await f.db.update(issues).set({ executionRunId: queued.id }).where(eq(issues.id, f.issueId)); + expect(await migrateNativeHarnessBackups(f.db, { storage: f.storage, writersStopped: true, signal: AbortSignal.timeout(10_000) })).toBe(1); + expect(await migrateNativeHarnessBackups(f.db, { storage: f.storage, writersStopped: true, signal: AbortSignal.timeout(10_000) })).toBe(1); + expect(await hasLocalNativeHarnessObligations(f.db)).toBe(false); + expect(await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest))).toHaveLength(1); + expect((await f.db.select().from(agentTaskSessions).where(eq(agentTaskSessions.id, selected.id)))[0]).toEqual(selected); + expect((await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0]).toEqual(preservedOwner); + expect((await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, queued.id)))[0]).toEqual(queued); + expect((await f.db.select().from(issues).where(eq(issues.id, f.issueId)))[0].executionRunId).toBe(queued.id); + expect(JSON.parse(await readFile(join(root, "manifest.json"), "utf8"))).toEqual(manifest); + const restored = join(base, "replacement"); + expect(await restoreDurableNativeHarnessBackup({ ...f, backupRoot: restored })).toBe(true); + expect(await readlink(join(restored, "current", "codex-home", "sessions", "history-link"))).toBe("history.json"); + const changedManifest = JSON.stringify({ ...manifest, completedAt: new Date(Date.now() + 1_000).toISOString() }); + await writeFile(join(root, "manifest.json"), changedManifest); + await expect(migrateNativeHarnessBackups(f.db, { storage: f.storage, writersStopped: true, signal: AbortSignal.timeout(10_000) })) + .rejects.toThrow("native_harness_migration_durable_source_conflict"); + expect(await readFile(join(root, "manifest.json"), "utf8")).toBe(changedManifest); + expect(await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest))).toHaveLength(1); + }); + it("preserves orphaned local backup files without blocking other enrollment", async () => { + const f = await fixture(); + const base = join(environment.current.runtimeRoot, randomUUID()); ownedHomes.push(base); await mkdir(base); + await rename(f.home, join(base, f.scopeDigest)); + vi.stubEnv("PAPERCLIP_RUNNER_STATE_DIR", base); + const root = join(base, f.scopeDigest, "failover-backups", "current"); + const orphan = JSON.stringify({ runId: randomUUID(), lifecycle: "suspended" }); + await writeFile(join(root, "runner", "runner-state.json"), orphan); + const ownedRuns = await f.db.select({ id: heartbeatRuns.id, issueId: heartbeatRuns.nativeIssueId }).from(heartbeatRuns) + .where(eq(heartbeatRuns.companyId, f.companyId)); + for (const run of ownedRuns) await settle({ ...f, runId: run.id, issueId: run.issueId! }); + expect(await migrateNativeHarnessBackups(f.db, { storage: f.storage, writersStopped: true, signal: AbortSignal.timeout(10_000) })).toBe(0); + expect(await readFile(join(root, "runner", "runner-state.json"), "utf8")).toBe(orphan); + expect(await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest))).toHaveLength(0); + }); + it("retains live work and retries an uncertain object deletion through the existing startup retention pass", async () => { + const f = await fixture(); await persistNativeHarnessBackup(f); + const old = new Date(Date.now() - 10 * 86_400_000); + await f.db.update(nativeHarnessBackups).set({ createdAt: old }).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest)); + const retention = nativeHarnessBackupExpirationService(f.db, { storage: f.storage }); + expect((await retention.tickCheckpoints()).expired).toBe(0); + await settle(f, old); + const drop = vi.spyOn(f.storage, "deleteObject").mockRejectedValueOnce(new Error("delete reply lost")); + expect((await retention.tickCheckpoints()).failed).toBe(1); + expect((await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest)))[0].status).toBe("retired"); + expect((await retention.tickCheckpoints()).expired).toBe(1); + expect(await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest))).toHaveLength(0); + expect(drop).toHaveBeenCalledTimes(2); + }); + it("expires obsolete objects without clearing an open conversation or its exact lease backup", async () => { + const f = await fixture(); await persistNativeHarnessBackup(f); + const [obsolete] = await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest)); + await writeFile(join(f.root, "manifest.json"), JSON.stringify({ revision: 2 })); + await persistNativeHarnessBackup(f); + const old = new Date(Date.now() - 10 * 86_400_000); + await f.db.update(nativeHarnessBackups).set({ createdAt: old, updatedAt: old }).where(eq(nativeHarnessBackups.id, obsolete.id)); + const [selected] = await f.db.insert(agentTaskSessions).values({ companyId: f.companyId, agentId: environment.current.agentId, + adapterType: "paperclip_runner", taskKey: f.issueId, lastRunId: f.runId }).returning(); + const [lease] = await f.db.insert(environmentLeases).values({ companyId: f.companyId, issueId: f.issueId, + heartbeatRunId: f.runId, status: "pending_cleanup", metadata: { nativeHarnessBackup: { + sessionScopeSha256: f.scopeDigest, manifestSha256: obsolete.manifestSha256 } } }).returning(); + const retention = nativeHarnessBackupExpirationService(f.db, { storage: f.storage }); + expect((await retention.tickCheckpoints()).expired).toBe(0); + await f.db.update(environmentLeases).set({ status: "expired" }).where(eq(environmentLeases.id, lease.id)); + expect((await retention.tickCheckpoints({ dryRun: true })).expired).toBe(0); + expect(await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.id, obsolete.id))).toHaveLength(1); + expect((await retention.tickCheckpoints()).expired).toBe(1); + const rows = await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest)); + expect(rows).toHaveLength(1); expect(rows[0].status).toBe("current"); + expect((await f.db.select().from(agentTaskSessions).where(eq(agentTaskSessions.id, selected.id)))[0]).toEqual(selected); + expect((await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0].status).toBe("running"); + }); + it("retains an in-flight pending upload and removes it once its owner settles", async () => { + const f = await fixture(); await persistNativeHarnessBackup(f); + const put = vi.spyOn(f.storage, "putFile").mockRejectedValueOnce(new Error("upload unavailable")); + await expect(persistNativeHarnessBackup(f)).rejects.toThrow("upload unavailable"); put.mockRestore(); + const rows = await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest)); + const pending = rows.find(row => row.status === "pending")!; + const old = new Date(Date.now() - 10 * 86_400_000); + await f.db.update(nativeHarnessBackups).set({ createdAt: old, updatedAt: old }).where(eq(nativeHarnessBackups.id, pending.id)); + const retention = nativeHarnessBackupExpirationService(f.db, { storage: f.storage }); + expect((await retention.tickCheckpoints()).expired).toBe(0); + await settle(f); + expect((await retention.tickCheckpoints()).expired).toBe(1); + const remaining = await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest)); + expect(remaining).toHaveLength(1); expect(remaining[0].status).toBe("current"); + }); + it("keeps deleted-source objects unchanged during a retention dry run", async () => { + const f = await fixture(); await persistNativeHarnessBackup(f); + const old = new Date(Date.now() - 10 * 86_400_000); + await f.db.update(nativeHarnessBackups).set({ createdAt: old, runId: randomUUID() }) + .where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest)); + const [before] = await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest)); + const retention = nativeHarnessBackupExpirationService(f.db, { storage: f.storage }); + expect((await retention.tickCheckpoints({ dryRun: true })).expired).toBe(0); + expect((await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.id, before.id)))[0]).toEqual(before); + expect(await verifyDurableNativeHarnessBackup(f)).toBe(true); + expect((await retention.tickCheckpoints()).expired).toBe(1); + expect(await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.id, before.id))).toHaveLength(0); + }); +}); diff --git a/server/src/services/native-runtime/native-harness-backup-storage.ts b/server/src/services/native-runtime/native-harness-backup-storage.ts new file mode 100644 index 00000000000..5bf06d58037 --- /dev/null +++ b/server/src/services/native-runtime/native-harness-backup-storage.ts @@ -0,0 +1,228 @@ +import { publishNativeHarnessBackup } from "./native-harness-backup-publication.js"; +import { createCipheriv, createDecipheriv, createHash, randomBytes, randomUUID } from "node:crypto"; +import { createReadStream, createWriteStream } from "node:fs"; +import { mkdtemp, readFile } from "node:fs/promises"; +import { join } from "node:path"; +import os from "node:os"; +import { pipeline } from "node:stream/promises"; +import { Transform } from "node:stream"; +import { and, eq, inArray, sql } from "drizzle-orm"; +import { heartbeatRuns, issues, nativeHarnessBackups, nativeRunFinalizations, type Db } from "@paperclipai/db"; +import { createLocalEncryptedCipher } from "../../secrets/local-encrypted-provider.js"; +import type { StorageService } from "../../storage/types.js"; +import { assertNativeRunnerCheckpointOwner, type NativeRunnerCheckpointOwner } from "./native-runner-checkpoint.js"; +import { loadConfig } from "../../config.js"; +import { getStorageService } from "../../storage/index.js"; +import { readNativeHarnessBackupStamp, verifyNativeHarnessBackupStamp } from "./native-harness-backup-stamp.js"; +import { captureNativeHarnessArchive, restoreNativeHarnessArchive, MAX_NATIVE_HARNESS_OBJECT_BYTES } from "./native-harness-backup-archive.js"; +import { removeNativeHarnessBackup } from "./native-harness-backup-cleanup.js"; +import { offlineMigrationExecutionWhere } from "../offline-migration-execution.js"; + +const SCHEMA = "paperclip.native-harness-object.v1"; +type Binding = { companyId: string; scopeDigest: string; id: string }; +type Reference = Binding & { + schema: typeof SCHEMA; objectKey: string; sha256: string; byteSize: number; + material: Record; manifestSha256: string; +}; +type Row = typeof nativeHarnessBackups.$inferSelect; +type OfflineOwner = { kind: "offline_migration"; companyId: string; issueId: string; runId: string; + writersStopped: true; terminalFailureOnly?: true }; +const fail = (reason: string): never => { throw new Error(`native_harness_storage_${reason}`); }; +const aad = (binding: Binding) => Buffer.from(JSON.stringify([SCHEMA, binding.companyId, binding.scopeDigest, binding.id])); +const scope = (binding: Pick) => and( + eq(nativeHarnessBackups.companyId, binding.companyId), eq(nativeHarnessBackups.scopeDigest, binding.scopeDigest)); + +export function durableNativeHarnessStorage(): StorageService | null { + if (loadConfig().storageProvider === "s3") return getStorageService(); + if (process.env.PAPERCLIP_EPHEMERAL_SERVER === "true") fail("durable_storage_required"); + return null; +} + +export async function verifyRecoverableNativeHarnessBackup(input: { + db: Db; companyId: string; stamp: unknown; providerLeaseId: string; +}): Promise { + const stamp = readNativeHarnessBackupStamp(input.stamp, input.providerLeaseId); + if (!stamp) return false; + const storage = durableNativeHarnessStorage(); + if (!storage) return verifyNativeHarnessBackupStamp(stamp, input.providerLeaseId); + const durable = await verifyDurableNativeHarnessBackup({ db: input.db, storage, companyId: input.companyId, + scopeDigest: stamp.sessionScopeSha256, manifestSha256: stamp.manifestSha256, signal: AbortSignal.timeout(300_000) }); + return durable || process.env.PAPERCLIP_EPHEMERAL_SERVER !== "true" && verifyNativeHarnessBackupStamp(stamp, input.providerLeaseId); +} +function reference(row: Row): Reference { + const ref = row.reference as Reference; + if (ref.schema !== SCHEMA || ref.id !== row.id || ref.companyId !== row.companyId || ref.scopeDigest !== row.scopeDigest + || !/^[a-f0-9]{64}$/.test(ref.scopeDigest) || !/^[a-f0-9]{64}$/.test(ref.sha256) + || ref.manifestSha256 !== row.manifestSha256 || !/^sha256:[a-f0-9]{64}$/.test(ref.manifestSha256) + || ref.objectKey !== `${row.companyId}/native-harness-backups/${row.scopeDigest}/${row.id}.pchn` + || !Number.isSafeInteger(ref.byteSize) || ref.byteSize <= 0 || ref.byteSize > MAX_NATIVE_HARNESS_OBJECT_BYTES + || !ref.material || typeof ref.material !== "object") fail("reference_invalid"); + return ref; +} +async function digest(file: string, signal?: AbortSignal) { + const hash = createHash("sha256"); let byteSize = 0; + for await (const bytes of createReadStream(file, { signal })) { hash.update(bytes); byteSize += bytes.length; } + return { sha256: hash.digest("hex"), byteSize }; +} +async function receive(storage: StorageService, ref: Reference, target?: string, signal?: AbortSignal) { + const object = await storage.getObject(ref.companyId, ref.objectKey, { signal }); + const hash = createHash("sha256"); let byteSize = 0; + const observed = new Transform({ transform(bytes, _encoding, callback) { + hash.update(bytes); byteSize += bytes.length; + callback(byteSize > ref.byteSize ? new Error("native_harness_storage_size_mismatch") : null, bytes); + } }); + try { + if (target) await pipeline(object.stream, observed, createWriteStream(target, { flags: "wx", mode: 0o600 }), { signal }); + else for await (const bytes of object.stream) { + signal?.throwIfAborted(); hash.update(bytes); byteSize += bytes.length; + if (byteSize > ref.byteSize) fail("size_mismatch"); + } + if (byteSize !== ref.byteSize || hash.digest("hex") !== ref.sha256) fail("hash_mismatch"); + } finally { object.stream.destroy(); } +} + +export async function nativeHarnessBackupOwner(db: Db, input: { companyId: string; issueId: string; runId: string; bootId: string }) { + const [row] = await db.select().from(nativeRunFinalizations).where(and( + eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.issueId, input.issueId), + eq(nativeRunFinalizations.runId, input.runId), eq(nativeRunFinalizations.controllerBootId, input.bootId))); + if (!row?.leaseOwner) fail("owner_missing"); + return { ...input, authorityRunId: input.runId, leaseOwner: row!.leaseOwner!, generation: row!.controllerGeneration } satisfies NativeRunnerCheckpointOwner; +} +async function fence(tx: Parameters[0]>[0], owner: NativeRunnerCheckpointOwner | OfflineOwner) { + const [task] = await tx.select({ executionRunId: issues.executionRunId }).from(issues) + .where(and(eq(issues.companyId, owner.companyId), eq(issues.id, owner.issueId))).for("share"); + const authorityRunId = "kind" in owner ? owner.runId : owner.authorityRunId; + if (!task || !("kind" in owner) && task.executionRunId && task.executionRunId !== authorityRunId) fail("owner_superseded"); + if ("kind" in owner) { + if (!owner.writersStopped) fail("offline_writers_required"); + const [active] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(offlineMigrationExecutionWhere()).limit(1); + if (active) fail("offline_writers_required"); + const [run] = await tx.select({ finishedAt: heartbeatRuns.finishedAt, status: heartbeatRuns.status }).from(heartbeatRuns).where(and( + eq(heartbeatRuns.id, owner.runId), eq(heartbeatRuns.companyId, owner.companyId), + eq(heartbeatRuns.nativeIssueId, owner.issueId), eq(heartbeatRuns.runtimeMode, "native"))).for("share"); + const [coordinator] = await tx.select({ leaseOwner: nativeRunFinalizations.leaseOwner, + leaseExpiresAt: nativeRunFinalizations.leaseExpiresAt, phase: nativeRunFinalizations.phase }) + .from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, owner.runId)).for("share"); + const [clock] = await tx.execute<{ now: Date }>(sql`select clock_timestamp() as now`); + if (!run?.finishedAt || !["succeeded", "failed", "cancelled", "interrupted", "timed_out"].includes(run.status) + || !coordinator || coordinator.leaseOwner && (!coordinator.leaseExpiresAt || coordinator.leaseExpiresAt > new Date(clock.now)) + || owner.terminalFailureOnly && (run.status !== "failed" || coordinator.phase !== "terminal_failure")) { + fail("offline_owner_unsettled"); + } + return; + } + await assertNativeRunnerCheckpointOwner(tx, owner, false); +} + +export async function persistNativeHarnessBackup(input: { + db: Db; storage: StorageService; owner: NativeRunnerCheckpointOwner | OfflineOwner; + scopeDigest: string; root: string; signal?: AbortSignal; +}): Promise { + if (!/^[a-f0-9]{64}$/.test(input.scopeDigest)) fail("scope_invalid"); + const manifestBytes = await readFile(join(input.root, "manifest.json")); + const manifestSha256 = `sha256:${createHash("sha256").update(manifestBytes).digest("hex")}`; + const binding = { companyId: input.owner.companyId, scopeDigest: input.scopeDigest, id: randomUUID() }; + const scratch = await mkdtemp(join(os.tmpdir(), "pc-harness-upload-")); + const key = randomBytes(32); + try { + const archive = join(scratch, "archive"), encrypted = join(scratch, "encrypted"); + await captureNativeHarnessArchive(input.root, archive, input.signal); + const plaintext = await digest(archive, input.signal), iv = randomBytes(12); + const cipher = createCipheriv("aes-256-gcm", key, iv); cipher.setAAD(aad(binding)); + await pipeline(createReadStream(archive), cipher, createWriteStream(encrypted, { flags: "wx", mode: 0o600 }), { signal: input.signal }); + const ref: Reference = { schema: SCHEMA, ...binding, manifestSha256, + objectKey: `${binding.companyId}/native-harness-backups/${binding.scopeDigest}/${binding.id}.pchn`, + ...await digest(encrypted, input.signal), + material: createLocalEncryptedCipher("write").encrypt(JSON.stringify({ schema: SCHEMA, ...binding, + key: key.toString("base64"), iv: iv.toString("base64"), tag: cipher.getAuthTag().toString("base64"), plaintext })) }; + if (ref.byteSize > MAX_NATIVE_HARNESS_OBJECT_BYTES) fail("size_limit_exceeded"); + await input.db.transaction(async tx => { + await fence(tx, input.owner); + await tx.insert(nativeHarnessBackups).values({ ...binding, issueId: input.owner.issueId, runId: input.owner.runId, + manifestSha256, reference: ref }); + }); + await input.storage.putFile({ companyId: ref.companyId, namespace: "native-harness-backups", objectKey: ref.objectKey, + originalFilename: null, contentType: "application/octet-stream", body: createReadStream(encrypted), + byteSize: ref.byteSize, sha256: ref.sha256, signal: input.signal }); + await receive(input.storage, ref, undefined, input.signal); + await input.db.transaction(async tx => { + await fence(tx, input.owner); + await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${`${ref.companyId}:${ref.scopeDigest}`}, 0))`); + const [pending] = await tx.select({ status: nativeHarnessBackups.status }).from(nativeHarnessBackups) + .where(eq(nativeHarnessBackups.id, ref.id)).for("update"); + if (pending?.status !== "pending") fail("publication_retired"); + const [successor] = await tx.select({ id: nativeHarnessBackups.id }).from(nativeHarnessBackups) + .innerJoin(heartbeatRuns, eq(heartbeatRuns.id, nativeHarnessBackups.runId)).where(and(scope(ref), + eq(nativeHarnessBackups.status, "current"), sql`${nativeHarnessBackups.runId} <> ${input.owner.runId}::uuid`, + sql`${heartbeatRuns.createdAt} >= (select created_at from heartbeat_runs where id = ${input.owner.runId}::uuid)`)).limit(1); + if (successor) fail("owner_superseded"); + await fence(tx, input.owner); + await tx.update(nativeHarnessBackups).set({ status: "superseded", updatedAt: new Date() }) + .where(and(scope(ref), eq(nativeHarnessBackups.status, "current"))); + await tx.update(nativeHarnessBackups).set({ status: "current", updatedAt: new Date() }) + .where(and(eq(nativeHarnessBackups.id, ref.id), eq(nativeHarnessBackups.status, "pending"))); + }); + } finally { key.fill(0); await removeNativeHarnessBackup(scratch); } +} + +export async function restoreDurableNativeHarnessBackup(input: { + db: Db; storage: StorageService; companyId: string; scopeDigest: string; + backupRoot: string; signal?: AbortSignal; +}): Promise { + const [row] = await input.db.select().from(nativeHarnessBackups).where(and(scope(input), eq(nativeHarnessBackups.status, "current"))); + if (!row) return false; + const ref = reference(row); + const wrapped = JSON.parse(createLocalEncryptedCipher("read").decrypt(ref.material)); + if (wrapped.schema !== SCHEMA || wrapped.id !== ref.id || wrapped.companyId !== ref.companyId + || wrapped.scopeDigest !== ref.scopeDigest) fail("binding_mismatch"); + const current = join(input.backupRoot, "current"); + const localManifest = await readFile(join(current, "manifest.json")).catch((error: NodeJS.ErrnoException) => { + if (error.code === "ENOENT") return null; throw error; + }); + if (localManifest && `sha256:${createHash("sha256").update(localManifest).digest("hex")}` === ref.manifestSha256) return true; + const scratch = await mkdtemp(join(os.tmpdir(), "pc-harness-restore-")); + const key = Buffer.from(wrapped.key, "base64"); + try { + if (key.length !== 32) fail("key_invalid"); + const encrypted = join(scratch, "encrypted"), archive = join(scratch, "archive"); + await receive(input.storage, ref, encrypted, input.signal); + const decipher = createDecipheriv("aes-256-gcm", key, Buffer.from(wrapped.iv, "base64")); + decipher.setAAD(aad(ref)); decipher.setAuthTag(Buffer.from(wrapped.tag, "base64")); + await pipeline(createReadStream(encrypted), decipher, createWriteStream(archive, { flags: "wx", mode: 0o600 }), { signal: input.signal }); + const actual = await digest(archive, input.signal); + if (actual.sha256 !== wrapped.plaintext?.sha256 || actual.byteSize !== wrapped.plaintext?.byteSize) fail("plaintext_mismatch"); + const restored = join(scratch, "restored"); + await restoreNativeHarnessArchive(archive, restored, input.signal); + const manifest = await readFile(join(restored, "manifest.json")); + if (`sha256:${createHash("sha256").update(manifest).digest("hex")}` !== ref.manifestSha256) fail("manifest_mismatch"); + const { mkdir, cp } = await import("node:fs/promises"); + await mkdir(input.backupRoot, { recursive: true, mode: 0o700 }); + const pending = join(input.backupRoot, `.durable-${randomUUID()}`); + try { + await cp(restored, pending, { recursive: true, preserveTimestamps: true, verbatimSymlinks: true, errorOnExist: true, force: false }); + await publishNativeHarnessBackup({ db: input.db, companyId: input.companyId, scopeDigest: input.scopeDigest, + backupRoot: input.backupRoot, pendingRoot: pending }); + } finally { await removeNativeHarnessBackup(pending); } + return true; + } finally { key.fill(0); await removeNativeHarnessBackup(scratch); } +} + +export async function verifyDurableNativeHarnessBackup(input: { + db: Db; storage: StorageService; companyId: string; scopeDigest: string; manifestSha256: string; signal?: AbortSignal; +}): Promise { + const [row] = await input.db.select().from(nativeHarnessBackups).where(and(scope(input), + eq(nativeHarnessBackups.manifestSha256, input.manifestSha256), inArray(nativeHarnessBackups.status, ["current", "superseded"]))).limit(1); + if (!row) return false; + await receive(input.storage, reference(row), undefined, input.signal); + return true; +} + +export async function deleteRetiredNativeHarnessBackup(db: Db, storage: StorageService, id: string, signal?: AbortSignal) { + const [row] = await db.select().from(nativeHarnessBackups).where(and( + eq(nativeHarnessBackups.id, id), eq(nativeHarnessBackups.status, "retired"))); + if (!row) return false; + const ref = reference(row); + await storage.deleteObject(ref.companyId, ref.objectKey, { allVersions: true, signal }); + await db.delete(nativeHarnessBackups).where(and(eq(nativeHarnessBackups.id, id), eq(nativeHarnessBackups.status, "retired"))); + return true; +} diff --git a/server/src/services/native-runtime/native-runner-checkpoint.ts b/server/src/services/native-runtime/native-runner-checkpoint.ts index 0a678a53bea..8c568cb6772 100644 --- a/server/src/services/native-runtime/native-runner-checkpoint.ts +++ b/server/src/services/native-runtime/native-runner-checkpoint.ts @@ -6,6 +6,8 @@ import { createLocalEncryptedCipher } from "../../secrets/local-encrypted-provid import { legacyControllerBootId } from "../legacy-controller-lease.js"; type Binding = { companyId: string; issueId: string; runId: string }; +export const NATIVE_CHECKPOINT_HEADER_INSPECTION_MAX_BYTES = 1024 * 1024; +export const NATIVE_CHECKPOINT_INSPECTION_TOTAL_MAX_BYTES = 8 * 1024 * 1024; export type NativeRunnerCheckpointOwner = Binding & { bootId: string; leaseOwner: string; generation: number; authorityRunId: string }; function encryptParts(binding: Binding, records: ReadonlyMap) { if (!records.size) return []; @@ -16,6 +18,11 @@ function encryptParts(binding: Binding, records: ReadonlyMap) { })); } +export async function insertMigratedNativeRunnerCheckpoint(tx: CheckpointTransaction, binding: Binding, checkpoint: DurablePrpCheckpoint) { + if (checkpoint.runId !== binding.runId) throw new Error("native_runner_checkpoint_binding_mismatch"); + await tx.insert(nativeRunnerCheckpoints).values(encryptParts(binding, checkpoint.records)); +} + function decryptPart(cipher: ReturnType, binding: Binding, part: string, material: Record): string { const envelope = JSON.parse(cipher.decrypt(material)); diff --git a/server/src/services/native-runtime/native-session-executor.test.ts b/server/src/services/native-runtime/native-session-executor.test.ts index e3611e34991..2de774f90ea 100644 --- a/server/src/services/native-runtime/native-session-executor.test.ts +++ b/server/src/services/native-runtime/native-session-executor.test.ts @@ -1632,6 +1632,18 @@ describe("native provider usage normalization", () => { it("does not treat a protocol-default price as free usage", () => { expect(nativeUsageCostUsd({ runDelta: { inputTokens: 20, outputTokens: 10, providerCostUsd: 0 } })).toBeUndefined(); }); + it("attributes Grok API usage while preserving managed billing identity", () => { + const provider = { kind: "acpx", agent: "grok", model: "grok-4.7" } as NativeExecutionInput["provider"]; + expect(nativeUsageBiller(provider)).toBe("xai"); + expect(resolveNativeBilling(provider)).toEqual({ provider: "xai", biller: "xai", billingType: "unknown" }); + expect(resolveNativeBilling(provider, { XAI_API_KEY: "fixture" })) + .toEqual({ provider: "xai", biller: "xai", billingType: "metered_api" }); + const subscription = { provider: "xai", biller: "xai", billingType: "subscription_included" as const }; + expect(resolveNativeBilling(provider, { XAI_API_KEY: "fixture" }, subscription)).toEqual(subscription); + expect(nativeUsageCostUsd({ runDelta: { inputTokens: 30787, outputTokens: 5763, cacheReadTokens: 475008, + cacheWriteTokens: 0, providerCostUsd: 0.336842 } }, provider)).toBe(0.336842); + }); + it("reports the actual runtime billing identity", () => { expect(resolveNativeBilling({ kind: "opencode", model: "openrouter/deepseek/test", permissionMode: "allow" })) .toEqual({ provider: "deepseek", biller: "openrouter", billingType: "unknown" }); diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index a2600506bd0..7c85342d3bf 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -4,6 +4,7 @@ import { providerFailureFromError } from "../../vendor/paperclip-runner/index.js import { nativeCancellationHasUnsettledOwner, notifyNativeCancellation, waitForNativeCancellation } from "./native-cancellation-delivery.js"; import { readClaimedNativeStopRecovery } from "./native-cancellation-request.js"; import { publishNativeHarnessBackup } from "./native-harness-backup-publication.js"; +import { durableNativeHarnessStorage, nativeHarnessBackupOwner, persistNativeHarnessBackup, restoreDurableNativeHarnessBackup } from "./native-harness-backup-storage.js"; import { rehydratePinnedSkillAssets } from "./runtime-context.js"; import { createNativeUsageDelivery } from "./native-usage-delivery.js"; import { createNativeRecoveryUsageDelivery } from "./native-recovery-usage-delivery.js"; @@ -5964,6 +5965,7 @@ function nativeSessionConfigDigest( execution: NativeExecutionInput, executionTargetKind: "local" | "remote" = "local", legacyProjectlessRunId?: string, + recordedToolContractFingerprint?: string, ): string { const executionLocation = { executionKind: "local_process", @@ -5999,13 +6001,25 @@ function nativeSessionConfigDigest( // changes so an older thread is rotated instead of falsely resuming // without newly required tools. nativeToolContractFingerprint: - nativeToolContractFingerprintForTarget(executionTargetKind, execution.provider.kind), + recordedToolContractFingerprint ?? nativeToolContractFingerprintForTarget(executionTargetKind, execution.provider.kind), runtimeContract: nativeRuntimeContractForProvider(execution.provider), }), ) .digest("hex")}`; } +export function nativeControllerMigrationDescriptor(execution: NativeExecutionInput, recordedToolContractFingerprint: unknown) { + if (typeof recordedToolContractFingerprint !== "string" || !/^sha256:[a-f0-9]{64}$/.test(recordedToolContractFingerprint)) { + throw new Error("native_controller_migration_tool_contract_missing"); + } + return { + root: scopedRunnerdStateRoot(execution), + supervisorPath: resolve(resolvePaperclipInstanceRoot(), "runtime", "paperclip-runner", "sessions", + `${createHash("sha256").update(nativeSessionScopeKey(execution)).digest("hex")}.json`), + configDigest: nativeSessionConfigDigest(execution, "remote", undefined, recordedToolContractFingerprint), + }; +} + function idleWarmNativeSessionOwnerRunId(input: { execution: NativeExecutionInput; runnerExecutionTarget?: AdapterExecutionTarget | null; @@ -6027,6 +6041,7 @@ async function restorePerTurnRunnerdController(input: { if (input.restartRecovery || input.runnerExecutionTarget?.kind !== "remote" || input.execution.session.lifecyclePolicy.mode !== "per_turn") return; const root = scopedRunnerdStateRoot(input.execution); + await restoreDurableHarnessForExecution(input.db, input.execution, root); const identity = readRunnerdDurableIdentity(root); const reference = identity ?? readRunnerdDurableIdentity(root, true); if (!reference) { @@ -6091,6 +6106,7 @@ async function restoreWarmRunnerdController(input: { if (input.restartRecovery || input.runnerExecutionTarget?.kind !== "remote" || input.execution.session.lifecyclePolicy.mode !== "warm") return; const root = scopedRunnerdStateRoot(input.execution); + await restoreDurableHarnessForExecution(input.db, input.execution, root); const identity = readRunnerdDurableIdentity(root); const reference = identity ?? readRunnerdDurableIdentity(root, true); if (!idleWarmNativeSessionOwnerRunId(input) && (!reference || !existsSync(nativeSessionCheckpointPath(input.execution)))) { @@ -6599,6 +6615,14 @@ function harnessBackupRoot(root: string): string { return resolve(root, "failover-backups"); } +async function restoreDurableHarnessForExecution(db: Db, execution: NativeExecutionInput, root: string) { + const storage = durableNativeHarnessStorage(); + if (!storage) return; + await restoreDurableNativeHarnessBackup({ db, storage, companyId: execution.binding.companyId, + scopeDigest: createHash("sha256").update(nativeSessionScopeKey(execution)).digest("hex"), + backupRoot: harnessBackupRoot(root), signal: AbortSignal.timeout(300_000) }); +} + function harnessBackupCandidates(root: string): string[] { const backupRoot = harnessBackupRoot(root); return [resolve(backupRoot, "current"), resolve(backupRoot, "previous")]; @@ -10987,6 +11011,9 @@ export function resolveNativeBilling(provider: NativeExecutionInput["provider"], const biller = nativeUsageBiller(provider); return { provider: biller, biller, billingType: "unknown" }; } + if (provider.kind === "acpx" && provider.agent === "grok") { + return { provider: "xai", biller: "xai", billingType: env.XAI_API_KEY ? "metered_api" : "unknown" }; + } if (provider.kind === "acpx" && provider.agent === "claude") { if (env.CLAUDE_CODE_USE_BEDROCK === "1") return { provider: "anthropic", biller: "aws_bedrock", billingType: "metered_api" }; return { provider: "anthropic", biller: "anthropic", billingType: env.ANTHROPIC_API_KEY ? "metered_api" : env.CLAUDE_CODE_OAUTH_TOKEN ? "subscription_included" : "unknown" }; @@ -14069,6 +14096,11 @@ async function createRunnerdBackendWithinSessionClaim( !remoteRunnerFilesystemRoot ) return; + const durableStorage = durableNativeHarnessStorage(); + if (durableStorage && !input.controllerPersistence) throw new Error("native_harness_storage_owner_missing"); + const backupOwner = durableStorage ? await nativeHarnessBackupOwner(input.db, { ...input.execution.binding, + runId: sessionToolAuthorityEpochs.get(sessionScopeId)?.runId ?? input.execution.binding.runId, + bootId: input.controllerPersistence!.bootId }) : null; // Transport release also runs when provider bootstrap failed. In that case // runnerd has no durable provider identity (and may not have created the // provider persistence directory at all), so attempting a failover backup @@ -14174,6 +14206,11 @@ async function createRunnerdBackendWithinSessionClaim( }); renameSync(temporaryManifest, manifestPath); + if (durableStorage && backupOwner) { + await persistNativeHarnessBackup({ db: input.db, storage: durableStorage, owner: backupOwner, + scopeDigest: createHash("sha256").update(sessionScopeId).digest("hex"), root: pendingRoot, + signal: AbortSignal.timeout(300_000) }); + } await publishNativeHarnessBackup({ db: input.db, companyId: input.execution.binding.companyId, scopeDigest: createHash("sha256").update(sessionScopeId).digest("hex"), backupRoot, pendingRoot, diff --git a/server/src/services/native-runtime/native-session-resume.ts b/server/src/services/native-runtime/native-session-resume.ts index 9e4fe58b486..fcf2a3e0856 100644 --- a/server/src/services/native-runtime/native-session-resume.ts +++ b/server/src/services/native-runtime/native-session-resume.ts @@ -225,7 +225,7 @@ type NativeSessionResumeSelection = { agentId: string; issueId: string; normalizedSessionId: string; - currentRunId: string; + currentRunId?: string; beforeCreatedAt: Date; /** Recovery discovery must retain an ineligible newest progress barrier. */ includeProgressBarrier?: boolean; @@ -306,7 +306,7 @@ async function selectNativeSessionResumeBarrier(db: Db, input: NativeSessionResu .from(heartbeatRuns) .where( and( - ne(heartbeatRuns.id, input.currentRunId), + input.currentRunId ? ne(heartbeatRuns.id, input.currentRunId) : undefined, eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.agentId, input.agentId), eq(heartbeatRuns.nativeIssueId, input.issueId), diff --git a/server/src/services/native-runtime/native-usage-accounting.ts b/server/src/services/native-runtime/native-usage-accounting.ts index 8ef04b1da17..e4e95f12489 100644 --- a/server/src/services/native-runtime/native-usage-accounting.ts +++ b/server/src/services/native-runtime/native-usage-accounting.ts @@ -51,6 +51,7 @@ export function nativeUsageBiller(provider: NativeExecutionInput["provider"]): s if (provider.agent === "cursor") return "cursor"; if (provider.agent === "copilot") return "github"; if (provider.agent === "pi") return "openrouter"; + if (provider.agent === "grok") return "xai"; } return "openai"; } diff --git a/server/src/services/remote-workspaces.integration.test.ts b/server/src/services/remote-workspaces.integration.test.ts index 0ef9e7ccdf3..566934be7dd 100644 --- a/server/src/services/remote-workspaces.integration.test.ts +++ b/server/src/services/remote-workspaces.integration.test.ts @@ -387,7 +387,8 @@ suite("remote workspace lifecycle with real Postgres and independent filesystem } }, 60_000); - it("prepares, saves and restores task files without a controller checkout or original allocation", async () => { + it.each(["same root", "different root", "destroyed allocation", "occupied root"] as const)( + "preserves the published workspace path and files on a cold restore (%s)", async mode => { const reserved = await reserve({ kind: "empty" }), first = await target("sandbox-A"); const prepared = await prepareRemoteWorkspace({ db, storage, workspace: reserved, runId: ids.run, lease: await lease(), target: first }); expect(readRemoteWorkspaceAuthority(prepared.workspace.metadata)?.snapshotId).toBeTruthy(); @@ -399,8 +400,16 @@ suite("remote workspace lifecycle with real Postgres and independent filesystem lease: prepared.lease, target: first, assertOwned: async () => undefined }); const current = await workspace(reserved.id); expect(readRemoteWorkspaceAuthority(current.metadata)).toMatchObject({ dirty: false, writerRunId: null, snapshotId: saved.snapshotId }); - await fs.rm(path.dirname(prepared.remoteRoot), { recursive: true, force: true }); - const second = await target("sandbox-B"); + if (mode === "destroyed allocation") { + const destroyed: EnvironmentLease = { ...prepared.lease, status: "released", releasedAt: new Date(), cleanupStatus: "success", + metadata: { ...prepared.lease.metadata, remoteExecutionTermination: remoteTerminationReceipt(prepared.lease, + { providerLeaseId: prepared.lease.providerLeaseId, state: "destroyed" }) } }; + await db.transaction(tx => clearRemoteWorkspaceLocationForReleasedLease({ db: tx, lease: destroyed })); + expect(readRemoteWorkspaceAuthority((await workspace(reserved.id)).metadata)?.location).toBeNull(); + } + await fs.rm(path.join(root, "sandbox-A"), { recursive: true, force: true }); + const second = await target(mode === "different root" ? "sandbox-B" : "sandbox-A"); + const expectedRemoteRoot = path.join(second.remoteCwd, ".paperclip-workspaces", path.basename(prepared.remoteRoot)); const nextRun = randomUUID(), nextLease = randomUUID(); await db.insert(heartbeatRuns).values({ id: nextRun, companyId: ids.company, agentId: ids.agent, nativeIssueId: ids.issue, runtimeMode: "native", status: "running" }); await db.insert(environmentLeases).values({ id: nextLease, companyId: ids.company, environmentId: ids.environment, @@ -408,7 +417,19 @@ suite("remote workspace lifecycle with real Postgres and independent filesystem const reservedAgain = await reserveRemoteWorkspace({ db, storage, companyId: ids.company, issueId: ids.issue, runId: nextRun, agentId: ids.agent, environmentId: ids.environment, projectId: null, projectWorkspaceId: null, existingWorkspaceId: reserved.id, source: { kind: "empty" } }); const newLease = (await db.select().from(environmentLeases).where(eq(environmentLeases.id, nextLease)))[0]! as EnvironmentLease; + if (mode === "occupied root") { + await fs.mkdir(expectedRemoteRoot, { recursive: true }); + await fs.writeFile(path.join(expectedRemoteRoot, "unpublished.txt"), "preserve unknown preparation\n"); + await expect(prepareRemoteWorkspace({ db, storage, workspace: reservedAgain, runId: nextRun, lease: newLease, target: second })) + .rejects.toThrow("remote_workspace_operation_failed"); + expect(await fs.readFile(path.join(expectedRemoteRoot, "unpublished.txt"), "utf8")).toBe("preserve unknown preparation\n"); + expect(readRemoteWorkspaceAuthority((await workspace(reserved.id)).metadata)) + .toEqual(readRemoteWorkspaceAuthority(reservedAgain.metadata)); + await fs.rename(expectedRemoteRoot, path.join(root, "retained-unknown-preparation")); + } const restored = await prepareRemoteWorkspace({ db, storage, workspace: reservedAgain, runId: nextRun, lease: newLease, target: second }); + expect(restored.remoteRoot).toBe(expectedRemoteRoot); + if (mode !== "different root") expect(restored.remoteRoot).toBe(prepared.remoteRoot); expect(await fs.readFile(path.join(restored.remoteRoot, "report.md"), "utf8")).toBe("useful saved result\n"); expect(await fs.readFile(path.join(restored.remoteRoot, "binary.bin"))).toEqual(Buffer.from([0, 1, 255])); expect((await fs.stat(path.join(restored.remoteRoot, "empty-folder"))).isDirectory()).toBe(true); @@ -619,8 +640,7 @@ suite("remote workspace lifecycle with real Postgres and independent filesystem existingWorkspaceId: reserved.id, source: { kind: "empty" } }); const nextTarget = await target("restore-cleanup-destination"); const sandbox = nextTarget as Extract; - const expectedRemoteRoot = path.join(sandbox.remoteCwd, ".paperclip-workspaces", - `${reserved.id}-${readRemoteWorkspaceAuthority(nextWorkspace.metadata)!.epoch}-${nextLease}`); + const expectedRemoteRoot = path.join(sandbox.remoteCwd, ".paperclip-workspaces", path.basename(prepared.remoteRoot)); const execute = sandbox.runner!.execute.bind(sandbox.runner!); let failCleanup = true, cleanupAttempts = 0; sandbox.runner!.execute = async input => { diff --git a/server/src/services/remote-workspaces.ts b/server/src/services/remote-workspaces.ts index 4712ae33568..d651a716096 100644 --- a/server/src/services/remote-workspaces.ts +++ b/server/src/services/remote-workspaces.ts @@ -73,28 +73,22 @@ export function shouldUseRemoteWorkspaces(input: { return configuredRemoteWorkspacePersistence() !== null; } -/** Deployment profile admission belongs beside workspace authority. A warm - * remote controller may replace its local projection; host-owned execution - * cannot make that promise. External conversation providers own no task disk. */ export function assertEphemeralWorkspaceExecution(input: { externalConversation: boolean; adapterType: string; pluginTaskWorkspace: boolean; remoteAuthority: boolean; }) { if (process.env.PAPERCLIP_EPHEMERAL_SERVER !== "true" || input.externalConversation) return; // Runtime selection has not promoted a fresh run yet. Validate storage here; - // the runtime selector separately admits only its resolved warm native policy. if (input.adapterType !== "paperclip_runner" || (!input.pluginTaskWorkspace && !input.remoteAuthority)) { - throw unprocessable("Volume-free servers require warm Paperclip Runner execution with remote workspace storage.", + throw unprocessable("Volume-free servers require native Paperclip Runner execution with remote workspace storage.", { code: "ephemeral_execution_configuration_required" }); } } -/** The runtime selector calls this after environment, credentials and saved - * execution policy have resolved. Raw agent settings are not lifecycle authority. */ -export function assertEphemeralNativeLifecycle(mode: "warm" | "per_turn" | undefined) { - if (process.env.PAPERCLIP_EPHEMERAL_SERVER === "true" && mode !== "warm") { - throw unprocessable("Volume-free servers require warm Paperclip Runner execution with remote workspace storage.", +export function assertEphemeralNativeRuntime(native: boolean) { + if (process.env.PAPERCLIP_EPHEMERAL_SERVER === "true" && !native) { + throw unprocessable("Volume-free servers require native Paperclip Runner execution with remote workspace storage.", { code: "ephemeral_execution_configuration_required" }); } } @@ -327,8 +321,22 @@ function isSavedWorkspaceRepository(workspace: Workspace, authority: RemoteWorks && (saved.readOnly ?? false) === (repository.readOnly ?? false)); } +function publishedWorkspaceDirectory(workspace: Workspace & Pick, + authority: RemoteWorkspaceAuthority) { + const publishedRoot = authority.snapshotId && (authority.location?.remoteCwd ?? workspace.providerRef); + if (!publishedRoot) return null; + const directory = path.posix.basename(publishedRoot); + const suffix = directory.slice(workspace.id.length); + if (!path.posix.isAbsolute(publishedRoot) || path.posix.basename(path.posix.dirname(publishedRoot)) !== ".paperclip-workspaces" + || !directory.startsWith(workspace.id) || (suffix !== "" && !/^-[1-9][0-9]*-[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/.test(suffix))) { + throw new Error("remote_workspace_published_path_invalid"); + } + return directory; +} + export async function prepareRemoteWorkspace(input: { - db: Db; workspace: Workspace; runId: string; lease: EnvironmentLease; target: AdapterExecutionTarget | null; + db: Db; workspace: Workspace & Pick; + runId: string; lease: EnvironmentLease; target: AdapterExecutionTarget | null; source?: RemoteWorkspaceSource; assertOwned?: () => Promise; provisionCommand?: string | null; runtimeProvisionCommand?: string | null; storage?: StorageService; }) { @@ -345,7 +353,7 @@ export async function prepareRemoteWorkspace(input: { && initial.location?.environmentId === input.lease.environmentId; if (initial.location && !sameAllocation && initial.dirty) throw conflict("The prior workspace has unsaved work; recover it before replacing its allocation."); const remoteRoot = sameAllocation ? initial.location!.remoteCwd : path.posix.join(target.remoteCwd, ".paperclip-workspaces", - `${input.workspace.id}-${initial.epoch}-${input.lease.id}`); + publishedWorkspaceDirectory(input.workspace, initial) ?? `${input.workspace.id}-${initial.epoch}-${input.lease.id}`); const preparationIdentity = createHash("sha256").update(JSON.stringify([input.workspace.companyId, input.workspace.id, input.runId, initial.epoch, input.lease.id, input.lease.providerLeaseId, remoteRoot, initial.source])).digest("hex"); let restoredExport: Awaited> | null = null; diff --git a/server/src/vendor/paperclip-runner/index.ts b/server/src/vendor/paperclip-runner/index.ts index 51ddeff3ea2..d28a01baba6 100644 --- a/server/src/vendor/paperclip-runner/index.ts +++ b/server/src/vendor/paperclip-runner/index.ts @@ -94,6 +94,7 @@ export const RUNNER_TOOL_SCOPE_TOKEN_HEADER = runner.RUNNER_TOOL_SCOPE_TOKEN_HEA export const DurablePrpControlPlane = runner.DurablePrpControlPlane; export const providerFailureFromError = runner.providerFailureFromError; export const decodeDurablePrpCheckpoint = runner.decodeDurablePrpCheckpoint; +export const captureLegacyDurablePrpCheckpoint = runner.captureLegacyDurablePrpCheckpoint; export const inspectDurablePrpCheckpointHeader = runner.inspectDurablePrpCheckpointHeader; export const RUNNER_PRP_PROTOCOL_MIN = runner.RUNNER_PRP_PROTOCOL_MIN; export const RUNNER_PRP_PROTOCOL_MAX = runner.RUNNER_PRP_PROTOCOL_MAX; diff --git a/tests/runner-e2e/README.md b/tests/runner-e2e/README.md index 49181a8d3ec..4974e9ed25f 100644 --- a/tests/runner-e2e/README.md +++ b/tests/runner-e2e/README.md @@ -104,6 +104,24 @@ its DB checkpoint alone cannot replace those bytes. Per-turn application qualification under this disk loss awaits the provider-backup owner's durable storage contract; native controller recovery does not waive that dependency. +`rolling-web-deployment` definition 4 selects every registered native Daytona +profile in disposable per-turn environments; `rolling-web-deployment-warm` +uses the same profiles in reusable warm environments. Their +provider-backup path uses encrypted object storage and a fenced database ledger. +The earlier per-turn failure remains failed evidence; this definition requires +a fresh passing run, including the strict post-replacement continuation oracle. +Its revision task includes a 180-second foreground CLI reliability soak, giving +the replacement time to become ready while useful agent work remains active. +The observer rechecks the original live owner immediately before switching +traffic and fails if that turn already settled. A completed turn cannot count +as active takeover. Per-turn file inspection requires the exact committed saved +snapshot; the warm lane still requires the live workspace. +Cold restores retain the published workspace directory name, including after +the old allocation is destroyed, because provider recovery binds the working +directory into session identity. Unpublished preparations retain distinct +epoch/lease directories, and a restore refuses to overwrite an occupied path +owned by another preparation. + The ordinary Everyday matrix retains its 125-second active outage default. The two explicit continuity suites require 300 seconds and reject a shorter override. @@ -2170,6 +2188,12 @@ an API-key pass must not be reported as subscription evidence. See [`doc/grok-native-runner.md`](../../doc/grok-native-runner.md) for installation, credential boundaries and repetition requirements. +Grok accounting uses the complete prompt ledger bound to the matching ACP +response. Its reasoning counter can exceed its output counter; it is not a +validation bound or an extra term in the ledger's input-plus-output total. +Reported cost remains authoritative. See the upstream +[`UsageTotals` ledger](https://github.com/xai-org/grok-build/blob/2bdd1d6a6369de0e8c68132ea4539e9abd9e14a8/crates/codegen/xai-chat-state/src/usage.rs). + ### Kimi Code qualification `kimi-qualification` is a separate explicit-only candidate suite for maintained diff --git a/tests/runner-e2e/catalog.test.ts b/tests/runner-e2e/catalog.test.ts index 3729583453d..32c66c5450c 100644 --- a/tests/runner-e2e/catalog.test.ts +++ b/tests/runner-e2e/catalog.test.ts @@ -312,11 +312,11 @@ describe("runner E2E catalog", () => { expect(localIntegrityTasks).toHaveLength(2); expect(openRouterBreadthTasks).toHaveLength(3); expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([ - 1, 1, 4, 2, 1, 3, 2, 2, 4, 4, 4, 4, 4, 4, 8, 10, 6, 63, 12, 6, 4, 2, 4, 2, 30, 3, 16, 16, 20, 2, 1, 6, 8, 46, 2, 23, 15, 52, 6, 6, 20, 26, 52, 5, 28, 18, 2, 6, 6, 13, 10, 4, 48, 16, 10, 2, 1, 1, 4, 116, 2, 1, 2, + 1, 9, 9, 4, 2, 1, 3, 2, 2, 4, 4, 4, 4, 4, 4, 8, 10, 6, 63, 12, 6, 4, 2, 4, 2, 30, 3, 16, 16, 20, 2, 1, 6, 8, 46, 2, 23, 15, 52, 6, 6, 20, 26, 52, 5, 28, 18, 2, 6, 6, 13, 10, 4, 48, 16, 10, 2, 1, 1, 4, 116, 2, 1, 2, ]); - expect(validateRunnerCatalog()).toHaveLength(802); - expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(802); + expect(validateRunnerCatalog()).toHaveLength(819); + expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(819); expect( runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"), ).toHaveLength(48); diff --git a/tests/runner-e2e/catalog.ts b/tests/runner-e2e/catalog.ts index 81e7e4fdf85..ecf24df00fd 100644 --- a/tests/runner-e2e/catalog.ts +++ b/tests/runner-e2e/catalog.ts @@ -1,4 +1,4 @@ -import { ROLLING_WEB_DEPLOYMENT_SUITE } from "./controller-rolling-deployment.js"; +import { ROLLING_WEB_DEPLOYMENT_SUITE, ROLLING_WEB_DEPLOYMENT_WARM_SUITE } from "./controller-rolling-deployment.js"; import { FEEDBACK_PERSISTENCE_SUITE, feedbackPersistenceTask, feedbackPersistenceDigest } from "./feedback-persistence.js"; import { followupStopTask, followupStopDigest } from "./followup-stop-resume.js"; import { finalizationRecoveryTask, finalizationRecoveryDigest } from "./finalization-recovery.js"; @@ -1233,15 +1233,45 @@ const remoteWorkspaceStorageSuite: RunnerSuiteFixture = { source: "projectless-synthetic-files-and-model-created-git", authentication: "api-key", stagingQualification: false }, }; +export const kimiRunnerProfile = nativeProfile({ + id: "runner-acpx-kimi", label: "Runner Kimi Code", + provider: "acpx", acpxAgent: "kimi", qualificationCandidate: "kimi", + credential: "KIMI_API_KEY", model: "kimi-for-coding", + modelQualification: { source: "qualified_runner_profile", qualificationId: "kimi:2.3.3:native-qualified-profile17" }, +}); + +const rollingDeploymentProfiles = [...runnerProfiles, ...extendedHarnessProfiles, kimiRunnerProfile] + .filter(profile => profile.generation === "native" && profile.supportedEnvironments.includes("daytona") + && (!profile.qualificationCandidate || QUALIFIED_ACPX_PROFILES[profile.qualificationCandidate].qualificationStatus !== "pending")) + .map(original => { + const { qualificationCandidate: _candidate, ...profile } = productionStoryProfile(original); + return { ...profile, buildAgent(input: AgentFixtureBuildInput) { + const agent = profile.buildAgent(input); + const adapterConfig = { ...(agent.adapterConfig as Record) }; + delete adapterConfig.cwd; + return { ...agent, adapterConfig }; + } }; + }); +const rollingDeploymentSuites: RunnerSuiteFixture[] = [ + { id: ROLLING_WEB_DEPLOYMENT_SUITE, environment: runnerEnvironments.find(environment => environment.id === "daytona")! }, + { id: ROLLING_WEB_DEPLOYMENT_WARM_SUITE, environment: daytonaControllerOutageEnvironment }, +].map(({ id, environment }) => ({ + ...remoteWorkspaceStorageSuite, id, label: `Rolling control-plane web deployment (${environment.configurationKey ? "warm" : "per-turn"})`, + description: "Overlap independent ephemeral web servers while a remote execution stays active; keep ordinary reads, saves, files and a later native continuation working.", + profiles: rollingDeploymentProfiles, environments: [environment], + expectedMatrixSize: rollingDeploymentProfiles.length * remoteWorkspaceStorageSuite.tasks.length, + definitionMetadata: { ...remoteWorkspaceStorageSuite.definitionMetadata, version: 4, controllerDowntimeMs: 0, + activeWorkBoundary: "foreground-180-second-cli-reliability-soak-and-live-owner-recheck", + workspaceFileSource: environment.configurationKey ? "live" : "saved-exact-snapshot-after-per-turn-retirement", + lifecycleCoverage: environment.configurationKey ? "warm-reusable" : "per-turn-disposable", providerCoverage: "registered-native-daytona-profiles", + replacement: "ready-before-traffic-switch-sigterm-delete-entire-home", + upgradeReadiness: "active-remote-workspace-recoverable-before-handoff", + grading: "same-active-execution-ordinary-reads-saves-and-second-empty-home-continuation" }, +})); + export const runnerSuites: readonly RunnerSuiteFixture[] = [ remoteWorkspaceStorageSuite, - { ...remoteWorkspaceStorageSuite, id: ROLLING_WEB_DEPLOYMENT_SUITE, label: "Rolling control-plane web deployment", - description: "Overlap independent ephemeral web servers while a remote execution stays active; keep ordinary reads, saves, files and a later native continuation working.", - definitionMetadata: { ...remoteWorkspaceStorageSuite.definitionMetadata, version: 1, controllerDowntimeMs: 0, - replacement: "ready-before-traffic-switch-sigterm-delete-entire-home", - upgradeReadiness: "active-remote-workspace-recoverable-before-handoff", - grading: "same-active-execution-ordinary-reads-saves-and-second-empty-home-continuation" }, - }, + ...rollingDeploymentSuites, { id: CURSOR_CLOUD_SUITE, label: "Native Cursor Cloud", manualOnly: true, groups: [], description: "Local Paperclip controller with hosted Cursor work: verified artifact, same-agent follow-up, real active-run Stop and semantic plan revision/approval.", @@ -1537,12 +1567,7 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [ id: "kimi-qualification", label: "Kimi Code Qualification", manualOnly: true, description: "Kimi Code replies, semantic planning and questions, verified file delivery, Stop and controller continuation through local and Daytona user journeys.", groups: ["native"], - profiles: [nativeProfile({ - id: "runner-acpx-kimi", label: "Runner Kimi Code", - provider: "acpx", acpxAgent: "kimi", qualificationCandidate: "kimi", - credential: "KIMI_API_KEY", model: "kimi-for-coding", - modelQualification: { source: "qualified_runner_profile", qualificationId: "kimi:2.3.3:native-qualified-profile17" }, - })], + profiles: [kimiRunnerProfile], environments: [localEnvironment, daytonaWarmEnvironment], environmentOverrides: Object.fromEntries(kimiLifecycleTasks.map(task => [task.id, { daytona: runnerEnvironments.find(environment => environment.id === "daytona")!, diff --git a/tests/runner-e2e/controller-recovery-observer.test.ts b/tests/runner-e2e/controller-recovery-observer.test.ts index 8597ee97a27..a84236c905a 100644 --- a/tests/runner-e2e/controller-recovery-observer.test.ts +++ b/tests/runner-e2e/controller-recovery-observer.test.ts @@ -1,10 +1,11 @@ import type { Db } from "../../packages/db/src/client.js"; import { describe, expect, it } from "vitest"; -import { advanceRecoveryCursor, parseRecoveryBinding, readControllerRecoverySnapshot, recoveryObservation, validControllerRecoveryReceipt, type RecoverySnapshot } from "./controller-recovery-observer.js"; +import { advanceRecoveryCursor, assertActiveRecoveryBoundary, parseRecoveryBinding, readControllerRecoverySnapshot, recoveryObservation, validControllerRecoveryReceipt, type RecoverySnapshot } from "./controller-recovery-observer.js"; import type { ControllerOutageReceipt } from "./controller-outage.js"; const binding = { companyId: "10000000-0000-0000-0000-000000000001", issueId: "10000000-0000-0000-0000-000000000002", runId: "10000000-0000-0000-0000-000000000003" }; const before: RecoverySnapshot = { runId: binding.runId, runnerInstanceId: "runner", nativeSessionId: "session", + status: "running", phase: "observed", resultId: null, bootId: "old", generation: 1, attempt: 1, leaseExpiresAt: "2026-10-24T00:01:00.000Z", sourceSeq: 42, history: [], authenticatedConnectionCount: 1, checkpointSourceSeq: 42 }; const fault: ControllerOutageReceipt = { schema: "paperclip.e2e.controller-outage/v1", requestId: "restart", previousPid: 1, @@ -13,6 +14,14 @@ const fault: ControllerOutageReceipt = { schema: "paperclip.e2e.controller-outag const after = (): RecoverySnapshot => ({ ...before, bootId: "new", generation: 2, sourceSeq: 47, authenticatedConnectionCount: 2, history: [{ controllerBootId: "new", controllerGeneration: 2, disposition: "reattach_remote_runner", at: "2026-10-24T00:01:00.001Z" }] }); describe("read-only controller recovery evidence", () => { + it("fails immediately when the original work settles while the replacement starts", () => { + const active = { ...before, leaseExpiresAt: new Date(Date.now() + 60_000).toISOString() }; + expect(() => assertActiveRecoveryBoundary(active)).not.toThrow(); + for (const patch of [{ status: "succeeded" }, { phase: "committed" }, { phase: "workspace_finalizing" }, + { resultId: "accepted" }, { leaseExpiresAt: null }, { leaseExpiresAt: new Date(0).toISOString() }]) { + expect(() => assertActiveRecoveryBoundary({ ...active, ...patch })).toThrow("boundary unexercised"); + } + }); it("does not count A's shutdown tail as B's processing even if B already claimed", () => { const stopped = { ...after(), checkpointSourceSeq: 48 }; const baseline = advanceRecoveryCursor(before, stopped); diff --git a/tests/runner-e2e/controller-recovery-observer.ts b/tests/runner-e2e/controller-recovery-observer.ts index a236b478c07..87f321425b6 100644 --- a/tests/runner-e2e/controller-recovery-observer.ts +++ b/tests/runner-e2e/controller-recovery-observer.ts @@ -13,11 +13,19 @@ import type { ControllerReplacementReceipt } from "./controller-replacement.js"; export interface RecoveryBinding { companyId: string; issueId: string; runId: string } export interface RecoverySnapshot { runId: string; runnerInstanceId: string; nativeSessionId: string; + status: string; phase: string; resultId: string | null; bootId: string; generation: number; attempt: number; leaseExpiresAt: string | null; sourceSeq: number; history: Array>; authenticatedConnectionCount: number; checkpointSourceSeq: number; } + +export function assertActiveRecoveryBoundary(snapshot: RecoverySnapshot): void { + if (snapshot.status !== "running" || snapshot.phase !== "observed" || snapshot.resultId !== null + || !snapshot.leaseExpiresAt || !Number.isFinite(Date.parse(snapshot.leaseExpiresAt)) || Date.parse(snapshot.leaseExpiresAt) <= Date.now()) { + throw new Error("Active controller handoff boundary unexercised: original execution settled before traffic switched"); + } +} export interface ControllerRecoveryReceipt { binding: RecoveryBinding; runnerInstanceId: string; @@ -98,7 +106,7 @@ export function validControllerRecoveryReceipt(fault: ControllerReplacementRecei export async function readControllerRecoverySnapshot(db: Db, binding: RecoveryBinding): Promise { return db.transaction(async tx => { - const [row] = await tx.execute(sql`select r.id as "runId", r.runner_instance_id as "runnerInstanceId", + const [row] = await tx.execute(sql`select r.id as "runId", r.status, f.phase, f.result_id as "resultId", r.runner_instance_id as "runnerInstanceId", r.native_session_id as "nativeSessionId", f.controller_boot_id as "bootId", f.controller_generation as generation, f.attempt, f.lease_expires_at::text as "leaseExpiresAt", f.recovery_history as history, coalesce((select max(e.source_seq) from heartbeat_run_events e where e.run_id = r.id @@ -139,7 +147,10 @@ export async function createControllerRecoveryObserver(configPath: string, bindi runnerInstanceId: before.runnerInstanceId, nativeSessionId: before.nativeSessionId }; }; return { - beforeHandoff: capture, + beforeHandoff: async (fault: ControllerReplacementReceipt) => { + await capture(fault); + assertActiveRecoveryBoundary(before!); + }, // Called only after the old server is confirmed dead. This also excludes // receipts committed while it was stopping from replacement evidence. stopped: capture, diff --git a/tests/runner-e2e/controller-rolling-deployment.test.ts b/tests/runner-e2e/controller-rolling-deployment.test.ts index 2dfa052eefb..2484abb0212 100644 --- a/tests/runner-e2e/controller-rolling-deployment.test.ts +++ b/tests/runner-e2e/controller-rolling-deployment.test.ts @@ -1,8 +1,10 @@ +import { QUALIFIED_ACPX_PROFILES } from "../../packages/paperclip-runner/src/drivers/acpx/qualified-profiles.js"; import { describe, expect, it } from "vitest"; import { rollController, validControllerRollingDeploymentReceipt } from "./controller-rolling-deployment.js"; import { validControllerReplacementReceipt } from "./controller-replacement.js"; import { controllerOutageForExecution, validControllerOutageReceipt } from "./controller-outage.js"; import { parseRunnerSelectors, selectRunnerExecutions } from "./selectors.js"; +import { runnerProfiles, extendedHarnessProfiles, kimiRunnerProfile, runnerSuites } from "./catalog.js"; function fixture() { const order: string[] = []; @@ -55,4 +57,30 @@ describe("rolling controller qualification", () => { const [outage] = selectRunnerExecutions(parseRunnerSelectors(["--suite", "remote-workspace-storage"])); expect(controllerOutageForExecution(outage!, {})).toEqual({ signal: "SIGKILL", holdMs: 300_000 }); }); + it("requires both lifecycle modes for every production-qualified native sandbox profile", () => { + const executions = ["rolling-web-deployment", "rolling-web-deployment-warm"] + .flatMap(suite => selectRunnerExecutions(parseRunnerSelectors(["--suite", suite]))); + const native = [...runnerProfiles, ...extendedHarnessProfiles, kimiRunnerProfile] + .filter(profile => profile.generation === "native" && profile.supportedEnvironments.includes("daytona") + && (!profile.qualificationCandidate || QUALIFIED_ACPX_PROFILES[profile.qualificationCandidate].qualificationStatus !== "pending")); + const kind = (profile: (typeof native)[number]) => { + const provider = profile.expectedRuntimeMetadata.provider; + const harness = "acpxAgent" in profile ? profile.acpxAgent + : provider === "acpx" && profile.provider !== "acpx" ? profile.provider : ""; + return `${provider}:${harness}`; + }; + const registeredKinds = new Set(runnerSuites.flatMap(suite => suite.profiles) + .filter(profile => profile.generation === "native" && profile.supportedEnvironments.includes("daytona") + && (!profile.qualificationCandidate || QUALIFIED_ACPX_PROFILES[profile.qualificationCandidate].qualificationStatus !== "pending")) + .map(kind)); + expect(new Set(native.map(kind))).toEqual(registeredKinds); + for (const profile of native) { + const cells = executions.filter(execution => execution.profile.id === profile.id); + expect(cells).toHaveLength(2); + expect(cells.every(cell => !cell.profile.qualificationCandidate)).toBe(true); + expect(new Set(cells.map(cell => cell.environment.configurationKey ?? "per_turn"))) + .toEqual(new Set(["per_turn", "warm-controller-outage-v1"])); + } + expect(executions.every(execution => execution.suite.definitionMetadata?.retainedControllerContinuation === true)).toBe(true); + }); }); diff --git a/tests/runner-e2e/controller-rolling-deployment.ts b/tests/runner-e2e/controller-rolling-deployment.ts index 75ace959092..46e1570304e 100644 --- a/tests/runner-e2e/controller-rolling-deployment.ts +++ b/tests/runner-e2e/controller-rolling-deployment.ts @@ -3,7 +3,9 @@ import type { MatrixExecution } from "./types.js"; import type { ControllerOutageReceipt } from "./controller-outage.js"; export const ROLLING_WEB_DEPLOYMENT_SUITE = "rolling-web-deployment"; -export const isRollingWebDeployment = (execution: Pick) => execution.suite.id === ROLLING_WEB_DEPLOYMENT_SUITE; +export const ROLLING_WEB_DEPLOYMENT_WARM_SUITE = "rolling-web-deployment-warm"; +export const isRollingWebDeployment = (execution: Pick) => + [ROLLING_WEB_DEPLOYMENT_SUITE, ROLLING_WEB_DEPLOYMENT_WARM_SUITE].includes(execution.suite.id); export interface ControllerRollingDeploymentReceipt extends Omit { schema: "paperclip.e2e.controller-rolling-deployment/v1"; diff --git a/tests/runner-e2e/everyday-cases.ts b/tests/runner-e2e/everyday-cases.ts index 153ac13096c..10dc07d9db4 100644 --- a/tests/runner-e2e/everyday-cases.ts +++ b/tests/runner-e2e/everyday-cases.ts @@ -29,6 +29,7 @@ export function productionStoryProfile( export const SLUGIFY_REQUIREMENTS = `Build a small dependency-free Python command-line tool in slugify.py. It accepts one positional string and prints its slug: trim whitespace, lowercase, replace each sequence outside ASCII a-z and 0-9 with one hyphen, then trim hyphens. Expose slugify(text) for reuse. Include a README and automated tests. Deliver the source and tests as a downloadable ZIP. Work in the project workspace.`; export const SLUGIFY_REVISION = `Add an optional --separator argument that accepts either - or _. Keep - as the default and preserve the earlier behavior. Deliver an updated ZIP, keeping the previous download available.`; +export const SLUGIFY_DEPLOYMENT_SOAK = `Before editing, run a short command that prints the current workspace path. After implementing the revision, run a foreground reliability soak for a full 180 seconds: repeatedly invoke the CLI on varied inputs and check its output, sleeping briefly between iterations to avoid excessive CPU use. Wait for that command to finish; do not background it or shorten the duration. Include a small soak report with measured elapsed time and check count in the updated ZIP. Finish only after the soak and the regular tests pass.`; export const LATE_REQUIREMENT = `Also support --max-length as a positive integer. Truncate the final slug to that length, then remove any trailing separator. In particular, input " Queue--Ready!! " with --max-length 7 must print "queue-r". Preserve the default behavior.`; const definitions = [ diff --git a/tests/runner-e2e/everyday-flow.ts b/tests/runner-e2e/everyday-flow.ts index a699646cbee..ccd87e9ef89 100644 --- a/tests/runner-e2e/everyday-flow.ts +++ b/tests/runner-e2e/everyday-flow.ts @@ -39,7 +39,7 @@ import { StoryDecisionError, type StoryInteraction, } from "./everyday-decisions.js"; -import { LATE_REQUIREMENT, SLUGIFY_REVISION, requiresEverydayArtifactOracle } from "./everyday-cases.js"; +import { LATE_REQUIREMENT, SLUGIFY_REVISION, SLUGIFY_DEPLOYMENT_SOAK, requiresEverydayArtifactOracle } from "./everyday-cases.js"; import { isActiveStoryRun, isStoryWorkspaceDeferral, @@ -186,7 +186,8 @@ export async function runEverydayFlow(input: Input) { const controllerOutage = controllerOutageForExecution(execution, process.env); const requireRemoteRestart = controllerOutage !== undefined || isRollingWebDeployment(execution); const remoteWorkspaceStorage = isRemoteWorkspaceStorageExecution(execution); - if (remoteWorkspaceStorage) ev.remoteWorkspaceStorage = { browserReadPhases: [] }; + if (remoteWorkspaceStorage) ev.remoteWorkspaceStorage = { browserReadPhases: [], + expectedFileSource: execution.suite.definitionMetadata?.lifecycleCoverage === "per-turn-disposable" ? "saved" : "live" }; const providerChoice = caseId === "provider-decline" || caseId === "provider-second"; const nativeProviderCase = caseId === "provider-native"; let aggregatorFixture: Awaited> | undefined; @@ -562,7 +563,8 @@ export async function runEverydayFlow(input: Input) { const point = await pollUntil({ label: `remote workspace ${phase} saved version`, deadlineAt: input.deadlineAt, load: () => readRemoteWorkspaceStorageCheckpoint({ api, companyId: fixtures.company.id, - agentId: fixtures.agent.id, issueId: parent!.id, environmentId: fixtures.environment.id }), + agentId: fixtures.agent.id, issueId: parent!.id, environmentId: fixtures.environment.id, + expectedFileSource: ev.remoteWorkspaceStorage!.expectedFileSource }), accept: observed => Boolean(observed.point), }); ev.remoteWorkspaceStorage![phase] = point.point!; @@ -570,7 +572,8 @@ export async function runEverydayFlow(input: Input) { const browserPhase = phase === "initial" ? "before-restart" : "after-restart"; await page.goto(`${taskUrl(parent!)}?browse=1&q=${REMOTE_WORKSPACE_PROOF_FILE}`); await expect(page.getByRole("combobox", { name: "Search workspace files" })).toBeVisible(); - await expect(page.getByLabel("Workspace file source", { exact: true })).toHaveText("Live workspace"); + await expect(page.getByLabel("Workspace file source", { exact: true })).toHaveText( + ev.remoteWorkspaceStorage!.expectedFileSource === "saved" ? /^Saved copy(?: from .+)?$/ : "Live workspace"); await page.getByRole("treeitem").filter({ hasText: REMOTE_WORKSPACE_PROOF_FILE }).last().click(); await expect(page.getByText(point.content!.trim(), { exact: true })).toBeVisible(); await input.capture(`remote-workspace-${browserPhase}`, `Remote workspace file ${browserPhase}`, `remote-workspace-${browserPhase}.png`); @@ -936,7 +939,7 @@ export async function runEverydayFlow(input: Input) { throw new Error( "Remote fault boundary unexercised: no verified saved project", ); - await reply(SLUGIFY_REVISION); + await reply(isRollingWebDeployment(execution) ? `${SLUGIFY_REVISION}\n\n${SLUGIFY_DEPLOYMENT_SOAK}` : SLUGIFY_REVISION); note("remote-revision-submitted"); await pollUntil({ label: "remote revision executing", diff --git a/tests/runner-e2e/everyday-remote-restart.test.ts b/tests/runner-e2e/everyday-remote-restart.test.ts index 43e41fd6b54..f1a45b4fa19 100644 --- a/tests/runner-e2e/everyday-remote-restart.test.ts +++ b/tests/runner-e2e/everyday-remote-restart.test.ts @@ -195,6 +195,18 @@ describe("remote restart lifecycle evidence", () => { expect(gradeRemoteRestart(evidence).every(check => check.passed)).toBe(true); }); + it("accepts the same start notification after reattachment and rejects a different turn or payload", () => { + const f = fixture(); + const replay = f.event(5, "turn.started"); + const after = [...f.rows, replay]; + expect(observeRemoteRestartBoundary(f.active, after, f.leases, f.scope, true).identity).toEqual(f.evidence.before.identity); + expect(() => observeRemoteRestartBoundary(f.active, after, f.leases, f.scope)).toThrow("active original provider turn"); + expect(() => observeRemoteRestartBoundary(f.active, [...f.rows, f.event(5, "turn.started", { changed: true })], f.leases, f.scope, true)) + .toThrow("active original provider turn"); + replay.payload.prpEvent.turnId = "another-turn"; + expect(() => observeRemoteRestartBoundary(f.active, after, f.leases, f.scope, true)).toThrow("active original provider turn"); + }); + it.each(["replacement", "second submission", "changed tool receipt", "changed start receipt", "missing terminal"])("rejects %s at completion", fault => { const { evidence } = fixture(); if (fault === "replacement") evidence.final.run.id = "replacement"; diff --git a/tests/runner-e2e/everyday-remote-restart.ts b/tests/runner-e2e/everyday-remote-restart.ts index 60b12c19de3..cb2fd98f3f7 100644 --- a/tests/runner-e2e/everyday-remote-restart.ts +++ b/tests/runner-e2e/everyday-remote-restart.ts @@ -125,7 +125,10 @@ export function observeRemoteRestartBoundary(run: StoryRun, rows: readonly Row[] const events = projectRemoteRestartEvents(run, rows); const starts = events.filter(event => event.eventType === "turn.started" && event.sourceKind === "runner"); const completed = events.filter(event => event.toolCompleted && event.sourceKind === "runner" && event.toolExecutionId); - if (starts.length !== 1 || !id(starts[0]!.turnId) || completed.length === 0 + const originalStart = starts[0]; + const sameStart = originalStart && id(originalStart.turnId) && starts.every(event => + event.turnId === originalStart.turnId && event.payloadSha256 === originalStart.payloadSha256); + if ((!recovered && starts.length !== 1) || !sameStart || completed.length === 0 || completed.some(event => event.turnId !== starts[0]!.turnId) || (!recovered && events.some(event => ["turn.completed", "turn.failed", "turn.cancelled", "turn.interrupted", "run.terminal"].includes(event.eventType)))) { throw new Error("Remote restart boundary has no completed tool in an active original provider turn"); diff --git a/tests/runner-e2e/installed-cli.test.ts b/tests/runner-e2e/installed-cli.test.ts index 7f3074e0fd8..a2ef8190188 100644 --- a/tests/runner-e2e/installed-cli.test.ts +++ b/tests/runner-e2e/installed-cli.test.ts @@ -1,3 +1,4 @@ +import { isRollingWebDeployment } from "./controller-rolling-deployment.js"; import { execFileSync } from "node:child_process"; import { createHash } from "node:crypto"; import { chmod, lstat, mkdtemp, mkdir, readFile, realpath, rm, symlink, writeFile } from "node:fs/promises"; @@ -7,7 +8,7 @@ import { afterEach, beforeEach, expect, it, vi } from "vitest"; import { runnerMatrix } from "./catalog.js"; import { assertInstalledCliSelection, installedCliKeys, probeInstalledCodexStartup, verifyInstalledCli } from "./installed-cli.js"; import { buildPaperclipServerEnvironment, buildRunnerE2EProcessEnvironment } from "./harness-env.js"; -const cells = runnerMatrix.filter(e => e.profile.id === "runner-acpx-pi"); +const cells = runnerMatrix.filter(e => !isRollingWebDeployment(e) && e.profile.id === "runner-acpx-pi"); const roots: string[] = []; // pnpm's source test wrapper sets NODE_PATH; the disposable public-install fixture must start closed. beforeEach(() => { vi.stubEnv("NODE_PATH", undefined); vi.stubEnv("NODE_OPTIONS", undefined); }); @@ -46,7 +47,7 @@ it("admits existing Codex onboarding, useful-file revision and restart journeys }); it("admits only the 20 agreed Copilot cells through normal installed resolution with no qualification override", async () => { const { env, server } = await fixture(); - const selected = runnerMatrix.filter(e => e.profile.id === "runner-acpx-copilot"); + const selected = runnerMatrix.filter(e => !isRollingWebDeployment(e) && e.profile.id === "runner-acpx-copilot"); expect(selected).toHaveLength(20); expect(selected.filter(e => e.environment.id === "local")).toHaveLength(10); expect(selected.filter(e => e.environment.id === "daytona")).toHaveLength(10); diff --git a/tests/runner-e2e/installed-daytona-plugin.test.ts b/tests/runner-e2e/installed-daytona-plugin.test.ts index 1f2ed5a29a8..953a314c905 100644 --- a/tests/runner-e2e/installed-daytona-plugin.test.ts +++ b/tests/runner-e2e/installed-daytona-plugin.test.ts @@ -1,3 +1,4 @@ +import { isRollingWebDeployment } from "./controller-rolling-deployment.js"; import { execFileSync } from "node:child_process"; import { createHash } from "node:crypto"; import { readFileSync } from "node:fs"; @@ -36,7 +37,7 @@ it("admits pinned published plugin and resolved SDK graph without claiming a ful }); it("admits all ten Copilot Daytona cells only with the pinned normal plugin graph", async () => { const f = await fixture(); - const selected = runnerMatrix.filter(e => e.profile.id === "runner-acpx-copilot" && e.environment.id === "daytona"); + const selected = runnerMatrix.filter(e => !isRollingWebDeployment(e) && e.profile.id === "runner-acpx-copilot" && e.environment.id === "daytona"); expect(selected).toHaveLength(10); for (const selectedCell of selected) { expect(await verifyInstalledDaytonaPlugin(f.env, [selectedCell])).toMatchObject({ packageRoot: f.authority.plugin.root, sourceFallback: false, fullGraphVerified: false, completeGraphAuthorityRequired: true }); diff --git a/tests/runner-e2e/pi-file-evidence.test.ts b/tests/runner-e2e/pi-file-evidence.test.ts index 0eda6cd77a4..a66111a3786 100644 --- a/tests/runner-e2e/pi-file-evidence.test.ts +++ b/tests/runner-e2e/pi-file-evidence.test.ts @@ -1,3 +1,4 @@ +import { isRollingWebDeployment } from "./controller-rolling-deployment.js"; import { execFile } from "node:child_process"; import { promisify } from "node:util"; import { createHash } from "node:crypto"; @@ -206,7 +207,7 @@ describe("Pi edit, validation and public artifact oracle", () => { expect(prompt).toContain(`title ${artifact.name},`); } } - expect(runnerMatrix.filter(c => c.profile.qualificationCandidate === "pi")).toHaveLength(26); + expect(runnerMatrix.filter(c => !isRollingWebDeployment(c) && c.profile.qualificationCandidate === "pi")).toHaveLength(26); }); it.each(["local", "daytona"] as const)("seeds once and collects public %s evidence", async environment => { const root = await mkdtemp(join(tmpdir(), "pi-file-evidence-")); diff --git a/tests/runner-e2e/remote-workspace-storage.test.ts b/tests/runner-e2e/remote-workspace-storage.test.ts index 64f9227b93c..4fa50e72462 100644 --- a/tests/runner-e2e/remote-workspace-storage.test.ts +++ b/tests/runner-e2e/remote-workspace-storage.test.ts @@ -56,6 +56,36 @@ describe("remote workspace Product E2E qualification", () => { expect(result.content).toBe(`${"a".repeat(32)}\n`); }); + it("reads the exact saved snapshot after per-turn sandbox retirement", async () => { + const f = checkpointApi(); + const originalGet = f.api.get; + const location = f.workspace.workspace.metadata.remoteWorkspace.location; + const detached = { ...f.workspace, workspace: { ...f.workspace.workspace, + metadata: { remoteWorkspace: { ...f.workspace.workspace.metadata.remoteWorkspace, location: null } } } }; + const file = { ...f.file, resource: { ...f.file.resource, source: { kind: "saved", snapshotId: f.receipt.snapshotId } } }; + const lease = { id: location.leaseId, companyId: f.context.companyId, issueId: f.context.issueId, + providerLeaseId: location.providerLeaseId }; + const api = { get: async (route: string): Promise => { + if (route.endsWith("/workspace")) return detached as T; + if (route.endsWith("/leases")) return [lease] as T; + if (route.includes("file-resources/content")) return file as T; + return originalGet(route); + } }; + const input = { ...f.context, api, expectedFileSource: "saved" as const }; + expect((await readRemoteWorkspaceStorageCheckpoint(input)).point).toMatchObject({ fileSource: "saved", fileSnapshotId: id("2") }); + expect(await readRemoteWorkspaceStorageCheckpoint({ ...input, expectedFileSource: "live" })).toEqual({}); + file.resource.source.snapshotId = id("5"); + expect(await readRemoteWorkspaceStorageCheckpoint(input)).toEqual({}); + file.resource.source.snapshotId = f.receipt.snapshotId; + lease.companyId = id("5"); + expect(await readRemoteWorkspaceStorageCheckpoint(input)).toEqual({}); + const evidence = goodEvidence(); evidence.expectedFileSource = "saved"; + for (const point of [evidence.initial!, evidence.final!]) Object.assign(point, { fileSource: "saved", fileSnapshotId: point.snapshotId }); + expect(gradeRemoteWorkspaceStorage(evidence).every(check => check.passed)).toBe(true); + evidence.final!.fileSnapshotId = id("9"); + expect(gradeRemoteWorkspaceStorage(evidence).every(check => check.passed)).toBe(false); + }); + it.each(["run-id", "company", "agent", "native-issue", "runtime", "status", "schema", "workspace", "epoch", "snapshot", "save-state", "lease", "provider", "remote-root"] as const)( "rejects a full-run %s mismatch even when its public summary succeeds", async mismatch => { diff --git a/tests/runner-e2e/remote-workspace-storage.ts b/tests/runner-e2e/remote-workspace-storage.ts index 7a820c60bd9..6275706a4e0 100644 --- a/tests/runner-e2e/remote-workspace-storage.ts +++ b/tests/runner-e2e/remote-workspace-storage.ts @@ -23,9 +23,11 @@ export interface RemoteWorkspaceStorageCheckpoint { providerType: string; proofSha256: string; fileSource: string; + fileSnapshotId?: string; } export interface RemoteWorkspaceStorageEvidence { + expectedFileSource?: "live" | "saved"; initial?: RemoteWorkspaceStorageCheckpoint; final?: RemoteWorkspaceStorageCheckpoint; browserReadPhases: string[]; @@ -47,6 +49,7 @@ export function remoteWorkspaceProofDigest(content: string): string { export async function readRemoteWorkspaceStorageCheckpoint(input: { api: Pick; companyId: string; agentId: string; issueId: string; environmentId: string; + expectedFileSource?: "live" | "saved"; }): Promise<{ point?: RemoteWorkspaceStorageCheckpoint; content?: string }> { const view = await input.api.get<{ issueId: string; workspace: ExecutionWorkspace | null }>(`/api/issues/${input.issueId}/workspace`); const workspace = view.workspace; @@ -54,8 +57,10 @@ export async function readRemoteWorkspaceStorageCheckpoint(input: { if (view.issueId !== input.issueId || !workspace || workspace.companyId !== input.companyId || workspace.providerType !== "cloud_sandbox" || !authorityResult.success) return {}; const authority = authorityResult.data; - if (!authority.snapshotId || authority.dirty || authority.writerRunId || !authority.location - || authority.location.environmentId !== input.environmentId) return {}; + const expectedSource = input.expectedFileSource ?? "live"; + if (!authority.snapshotId || authority.dirty || authority.writerRunId) return {}; + if (expectedSource === "live" && (!authority.location || authority.location.environmentId !== input.environmentId)) return {}; + if (expectedSource === "saved" && authority.location !== null) return {}; const runs = await input.api.get(`/api/issues/${input.issueId}/runs`); const latest = runs.filter(run => run.runtimeMode === "native" && run.status === "succeeded" && run.agentId === input.agentId && typeof run.runId === "string") @@ -68,19 +73,27 @@ export async function readRemoteWorkspaceStorageCheckpoint(input: { if (!receiptResult.success) return {}; const receipt = receiptResult.data; if (receipt.state !== "saved" || receipt.snapshotId !== authority.snapshotId - || receipt.workspaceId !== workspace.id || receipt.epoch !== authority.epoch - || receipt.leaseId !== authority.location.leaseId || receipt.providerLeaseId !== authority.location.providerLeaseId - || receipt.remoteCwd !== authority.location.remoteCwd) return {}; + || receipt.workspaceId !== workspace.id || receipt.epoch !== authority.epoch) return {}; + if (authority.location && (receipt.leaseId !== authority.location.leaseId + || receipt.providerLeaseId !== authority.location.providerLeaseId || receipt.remoteCwd !== authority.location.remoteCwd)) return {}; + if (expectedSource === "saved") { + const leases = await input.api.get>( + `/api/environments/${input.environmentId}/leases`); + if (!leases.some(lease => lease.id === receipt.leaseId && lease.companyId === input.companyId + && lease.issueId === input.issueId && lease.providerLeaseId === receipt.providerLeaseId)) return {}; + } const file = await input.api.get(`/api/issues/${input.issueId}/file-resources/content?workspace=execution&path=${REMOTE_WORKSPACE_PROOF_FILE}`); if (file.content?.encoding !== "utf8" || typeof file.content.data !== "string") throw new Error("Remote workspace file oracle did not return text bytes"); - if (file.resource?.source?.kind !== "live" || file.resource.workspaceKind !== "execution_workspace" + if (file.resource?.source?.kind !== expectedSource || file.resource.workspaceKind !== "execution_workspace" || file.resource.workspaceId !== workspace.id) return {}; + const fileSnapshotId = file.resource.source.kind === "saved" ? file.resource.source.snapshotId ?? undefined : undefined; + if (expectedSource === "saved" && fileSnapshotId !== authority.snapshotId) return {}; return { content: file.content.data, point: { workspaceId: workspace.id, snapshotId: authority.snapshotId, runId: run.id, runSnapshotId: receipt.snapshotId, runSaveState: receipt.state, authoritySchema: authority.schema, authorityDirty: authority.dirty, authorityWriter: authority.writerRunId, providerType: workspace.providerType, - proofSha256: remoteWorkspaceProofDigest(file.content.data), fileSource: file.resource?.source?.kind, + proofSha256: remoteWorkspaceProofDigest(file.content.data), fileSource: file.resource?.source?.kind, fileSnapshotId, } }; } @@ -91,7 +104,9 @@ export function gradeRemoteWorkspaceStorage(evidence: RemoteWorkspaceStorageEvid && point.snapshotId === point.runSnapshotId && point.runSaveState === "saved" && point.authoritySchema === "paperclip.remote-workspace.v1" && point.authorityDirty === false && point.authorityWriter === null && point.providerType === "cloud_sandbox" - && point.fileSource === "live" && /^[a-f0-9]{64}$/.test(point.proofSha256)); + && point.fileSource === (evidence.expectedFileSource ?? "live") + && (point.fileSource !== "saved" || point.fileSnapshotId === point.snapshotId) + && /^[a-f0-9]{64}$/.test(point.proofSha256)); return [ { id: "remote-workspace-initial-save", passed: committed(evidence.initial), detail: "The initial task admission commits a remote workspace snapshot and a matching saved run receipt." }, { id: "remote-workspace-final-save", passed: committed(evidence.final), detail: "The later admission commits its files before finalization succeeds." },