Jannik/PaperClipAI
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-11 13:24:55 +01:00
Code Issues Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'origin/master' into chat-eval/master-candidate

Browse Source 
* origin/master: (21 commits)
  fix(secrets): delete a secret that a binding proposal referenced (#20193)
  perf(runner): save provider state once per poll instead of once per frame (#20217)
  fix(native): an inherited checkpoint with a pending runtime request is evidence (#20226)
  fix(chat): post a turn's answer when a card it did not create is pending (#20220)
  fix(native): roll back a warm rotation cut off before its new control plane once its runner is proven dead (#20218)
  fix(native): restart an unclaimed remote follow-up that carries only its inherited checkpoint (#20214)
  fix(native-runtime): a stopped run cannot close a task another run continues; deliver a pre-restart comment at once (#20216)
  fix(daytona): wait out a sandbox state change before starting it for workspace realize (#20212)
  fix(authz): require board authority for agent budget changes on every route (#20213)
  fix(native): retire a finished remote owner whose suspend never finished once its sandbox leases ended (#20211)
  fix(guidance): tell agents who acts and which page to link for person-only steps (#20206)
  fix(runner): keep discarded OpenCode token deltas from delaying tool calls (#20209)
  fix(authz): say every plugin management step is board-only in plugin denials (#20207)
  perf(budgets): judge Paperclip's share of restart recovery; report the provider's session load (#20183)
  fix(runner): keep a restart continuation when the lost turn's terminal is delivered again (#20196)
  fix(chat): recover a held conversation turn's answer after a restart (#20204)
  test(chaos): honor 429 retry delays on harness reads (#20194)
  perf(environments): release a stopped plugin task once (#20203)
  perf(native-runtime): claim a parked Runner while the follow-up run prepares (#20195)
  perf(plugins): read runtime identity in one statement without a row lock (#20192)
  ...

# Conflicts:
#	maps/agent-access-map/agents.md
#	server/src/__tests__/agent-budget-configuration.test.ts
#	server/src/__tests__/agent-permissions-routes.test.ts
#	server/src/__tests__/built-in-agent-routes.test.ts
#	server/src/routes/agents.ts
#	server/src/routes/authz.ts
#	server/src/routes/built-in-agents.ts
#	server/src/routes/openapi.ts
#	server/src/services/heartbeat/run-completion.ts
#	server/src/services/native-runtime/agent-chat-response-recovery.test.ts
#	server/src/services/native-runtime/native-run-finalizer.ts
This commit is contained in:
Dotta committed 2026-11-09 04:06:37 -06:00
2 parents 8a1bea59ac d2fe85e57c
commit 59c64f78de
75 files changed
+2931 -323
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

doc/execution-semantics.md
+6 -2
View File
@@ -2276,8 +2276,12 @@ In an Agent Chat conversation, a successful turn whose status decision committed
posts its final answer to the conversation even when the task stays held: a
pending or revision-requested approval, a pending execution stage, or a
dependency blocker. A held task does not present the turn. Only a pending
thread interaction (a question or confirmation card) presents the turn in place
of its prose.
thread interaction (a question or confirmation card) that the turn's own run
created presents the turn in place of its prose. A card from an earlier turn,
or one the board addressed to another person, can hold the task but does not
present the turn, so its answer is still posted. Background presentation recovery applies the same rule, so a
restart between the status commit and the live presentation posts the held
turn's answer once, or records that its card presented the turn.
A delegated-task completion callback can finish native execution before its
accepted final reply is saved. The completion outbox must retain that callback's
doc/plans/2026-11-07-durable-ownership-and-reconciliation.md
+6
View File
@@ -99,6 +99,12 @@ Restart recovery latency, from `ready` to the recovered run's next provider even
| codex | 6.5 / 8.0 |
| opencode | 25.5 / 38.5 |
Since 2026-11-09 this full latency is reported, not judged: the provider's saved-session load (`provider.session.load`) grows with session size and with how many sessions resume at once (Codex at concurrency 6: 3.0–6.3 s of a 5.5–8.5 s recovery). The judged row is Paperclip's own share, `restart_recovery_control_plane`: from `ready` to the start of the provider's session load, or to the next provider event when nothing reloads. It covers claim, prepare, runtime stage and runner launch, which measured 1.2–2.0 s per run with six Codex runs resuming at once (wave `w-20261109T045320-s9807`).
| Adapter | Paperclip share budget (s, p50 / p95) |
|---|---|
| claude, codex, opencode | 2.5 / 4.0 |
These budgets measure the product default `lifecycleMode: per_turn`. Phase 1's TTFT work (runner and provider-session reuse, baseline finding 1) should lower them. Lower budgets go in the same table.
### Phase 1 report-only data: local chaos waves (2026-11-07)
doc/plugins/PLUGIN_RUNNER_EXECUTION.md
+9
View File
@@ -344,6 +344,15 @@ unsettled Core tool result, other close error, or continuation of the same
session still blocks replacement.
A controller restart is not required for this stopped-owner reset path.
Core ends a dispatched task lease with `stop`, `status`, and `complete`. When
that sequence confirms the stop and completes the same task, Core does not also
call `onEnvironmentReleaseLease`. Core still calls it when the release did not
complete a dispatched task, and the cleanup retry calls it after any failure.
When `onEnvironmentResumeLease` returns no provider lease, Core sends `status`
first. It sends `stop` only when that `status` does not report
`executionStopped: true`. A driver must release task-scoped resources in its
`status` or `complete` handling, not only in `onEnvironmentReleaseLease`.
Termination receipts cover all run leases that owned the retired physical task.
A stopped task gets a new provider task ID for checkpoint resumption. Inspection
precedes in-memory session selection so a replacement cannot inherit the stopped
doc/run-log-events.md
+15 -3
View File
@@ -64,6 +64,11 @@ instance run log and add no first-party Telemetry or OpenTelemetry export.
## Provider-pack preparation timing
Every `run.performance.span` payload also carries `wallStartedAtMs`, the span's own
start time in epoch milliseconds (since 2026-11-09). Span rows can be written late
through deferred diagnostics, so readers take a span's start from this field (or
from the trace start plus `startOffsetMs`), never from the row's write time.
`run.performance.span` also records `task.time_to_first_token` once per native
dispatch at the first newly committed deliverable text: assistant or reasoning
`item.delta`/`item.completed`, a redaction tail released at completion, or an
@@ -97,8 +102,15 @@ budgets come from the Phase 0 local baseline and are listed in
the first text delta, tool-call start or reasoning item; named `ttft` and
judged on `task.time_to_first_token` before 2026-11-08), `agent_boot` (end of
`heartbeat.queue` to the end of `runner.turn.submit` or `runner.session.resume`)
or `restart_recovery` (startup recovery `ready` to the recovered run's first
provider event). The payload names `adapter` (`codex`, `claude` or `opencode`),
or `restart_recovery_control_plane` (startup recovery `ready` to the start of the
provider's saved-session load, `provider.session.load`, or to the recovered run's
first provider event when nothing reloads). That is Paperclip's own share of a
recovery: claim, prepare, runtime stage and runner launch. The full recovery
(`ready` to the first provider event) and the provider's session load grow with
session size and with how many sessions resume at once, so they are reported in
the payload as `restartTotalMs` and `providerLoadMs` and are not judged; events
written before 2026-11-09 judged the full latency as `restart_recovery`.
The payload names `adapter` (`codex`, `claude` or `opencode`),
`environment` (`local`), `startKind` (`cold`, `warm` or `unknown`; an unknown
start uses the looser budget), `measuredMs`, `budgetP50Ms`, `budgetP95Ms`,
`percentile` (`p95`), `baseline`, and, for restart recovery, `disposition`.
@@ -106,7 +118,7 @@ For `time_to_first_activity`, `measuredMs` counts from run creation: when the in
run to exist (for example a comment posted while the previous turn was still
running), that wait is excluded from the judgment and reported as `inputWaitMs`.
Each metric is checked at most once per dispatch. A restart-recovery dispatch is
judged only on `restart_recovery`, because its first activity and boot span the outage. Time to first text (`task.time_to_first_token`) is reported without a budget. Remote runs and other adapters have
judged only on `restart_recovery_control_plane`, because its first activity and boot span the outage. Time to first text (`task.time_to_first_token`) is reported without a budget. Remote runs and other adapters have
no budget yet. The event is written in detached diagnostic work after the
measurement, so it adds no wait before `turn.start`. It is a run-log row only
and never goes to Telemetry.
maps/agent-access-map/agents.md
+1 -1
View File
@@ -212,7 +212,7 @@ CLI.
Policy: Agent creation/configuration uses action grants, resource policies and change
consent; some controls require a board actor.
Changes to the legacy `budgetMonthlyCents` field and built-in provision budget overrides require a board actor, including budget-changing rollbacks. Agent-authored rollbacks with an unchanged amount do not write budget policy or policy audit. Board mutations retain the actual operator identity.
Changes to the legacy `budgetMonthlyCents` field and built-in provision budget overrides require a board actor, including budget-changing rollbacks.
The roster intent preserves unrelated settings, rejects stale model/connection
previews and harness changes, and checks the full execution configuration under
packages/db/src/migrations/0452_secret_proposal_secret_cascade.sql
+3
View File
@@ -0,0 +1,3 @@
ALTER TABLE "company_secret_proposals" DROP CONSTRAINT IF EXISTS "company_secret_proposals_secret_id_company_secrets_id_fk";
--> statement-breakpoint
ALTER TABLE "company_secret_proposals" ADD CONSTRAINT "company_secret_proposals_secret_id_company_secrets_id_fk" FOREIGN KEY ("secret_id") REFERENCES "public"."company_secrets"("id") ON DELETE cascade ON UPDATE no action;
packages/db/src/migrations/meta/0447_snapshot.json → packages/db/src/migrations/meta/0452_snapshot.json
Generated
+287 -3
View File
@@ -1,6 +1,6 @@
{
"id": "76f06e29-53a8-4ecc-809a-d52b6bbb6520",
"prevId": "782688f1-7843-4a93-8ea3-2cec362d269e",
"id": "64e8be1e-e0cb-4cd8-85e0-0fdcce5eafde",
"prevId": "b5699366-77c2-48c8-b7dc-658a75de7658",
"version": "7",
"dialect": "postgresql",
"tables": {
@@ -4102,6 +4102,28 @@
"method": "btree",
"with": {}
},
"agent_wakeup_requests_issue_blocker_stalled_uq": {
"name": "agent_wakeup_requests_issue_blocker_stalled_uq",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "idempotency_key",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": true,
"where": "\"agent_wakeup_requests\".\"idempotency_key\" LIKE 'issue_blocker_stalled:%' AND \"agent_wakeup_requests\".\"status\" IN ('queued', 'deferred_issue_execution', 'claimed', 'coalesced', 'completed')",
"concurrently": true,
"method": "btree",
"with": {}
},
"agent_wakeup_requests_company_payload_issue_idx": {
"name": "agent_wakeup_requests_company_payload_issue_idx",
"columns": [
@@ -4252,6 +4274,18 @@
"primaryKey": false,
"notNull": false
},
"harness_verified_digest": {
"name": "harness_verified_digest",
"type": "text",
"primaryKey": false,
"notNull": false
},
"harness_verified_at": {
"name": "harness_verified_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"reports_to": {
"name": "reports_to",
"type": "uuid",
@@ -7848,6 +7882,150 @@
},
"isRLSEnabled": false
},
"public.budget_resume_claims": {
"name": "budget_resume_claims",
"schema": "",
"columns": {
"run_id": {
"name": "run_id",
"type": "uuid",
"primaryKey": true,
"notNull": true
},
"company_id": {
"name": "company_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"incident_id": {
"name": "incident_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"actor_user_id": {
"name": "actor_user_id",
"type": "text",
"primaryKey": false,
"notNull": true
},
"actor_source": {
"name": "actor_source",
"type": "text",
"primaryKey": false,
"notNull": false
},
"state": {
"name": "state",
"type": "text",
"primaryKey": false,
"notNull": true,
"default": "'pending'"
},
"claimed_at": {
"name": "claimed_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"settled_at": {
"name": "settled_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"reason": {
"name": "reason",
"type": "text",
"primaryKey": false,
"notNull": false
},
"request_id": {
"name": "request_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {
"budget_resume_claims_pending_idx": {
"name": "budget_resume_claims_pending_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "run_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"where": "\"budget_resume_claims\".\"state\" = 'pending'",
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {
"budget_resume_claims_run_id_heartbeat_runs_id_fk": {
"name": "budget_resume_claims_run_id_heartbeat_runs_id_fk",
"tableFrom": "budget_resume_claims",
"tableTo": "heartbeat_runs",
"columnsFrom": [
"run_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"budget_resume_claims_company_id_companies_id_fk": {
"name": "budget_resume_claims_company_id_companies_id_fk",
"tableFrom": "budget_resume_claims",
"tableTo": "companies",
"columnsFrom": [
"company_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"budget_resume_claims_incident_id_budget_incidents_id_fk": {
"name": "budget_resume_claims_incident_id_budget_incidents_id_fk",
"tableFrom": "budget_resume_claims",
"tableTo": "budget_incidents",
"columnsFrom": [
"incident_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.built_in_managed_resources": {
"name": "built_in_managed_resources",
"schema": "",
@@ -17221,7 +17399,7 @@
"columnsTo": [
"id"
],
"onDelete": "set null",
"onDelete": "cascade",
"onUpdate": "no action"
},
"company_secret_proposals_secret_proposal_id_company_secret_proposals_id_fk": {
@@ -21965,6 +22143,112 @@
"checkConstraints": {},
"isRLSEnabled": false
},
"public.controller_instances": {
"name": "controller_instances",
"schema": "",
"columns": {
"boot_id": {
"name": "boot_id",
"type": "text",
"primaryKey": true,
"notNull": true
},
"epoch": {
"name": "epoch",
"type": "bigserial",
"primaryKey": false,
"notNull": true
},
"process_space": {
"name": "process_space",
"type": "text",
"primaryKey": false,
"notNull": true
},
"pid": {
"name": "pid",
"type": "integer",
"primaryKey": false,
"notNull": true
},
"started_at": {
"name": "started_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
},
"renewed_at": {
"name": "renewed_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
},
"lease_expires_at": {
"name": "lease_expires_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true
},
"fenced_at": {
"name": "fenced_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"fenced_by_epoch": {
"name": "fenced_by_epoch",
"type": "bigint",
"primaryKey": false,
"notNull": false
},
"stopped_at": {
"name": "stopped_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
}
},
"indexes": {
"controller_instances_epoch_idx": {
"name": "controller_instances_epoch_idx",
"columns": [
{
"expression": "epoch",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": true,
"concurrently": false,
"method": "btree",
"with": {}
},
"controller_instances_lease_idx": {
"name": "controller_instances_lease_idx",
"columns": [
{
"expression": "lease_expires_at",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.cost_events": {
"name": "cost_events",
"schema": "",
packages/db/src/migrations/meta/_journal.json
Generated
+7
View File
@@ -3151,6 +3151,13 @@
"when": 1794185816322,
"tag": "0451_budget_resume_claims",
"breakpoints": true
},
{
"idx": 452,
"version": "7",
"when": 1794205579544,
"tag": "0452_secret_proposal_secret_cascade",
"breakpoints": true
}
]
}
packages/db/src/schema/company_secret_proposals.ts
+3 -1
View File
@@ -21,7 +21,9 @@ export const companySecretProposals = pgTable(
valueCiphertext: jsonb("value_ciphertext").$type<Record<string, unknown> | null>(),
valueFingerprintSha256: text("value_fingerprint_sha256"),
valueLength: integer("value_length"),
secretId: uuid("secret_id").references(() => companySecrets.id, { onDelete: "set null" }),
// Binding proposals require exactly one source reference (shape check), so
// they go with the secret they reference, like secretProposalId below.
secretId: uuid("secret_id").references(() => companySecrets.id, { onDelete: "cascade" }),
secretProposalId: uuid("secret_proposal_id").references((): AnyPgColumn => companySecretProposals.id, { onDelete: "cascade" }),
targetType: text("target_type"),
targetId: uuid("target_id").references(() => agents.id, { onDelete: "cascade" }),
packages/paperclip-runner/runner/CODEX_PROVIDER.md
+4 -1
View File
@@ -29,7 +29,10 @@ execution and finalization path.
The provider descriptor, Codex thread ID, account session ID, active Codex turn
ID, and unacknowledged normalized-event prefix are written to a private,
bounded, atomically replaced sidecar in the runner state directory. On restart,
bounded, atomically replaced sidecar in the runner state directory. runnerd
writes the sidecar once for the provider frames read in one poll, not once per
frame. The write always comes before an event, a command result, or a reply to
the provider that depends on those frames leaves runnerd. On restart,
runnerd resumes that exact thread and reconciles the active turn from
`thread/read`. An unexpected provider exit retains the last active turn until
that reconciliation proves whether it is still running, so cancellation and a
packages/paperclip-runner/runner/crates/runner-core/src/codex_provider.rs
+45
View File
@@ -869,6 +869,11 @@ pub struct CodexProvider {
quarantined: bool,
trace: Option<ProviderTraceSink>,
last_trace_frame_id: Option<u64>,
held_courtesy_responses: Option<Vec<Value>>,
#[cfg(test)]
pub(crate) sent_frame_observer: Option<Box<dyn FnMut(&Value) + Send>>,
#[cfg(test)]
pub(crate) interpretation_observer: Option<Box<dyn FnMut(&[String]) + Send>>,
opencode_launch_profile: Option<OpenCodeLaunchProfile>,
cursor_cloud_launch_profile: Option<CursorCloudLaunchProfile>,
provider_termination_receipt: Option<Value>,
@@ -1300,6 +1305,11 @@ impl CodexProvider {
quarantined: false,
trace: ProviderTraceSink::from_environment(),
last_trace_frame_id: None,
held_courtesy_responses: None,
#[cfg(test)]
sent_frame_observer: None,
#[cfg(test)]
interpretation_observer: None,
opencode_launch_profile: opencode_launch_profile.cloned(),
cursor_cloud_launch_profile: cursor_cloud_launch_profile.cloned(),
provider_termination_receipt: None,
@@ -1467,6 +1477,11 @@ impl CodexProvider {
self.last_trace_frame_id.take()
}
#[cfg(test)]
pub(crate) fn enable_trace_for_test(&mut self, trace_path: PathBuf) {
self.trace = Some(ProviderTraceSink::at_path(trace_path, 64 * 1024 * 1024));
}
pub(crate) fn record_provider_trace_interpretation(
&mut self,
frame_id: u64,
@@ -1475,6 +1490,10 @@ impl CodexProvider {
emitted_event_ids: Vec<String>,
reason: &str,
) {
#[cfg(test)]
if let Some(observer) = self.interpretation_observer.as_mut() {
observer(&emitted_event_ids);
}
if let Some(trace) = self.trace.as_mut() {
trace.interpretation(
frame_id,
@@ -3831,10 +3850,36 @@ impl CodexProvider {
{
return Ok(());
}
if let Some(held) = self.held_courtesy_responses.as_mut() {
held.push(value.clone());
return Ok(());
}
self.send_frame(value)
}
pub(crate) fn hold_courtesy_responses(&mut self) {
self.held_courtesy_responses.get_or_insert_with(Vec::new);
}
pub(crate) fn release_courtesy_responses(&mut self, keep_holding: bool) {
let held = if keep_holding {
self.held_courtesy_responses
.as_mut()
.map(std::mem::take)
.unwrap_or_default()
} else {
self.held_courtesy_responses.take().unwrap_or_default()
};
for value in held {
let _ = self.send_frame(&value);
}
}
fn send_frame(&mut self, value: &Value) -> Result<(), LocalRunnerError> {
#[cfg(test)]
if let Some(observer) = self.sent_frame_observer.as_mut() {
observer(value);
}
if let Some(trace) = self.trace.as_mut() {
let raw = serde_json::to_vec(value).unwrap_or_default();
if let Some(frame_id) = trace.frame("client_to_provider", &raw) {
packages/paperclip-runner/runner/crates/runner-core/src/provider_backend.rs
+149 -41
View File
@@ -79,6 +79,18 @@ fn invalid_tool_call_result(
}
}
fn serialize_provider_state(state: &CodexProviderState) -> Result<Vec<u8>, DurableRunnerError> {
let bytes = serde_json::to_vec_pretty(state).map_err(|error| {
DurableRunnerError::invalid(format!("failed to serialize Codex provider state: {error}"))
})?;
if bytes.len() as u64 > MAX_PROVIDER_STATE_BYTES {
return Err(DurableRunnerError::invalid(
"Codex provider state exceeds the 16 MiB limit",
));
}
Ok(bytes)
}
fn receipt_limit_deadline_after(timeout_ms: u64) -> Result<u64, DurableRunnerError> {
current_unix_ms()?.checked_add(timeout_ms).ok_or_else(|| {
DurableRunnerError::invalid("Codex receipt-limit interruption deadline overflowed")
@@ -1973,6 +1985,13 @@ impl CodexProviderState {
}
}
struct NormalizationTrace {
frame_id: u64,
rule_id: String,
mapped: bool,
emitted_event_ids: Vec<String>,
}
pub struct CodexCommandExecutor {
state_dir: PathBuf,
state: Option<CodexProviderState>,
@@ -1985,6 +2004,10 @@ pub struct CodexCommandExecutor {
startup_command: Option<ProviderStartupCommand>,
startup_evidence_error: Option<DurableRunnerError>,
recovery_pending_effects: Option<usize>,
unsaved_frame_state: std::cell::Cell<bool>,
unsaved_trace_interpretations: Vec<NormalizationTrace>,
#[cfg(test)]
persist_count: std::cell::Cell<usize>,
}
impl CodexCommandExecutor {
@@ -2001,6 +2024,10 @@ impl CodexCommandExecutor {
startup_command: None,
startup_evidence_error: None,
recovery_pending_effects: None,
unsaved_frame_state: std::cell::Cell::new(false),
unsaved_trace_interpretations: Vec::new(),
#[cfg(test)]
persist_count: std::cell::Cell::new(0),
}
}
@@ -2960,11 +2987,69 @@ impl CodexCommandExecutor {
.state
.as_ref()
.ok_or_else(|| DurableRunnerError::invalid("Codex provider state is unavailable"))?;
self.persist_state(state)
self.persist_state(state)?;
self.unsaved_frame_state.set(false);
Ok(())
}
fn save_unsaved_frame_state(&self) -> Result<(), DurableRunnerError> {
if !self.unsaved_frame_state.get() {
return Ok(());
}
let state = self
.state
.as_ref()
.ok_or_else(|| DurableRunnerError::invalid("Codex provider state is unavailable"))?;
let bytes = serialize_provider_state(state)?;
if fs::read(self.state_path()).is_ok_and(|saved| saved == bytes) {
self.unsaved_frame_state.set(false);
return Ok(());
}
state.validate()?;
self.write_state_bytes(&bytes)?;
self.unsaved_frame_state.set(false);
Ok(())
}
fn record_saved_trace_interpretations(&mut self) {
let traces = std::mem::take(&mut self.unsaved_trace_interpretations);
let Some(provider) = self.provider.as_mut() else {
return;
};
for trace in traces {
provider.record_provider_trace_interpretation(
trace.frame_id,
&trace.rule_id,
if trace.mapped { "mapped" } else { "ignored" },
trace.emitted_event_ids,
if trace.mapped {
"Provider notification normalized into durable PRP events"
} else {
"Provider notification did not produce a durable PRP event"
},
);
}
}
fn save_before_provider_writes(
&mut self,
still_polling: bool,
) -> Result<(), DurableRunnerError> {
self.save_unsaved_frame_state()?;
self.record_saved_trace_interpretations();
if let Some(provider) = self.provider.as_mut() {
provider.release_courtesy_responses(still_polling);
}
Ok(())
}
fn persist_state(&self, state: &CodexProviderState) -> Result<(), DurableRunnerError> {
state.validate()?;
let bytes = serialize_provider_state(state)?;
self.write_state_bytes(&bytes)
}
fn write_state_bytes(&self, bytes: &[u8]) -> Result<(), DurableRunnerError> {
fs::create_dir_all(&self.state_dir).map_err(|error| {
DurableRunnerError::invalid(format!(
"failed to create provider state directory: {error}"
@@ -2980,19 +3065,9 @@ impl CodexCommandExecutor {
)?;
verify_private_directory(&self.state_dir)?;
let path = self.state_path();
let bytes = serde_json::to_vec_pretty(state).map_err(|error| {
DurableRunnerError::invalid(format!(
"failed to serialize Codex provider state: {error}"
))
})?;
if bytes.len() as u64 > MAX_PROVIDER_STATE_BYTES {
return Err(DurableRunnerError::invalid(
"Codex provider state exceeds the 16 MiB limit",
));
}
let (temporary, mut file) = create_private_temporary_file(&path)?;
let result = (|| -> std::io::Result<()> {
file.write_all(&bytes)?;
file.write_all(bytes)?;
file.sync_all()?;
drop(file);
fs::rename(&temporary, &path)?;
@@ -3010,6 +3085,8 @@ impl CodexCommandExecutor {
fs::set_permissions(&path, fs::Permissions::from_mode(0o600)).map_err(|error| {
DurableRunnerError::invalid(format!("failed to protect provider state: {error}"))
})?;
#[cfg(test)]
self.persist_count.set(self.persist_count.get() + 1);
Ok(())
}
@@ -4515,6 +4592,7 @@ impl CodexCommandExecutor {
call_id: String,
operation_id: String,
) -> Result<(), DurableRunnerError> {
self.save_before_provider_writes(true)?;
let deadline_unix_ms =
receipt_limit_deadline_after(RECEIPT_LIMIT_INTERRUPT_TERMINAL_DEADLINE_MS)?;
let state = self
@@ -4562,7 +4640,12 @@ impl CodexCommandExecutor {
.expect("Codex state remains available before receipt-limit interruption")
.record_receipt_limit_interrupt_attempt()?;
self.save_state()?;
match self.interrupt_turn("semantic_tool_turn_receipt_limit") {
self.save_before_provider_writes(false)?;
let interrupted = self.interrupt_turn("semantic_tool_turn_receipt_limit");
if let Some(provider) = self.provider.as_mut() {
provider.hold_courtesy_responses();
}
match interrupted {
Ok(_) => {
let accepted_deadline_unix_ms =
receipt_limit_deadline_after(RECEIPT_LIMIT_ACCEPTED_TERMINAL_DEADLINE_MS)?;
@@ -4782,6 +4865,7 @@ impl CodexCommandExecutor {
.admit_tool_call(&call_id, &operation_id, &input);
match admission {
Ok(ToolCallAdmission::CompletedReplay(result)) => {
self.save_before_provider_writes(true)?;
self.provider
.as_mut()
.expect("provider remains present while handling its tool call")
@@ -5113,6 +5197,25 @@ impl CodexCommandExecutor {
if self.provider.is_none() {
return self.settle_receipt_limit_interrupt_if_deadline_elapsed();
}
if let Some(provider) = self.provider.as_mut() {
provider.hold_courtesy_responses();
}
let polled = self.poll_provider_frames(allow_control, receipt_limit_terminal_poll);
let saved = self.save_before_provider_writes(false);
polled?;
saved?;
// Check the durable deadline only after a complete provider poll. A
// terminal that arrived after interruption acceptance but before this
// observation remains authoritative even when several fast controller
// polls have already exhausted the interruption-attempt budget.
self.settle_receipt_limit_interrupt_if_deadline_elapsed()
}
fn poll_provider_frames(
&mut self,
allow_control: bool,
receipt_limit_terminal_poll: bool,
) -> Result<(), DurableRunnerError> {
for _ in 0..MAX_EVENTS_PER_POLL {
let provider = self
.provider
@@ -5175,6 +5278,7 @@ impl CodexCommandExecutor {
))?;
// Commit the authoritative failure before best-effort provider cleanup.
self.save_state()?;
self.record_saved_trace_interpretations();
if let Some(mut provider) = self.provider.take() {
if let Some(frame_id) = trace_frame_id {
provider.record_provider_trace_interpretation(
@@ -5225,7 +5329,7 @@ impl CodexCommandExecutor {
"method": method.chars().take(128).collect::<String>(), "receivedThreadId": child, "expectedThreadId": root_thread,
"receivedTurnId": child_turn, "expectedTurnId": root_turn }),
}])?;
self.save_state()?;
self.unsaved_frame_state.set(true);
if let (Some(frame_id), Some(provider)) =
(trace_frame_id, self.provider.as_mut())
{
@@ -5245,6 +5349,9 @@ impl CodexCommandExecutor {
};
let normalized_terminal_type =
normalized_codex_terminal_event_type(&method, &params);
if normalized_terminal_type.is_some() {
self.save_before_provider_writes(true)?;
}
let cursor_cloud_delivery = if self
.state
.as_ref()
@@ -5486,12 +5593,16 @@ impl CodexCommandExecutor {
.collect::<Vec<_>>()
})
.unwrap_or_default();
self.save_state()?;
if self
.state
.as_ref()
.is_some_and(|state| state.lifecycle == "reconciliation_required")
let reconciliation_required = state.lifecycle == "reconciliation_required";
if terminal_event_type.is_some()
|| cursor_cloud_delivery.is_some()
|| reconciliation_required
{
self.save_state()?;
} else {
self.unsaved_frame_state.set(true);
}
if reconciliation_required {
break;
}
if let (Some(delivery), Some(provider)) =
@@ -5499,24 +5610,16 @@ impl CodexCommandExecutor {
{
let _ = provider.acknowledge_cursor_cloud_terminal(delivery);
}
if let (Some(frame_id), Some(provider)) =
(trace_frame_id, self.provider.as_mut())
{
provider.record_provider_trace_interpretation(
if let Some(frame_id) = trace_frame_id {
self.unsaved_trace_interpretations.push(NormalizationTrace {
frame_id,
&format!("codex.normalize.{}", method.replace('/', ".")),
if normalized_event_count > 0 {
"mapped"
} else {
"ignored"
},
trace_emitted_event_ids,
if normalized_event_count > 0 {
"Provider notification normalized into durable PRP events"
} else {
"Provider notification did not produce a durable PRP event"
},
);
rule_id: format!("codex.normalize.{}", method.replace('/', ".")),
mapped: normalized_event_count > 0,
emitted_event_ids: trace_emitted_event_ids,
});
}
if !self.unsaved_frame_state.get() {
self.record_saved_trace_interpretations();
}
}
CodexProviderEvent::McpPermissionRequest { request_id, params } => {
@@ -5582,6 +5685,8 @@ impl CodexCommandExecutor {
process_generation,
completed_turn_process_generation,
} => {
self.save_unsaved_frame_state()?;
self.record_saved_trace_interpretations();
self.provider = None;
if !success {
let state = self
@@ -5624,16 +5729,13 @@ impl CodexCommandExecutor {
}
}
}
// Check the durable deadline only after a complete provider poll. A
// terminal that arrived after interruption acceptance but before this
// observation remains authoritative even when several fast controller
// polls have already exhausted the interruption-attempt budget.
self.settle_receipt_limit_interrupt_if_deadline_elapsed()
Ok(())
}
}
impl CommandExecutor for CodexCommandExecutor {
fn host_park_ready(&mut self) -> Result<bool, DurableRunnerError> {
self.save_before_provider_writes(false)?;
let Some(state) = self.state.as_ref() else {
return Ok(false);
};
@@ -5701,6 +5803,7 @@ impl CommandExecutor for CodexCommandExecutor {
command_type: command.command_type.clone(),
});
let outcome = (|| {
self.save_before_provider_writes(false)?;
self.restore()?;
if self
.state
@@ -5805,6 +5908,7 @@ impl CommandExecutor for CodexCommandExecutor {
if let Some(error) = &self.startup_evidence_error {
return Err(error.clone());
}
self.save_before_provider_writes(false)?;
Ok(self
.state
.as_ref()
@@ -9312,6 +9416,10 @@ done
#[path = "tests/provider_api_validation.rs"]
mod api_validation_tests;
#[cfg(test)]
#[path = "tests/provider_frame_save_batching.rs"]
mod frame_save_batching_tests;
#[cfg(test)]
#[path = "tests/provider_assigned_tool_validation.rs"]
mod assigned_tool_validation_tests;
packages/paperclip-runner/runner/crates/runner-core/src/tests/provider_frame_save_batching.rs
+579
View File
@@ -0,0 +1,579 @@
use super::*;
fn frame_test_state() -> CodexProviderState {
let mut state = CodexProviderState::new(
CodexProviderConfig {
provider: "opencode".to_owned(),
driver: "opencode_server".to_owned(),
provider_version: "1.18.34".to_owned(),
command: PathBuf::from("node"),
args: Vec::new(),
cwd: std::env::current_dir()
.unwrap()
.to_string_lossy()
.into_owned(),
model: Some("openrouter/model".to_owned()),
provider_session_id: None,
instructions: String::new(),
approval_policy: "never".to_owned(),
externally_sandboxed: false,
read_only: false,
include_skill_instructions: None,
conversation_mode: None,
},
Some(CompletionContractBinding {
revision: "revision-1".to_owned(),
criterion_ids: vec!["criterion-1".to_owned()],
}),
ProviderToolBridge::default(),
);
state.thread_id = Some("thread-1".to_owned());
state.active_provider_turn_id = Some("turn-1".to_owned());
state.lifecycle = "turn_active".to_owned();
state
}
fn streamed_task_context_operation() -> crate::provider_bridge::AuthorizedTool {
crate::provider_bridge::AuthorizedTool {
operation_id: "get_task_context".to_owned(),
version: 1,
description: "Read the active task context.".to_owned(),
input_schema: json!({"type": "object"}),
response_schema: json!({"type": "object"}),
}
}
#[cfg(unix)]
fn streamed_turn_before_tool_call(
frames: usize,
retained_results: usize,
tool: &str,
wait_for_stream: bool,
) -> (PathBuf, CodexCommandExecutor) {
let tail = format!(
r#"{{"id":"tool-1","method":"item/tool/call","params":{{"threadId":"thread-1","turnId":"turn-1","callId":"call-streamed","tool":"{tool}","arguments":{{}}}}}}"#
);
streamed_turn(frames, retained_results, &tail, wait_for_stream, |_| {})
}
#[cfg(unix)]
fn streamed_turn(
frames: usize,
retained_results: usize,
tail: &str,
wait_for_stream: bool,
prepare_provider: impl FnOnce(&mut CodexProvider),
) -> (PathBuf, CodexCommandExecutor) {
streamed_turn_then(
frames,
retained_results,
tail,
":",
wait_for_stream,
prepare_provider,
)
}
#[cfg(unix)]
fn streamed_turn_then(
frames: usize,
retained_results: usize,
tail: &str,
after: &str,
wait_for_stream: bool,
prepare_provider: impl FnOnce(&mut CodexProvider),
) -> (PathBuf, CodexCommandExecutor) {
let directory = std::env::temp_dir().join(format!(
"paperclip-streamed-frames-{}",
uuid::Uuid::new_v4()
));
fs::create_dir_all(&directory).unwrap();
let streamed = directory.join("streamed");
let script = r#"
while IFS= read -r line; do
case "$line" in
*'"method":"initialize"'*) printf '%s\n' '{"id":1,"result":{}}' ;;
*'"method":"thread/start"'*) printf '%s\n' '{"id":2,"result":{"thread":{"id":"thread-1"}}}' ;;
*'"method":"turn/start"'*)
printf '%s\n' '{"id":3,"result":{"turn":{"id":"turn-1"}}}'
printf '%s\n' '{"method":"turn/started","params":{"threadId":"thread-1","turn":{"id":"turn-1","status":"inProgress"}}}'
i=0
while [ "$i" -lt FRAMES ]; do
printf '%s\n' '{"method":"item/agentMessage/delta","params":{"threadId":"thread-1","turnId":"turn-1","itemId":"message-1","delta":"x"}}'
printf '%s\n' '{"method":"item/updated","params":{"threadId":"thread-1","turnId":"turn-1","item":{"id":"reasoning-1","type":"reasoning","summary":[],"content":[]}}}'
i=$((i+1))
done
printf '%s\n' 'TAIL'
touch 'STREAMED'
AFTER ;;
esac
done
"#
.replace("FRAMES", &frames.to_string())
.replace("TAIL", tail)
.replace("AFTER", after)
.replace("STREAMED", &streamed.display().to_string());
let operation = streamed_task_context_operation();
let mut bridge = ProviderToolBridge::default();
bridge
.prepare(AuthorizedToolSet {
schema: TOOL_SET_SCHEMA.to_owned(),
schema_version: 1,
catalog_digest: authorized_tool_catalog_digest(std::slice::from_ref(&operation))
.unwrap(),
operations: vec![operation.clone()],
})
.unwrap();
for index in 0..retained_results {
let call_id = format!("retained-{index}");
bridge
.begin_call(call_id.clone(), "get_task_context".to_owned(), json!({}))
.unwrap();
bridge
.apply_result(ToolResult {
call_id,
operation_id: "get_task_context".to_owned(),
result: json!({"body": "x".repeat(10_000)}),
is_error: false,
})
.unwrap();
}
let mut state = frame_test_state();
state.config.provider = "codex".to_owned();
state.config.driver = "codex_app_server".to_owned();
state.config.command = PathBuf::from("/bin/sh");
state.config.args = vec!["-c".to_owned(), script];
state.config.model = None;
state.tool_bridge = bridge;
let mut provider = CodexProvider::start_with_tools(&state.config, [operation], None).unwrap();
prepare_provider(&mut provider);
provider.start_turn("Test", &state.config.cwd).unwrap();
state.provider_process_generation = provider.process_generation();
let mut executor = CodexCommandExecutor::new(&directory);
executor.event_identity = Some(ProviderEventIdentity {
runner_instance_id: "runner-1".to_owned(),
run_id: "run-1".to_owned(),
normalized_session_id: "session-1".to_owned(),
turn_id: "turn-1".to_owned(),
item_id: "item-1".to_owned(),
});
executor.state = Some(state);
executor.provider = Some(provider);
executor.restore_checked = true;
executor.save_state().unwrap();
let deadline = std::time::Instant::now() + std::time::Duration::from_secs(10);
while wait_for_stream && !streamed.exists() {
assert!(std::time::Instant::now() < deadline);
std::thread::sleep(std::time::Duration::from_millis(5));
}
(directory, executor)
}
fn poll_until_semantic_tool_call(executor: &mut CodexCommandExecutor) -> Vec<PolledEvent> {
let deadline = std::time::Instant::now() + std::time::Duration::from_secs(30);
let mut observed = Vec::new();
loop {
assert!(std::time::Instant::now() < deadline);
let events = executor.poll_events().unwrap();
let count = events.len();
let delivered = events
.iter()
.any(|event| event.event_type == "semantic_tool.input");
observed.extend(events);
if delivered {
return observed;
}
executor.acknowledge_events(count).unwrap();
}
}
#[cfg(unix)]
#[test]
fn streamed_frames_do_not_multiply_durable_saves_before_a_tool_call() {
let mut saves = Vec::new();
for frames in [8, 48] {
let (directory, mut executor) =
streamed_turn_before_tool_call(frames, 4, "get_task_context", true);
let before = executor.persist_count.get();
let observed = poll_until_semantic_tool_call(&mut executor);
saves.push(executor.persist_count.get() - before);
assert_eq!(
observed
.iter()
.filter(|event| event.event_type == "item.delta")
.count(),
frames
);
executor.provider.as_mut().unwrap().shutdown().unwrap();
fs::remove_dir_all(directory).unwrap();
}
eprintln!("durable saves before the tool call: {saves:?}");
assert_eq!(
saves[0], saves[1],
"saves grew with streamed frames: {saves:?}"
);
}
fn saved_provider_state(executor: &CodexCommandExecutor) -> CodexProviderState {
serde_json::from_slice(&fs::read(executor.state_path()).unwrap()).unwrap()
}
#[cfg(unix)]
#[test]
fn batched_frame_state_is_saved_before_events_leave_and_survives_a_restart() {
let (directory, mut executor) =
streamed_turn_before_tool_call(300, 4, "get_task_context", false);
let deadline = std::time::Instant::now() + std::time::Duration::from_secs(30);
loop {
assert!(std::time::Instant::now() < deadline);
let events = executor.poll_events().unwrap();
assert!(!executor.unsaved_frame_state.get());
let saved = saved_provider_state(&executor);
assert_eq!(
serde_json::to_value(&saved).unwrap(),
serde_json::to_value(executor.state.as_ref().unwrap()).unwrap()
);
assert_eq!(saved.pending_events, events);
if events
.iter()
.any(|event| event.event_type == "semantic_tool.input")
{
break;
}
let count = events.len();
executor.acknowledge_events(count).unwrap();
}
let live = executor.state.clone().unwrap();
executor.provider.as_mut().unwrap().shutdown().unwrap();
drop(executor);
let mut resumed = CodexCommandExecutor::new(&directory);
resumed.load_state_without_provider().unwrap();
let resumed_state = resumed.state.as_ref().unwrap();
assert_eq!(
serde_json::to_value(resumed_state).unwrap(),
serde_json::to_value(&live).unwrap()
);
assert!(resumed_state
.tool_bridge
.pending_calls()
.any(|call| call.call_id == "call-streamed"));
assert_eq!(
resumed.retained_events().unwrap(),
live.pending_events.iter().cloned().collect::<Vec<_>>()
);
fs::remove_dir_all(directory).unwrap();
}
#[cfg(unix)]
#[test]
fn batched_frame_state_is_saved_when_a_later_frame_fails() {
let (directory, mut executor) =
streamed_turn_before_tool_call(20, 0, "unauthorized_tool", true);
assert!(executor.poll_events().is_err());
assert!(!executor.unsaved_frame_state.get());
let saved = saved_provider_state(&executor);
assert_eq!(
serde_json::to_value(&saved).unwrap(),
serde_json::to_value(executor.state.as_ref().unwrap()).unwrap()
);
assert_eq!(
saved
.pending_events
.iter()
.filter(|event| event.event_type == "item.delta")
.count(),
20
);
let _ = executor.provider.as_mut().unwrap().shutdown();
fs::remove_dir_all(directory).unwrap();
}
fn previous_release_state() -> CodexProviderState {
let mut state = frame_test_state();
state.config.provider = "codex".to_owned();
state.config.driver = "codex_app_server".to_owned();
state.config.model = None;
state
}
#[test]
fn previous_release_state_is_rewritten_by_the_first_batched_save() {
let directory = std::env::temp_dir().join(format!(
"paperclip-batched-save-upgrade-{}",
uuid::Uuid::new_v4()
));
fs::create_dir_all(&directory).unwrap();
#[cfg(unix)]
fs::set_permissions(&directory, fs::Permissions::from_mode(0o700)).unwrap();
let mut previous = serde_json::to_value(previous_release_state()).unwrap();
let object = previous.as_object_mut().unwrap();
assert!(object.remove("activeTerminalToolRejected").is_some());
assert!(object.remove("descendantThreadIds").is_some());
let previous_bytes = serde_json::to_vec(&previous).unwrap();
let path = directory.join(CODEX_PROVIDER_STATE_FILE);
fs::write(&path, &previous_bytes).unwrap();
#[cfg(unix)]
fs::set_permissions(&path, fs::Permissions::from_mode(0o600)).unwrap();
let mut resumed = CodexCommandExecutor::new(&directory);
resumed.load_state_without_provider().unwrap();
assert_eq!(resumed.state.as_ref().unwrap(), &previous_release_state());
resumed.unsaved_frame_state.set(true);
resumed.save_unsaved_frame_state().unwrap();
assert_eq!(resumed.persist_count.get(), 1);
assert_ne!(fs::read(&path).unwrap(), previous_bytes);
assert_eq!(
serde_json::to_value(saved_provider_state(&resumed)).unwrap(),
serde_json::to_value(resumed.state.as_ref().unwrap()).unwrap()
);
resumed.unsaved_frame_state.set(true);
resumed.save_unsaved_frame_state().unwrap();
assert_eq!(resumed.persist_count.get(), 1);
resumed.state.as_mut().unwrap().last_agent_message = Some("streamed".to_owned());
resumed.unsaved_frame_state.set(true);
resumed.save_unsaved_frame_state().unwrap();
assert_eq!(resumed.persist_count.get(), 2);
assert_eq!(
saved_provider_state(&resumed).last_agent_message.as_deref(),
Some("streamed")
);
fs::remove_dir_all(directory).unwrap();
}
#[cfg(unix)]
#[test]
#[ignore = "timing measurement; run with --ignored --nocapture"]
fn measure_streamed_frame_cost_before_a_tool_call() {
for retained in [0, 20, 80] {
for frames in [50, 200] {
let (directory, mut executor) =
streamed_turn_before_tool_call(frames, retained, "get_task_context", false);
let state_bytes = fs::metadata(executor.state_path()).unwrap().len();
let before = executor.persist_count.get();
let started = std::time::Instant::now();
poll_until_semantic_tool_call(&mut executor);
let elapsed = started.elapsed();
eprintln!(
"retained={retained} stateBytes={state_bytes} frames={} saves={} toolCallLatencyMs={:.1}",
frames * 2,
executor.persist_count.get() - before,
elapsed.as_secs_f64() * 1000.0
);
executor.provider.as_mut().unwrap().shutdown().unwrap();
fs::remove_dir_all(directory).unwrap();
}
}
}
#[cfg(unix)]
fn event_identity_for_tests() -> ProviderEventIdentity {
ProviderEventIdentity {
runner_instance_id: "runner-1".to_owned(),
run_id: "run-1".to_owned(),
normalized_session_id: "session-1".to_owned(),
turn_id: "turn-1".to_owned(),
item_id: "item-1".to_owned(),
}
}
#[cfg(unix)]
fn read_saved_state(path: &std::path::Path) -> CodexProviderState {
serde_json::from_slice(&fs::read(path).unwrap()).unwrap()
}
#[cfg(unix)]
#[test]
fn courtesy_rejections_reach_the_provider_only_after_earlier_frames_are_saved() {
let directory_slot = std::sync::Arc::new(std::sync::Mutex::new(None::<PathBuf>));
let saved_at_rejection = std::sync::Arc::new(std::sync::Mutex::new(None::<CodexProviderState>));
let observer_directory = std::sync::Arc::clone(&directory_slot);
let observer_saved = std::sync::Arc::clone(&saved_at_rejection);
let tail = r#"{"id":"courtesy-1","method":"paperclip/test/unsupported","params":{"threadId":"thread-1","turnId":"turn-1"}}"#;
let (directory, mut executor) = streamed_turn(20, 0, tail, true, move |provider| {
provider.sent_frame_observer = Some(Box::new(move |frame: &Value| {
if frame["id"] != "courtesy-1" {
return;
}
let directory = observer_directory.lock().unwrap().clone().unwrap();
*observer_saved.lock().unwrap() =
Some(read_saved_state(&directory.join(CODEX_PROVIDER_STATE_FILE)));
}));
});
*directory_slot.lock().unwrap() = Some(directory.clone());
let events = executor.poll_events().unwrap();
assert!(events.iter().any(|event| event.event_type == "item.delta"));
let resumed = saved_at_rejection
.lock()
.unwrap()
.take()
.expect("runnerd sent the courtesy rejection");
assert_eq!(
resumed
.pending_events
.iter()
.filter(|event| event.event_type == "item.delta")
.count(),
20
);
assert_eq!(
serde_json::to_value(&resumed).unwrap(),
serde_json::to_value(executor.state.as_ref().unwrap()).unwrap()
);
executor.provider.as_mut().unwrap().shutdown().unwrap();
fs::remove_dir_all(directory).unwrap();
}
#[cfg(unix)]
#[test]
fn trace_interpretations_name_only_saved_events() {
let directory_slot = std::sync::Arc::new(std::sync::Mutex::new(None::<PathBuf>));
let checked = std::sync::Arc::new(std::sync::Mutex::new((0usize, Vec::<String>::new())));
let observer_directory = std::sync::Arc::clone(&directory_slot);
let observer_checked = std::sync::Arc::clone(&checked);
let trace_path = std::env::temp_dir().join(format!(
"paperclip-frame-trace-{}.jsonl",
uuid::Uuid::new_v4()
));
let provider_trace_path = trace_path.clone();
let (directory, mut executor) = streamed_turn_before_tool_call_with(40, move |provider| {
provider.enable_trace_for_test(provider_trace_path);
provider.interpretation_observer = Some(Box::new(move |ids: &[String]| {
if ids.is_empty() {
return;
}
let directory = observer_directory.lock().unwrap().clone().unwrap();
let saved = read_saved_state(&directory.join(CODEX_PROVIDER_STATE_FILE));
let identity = event_identity_for_tests();
let saved_ids = (0..saved.next_provider_event_seq)
.map(|sequence| identity.source_event_id(&provider_event_id(sequence)))
.collect::<HashSet<_>>();
let mut checked = observer_checked.lock().unwrap();
checked.0 += ids.len();
checked
.1
.extend(ids.iter().filter(|id| !saved_ids.contains(*id)).cloned());
}));
});
*directory_slot.lock().unwrap() = Some(directory.clone());
poll_until_semantic_tool_call(&mut executor);
let (count, unsaved) = checked.lock().unwrap().clone();
assert!(count >= 40, "trace named {count} events");
assert!(
unsaved.is_empty(),
"trace named unsaved events: {unsaved:?}"
);
executor.provider.as_mut().unwrap().shutdown().unwrap();
fs::remove_dir_all(directory).unwrap();
let _ = fs::remove_file(trace_path);
}
#[cfg(unix)]
#[test]
fn a_crash_inside_an_unsaved_batch_leaves_saved_state_and_provider_output_unchanged() {
let sent = std::sync::Arc::new(std::sync::Mutex::new(Vec::<Value>::new()));
let observer_sent = std::sync::Arc::clone(&sent);
let tail = r#"{"method":"item/agentMessage/delta","params":{"threadId":"thread-1","turnId":"turn-1","itemId":"message-1","delta":"last"}}"#;
let (directory, mut executor) = streamed_turn(60, 0, tail, true, move |provider| {
provider.sent_frame_observer = Some(Box::new(move |frame: &Value| {
observer_sent.lock().unwrap().push(frame.clone());
}));
});
let path = executor.state_path();
let saved_before = fs::read(&path).unwrap();
let sent_before = sent.lock().unwrap().len();
executor
.provider
.as_mut()
.unwrap()
.hold_courtesy_responses();
executor.poll_provider_frames(true, false).unwrap();
assert!(executor.unsaved_frame_state.get());
let live_next = executor.state.as_ref().unwrap().next_provider_event_seq;
let saved_next = read_saved_state(&path).next_provider_event_seq;
assert!(live_next >= saved_next + 121);
assert_eq!(fs::read(&path).unwrap(), saved_before);
assert_eq!(sent.lock().unwrap().len(), sent_before);
executor.provider.as_mut().unwrap().shutdown().unwrap();
drop(executor);
let mut resumed = CodexCommandExecutor::new(&directory);
resumed.load_state_without_provider().unwrap();
let resumed_state = resumed.state.as_ref().unwrap();
assert_eq!(resumed_state.next_provider_event_seq, saved_next);
assert!(resumed_state.pending_events.is_empty());
fs::remove_dir_all(directory).unwrap();
}
#[cfg(unix)]
fn streamed_turn_before_tool_call_with(
frames: usize,
prepare_provider: impl FnOnce(&mut CodexProvider),
) -> (PathBuf, CodexCommandExecutor) {
let tail = r#"{"id":"tool-1","method":"item/tool/call","params":{"threadId":"thread-1","turnId":"turn-1","callId":"call-streamed","tool":"get_task_context","arguments":{}}}"#;
streamed_turn(frames, 0, tail, true, prepare_provider)
}
#[cfg(unix)]
#[test]
fn a_held_cancellation_that_cannot_reach_the_provider_keeps_the_completed_turn() {
let tail = r#"{"id":"tool-1","method":"item/tool/call","params":{"threadId":"thread-1","turnId":"turn-1","callId":"call-streamed","tool":"get_task_context","arguments":{}}}"#;
let after = r#"exec 0<&-; printf '%s\n' '{"method":"turn/completed","params":{"threadId":"thread-1","turn":{"id":"turn-1","status":"completed"}}}'; sleep 2; exit 0"#;
let (directory, mut executor) = streamed_turn_then(10, 0, tail, after, true, |_| {});
std::thread::sleep(std::time::Duration::from_millis(200));
let deadline = std::time::Instant::now() + std::time::Duration::from_secs(30);
let mut observed = Vec::new();
while !observed
.iter()
.any(|event: &PolledEvent| event.event_type == "turn.completed")
{
assert!(std::time::Instant::now() < deadline);
let events = executor.poll_events().unwrap();
let count = events.len();
observed.extend(events);
executor.acknowledge_events(count).unwrap();
}
assert!(observed
.iter()
.any(|event| event.event_type == "semantic_tool.input"));
let saved = read_saved_state(&executor.state_path());
assert!(saved.completed_turn_authoritative);
if let Some(provider) = executor.provider.as_mut() {
let _ = provider.shutdown();
}
fs::remove_dir_all(directory).unwrap();
}
#[cfg(unix)]
#[test]
fn trace_interpretations_survive_a_provider_exit_in_the_same_poll() {
let named = std::sync::Arc::new(std::sync::Mutex::new(0usize));
let observer_named = std::sync::Arc::clone(&named);
let tail = r#"{"method":"item/agentMessage/delta","params":{"threadId":"thread-1","turnId":"turn-1","itemId":"message-1","delta":"last"}}"#;
let trace_path = std::env::temp_dir().join(format!(
"paperclip-frame-trace-exit-{}.jsonl",
uuid::Uuid::new_v4()
));
let provider_trace_path = trace_path.clone();
let (directory, mut executor) =
streamed_turn_then(20, 0, tail, "exit 0", true, move |provider| {
provider.enable_trace_for_test(provider_trace_path);
provider.interpretation_observer = Some(Box::new(move |ids: &[String]| {
*observer_named.lock().unwrap() += ids.len();
}));
});
std::thread::sleep(std::time::Duration::from_millis(200));
let deadline = std::time::Instant::now() + std::time::Duration::from_secs(30);
while executor.provider.is_some() {
assert!(std::time::Instant::now() < deadline);
let events = executor.poll_events().unwrap();
let count = events.len();
executor.acknowledge_events(count).unwrap();
}
assert!(*named.lock().unwrap() >= 41);
fs::remove_dir_all(directory).unwrap();
let _ = fs::remove_file(trace_path);
}
packages/paperclip-runner/src/backends/harness-driver-backend.ts
+8
View File
@@ -480,6 +480,14 @@ class HarnessNativeSession implements NativeSession {
};
}
beginRunAttachment(input: { identity: OpenNativeSessionInput["identity"] }): boolean {
const currentIdentity = this.#input.identity;
if (this.#protocolIntegrityFailure !== null || this.#session.beginRunAttachment === undefined
|| input.identity.sessionId !== currentIdentity.sessionId || input.identity.companyId !== currentIdentity.companyId
|| input.identity.issueId !== currentIdentity.issueId || input.identity.agentId !== currentIdentity.agentId) return false;
return this.#session.beginRunAttachment({ runId: input.identity.runId });
}
async attachRun(input: {
identity: OpenNativeSessionInput["identity"];
currentRunGrant?: NativeRunRuntimeGrant;
packages/paperclip-runner/src/contracts/harness-driver.ts
+1
View File
@@ -564,6 +564,7 @@ export interface HarnessSession {
displayId?: string | null;
};
events(): AsyncIterable<PrpEvent>;
beginRunAttachment?(input: { runId: string }): boolean;
attachRun?(input: { runId: string; currentRunGrant?: NativeRunRuntimeGrant; completionContract?: { revision: string; criterionIds: readonly string[] } }): Promise<void> | void;
startTurn(input: {
message: NativeUserMessage;
packages/paperclip-runner/src/contracts/native-session-backend.ts
+1
View File
@@ -217,6 +217,7 @@ export interface NativeSession {
capabilities(): Promise<NativeSessionCapabilities>;
/** completionContract is the attaching run's own contract; a retained session must not keep the prior run's. */
attachRun?(input: { identity: NativeRunIdentity; currentRunGrant?: NativeRunRuntimeGrant; completionContract?: { revision: string; criterionIds: readonly string[] } }): Promise<void>;
beginRunAttachment?(input: { identity: NativeRunIdentity }): boolean;
/** Return the confirmed released source, or false when active use was not released. */
release?(): Promise<DurableRecoveryIdentity | false>;
/** Relinquish controller authority without suspending provider execution. */
packages/paperclip-runner/src/drivers/codex/app-server-transport.ts
+1
View File
@@ -80,6 +80,7 @@ export interface CodexAppServerTransport {
/** Relinquish controller authority while leaving durable runner work alive. */
detachControllerForRestart?(): Promise<void>;
processInfo?(): CodexTransportProcessInfo;
beginRunAttachment?(input: { runId: string }): boolean;
attachRun?(input: {
runId: string;
turnId: string;
packages/paperclip-runner/src/drivers/codex/codex-harness-session.ts
+5
View File
@@ -110,6 +110,11 @@ export class CodexHarnessSession
};
}
beginRunAttachment(input: { runId: string }): boolean {
if (this.protocolIntegrityFailure !== null || this.turnStartPending || this.pendingRunAttachment !== null || !input.runId) return false;
return this.transport.beginRunAttachment?.(input) ?? false;
}
async attachRun(input: { runId: string; currentRunGrant?: NativeRunRuntimeGrant; completionContract?: { revision: string; criterionIds: readonly string[] } }): Promise<void> {
this.assertProtocolIntegrity();
const transportOwnsQuiescence = this.transport.attachRun !== undefined;
packages/paperclip-runner/src/live/runnerd-codex-transport-release-resume.test.ts
+47 -7
View File
@@ -78,7 +78,7 @@ async function nativeWire(port: number, path: string, closeGate?: Promise<void>)
});
}
it.skipIf(process.platform === "win32").each(["cached", "lost claim reply", "cache adoption", "acpx cache adoption", "unavailable", "claim cancellation", "expired ticket", "late claim reply", "held past deadline", "release admission", "no cleanup callback", "adoption no cleanup callback", "resume no cleanup callback", "deferred turn after close", "late wire close", "empty goal snapshot"] as const)(
it.skipIf(process.platform === "win32").each(["cached", "lost claim reply", "cache adoption", "acpx cache adoption", "unavailable", "claim cancellation", "expired ticket", "late claim reply", "held past deadline", "release admission", "no cleanup callback", "adoption no cleanup callback", "resume no cleanup callback", "deferred turn after close", "late wire close", "empty goal snapshot", "early claim", "early claim abandoned", "early claim cancellation", "early claim expired", "early claim stale ticket"] as const)(
"releases and resumes the native Codex source before attachment (%s)", async mode => {
const root = await mkdtemp(join(tmpdir(), "runner-release-resume-"));
const acpx = mode === "acpx cache adoption";
@@ -119,7 +119,7 @@ it.skipIf(process.platform === "win32").each(["cached", "lost claim reply", "cac
const heldMeasuredTurn = new Promise<void>(resolve => { finishMeasuredTurn = resolve; });
const sockets = new Set<PrpWireConnection>();
const registration = async (core: DurablePrpControlPlane, source = core.store.state.identity) => {
if ((mode === "late claim reply" || mode === "held past deadline") && claimAccepted) throw new Error("Fixture source authentication started after expiry");
if ((mode === "late claim reply" || mode === "held past deadline" || mode === "early claim expired") && claimAccepted) throw new Error("Fixture source authentication started after expiry");
authority = core;
let socket: PrpWireConnection | undefined;
return {
@@ -165,8 +165,8 @@ it.skipIf(process.platform === "win32").each(["cached", "lost claim reply", "cac
parkId: parkId(source), bootstrapTicket: ticket });
if (result.status !== "claimed") throw new Error("Fixture claim receipt unavailable");
claimAccepted = true; acceptClaim();
if (mode === "claim cancellation" || mode === "held past deadline") await claimReceipt;
if (mode === "expired ticket" || mode === "late claim reply") {
if (mode === "claim cancellation" || mode === "held past deadline" || mode === "early claim" || mode === "early claim cancellation") await claimReceipt;
if (mode === "expired ticket" || mode === "late claim reply" || mode === "early claim expired") {
expireClock = vi.spyOn(Date, "now").mockReturnValue(Date.now() + 61_000);
if (mode === "expired ticket") throw new Error("Fixture lost claim reply past ticket lifetime");
}
@@ -347,12 +347,51 @@ it.skipIf(process.platform === "win32").each(["cached", "lost claim reply", "cac
await expect(bundle.transport.request("thread/goal/set", { threadId, objective: "Must stay fenced" })).rejects.toThrow("release_resume_pending");
}
if (mode === "unavailable") await fixture.close();
if (mode.startsWith("early claim")) {
expect(bundle.transport.beginRunAttachment!({ runId: claimantRunId })).toBe(true);
expect(bundle.transport.beginRunAttachment!({ runId: claimantRunId })).toBe(false);
await vi.waitFor(() => expect(pair.resume).toHaveBeenCalledOnce());
await expect(bundle.transport.release!()).rejects.toThrow("run_attachment_in_progress");
await expect(bundle.transport.attachRun!({ runId: "other-run", turnId: "other-turn", itemId: "other-item" })).rejects.toThrow("run_attachment_in_progress");
await expect(bundle.transport.request("turn/start", { input: [] })).rejects.toThrow("run_attachment_in_progress");
expect(routeRuns).toEqual([identity.runId]);
}
if (mode === "early claim cancellation") {
await vi.waitFor(() => expect(claimAccepted).toBe(true));
const closing = bundle.transport.close();
releaseClaim();
await closing;
await expect(bundle.transport.attachRun!({ runId: claimantRunId, turnId: "next-turn", itemId: "next-item" })).rejects.toThrow("closed");
}
if (mode === "early claim abandoned" || mode === "early claim stale ticket") {
await vi.waitFor(async () => expect(await pair.resume.mock.results[0]?.value).toBe("resumed"));
if (mode === "early claim abandoned") await bundle.transport.close();
else {
expireClock = vi.spyOn(Date, "now").mockReturnValue(Date.now() + 61_000);
await expect(bundle.transport.attachRun!({ runId: claimantRunId, turnId: "next-turn", itemId: "next-item" })).rejects.toThrow("reconciliation_expired");
expireClock.mockRestore(); expireClock = undefined;
await bundle.transport.close();
}
expect(routeRuns).toEqual([identity.runId]);
expect(pair.release).toHaveBeenCalledOnce();
}
if (mode === "held past deadline") vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] });
const attaching = Promise.resolve(recoveredSession
? recoveredSession.attachRun!({ runId: claimantRunId })
: bundle.transport.attachRun!({ runId: claimantRunId, turnId: "next-turn", itemId: "next-item" }));
void attaching.catch(() => undefined);
if (mode === "held past deadline") {
if (mode === "early claim cancellation" || mode === "early claim abandoned" || mode === "early claim stale ticket") {
await expect(attaching).rejects.toThrow("closed");
} else if (mode === "early claim") {
await vi.waitFor(() => expect(claimAccepted).toBe(true));
let attached = false;
void attaching.then(() => { attached = true; }, () => undefined);
await new Promise(resolve => setTimeout(resolve, 100));
expect(attached).toBe(false);
expect(routeRuns).toEqual([identity.runId]);
releaseClaim();
await attaching;
} else if (mode === "held past deadline") {
await acceptedClaim;
let failure: unknown;
const settled = attaching.catch(error => { failure = error; });
@@ -372,13 +411,14 @@ it.skipIf(process.platform === "win32").each(["cached", "lost claim reply", "cac
releaseClaim();
await expect(attaching).rejects.toThrow("closed");
await closing;
} else if (mode === "unavailable" || mode === "expired ticket" || mode === "late claim reply" || mode === "resume no cleanup callback") {
} else if (mode === "unavailable" || mode === "expired ticket" || mode === "late claim reply" || mode === "resume no cleanup callback" || mode === "early claim expired") {
await expect(attaching).rejects.toThrow(mode === "unavailable" ? "reuse_unavailable" : mode === "resume no cleanup callback" ? "released_cleanup_unavailable" : "reconciliation_expired");
expireClock?.mockRestore(); expireClock = undefined;
await bundle.transport.close();
} else await attaching;
expect(new Set(claimTicketDigests).size).toBe(1);
if (["unavailable", "claim cancellation", "expired ticket", "late claim reply", "held past deadline", "resume no cleanup callback"].includes(mode)) {
if (["unavailable", "claim cancellation", "expired ticket", "late claim reply", "held past deadline", "resume no cleanup callback",
"early claim abandoned", "early claim cancellation", "early claim expired", "early claim stale ticket"].includes(mode)) {
expect(pair.resume).toHaveBeenCalledOnce();
expect(authority!.store.state.identity.runId).toBe(identity.runId);
expect(launch).toHaveBeenCalledOnce();
packages/paperclip-runner/src/live/runnerd-codex-transport.ts
+41 -12
View File
@@ -4000,6 +4000,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
#detachPromise: Promise<void> | null = null;
#startupOperation: Promise<unknown> | null = null;
#attachmentOperation: Promise<void> | null = null;
#earlyClaim: { runId: string; operation: Promise<void> } | null = null;
#releaseEpisode: {
source: DurableRecoveryIdentity; confirmed: boolean; claimed?: boolean;
claimantRunId?: string; ticket?: string; ticketFlushed?: boolean; deadline?: number;
@@ -4721,6 +4722,21 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
if (this.options.adoptExistingRunner && this.#adoptedRunnerMonitor === null) this.#watchAdoptedRunner(this.options.adoptExistingRunner);
}
beginRunAttachment(input: { runId: string }): boolean {
if (this.#attachmentOperation !== null || !this.#releaseEpisode?.confirmed || this.#releaseEpisode.claimed || !input.runId) return false;
try {
this.#assertStartupOpen();
this.#assertRunAttachmentAdmission();
} catch {
return false;
}
const operation = this.#claimReleasedRunner(input.runId);
void operation.catch(() => undefined);
this.#attachmentOperation = operation;
this.#earlyClaim = { runId: input.runId, operation };
return true;
}
attachRun(input: {
runId: string;
turnId: string;
@@ -4728,12 +4744,14 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
currentRunGrant?: NativeRunRuntimeGrant;
completionContract?: { revision: string; criterionIds: readonly string[] };
}): Promise<void> {
if (this.#attachmentOperation !== null) {
const earlyClaim = this.#earlyClaim?.runId === input.runId ? this.#earlyClaim.operation : undefined;
if (this.#attachmentOperation !== null && earlyClaim === undefined) {
return Promise.reject(new Error("native_runner_run_attachment_in_progress"));
}
this.#earlyClaim = null;
const operation = Promise.resolve().then(() => {
this.#assertStartupOpen();
return this.#attachRun(input);
return this.#attachRun(input, earlyClaim);
}).finally(() => { this.#attachmentOperation = null; });
this.#attachmentOperation = operation;
return operation;
@@ -4745,20 +4763,14 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
itemId: string;
currentRunGrant?: NativeRunRuntimeGrant;
completionContract?: { revision: string; criterionIds: readonly string[] };
}): Promise<void> {
}, earlyClaim?: Promise<void>): Promise<void> {
const previous = record(this.#runAttachTemplate?.provider).runtimeContext;
const current = input.currentRunGrant?.runtimeContext;
return this.options.measureOperation ? this.options.measureOperation("runner.transport.attach", () => this.#attachRunWork(input),
previous === undefined ? undefined : {contextReused:current === undefined || commandDigest(current) === commandDigest(previous)}) : this.#attachRunWork(input);
return this.options.measureOperation ? this.options.measureOperation("runner.transport.attach", () => this.#attachRunWork(input, earlyClaim),
previous === undefined ? undefined : {contextReused:current === undefined || commandDigest(current) === commandDigest(previous)}) : this.#attachRunWork(input, earlyClaim);
}
async #attachRunWork(input: {
runId: string;
turnId: string;
itemId: string;
currentRunGrant?: NativeRunRuntimeGrant;
completionContract?: { revision: string; criterionIds: readonly string[] };
}): Promise<void> {
#assertRunAttachmentAdmission(): { core: DurablePrpControlPlane; retainingSource: boolean } {
if (this.#pendingWarmRecoveryCompletion !== null) {
throw new Error("native_runner_warm_transition_completion_pending");
}
@@ -4770,6 +4782,23 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
if (retainingSource && this.#bridgedRuntimeInputs.size > 0) {
throw new NativeRunnerSourceReconciliationRequiredError("runtime");
}
return { core, retainingSource };
}
async #attachRunWork(input: {
runId: string;
turnId: string;
itemId: string;
currentRunGrant?: NativeRunRuntimeGrant;
completionContract?: { revision: string; criterionIds: readonly string[] };
}, earlyClaim?: Promise<void>): Promise<void> {
const { core, retainingSource } = this.#assertRunAttachmentAdmission();
if (earlyClaim) {
await earlyClaim;
if (this.#releaseEpisode && Date.now() >= this.#releaseEpisode.deadline!) {
throw new Error("native_runner_resume_reconciliation_expired");
}
}
if (this.#releaseEpisode) await this.#resumeReleasedRunner(input.runId);
await this.#awaitWarmRunAttachmentReady();
if (retainingSource && !core.semanticToolResultsSettled()) {
packages/paperclip-runner/src/native-session-runtime.test.ts
+56 -5
View File
@@ -587,9 +587,10 @@ describe("executeNativeSession recovery", () => {
});
it.each([
{ label: "after the continuation starts", staleFirst: false },
{ label: "before the continuation starts", staleFirst: true },
])("keeps the restart continuation when the restored runner reports an earlier unrecorded continuation lost $label", async ({ staleFirst }) => {
{ label: "after the continuation starts", staleFirst: false, staleTurnId: "turn-unrecorded-continuation" },
{ label: "before the continuation starts", staleFirst: true, staleTurnId: "turn-unrecorded-continuation" },
{ label: "as the checkpointed interrupted turn before the continuation starts", staleFirst: true, staleTurnId: "turn-recovery" },
])("keeps the restart continuation when the restored runner reports an earlier unrecorded continuation lost $label", async ({ staleFirst, staleTurnId }) => {
let staleAppended!: () => void;
const staleTerminal = new Promise<void>(resolve => { staleAppended = resolve; });
const failure = { code: "provider_turn_lost_on_restore", recoverable: true };
@@ -607,7 +608,7 @@ describe("executeNativeSession recovery", () => {
identity: () => identity,
async capabilities() { return { resume: true, typedEvents: true, steering: false, interruption: true, structuredResult: true }; },
async *events() {
yield { ...runnerEvent(1, "turn.failed", { status: "failed", error: failure }), turnId: "turn-unrecorded-continuation" };
yield { ...runnerEvent(1, "turn.failed", { status: "failed", error: failure }), turnId: staleTurnId };
yield { ...runnerEvent(2, "turn.completed", { status: "completed" }), turnId: "turn-continuation" };
},
startTurn,
@@ -639,7 +640,7 @@ describe("executeNativeSession recovery", () => {
})).resolves.toMatchObject({ result, turnId: "turn-continuation", terminal: { runTerminalState: "succeeded" } });
expect(startTurn).toHaveBeenCalledOnce();
expect((startTurn.mock.calls[0]![0] as Parameters<NativeSession["startTurn"]>[0]).continuation).toBe(true);
expect(events.filter(event => event.eventType === "turn.failed").map(event => event.turnId)).toEqual(["turn-unrecorded-continuation"]);
expect(events.filter(event => event.eventType === "turn.failed").map(event => event.turnId)).toEqual([staleTurnId]);
});
it("recovers a restart continuation whose checkpoint still carries the lost turn's failed terminal", async () => {
@@ -689,6 +690,56 @@ describe("executeNativeSession recovery", () => {
.toBe(false);
});
it.each([
{ label: "after it starts", proposedFirst: false },
{ label: "after the continuation proposed its result", proposedFirst: true },
])("keeps the restart continuation when the lost turn's terminal is delivered again $label", async ({ proposedFirst }) => {
const failure = { code: "provider_turn_lost_on_restore", recoverable: true };
const checkpoint: PersistedNativeSession = {
backendKind: "runner", driverKind: "acpx_runtime", sessionId: "record-recovery", identity,
providerSessionId: "provider-recovery", cursor: "0", activeTurnId: "turn-recovery",
terminalTurns: [], semanticResult: null, pendingRuntimeRequests: [], lineage: [],
};
const interrupted: PersistedNativeSession = {
...checkpoint, activeTurnId: null,
terminalTurns: [{ turnId: "turn-recovery", fingerprint: JSON.stringify({ terminalState: "failed", error: failure, result: null }) }],
};
const startTurn = vi.fn<NativeSession["startTurn"]>(async () => ({ turnId: "turn-continuation" }));
const session: NativeSession = {
identity: () => identity,
async capabilities() { return { resume: true, typedEvents: true, steering: false, interruption: true, structuredResult: true }; },
async *events() {
yield runnerEvent(1, "turn.failed", { status: "failed", error: failure });
if (!startTurn.mock.calls.length) return;
if (proposedFirst) yield { ...runnerEvent(2, "run.result.proposed", result), turnId: "turn-continuation" };
yield runnerEvent(3, "turn.failed", { status: "failed", error: failure });
yield { ...runnerEvent(4, "turn.completed", { status: "completed" }), turnId: "turn-continuation" };
},
startTurn,
async result() { return startTurn.mock.calls.length ? { result, terminal, turnId: "turn-continuation" } : null; },
async snapshot() { return structuredClone(interrupted); },
async close() {},
};
const backend: NativeSessionBackend = {
async descriptor() { return { kind: "runner", name: "restart", version: "1", capabilities: await session.capabilities() }; },
async openSession() { throw new Error("must preserve the provider session"); },
async recoverSession() { return { recovered: true, session }; },
};
const port: ControlPlanePort = {
async openRun() {},
async checkpointSession() {},
async appendEvent(event: PrpEvent) { return { cursor: event.sourceSeq, highestContiguousSourceSeq: event.sourceSeq, disposition: "committed" }; },
async replayEvents() { return { events: [], highestContiguousSourceSeq: 0 }; },
async completeRun() {},
};
await expect(executeNativeSession({
input, backend, controlPlane: port, persistedSession: checkpoint,
runnerInstanceId: "runner-recovery", controlPlaneInstanceId: "control-recovery",
resumeInterruptedTurn: true,
})).resolves.toMatchObject({ result, turnId: "turn-continuation", terminal: { runTerminalState: "succeeded" } });
expect(startTurn).toHaveBeenCalledOnce();
});
it.each(["acpx_runtime", "opencode_server"] as const)("continues a restart-lost %s turn once in the same conversation", async (driverKind) => {
const qualified = resolveQualifiedAcpxProfile("claude", "claude-sonnet-5");
const acpxInput: NativeExecutionInputV1 = (driverKind === "acpx_runtime" ? {
packages/paperclip-runner/src/native-session-runtime.ts
+9 -6
View File
@@ -1111,6 +1111,7 @@ async function consumeTurn(
initialGovernedWait?: PersistedNativeSession["governedWait"],
/** Settles a compaction-only turn, which reports no semantic result. */
compactionResult?: PrpStructuredRunResult,
supersededTerminal?: (event: PrpEvent, signal: AbortSignal) => Promise<boolean>,
) {
let timer: ReturnType<typeof setTimeout> | undefined;
const appendAbort = new AbortController();
@@ -1504,6 +1505,7 @@ async function consumeTurn(
}
}
if (isTurnTerminal(event)) {
if (await supersededTerminal?.(event, appendAbort.signal)) continue;
if (providerFailure) throw providerFailure;
if (compactionResult && event.eventType === "turn.completed") {
return {
@@ -2752,11 +2754,12 @@ export async function executeNativeSession(
return true;
};
const supersededByRestartContinuation = async (event: PrpEvent, signal: AbortSignal) => {
if (!recovered || !checkpointedInterruption || event.eventType !== "turn.failed" ||
!isNativeRestartInterruption(event.payload.error) || !event.turnId ||
event.turnId === checkpointedInterruption) return false;
await startupDecision;
signal.throwIfAborted();
if (!recovered || event.eventType !== "turn.failed" ||
!isNativeRestartInterruption(event.payload.error) || !event.turnId) return false;
if (checkpointedInterruption) {
await startupDecision;
signal.throwIfAborted();
}
if (!restartContinuationStarted) return false;
await continuationSetup?.catch(() => undefined);
signal.throwIfAborted();
@@ -2764,7 +2767,6 @@ export async function executeNativeSession(
};
const resumeInterruptedTurn = async (event: PrpEvent, signal: AbortSignal) => {
if (await requestContinuationDisposition(event, signal)) return true;
if (await supersededByRestartContinuation(event, signal)) return true;
if (restartContinuationStarted || !recovered || !options.resumeInterruptedTurn ||
// A compaction-only run never continues task work after a restart.
// Runnerd records a restart-lost turn only for these providers, after
@@ -2948,6 +2950,7 @@ export async function executeNativeSession(
options.sessionCompaction
? sessionCompactionResult(input, options.sessionCompaction.requestId)
: undefined,
supersededByRestartContinuation,
)
: Promise.resolve({
event: recoveryTerminal,
packages/plugins/sandbox-providers/daytona/src/plugin.test.ts
+81
View File
@@ -278,6 +278,67 @@ describe("Daytona idle eligibility", () => {
expect(await plugin.definition.onIdleDrain!(signal)).toBe("none");
});
describe("a sandbox in a state change during workspace realize", () => {
beforeEach(() => { vi.stubEnv("DAYTONA_API_KEY", "host-key"); });
const realize = (timeoutMs = 300_000) => plugin.definition.onEnvironmentRealizeWorkspace!({
driverKey: "daytona", companyId: "company-1", environmentId: "env-1",
config: { timeoutMs, reuseLease: false },
lease: { providerLeaseId: "sandbox-transition", metadata: { remoteCwd: "/home/daytona/paperclip-workspace" } },
workspace: { localPath: "/tmp/workspace" },
} as never);
const stateChangeInProgress = () => Object.assign(new Error("Sandbox state change in progress"), { statusCode: 409 });
it("waits for a stopping sandbox to settle, then starts it", async () => {
const sandbox = createMockSandbox({ id: "sandbox-transition", state: "stopping" });
mockGet.mockResolvedValue(sandbox);
sandbox.refreshData.mockImplementation(async () => { sandbox.state = "stopped"; });
sandbox.start.mockImplementation(async () => {
if (sandbox.state !== "stopped") throw stateChangeInProgress();
sandbox.state = "started";
});
await expect(realize()).resolves.toMatchObject({ cwd: "/home/daytona/paperclip-workspace" });
expect(sandbox.start).toHaveBeenCalledTimes(1);
expect(sandbox.fs.createFolder).toHaveBeenCalledWith("/home/daytona/paperclip-workspace", "755");
});
it("retries a start the provider rejected while its state was changing", async () => {
const sandbox = createMockSandbox({ id: "sandbox-transition", state: "stopped" });
mockGet.mockResolvedValue(sandbox);
sandbox.start.mockRejectedValueOnce(stateChangeInProgress())
.mockImplementationOnce(async () => { sandbox.state = "started"; });
await expect(realize()).resolves.toMatchObject({ cwd: "/home/daytona/paperclip-workspace" });
expect(sandbox.start).toHaveBeenCalledTimes(2);
});
it("gives up when the state change outlasts the start deadline", async () => {
const sandbox = createMockSandbox({ id: "sandbox-transition", state: "stopping" });
mockGet.mockResolvedValue(sandbox);
sandbox.start.mockRejectedValue(stateChangeInProgress());
await expect(realize(1_000)).rejects.toThrow("is still stopping at its start deadline");
expect(sandbox.start).not.toHaveBeenCalled();
expect(sandbox.fs.createFolder).not.toHaveBeenCalled();
});
it("gives the start only the budget the wait left", async () => {
const sandbox = createMockSandbox({ id: "sandbox-transition", state: "stopping" });
mockGet.mockResolvedValue(sandbox);
let reads = 0;
sandbox.refreshData.mockImplementation(async () => { if (++reads === 2) sandbox.state = "stopped"; });
sandbox.start.mockImplementation(async () => { sandbox.state = "started"; });
await expect(realize(4_000)).resolves.toMatchObject({ cwd: "/home/daytona/paperclip-workspace" });
expect(reads).toBe(2);
expect(sandbox.start.mock.calls[0]![0]).toBeLessThanOrEqual(2);
});
it("fails other start errors at once", async () => {
const sandbox = createMockSandbox({ id: "sandbox-transition", state: "stopped" });
mockGet.mockResolvedValue(sandbox);
sandbox.start.mockRejectedValue(new Error("quota exceeded"));
await expect(realize()).rejects.toThrow("quota exceeded");
expect(sandbox.start).toHaveBeenCalledTimes(1);
});
});
it("keeps a stale stopped reply from clearing a start that finished during the read", async () => {
const sandbox = createMockSandbox({ state: "stopped" });
mockGet.mockResolvedValue(sandbox);
@@ -3389,6 +3450,26 @@ describe("Daytona sandbox provider plugin", () => {
expect(lastExec[0]).toBe(freshSessionId);
});
it("opens a fresh session after a restart outside the plugin that realize waited out", async () => {
vi.stubEnv("DAYTONA_API_KEY", "host-key");
const sandbox = autoStoppedSandbox();
mockGet.mockResolvedValue(sandbox);
await plugin.definition.onEnvironmentExecute?.(sessionExecParams());
const staleSessionId = sandbox.process.createSession.mock.calls[0]![0] as string;
sandbox.state = "starting";
sandbox.refreshData.mockImplementationOnce(async () => { sandbox.state = "started"; });
await plugin.definition.onEnvironmentRealizeWorkspace?.({
driverKey: "daytona", companyId: "company-1", environmentId: "env-1",
config: { timeoutMs: 300000, reuseLease: false },
lease: { providerLeaseId: "sandbox-123", metadata: { remoteCwd: "/home/daytona/paperclip-workspace" } },
workspace: { localPath: "/tmp/workspace" },
} as never);
expect(sandbox.start).not.toHaveBeenCalled();
await plugin.definition.onEnvironmentExecute?.(sessionExecParams());
expect(sandbox.process.createSession).toHaveBeenCalledTimes(2);
expect(sandbox.process.createSession.mock.calls[1]![0]).not.toBe(staleSessionId);
});
it("opens a fresh session after sync-in restarts an auto-stopped sandbox", async () => {
process.env.DAYTONA_API_KEY = "host-key";
const sandbox = autoStoppedSandbox();
packages/plugins/sandbox-providers/daytona/src/plugin.ts
+40
View File
@@ -637,7 +637,47 @@ async function terminateAtProvider<T>(scope: SandboxScope, operation: string, ac
// restart, so the next command opens a fresh session instead of sending a dead
// id and failing with "session not found". A running sandbox keeps its live
// session, so the id is left in place when no restart happens.
const SANDBOX_TRANSITION_STATES: ReadonlySet<string> = new Set([
"starting", "stopping", "restoring", "archiving", "creating", "pending_build", "pulling_snapshot", "building_snapshot", "resizing",
]);
const SANDBOX_TRANSITION_POLL_MS = 1_000;
function isSandboxStateChangeInProgress(error: unknown): boolean {
return /state change in progress/i.test(error instanceof Error ? error.message : String(error));
}
async function waitOutSandboxTransition(sandbox: Sandbox, deadline: number, readTimeoutMs: number): Promise<void> {
while (SANDBOX_TRANSITION_STATES.has(sandbox.state ?? "") && Date.now() + SANDBOX_TRANSITION_POLL_MS < deadline) {
await new Promise(resolve => setTimeout(resolve, SANDBOX_TRANSITION_POLL_MS));
await withLivenessTimeout("sandbox.refreshData", Math.min(readTimeoutMs, Math.max(1, deadline - Date.now())),
() => sandbox.refreshData());
}
}
async function ensureSandboxStarted(sandbox: Sandbox, timeoutSeconds: number, scope: SandboxScope): Promise<void> {
if (sandbox.state === "started") return;
sandboxHandleSessionStore.clear(scope);
const deadline = Date.now() + timeoutSeconds * 1_000;
const readTimeoutMs = Math.min(scope.config.livenessTimeoutMs, 30_000);
for (;;) {
await waitOutSandboxTransition(sandbox, deadline, readTimeoutMs);
if (SANDBOX_TRANSITION_STATES.has(sandbox.state ?? "") || Date.now() >= deadline) {
throw new Error(`Daytona sandbox ${sandbox.id} is still ${sandbox.state ?? "changing state"} at its start deadline`);
}
const remainingSeconds = Math.max(1, Math.ceil((deadline - Date.now()) / 1_000));
try {
await startSettledSandbox(sandbox, remainingSeconds, scope);
return;
} catch (error) {
if (!isSandboxStateChangeInProgress(error) || Date.now() + SANDBOX_TRANSITION_POLL_MS >= deadline) throw error;
await new Promise(resolve => setTimeout(resolve, SANDBOX_TRANSITION_POLL_MS));
await withLivenessTimeout("sandbox.refreshData", Math.min(readTimeoutMs, Math.max(1, deadline - Date.now())),
() => sandbox.refreshData());
}
}
}
async function startSettledSandbox(sandbox: Sandbox, timeoutSeconds: number, scope: SandboxScope): Promise<void> {
if (sandbox.state === "started") return;
sandboxHandleSessionStore.clear(scope);
// Bound the lifecycle call just past its own SDK deadline. A normal slow start
server/src/__tests__/agent-budget-configuration.test.ts
+5 -5
View File
@@ -3,19 +3,19 @@ import type { Db } from "@paperclipai/db";
import type { AgentConfigurationRecord } from "../lib/agent-records.js";
import { prepareAgentConfiguration } from "../services/agent-configuration.js";
// Configuration preparation now publishes passive work hints; this unit only checks budget authority.
const hintDb = () => ({ execute: async () => [] }) as unknown as Db;
const existing = { id: "agent", companyId: "company", status: "idle", budgetMonthlyCents: 500 } as AgentConfigurationRecord;
// Preparation queues retention work through the database before reading the patch.
const fakeDb = () => ({ execute: async () => [] }) as unknown as Db;
describe("budget authority in configuration preparation", () => {
it("rejects agent-originated budget changes before persistence or policy audit", async () => {
const db = hintDb();
const db = fakeDb();
await expect(prepareAgentConfiguration(db, existing, { budgetMonthlyCents: 5000 }, { recordRevision: { createdByAgentId: "ceo" } })).rejects.toMatchObject({ status: 403 });
await expect(prepareAgentConfiguration(db, existing, { budgetMonthlyCents: 0 }, { recordRevision: { createdByAgentId: "ceo" } })).rejects.toMatchObject({ status: 403 });
});
it("removes an unchanged legacy budget from agent rollback so it cannot rewrite policy or invent a board actor", async () => {
expect(await prepareAgentConfiguration(hintDb(), existing, { budgetMonthlyCents: 500 }, { recordRevision: { createdByAgentId: "ceo", source: "rollback" } })).toEqual({});
expect(await prepareAgentConfiguration(fakeDb(), existing, { budgetMonthlyCents: 500 }, { recordRevision: { createdByAgentId: "ceo", source: "rollback" } })).toEqual({});
});
it("preserves the actual board-authorized amount for the existing policy writer", async () => {
expect(await prepareAgentConfiguration(hintDb(), existing, { budgetMonthlyCents: 5000 }, { recordRevision: { createdByUserId: "operator-user" } })).toEqual({ budgetMonthlyCents: 5000 });
expect(await prepareAgentConfiguration(fakeDb(), existing, { budgetMonthlyCents: 5000 }, { recordRevision: { createdByUserId: "operator-user" } })).toEqual({ budgetMonthlyCents: 5000 });
});
});
server/src/__tests__/agent-instance-secrets.test.ts
+26 -3
View File
@@ -1,7 +1,7 @@
import { randomBytes, randomUUID } from "node:crypto";
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
import { eq, sql } from "drizzle-orm";
import { agents, companies, companySecrets, companySecretVersions, activityLog, plugins, toolApplications, toolConnections, createDb, type Db } from "@paperclipai/db";
import { afterAll, beforeAll, describe, expect, it, onTestFinished, vi } from "vitest";
import { and, eq, sql } from "drizzle-orm";
import { agents, companies, companySecrets, companySecretVersions, activityLog, plugins, secretAccessEvents, toolApplications, toolConnections, createDb, type Db } from "@paperclipai/db";
import type { AgentLifecycleRequest } from "@paperclipai/shared";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { pluginAgentRuntimeIdentity } from "../services/plugin-agent-runtime-state.js";
@@ -118,6 +118,29 @@ const support = await getEmbeddedPostgresTestSupport();
await expect(secrets.createBinding({ companyId, secretId: stored.id, targetType: "agent", targetId: f.agent.id, configPath: "env.RUNTIME" })).rejects.toThrow("company-scoped");
});
it("reads each secret with its latest version in one statement, auditing every read and keeping version failures", async () => {
const f = await fixture(); onTestFinished(async () => { await db.delete(agents).where(eq(agents.id, f.agent.id)); });
await f.service.apply(f.request, pluginId, { runtime, launcher: "launcher value" });
await f.service.apply(f.request, pluginId, { launcher: "rotated launcher" });
const [stored] = await db.select().from(companySecrets).where(and(eq(companySecrets.ownerAgentId, f.agent.id), eq(companySecrets.name, "launcher")));
expect(stored.latestVersion).toBe(2);
const audit = () => db.select().from(secretAccessEvents).where(eq(secretAccessEvents.secretId, stored.id)).orderBy(secretAccessEvents.createdAt);
const selects = vi.spyOn(db, "select");
expect(await f.service.read(f.scope)).toEqual({ runtime, launcher: "rotated launcher" });
expect(selects).toHaveBeenCalledTimes(1);
selects.mockRestore();
expect((await audit()).at(-1)).toMatchObject({ outcome: "success", version: 2, consumerType: "plugin_worker", consumerId: pluginId, pluginId, actorType: "plugin" });
const audited = (await audit()).length;
expect(await f.service.read(f.scope)).toEqual({ runtime, launcher: "rotated launcher" });
expect(await audit()).toHaveLength(audited + 1);
await db.update(companySecretVersions).set({ revokedAt: new Date() }).where(and(eq(companySecretVersions.secretId, stored.id), eq(companySecretVersions.version, 2)));
await expect(f.service.read(f.scope)).rejects.toThrow("Agent instance secret unavailable");
expect((await audit()).at(-1)).toMatchObject({ outcome: "failure", errorCode: "version_inactive", version: 2 });
await db.delete(companySecretVersions).where(and(eq(companySecretVersions.secretId, stored.id), eq(companySecretVersions.version, 2)));
await expect(f.service.read(f.scope)).rejects.toThrow("Agent instance secret unavailable");
expect((await audit()).at(-1)).toMatchObject({ outcome: "failure", errorCode: "version_missing", version: 2 });
});
it("fences stale writes and removals, and cascades credentials when the owning agent is deleted", async () => {
const f = await fixture(); await f.service.apply(f.request, pluginId, { runtime });
for (const patch of [{ version: 2 }, { operationId: randomUUID() }, { phase: "cleaning_up" }, { companyId: randomUUID() }, { agentId: randomUUID() }]) {
server/src/__tests__/agent-permissions-routes.test.ts
+29 -5
View File
@@ -2642,15 +2642,37 @@ describe("agent permission routes", () => {
});
});
it.each([0, 5000])("budget authority: rejects agent policy changes through the legacy field (%i)", async amount => {
mockAgentService.getById.mockResolvedValue({ ...baseAgent, budgetMonthlyCents: 500 });
const app = createApp({ type: "agent", agentId, companyId, source: "agent_key", runId: "run-1" });
it.each([
["its own budget", agentId, 0],
["its own budget", agentId, 5000],
["another agent's budget as a CEO", "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa", 5000],
])("budget authority: PATCH /agents/:id never lets an agent change %s (%s, %i)", async (_case, actorAgentId, amount) => {
mockAgentService.getById.mockImplementation(async (id: string) => id === agentId
? { ...baseAgent, budgetMonthlyCents: 500 }
: { ...baseAgent, id, role: "ceo", permissions: { canCreateAgents: true } });
const app = createApp({ type: "agent", agentId: actorAgentId, companyId, source: "agent_key", runId: "run-1" });
const response = await requestApp(app, url => request(url).patch(`/api/agents/${agentId}`).send({ budgetMonthlyCents: amount }));
expect(response.status).toBe(403);
expect(response.status, JSON.stringify(response.body)).toBe(403);
expect(response.body.code).toBe("board_access_required");
expect(response.body.remediation).toContain("this denial is final");
expect(response.body.remediation).toContain(`[Open the agent's budget](/<prefix>/agents/${agentId}/budgets)`);
expect(mockAgentService.update).not.toHaveBeenCalled();
expect(mockBudgetService.upsertPolicy).not.toHaveBeenCalled();
expect(mockLogActivity).not.toHaveBeenCalled();
});
it("budget authority: a configuration rollback never lets an agent restore a different budget", async () => {
mockAgentService.getById.mockResolvedValue({ ...baseAgent, budgetMonthlyCents: 500 });
mockAgentService.getConfigRevision.mockResolvedValue({
id: "33333333-3333-4333-8333-333333333333",
afterConfig: { adapterType: "process", adapterConfig: {}, runtimeConfig: {}, budgetMonthlyCents: 50000 },
});
const app = createApp({ type: "agent", agentId, companyId, source: "agent_key", runId: "run-1" });
const response = await requestApp(app, url => request(url).post(`/api/agents/${agentId}/config-revisions/33333333-3333-4333-8333-333333333333/rollback`));
expect(response.status, JSON.stringify(response.body)).toBe(403);
expect(response.body.code).toBe("board_access_required");
expect(response.body.remediation).toContain(`[Open the agent's budget](/<prefix>/agents/${agentId}/budgets)`);
expect(mockAgentService.rollbackConfigRevision).not.toHaveBeenCalled();
});
it("budget authority: accepts an unchanged legacy amount with an unrelated title edit", async () => {
const app = createApp({ type: "agent", agentId, companyId, source: "agent_key" });
const response = await requestApp(app, url => request(url).patch(`/api/agents/${agentId}`).send({ budgetMonthlyCents: 0, title: "New title" }));
@@ -2661,7 +2683,9 @@ describe("agent permission routes", () => {
mockAccessService.hasPermission.mockResolvedValue(true);
const app = createApp({ type: "agent", agentId, companyId, source: "agent_key" });
const response = await requestApp(app, url => request(url).post(`/api/companies/${companyId}/${route}`).send({ name: "New employee", role: "engineer", adapterType: "process", budgetMonthlyCents: 5000 }));
expect(response.status).toBe(403);
expect(response.status, JSON.stringify(response.body)).toBe(403);
expect(response.body.code).toBe("board_access_required");
expect(response.body.remediation).toContain("[Open budgets](/<prefix>/activity/budgets)");
expect(mockAgentService.create).not.toHaveBeenCalled();
expect(mockBudgetService.upsertPolicy).not.toHaveBeenCalled();
expect(mockLogActivity).not.toHaveBeenCalled();
server/src/__tests__/agent-runtime-hooks.test.ts
+79 -2
View File
@@ -1,12 +1,12 @@
import * as databaseModule from "@paperclipai/db";
import { randomUUID } from "node:crypto";
import { randomBytes, randomUUID } from "node:crypto";
import { createServer, connect, type Socket } from "node:net";
import { EventEmitter } from "node:events";
import fs from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { and, eq, sql } from "drizzle-orm";
import { activityLog, agents, companies, applyPendingMigrations, closeFileStorageDb, createDb, pluginCompanySettings, pluginConfig, plugins, resourceLifecycleEvents, type Db } from "@paperclipai/db";
import { activityLog, agents, companies, applyPendingMigrations, closeFileStorageDb, createDb, pluginCompanySettings, pluginConfig, plugins, resourceLifecycleEvents, secretAccessEvents, type Db } from "@paperclipai/db";
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import * as instructionModule from "../services/agent-instructions.js";
@@ -18,6 +18,7 @@ import { idleWorkSnapshot } from "../services/task-admission.js";
import { resolveManagedInstructionsRoot } from "../services/agent-instructions.js";
import { agentFileStore } from "../services/agent-file-store.js";
import { agentInstructionRevisionService } from "../services/agent-instruction-revisions.js";
import { secretService } from "../services/secrets.js";
const externalUrl = process.env.PAPERCLIP_AGENT_RUNTIME_HOOKS_TEST_DATABASE_URL;
const expectedDatabase = process.env.PAPERCLIP_AGENT_RUNTIME_HOOKS_EXPECTED_DATABASE;
@@ -221,6 +222,82 @@ describeDatabase("agent runtime synchronization hooks", () => {
await expect(pluginAgentRuntimeIdentity(db, pluginId, companyId, agentId)).rejects.toMatchObject({ status: 403 });
});
it("returns the same identity, revision, secrets and failures as the full runtime read", async () => {
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", randomBytes(32).toString("base64"));
const scope = { companyId, agentId, pluginId };
const outcome = (read: Promise<unknown>) => read.then(value => value, (error: { status?: number }) => ({ status: error.status }));
const same = async (company = companyId, agent = agentId, plugin = pluginId) => {
const state = await outcome(pluginAgentRuntimeState(db, plugin, company, agent)) as Awaited<ReturnType<typeof pluginAgentRuntimeState>> | { status: number };
const identity = await outcome(pluginAgentRuntimeIdentity(db, plugin, company, agent));
expect(identity).toEqual(state && "revision" in state ? { revision: state.revision, instanceSecrets: state.instanceSecrets, agent: {
id: state.agent.id, companyId: state.agent.companyId, status: state.agent.status, lifecycleState: state.agent.lifecycleState, lifecycleVersion: state.agent.lifecycleVersion } } : state);
return identity as Awaited<ReturnType<typeof pluginAgentRuntimeIdentity>>;
};
await secretService(db).agentInstance.set(scope, "launcher", "first launcher");
const first = await same();
expect(first).toEqual({ revision: String((await events()).at(-1)!.id), instanceSecrets: { launcher: "first launcher" },
agent: { id: agentId, companyId, status: "idle", lifecycleState: "ready", lifecycleVersion: 0 } });
await db.insert(resourceLifecycleEvents).values([{ companyId, resourceType: "agent", resourceId: agentId, action: "pause" },
{ companyId, resourceType: "agent_runtime", resourceId: randomUUID(), action: "update" }]);
expect((await same())!.revision).toBe(first!.revision);
const [latest] = await db.insert(resourceLifecycleEvents).values({ companyId, resourceType: "agent_runtime", resourceId: agentId, action: "update" }).returning();
expect((await same())!.revision).toBe(String(latest.id));
await db.update(agents).set({ lifecycleState: "preparing", lifecycleVersion: 3 }).where(eq(agents.id, agentId));
expect((await same())!.agent).toMatchObject({ lifecycleState: "preparing", lifecycleVersion: 3 });
await secretService(db).agentInstance.set(scope, "launcher", "second launcher");
expect((await same())!.instanceSecrets).toEqual({ launcher: "second launcher" });
await secretService(db).agentInstance.set(scope, "launcher", null);
expect((await same())!.instanceSecrets).toEqual({});
await secretService(db).agentInstance.set(scope, "launcher", "third launcher");
const accessEvents = async () => (await db.select().from(secretAccessEvents).where(eq(secretAccessEvents.companyId, companyId))).length;
const audited = await accessEvents();
expect(await same(randomUUID())).toBeNull();
expect(await same(companyId, randomUUID())).toBeNull();
expect(await same(companyId, agentId, randomUUID())).toEqual({ status: 403 });
const manifest = { resourceLifecycle: ["agent_runtime"], capabilities: ["events.subscribe", "agents.lifecycle.manage"] };
const denials: Array<[() => Promise<unknown>, () => Promise<unknown>]> = [
[() => db.update(plugins).set({ status: "error" }).where(eq(plugins.id, pluginId)), () => db.update(plugins).set({ status: "ready" }).where(eq(plugins.id, pluginId))],
[() => db.update(plugins).set({ manifestJson: { ...manifest, capabilities: ["events.subscribe"] } as never }).where(eq(plugins.id, pluginId)),
() => db.update(plugins).set({ manifestJson: manifest as never }).where(eq(plugins.id, pluginId))],
[() => db.update(plugins).set({ manifestJson: { ...manifest, resourceLifecycle: [] } as never }).where(eq(plugins.id, pluginId)),
() => db.update(plugins).set({ manifestJson: manifest as never }).where(eq(plugins.id, pluginId))],
[() => db.insert(pluginCompanySettings).values({ companyId, pluginId, enabled: false }), () => db.delete(pluginCompanySettings).where(eq(pluginCompanySettings.pluginId, pluginId))],
[() => db.update(agents).set({ lifecycleRequiredPluginIds: [randomUUID()] }).where(eq(agents.id, agentId)),
() => db.update(agents).set({ lifecycleRequiredPluginIds: [pluginId] }).where(eq(agents.id, agentId))],
[() => db.update(agents).set({ lifecycleRequiredPluginIds: null }).where(eq(agents.id, agentId)),
() => db.update(agents).set({ lifecycleRequiredPluginIds: [pluginId] }).where(eq(agents.id, agentId))],
];
for (const [deny, restore] of denials) {
await deny();
expect(await same()).toEqual({ status: 403 });
await restore();
}
expect(await accessEvents()).toBe(audited);
await db.insert(pluginCompanySettings).values({ companyId, pluginId, enabled: true });
expect((await same())!.instanceSecrets).toEqual({ launcher: "third launcher" });
});
it("reads runtime identity from a snapshot without waiting for an agent row lock", async () => {
let locked!: () => void;
let release!: () => void;
const held = new Promise<void>(resolve => { locked = resolve; });
const writer = db.transaction(async tx => {
await tx.select().from(agents).where(eq(agents.id, agentId)).for("update");
await tx.update(agents).set({ lifecycleState: "pausing", lifecycleVersion: 5 }).where(eq(agents.id, agentId));
await new Promise<void>(resolve => { release = resolve; locked(); });
});
await held;
try {
const read = pluginAgentRuntimeIdentity(db, pluginId, companyId, agentId);
const timeout = new Promise<never>((_, reject) => setTimeout(() => reject(new Error("identity read waited for the agent lock")), 5_000));
expect((await Promise.race([read, timeout]))!.agent).toMatchObject({ lifecycleState: "ready", lifecycleVersion: 0 });
} finally {
release();
await writer;
}
expect((await pluginAgentRuntimeIdentity(db, pluginId, companyId, agentId))!.agent).toMatchObject({ lifecycleState: "pausing", lifecycleVersion: 5 });
});
it("uses the company reporting graph for runtime health data", async () => {
const [manager] = await db.insert(agents).values({ companyId, name: "Manager" }).returning();
await db.update(agents).set({ reportsTo: manager.id }).where(eq(agents.id, agentId));
server/src/__tests__/built-in-agent-routes.test.ts
+2
View File
@@ -208,6 +208,8 @@ describe("built-in agent routes", () => {
const app = await createApp({ type: "agent", agentId: "agent-1", companyId, source: "agent_key" });
const response = await request(app).post(`/api/companies/${companyId}/built-in-agents/briefs/provision`).send({ budgetMonthlyCents });
expect(response.status).toBe(403);
expect(response.body.code).toBe("board_access_required");
expect(response.body.remediation).toContain("[Open budgets](/<prefix>/activity/budgets)");
expect(mockBuiltInAgentService.provision).not.toHaveBeenCalled();
expect(mockLogActivity).not.toHaveBeenCalled();
});
server/src/__tests__/environment-task-runtime.test.ts
+17 -1
View File
@@ -85,9 +85,25 @@ describe("plugin-provided Runner execution", () => {
expect(operations).not.toContain("complete");
release();
const result = await checked;
if (preparationStopped) { expect(result).toEqual({ value: true, error: null }); expect(operations).toEqual(["stop", "status", "complete"]); }
if (preparationStopped) { expect(result).toEqual({ value: "completed", error: null }); expect(operations).toEqual(["stop", "status", "complete"]); }
else { expect(result.error?.message).toContain("termination is unconfirmed"); expect(fixture.sets).toContainEqual(expect.objectContaining({ status: "pending_cleanup", cleanupStatus: "failed" })); }
});
it("still stops a startup task first when resume already observed the stop", async () => {
const value = row();
Object.assign(value.lease.metadata, { runnerTaskDispatched: false,
runnerTaskStartupCleanup: { contextId: "startup", requiresPreparationStop: true } });
const fixture = database(value), workers = worker();
fixture.db.select = ((...args: unknown[]) => (database(value).db.select as (...values: unknown[]) => unknown)(...args)) as Db["select"];
Object.assign(workers, { drainStartup: vi.fn(async () => undefined) });
const operations: string[] = [];
workers.call.mockImplementation(async (...args: any[]) => {
const operation = args[2].operation.kind; operations.push(operation);
return operation === "status" ? { kind: "status", taskId: "attempt-1", phase: "cancelled", executionStopped: true, preparationStopped: true }
: { kind: "accepted", taskId: "attempt-1" };
});
await expect(settleDispatchedEnvironmentTask(fixture.db, workers as never, { companyId: "company", id: leaseId }, { executionStopObserved: true })).resolves.toBe("completed");
expect(operations).toEqual(["stop", "status", "complete"]);
});
it("negotiates the transient context without reading transport for an older plugin", async () => {
const context = new NativeExecutionContext({ binding: { companyId: "company", agentId: "agent", runId: "run" } } as never, 1_000);
const workers = worker();
server/src/__tests__/ownership-reconciliation-report.test.ts
+21 -2
View File
@@ -457,17 +457,36 @@ describe("report-only ownership reconciliation", () => {
body: "self", createdAt: minutesAgo(2) });
await db.insert(agentWakeupRequests).values({ companyId, agentId, source: "automation", status: "queued",
payload: { issueId, commentId: carried.id } });
const report = await runOwnershipReconciliationPass(db, { state: createOwnershipReportState(), record: false });
const report = await runOwnershipReconciliationPass(db, { state: createOwnershipReportState(), record: false,
processStartedAt: minutesAgo(10) });
expect(report.violations.filter(row => row.reason === "user_comment_without_wake").map(row => row.recordId)).toEqual([lost.id]);
// The young comment may still get its normal wake: a deadline, not a report.
expect(report.nextDueAt?.getTime()).toBe(young.createdAt.getTime() + 30_000);
// A run that started after the comment read it.
await db.insert(heartbeatRuns).values({ companyId, agentId, status: "succeeded", invocationSource: "manual",
contextSnapshot: { issueId }, startedAt: minutesAgo(1), createdAt: minutesAgo(1), finishedAt: new Date() });
const later = await runOwnershipReconciliationPass(db, { state: createOwnershipReportState(), record: false });
const later = await runOwnershipReconciliationPass(db, { state: createOwnershipReportState(), record: false,
processStartedAt: minutesAgo(10) });
expect(later.violations.filter(row => row.reason === "user_comment_without_wake")).toEqual([]);
});
it("finds at once a young user comment whose wake a restart lost after the board stopped the run", async () => {
const issueId = randomUUID();
await db.insert(issues).values({ id: issueId, companyId, title: "Stopped then continued", status: "todo", assigneeAgentId: agentId });
await db.insert(heartbeatRuns).values({ companyId, agentId, status: "cancelled", invocationSource: "assignment",
runtimeMode: "native", contextSnapshot: { issueId }, startedAt: minutesAgo(4), createdAt: minutesAgo(4), finishedAt: minutesAgo(1) });
const [comment] = await db.insert(issueComments).values({ companyId, issueId, authorUserId: "user-1",
body: "I stopped your previous run on purpose. Please continue.", createdAt: new Date(Date.now() - 12_000) }).returning();
const report = await runOwnershipReconciliationPass(db, { state: createOwnershipReportState(), record: false,
processStartedAt: new Date(Date.now() - 2_000) });
expect(report.violations.filter(row => row.reason === "user_comment_without_wake").map(row => row.recordId))
.toEqual([comment!.id]);
const sameProcess = await runOwnershipReconciliationPass(db, { state: createOwnershipReportState(), record: false,
processStartedAt: minutesAgo(10) });
expect(sameProcess.violations.filter(row => row.reason === "user_comment_without_wake")).toEqual([]);
expect(sameProcess.nextDueAt?.getTime()).toBe(comment!.createdAt.getTime() + 30_000);
});
it("finds a user comment that a restart left without a wake on a task its agent then closed", async () => {
const closedTask = async (actorType: "agent" | "user") => {
const issueId = randomUUID();
server/src/__tests__/plugin-runner-task.integration.test.ts
+1 -1
View File
@@ -416,7 +416,7 @@ databaseSuite("Core plugin Runner task lifecycle", () => {
return;
}
const next = await acquire;
expect(operations).toEqual(["stop", "status", "complete", "environmentReleaseLease", "environmentAcquireLease"]);
expect(operations).toEqual(["stop", "status", "complete", "environmentAcquireLease"]);
expect(next.lease.id).not.toBe(oldLeaseId);
await expect(resolvePluginTaskResume(db, { companyId, leaseId: next.lease.id, priorIdentity })).resolves.toBe("old-task");
await expect(runtime.task({ companyId, leaseId: next.lease.id, operation: { kind: "submit", projectIds: [], bootstrapTicket: "fixture", runner } })).rejects.toThrow("identity mismatch");
server/src/__tests__/plugin-task-credential-writeback.test.ts
+24 -6
View File
@@ -65,11 +65,11 @@ suite("durable plugin task credential handoff", () => {
const lease = () => db.select().from(environmentLeases).where(and(eq(environmentLeases.id, leaseId), eq(environmentLeases.companyId, companyId))).then(rows => rows[0]!);
return { companyId, agentId, runId, environmentId, leaseId, owner, selection: selection!, managed, workers, calls, lease,
failExport(value: boolean) { failExport = value; }, credential: () => service.credential(selection!),
settle: () => settleDispatchedEnvironmentTask(db, workers, { companyId, id: leaseId }) };
settle: (executionStopObserved = false) => settleDispatchedEnvironmentTask(db, workers, { companyId, id: leaseId }, { executionStopObserved }) };
}
it("persists refreshed credentials before completing a stopped task, after controller files disappear", async () => {
const f = await fixture();
expect(await f.settle()).toBe(true);
expect(await f.settle()).toBe("completed");
expect(await f.credential()).toBe(auth("fixture-refreshed", 2021));
expect(f.calls.indexOf("credentials")).toBeLessThan(f.calls.indexOf("complete"));
const saved = await f.lease();
@@ -77,6 +77,24 @@ suite("durable plugin task credential handoff", () => {
expect(JSON.stringify(saved.metadata)).not.toContain("fixture-refreshed");
expect(JSON.stringify(saved.metadata)).not.toContain(f.managed.home!);
});
it("writes back credentials before completing a task whose stop was already observed, without stopping it again", async () => {
const f = await fixture();
expect(await f.settle(true)).toBe("completed");
expect(f.calls).toEqual(["status", "credentials", "complete"]);
expect(await f.credential()).toBe(auth("fixture-refreshed", 2021));
expect((await f.lease()).metadata?.runnerTaskCredentialHandoff).toMatchObject({ state: "complete" });
});
it("leaves an observed-stop task for cleanup when credential writeback fails", async () => {
const f = await fixture(); f.failExport(true);
await expect(f.settle(true)).rejects.toThrow("termination is unconfirmed");
expect(f.calls).toEqual(["status", "credentials"]);
expect(await f.lease()).toMatchObject({ status: "pending_cleanup", cleanupStatus: "failed" });
expect((await f.lease()).metadata?.runnerTaskCredentialHandoff).toMatchObject({ state: "pending" });
f.failExport(false);
expect(await f.settle()).toBe("completed");
expect(f.calls.slice(2)).toEqual(["stop", "status", "credentials", "complete"]);
expect(await f.credential()).toBe(auth("fixture-refreshed", 2021));
});
it("keeps a failed transfer pending and recovers through a fresh cleanup after run and environment deletion", async () => {
const f = await fixture(); f.failExport(true);
await expect(f.settle()).rejects.toThrow("termination is unconfirmed");
@@ -86,7 +104,7 @@ suite("durable plugin task credential handoff", () => {
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
await db.delete(environments).where(eq(environments.id, f.environmentId));
f.failExport(false);
expect(await f.settle()).toBe(true);
expect(await f.settle()).toBe("completed");
expect(await f.credential()).toBe(auth("fixture-refreshed", 2021));
expect(f.calls.at(-1)).toBe("complete");
});
@@ -104,7 +122,7 @@ suite("durable plugin task credential handoff", () => {
expect((await f.lease()).metadata?.runnerTaskCredentialHandoff).toMatchObject({ state: "pending" });
} finally { await db.execute(sql`drop trigger reject_runner_credential_save on company_secret_versions; drop function reject_runner_credential_save()`); }
const receiver = createDb(database.connectionString);
expect(await settleDispatchedEnvironmentTask(receiver, f.workers, { companyId: f.companyId, id: f.leaseId })).toBe(true);
expect(await settleDispatchedEnvironmentTask(receiver, f.workers, { companyId: f.companyId, id: f.leaseId })).toBe("completed");
expect(await f.credential()).toBe(auth("fixture-refreshed", 2021));
expect(f.calls.filter(call => call === "credentials")).toHaveLength(2);
expect(f.calls.at(-1)).toBe("complete");
@@ -113,13 +131,13 @@ suite("durable plugin task credential handoff", () => {
it("does not overwrite an explicit account replacement", async () => {
const f = await fixture();
await secretService(db).rotate(f.managed.credentialWriteback!.secretId, { value: auth("fixture-reconnected", 2022) }, { userId: f.owner });
expect(await f.settle()).toBe(true);
expect(await f.settle()).toBe("completed");
expect(f.calls).not.toContain("credentials");
expect(await f.credential()).toBe(auth("fixture-reconnected", 2022));
});
it("allows concurrent cleanup to complete the same durable handoff", async () => {
const f = await fixture();
expect(await Promise.all([f.settle(), f.settle()])).toEqual([true, true]);
expect(await Promise.all([f.settle(), f.settle()])).not.toContain(false);
expect(await f.credential()).toBe(auth("fixture-refreshed", 2021));
expect((await f.lease()).metadata?.runnerTaskCredentialHandoff).toMatchObject({ state: "complete" });
});
server/src/__tests__/plugin-warm-tasks.integration.test.ts
+37 -4
View File
@@ -7,6 +7,7 @@ import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
import { eq, sql } from "drizzle-orm";
import { agents, companies, createDb, applyPendingMigrations, environmentLeases, environments, heartbeatRuns, issues, plugins, projects, sandboxCreationOperations } from "@paperclipai/db";
import { environmentRuntimeService } from "../services/environment-runtime.js";
import { environmentService } from "../services/environments.js";
import { recordSandboxCreationResource } from "../services/sandbox-creation-operations.js";
import { resolvePluginTaskResume } from "../services/environment-task-runtime.js";
import type { PluginWorkerManager } from "../services/plugin-worker-manager.js";
@@ -20,8 +21,8 @@ const support=external ? {supported:true} : await getEmbeddedPostgresTestSupport
afterAll(async()=>{await stop?.();});
async function fixture(initialProjects:string[]=[],initialExpiry?:string,deadline?:Date){
const companyId=randomUUID(),agentId=randomUUID(),issueId=randomUUID(),environmentId=randomUUID(),pluginId=randomUUID();
let runId=randomUUID(),allocation=0,stopped=false,outage=false,releaseFails=false;
const calls:string[]=[];
let runId=randomUUID(),allocation=0,stopped=false,outage=false,releaseFails=false,unavailable=false;
const calls:string[]=[],dispatched:Array<[string,string]>=[],failures=new Set<string>();
let projectIds=[...initialProjects],providerExpiry=initialExpiry;
const taskExpiries=new Map<string,string|undefined>(),acquireRequests:any[]=[];
let resumeGate:Promise<void>|undefined,releaseResume:(()=>void)|undefined,resumes=0;
@@ -38,12 +39,14 @@ const support=external ? {supported:true} : await getEmbeddedPostgresTestSupport
const workers={isRunning:()=>true,getWorker:()=>({supportedMethods:["environmentTask","environmentAcquireLease","environmentResumeLease","environmentReleaseLease"]}),
call:async(_id:string,method:string,p:any)=>{
calls.push(method==="environmentTask" ? p.operation.kind : method);
if(method==="environmentTask") dispatched.push([p.operation.kind,p.taskId]);
if(method==="environmentTask" && failures.delete(p.operation.kind)) throw Error(`${p.operation.kind} unavailable`);
if(method==="environmentAcquireLease") {
const id=`task-${++allocation}`;taskExpiries.set(id,providerExpiry);acquireRequests.push(p);
return {providerLeaseId:id,...(providerExpiry ? {expiresAt:providerExpiry}:{}),metadata:{runId:p.runId}};
}
if(method==="environmentResumeLease") {if(outage) throw Error("unreachable");
if(resumeGate){if(++resumes===2)releaseResume!();await resumeGate;}return {providerLeaseId:stopped?null:p.providerLeaseId,expiresAt:taskExpiries.get(p.providerLeaseId),metadata:{runId:p.runId}};}
if(resumeGate){if(++resumes===2)releaseResume!();await resumeGate;}return {providerLeaseId:stopped||unavailable?null:p.providerLeaseId,expiresAt:taskExpiries.get(p.providerLeaseId),metadata:{runId:p.runId}};}
if(method==="environmentReleaseLease") {if(releaseFails) throw Error("release unavailable");return {providerLeaseId:p.providerLeaseId,state:"stopped"};}
if(p.operation.kind==="status") return {kind:"status",taskId:p.taskId,phase:stopped?"completed":"running",executionStopped:stopped,sessionStateAvailable:stopped};
if(p.operation.kind==="read_state") return {kind:"state",taskId:p.taskId,state:{...p.operation.identity,lifecycle:"ready"}};
@@ -63,7 +66,8 @@ const support=external ? {supported:true} : await getEmbeddedPostgresTestSupport
launch:{runnerVersion:"fixture",runnerDigest:`sha256:${"a".repeat(64)}`,maxOutboxBytes:262144,p0ReserveBytes:65536,lifecyclePolicy:{mode:"warm",idleTimeoutMs:300000}}}});
await runtime().releaseRunLeases(runId,"released",undefined,"keep_running");
await db.update(heartbeatRuns).set({status:"succeeded"}).where(eq(heartbeatRuns.id,runId));
return {companyId,first,identity,calls,runtime,acquire,acquireRequests,
return {companyId,first,identity,calls,dispatched,runtime,acquire,acquireRequests,
fail(kind:string){failures.add(kind);},unavailable(){unavailable=true;},
setExpiry(next?:string){providerExpiry=next;},
async nextRun(){runId=randomUUID();await db.insert(heartbeatRuns).values({id:runId,companyId,agentId,nativeIssueId:issueId,invocationSource:"manual",status:"running"});return runId;},
setProjects(next:string[]){projectIds=next;},stopped(){stopped=true;},failRelease(){releaseFails=true;},outage(){outage=true;},
@@ -109,6 +113,35 @@ const support=external ? {supported:true} : await getEmbeddedPostgresTestSupport
expect(f.calls).toContain("complete");
await expect(resolvePluginTaskResume(db,{companyId:f.companyId,leaseId:next.lease.id,priorIdentity:f.identity})).resolves.toBe(f.first.lease.providerLeaseId);
});
it("retires a stopped retained task with one status and complete before allocating its replacement",async()=>{
const f=await fixture();f.stopped();await f.nextRun();
const calls=f.calls.length,dispatched=f.dispatched.length;
const next=await f.acquire();
expect(f.calls.slice(calls)).toEqual(["environmentResumeLease","status","complete","environmentAcquireLease"]);
expect(f.dispatched.slice(dispatched)).toEqual([["status",f.first.lease.providerLeaseId],["complete",f.first.lease.providerLeaseId]]);
expect(next.lease.providerLeaseId).not.toBe(f.first.lease.providerLeaseId);
const [old]=await db.select().from(environmentLeases).where(eq(environmentLeases.id,f.first.lease.id));
expect(old.status).toBe("released");
expect(old.metadata).toMatchObject({runnerTaskExecutionStopped:true,runnerTaskDispatched:false,remoteExecutionTermination:{providerLeaseId:f.first.lease.providerLeaseId,state:"stopped"}});
});
it("stops a retained task that resume could not attach but that is still running",async()=>{
const f=await fixture();f.unavailable();await f.nextRun();
const calls=f.calls.length;
await f.acquire();
expect(f.calls.slice(calls)).toEqual(["environmentResumeLease","status","stop","status","complete","environmentAcquireLease"]);
});
it.each([["status",false],["complete",false],["stop",true]] as const)("keeps a stopped retained task for cleanup retry when %s fails",async(kind,running)=>{
const f=await fixture();if(running) f.unavailable(); else f.stopped();await f.nextRun();
f.fail(kind);
await expect(f.acquire()).rejects.toThrow("termination is unconfirmed");
expect(f.calls.filter(call=>call==="environmentAcquireLease")).toHaveLength(1);
expect(f.calls).not.toContain("environmentReleaseLease");
const pending=(await environmentService(db).getLeaseById(f.first.lease.id))!;
expect(pending).toMatchObject({status:"pending_cleanup",cleanupStatus:"failed"});
const calls=f.calls.length;
await expect(f.runtime().retryPendingSandboxTeardown({environment:null,lease:pending})).resolves.toEqual({providerLeaseId:f.first.lease.providerLeaseId,state:"stopped"});
expect(f.calls.slice(calls)).toEqual(["stop","status","complete","environmentReleaseLease"]);
});
it("does not allocate a replacement when the retained computer cannot be inspected",async()=>{
const f=await fixture();f.outage();await f.nextRun();
await expect(f.acquire()).rejects.toThrow("unreachable");
server/src/__tests__/secret-proposals-routes.test.ts
+67
View File
@@ -865,6 +865,73 @@ describeEmbeddedPostgres("secret proposal routes", () => {
]);
});
it.each(["rejected", "approved", "pending"] as const)("deletes a secret whose %s binding proposal referenced it", async (outcome) => {
const fixture = await seedRun();
const secret = await secretService(db).create(fixture.companyId, {
name: "dev/deploy/publishing",
key: "DEPLOY_PUBLISHING",
provider: "local_encrypted",
value: "publishing-secret",
});
const proposals = createSecretProposalsService(db);
const proposal = await proposals.createBinding({ companyId: fixture.companyId, heartbeatRunId: fixture.heartbeatRunId }, {
secretId: secret.id,
configPath: "env.PUBLISHING_API_KEY",
justification: "Publish the site",
bindingTargetPolicy: "self_and_reports",
});
if (outcome === "rejected") await proposals.transition(fixture.companyId, proposal.id, "rejected", { resolvedByUserId: "board-user" });
if (outcome === "approved") {
await proposals.approve(fixture.companyId, proposal.id, { resolvedByUserId: "board-user" });
await db.delete(companySecretBindings).where(eq(companySecretBindings.secretId, secret.id));
}
// The proposal's secret reference used ON DELETE SET NULL, which the binding
// shape check rejects, so deleting the secret failed with a 500.
await db.update(issues).set({ assigneeAgentId: fixture.agentId }).where(eq(issues.id, fixture.issueId));
const wakeup = vi.fn().mockResolvedValue(null);
const persistedIssues = issueService(db);
const addComment = vi.fn().mockResolvedValue(undefined);
const deleted = await request(createBoardApp(fixture, {
heartbeat: { wakeup },
issues: { getById: persistedIssues.getById, addComment },
})).delete(`/api/secrets/${secret.id}`);
expect(deleted.status).toBe(200);
// Only a pending proposal is rejected by the delete; it gets the normal rejection signal.
expect(addComment).toHaveBeenCalledTimes(outcome === "pending" ? 1 : 0);
if (outcome === "pending") {
expect(addComment.mock.calls[0]?.[1]).toContain("Status: **rejected**");
expect(addComment.mock.calls[0]?.[1]).toContain("The referenced secret was deleted.");
}
expect(wakeup).toHaveBeenCalledTimes(outcome === "pending" ? 1 : 0);
expect(await db.select().from(companySecrets).where(eq(companySecrets.id, secret.id))).toEqual([]);
expect(await db.select().from(companySecretProposals).where(eq(companySecretProposals.id, proposal.id))).toEqual([]);
// A pending review card closes as rejected instead of pointing at a deleted proposal.
const [card] = await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, proposal.interactionId!));
expect(card?.status).toBe(outcome === "approved" ? "accepted" : "rejected");
});
it("refuses a new binding proposal for a secret that a delete already marked deleted", async () => {
const fixture = await seedRun();
const secret = await secretService(db).create(fixture.companyId, {
name: "dev/deploy/deleting",
key: "DEPLOY_DELETING",
provider: "local_encrypted",
value: "deleting-secret",
});
let proposalDuringDelete: unknown;
await secretService(db).remove(secret.id, {
beforeDelete: async () => {
proposalDuringDelete = await createSecretProposalsService(db).createBinding(
{ companyId: fixture.companyId, heartbeatRunId: fixture.heartbeatRunId },
{ secretId: secret.id, configPath: "env.LATE_KEY", justification: "Late request", bindingTargetPolicy: "self_and_reports" },
).catch((error: unknown) => error);
},
});
expect(proposalDuringDelete).toMatchObject({ status: 404 });
expect(await db.select().from(companySecretProposals)).toEqual([]);
});
it("proposes and approves a binding by resolving the proposer's own sourceConfigPath", async () => {
const fixture = await seedRun();
const sourceSecret = await secretService(db).create(fixture.companyId, {
server/src/__tests__/secrets-routes.test.ts
+1
View File
@@ -1107,6 +1107,7 @@ describe("secret routes", () => {
expect(res.body).toEqual({ ok: true });
expect(mockSecretService.remove).toHaveBeenCalledWith(
"33333333-3333-4333-8333-333333333333",
{ beforeDelete: expect.any(Function) },
);
expect(mockLogActivity).toHaveBeenCalledWith(
expect.anything(),
server/src/routes/agents.ts
+11 -8
View File
@@ -140,7 +140,7 @@ import {
import { badRequest, conflict, forbidden, HttpError, notFound, unprocessable } from "../errors.js";
import { ONBOARDING_FIRST_TASK_SKILL_KEY, PAPERCLIP_CORE_SKILL_KEYS } from "../services/company-skills.js";
import { createRunSecretRedactionRegistry } from "../services/run-secret-redaction.js";
import { assertAuthenticated, assertBoard, assertCompanyAccess, assertInstanceAdmin, buildActorSecretContext, getAccessibleResource, getActorInfo, hasCompanyAccess } from "./authz.js";
import { agentBudgetPage, assertAuthenticated, assertBoard, assertCompanyAccess, COMPANY_BUDGETS_PAGE, assertInstanceAdmin, buildActorSecretContext, getAccessibleResource, getActorInfo, hasCompanyAccess } from "./authz.js";
import { runAdapterLoginStartSpine } from "./adapter-login-route-spine.js";
import { isLoginCommandSupportedAdapterType } from "../services/login-command.js";
import {
@@ -4316,7 +4316,7 @@ export function agentRoutes(
throw unprocessable("Invalid revision snapshot");
}
if (typeof rollbackConfig.budgetMonthlyCents === "number" && rollbackConfig.budgetMonthlyCents !== existing.budgetMonthlyCents) {
assertBoard(req);
assertBoard(req, agentBudgetPage(existing.id));
}
assertProviderTraceSettingTransition(
req,
@@ -4460,7 +4460,9 @@ export function agentRoutes(
return;
}
if (!company.requireBoardApprovalForNewAgents && req.body.budgetMonthlyCents > 0) assertBoard(req);
// Budgets are board-only. An agent may propose one in a hire that waits for
// board approval; the approving operator then activates it.
if (!company.requireBoardApprovalForNewAgents && req.body.budgetMonthlyCents > 0) assertBoard(req, COMPANY_BUDGETS_PAGE);
const sourceIssueIds = parseSourceIssueIds(req.body);
const {
desiredSkills: requestedDesiredSkills,
@@ -4712,7 +4714,6 @@ export function agentRoutes(
let approval: Awaited<ReturnType<typeof approvalsSvc.getById>> | null = null;
const actor = getActorInfo(req);
if (requiresApproval) {
const requestedAdapterType = normalizedHireInput.adapterType ?? agent.adapterType;
const requestedAdapterConfig =
@@ -4837,7 +4838,7 @@ export function agentRoutes(
router.post("/companies/:companyId/agents", validate(createAgentSchema), async (req, res) => {
const companyId = req.params.companyId as string;
await assertCanCreateAgentsForCompany(req, companyId, { hiring: true });
if (req.body.budgetMonthlyCents > 0) assertBoard(req);
if (req.body.budgetMonthlyCents > 0) assertBoard(req, COMPANY_BUDGETS_PAGE);
const company = await db
.select()
@@ -5022,8 +5023,6 @@ export function agentRoutes(
);
await builtInAgentService(db).ensureCompanyDefaultAgentGrants(companyId);
res.status(201).json({ ...redactAgentRowForResponse(agent), ...(configurationWarnings.length ? { configurationWarnings } : {}) });
});
@@ -5520,7 +5519,11 @@ export function agentRoutes(
const id = req.params.id as string;
const existing = await getAccessibleResource(req, res, svc.getById(id), "Agent not found");
if (!existing) return;
if (hasOwn(req.body as object, "budgetMonthlyCents") && req.body.budgetMonthlyCents !== existing.budgetMonthlyCents) assertBoard(req);
// A budget change is the board-only budget control (PATCH /agents/:id/budgets),
// so PATCH is never a way around it.
if (hasOwn(req.body as object, "budgetMonthlyCents") && req.body.budgetMonthlyCents !== existing.budgetMonthlyCents) {
assertBoard(req, agentBudgetPage(existing.id));
}
if (hasOwn(req.body as object, "permissions")) {
res.status(422).json({ error: "Use /api/agents/:id/permissions for permission changes" });
server/src/routes/authz.ts
+18 -5
View File
@@ -44,13 +44,22 @@ const AGENT_PROFILE_TABS: Record<string, [string, string]> = {
"runtime-state": ["harness settings", "runtime"],
};
/** A board page as [label, path], where the path keeps <prefix> for the agent to fill. */
export type BoardControlPage = [label: string, path: string];
/** Budgets are board-only. These pages hold the budget controls behind any route that writes one. */
export const COMPANY_BUDGETS_PAGE: BoardControlPage = ["budgets", "/<prefix>/activity/budgets"];
export function agentBudgetPage(agentId: string): BoardControlPage {
return [`the agent's ${AGENT_PROFILE_TABS.budgets![0]}`, `/<prefix>/agents/${agentId}/${AGENT_PROFILE_TABS.budgets![1]}`];
}
// The board page that holds the control behind a board-only API route, when one
// is known. Paths keep <prefix> for the agent to fill from its own task identifier.
const UUID_PATTERN = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;
const BOARD_CONTROL_PAGES: Array<[RegExp, (match: RegExpExecArray, req: Request) => [string, string] | null]> = [
[/^\/(?:companies\/[^/]+\/)?secrets?(?:[-/]|$)/, () => ["company secrets", "/<prefix>/company/settings/secrets"]],
[/^\/companies\/[^/]+\/(?:budgets|budget-incidents|costs)(?:\/|$)/, () => ["budgets", "/<prefix>/activity/budgets"]],
[/^\/companies\/[^/]+\/(?:budgets|budget-incidents|costs)(?:\/|$)/, () => COMPANY_BUDGETS_PAGE],
[/^\/companies\/[^/]+\/(?:prompt-injection-prevention|decision-model)(?:\/|$)/, () => ["connection settings", "/<prefix>/company/settings/connections"]],
[/^\/companies\/[^/]+\/(?:members|invites|join-requests)(?:\/|$)/, () => ["company members", "/<prefix>/company/settings/members"]],
[/^\/companies\/[^/]+\/(?:tools\/)?gateways(?:\/|$)|^\/tool-gateway\//, () => ["tool gateways", "/<prefix>/apps/gateways"]],
@@ -105,9 +114,9 @@ function boardControlPage(req: Request) {
* A board-only denial. Agents get a final, actionable remediation so they hand the
* action to the person instead of searching other routes for one that does not exist.
*/
export function boardAccessRequired(req: Request) {
export function boardAccessRequired(req: Request, controlPage?: BoardControlPage) {
if (req.actor.type !== "agent") return forbidden("Board access required");
const page = boardControlPage(req);
const page = controlPage ?? boardControlPage(req);
const scope = PLUGIN_ROUTE.test(deniedApiPath(req)) ? `${PLUGIN_ADMINISTRATION_SCOPE} ` : "";
const where = page
? `They can do it in the Paperclip web app: link [Open ${page[0]}](${page[1]}), with this conversation's company prefix as <prefix>.`
@@ -135,9 +144,13 @@ export function issueNotFoundBody(req: Request): Record<string, unknown> {
};
}
export function assertBoard(req: Request) {
/**
* Board-only check. Pass `controlPage` when the board control lives on a different
* page from the one the route path implies, e.g. a budget field on a general route.
*/
export function assertBoard(req: Request, controlPage?: BoardControlPage) {
if (req.actor.type !== "board") {
throw boardAccessRequired(req);
throw boardAccessRequired(req, controlPage);
}
}
server/src/routes/built-in-agents.ts
+2 -2
View File
@@ -6,7 +6,7 @@ import { forbidden, notFound } from "../errors.js";
import { accessService, instanceSettingsService, logActivity } from "../services/index.js";
import { builtInAgentService } from "../services/built-in-agents.js";
import { authorizationDeniedDetails } from "../services/authorization.js";
import { assertBoard, assertCompanyAccess, getActorInfo } from "./authz.js";
import { assertBoard, assertCompanyAccess, COMPANY_BUDGETS_PAGE, getActorInfo } from "./authz.js";
import type { BuiltInAgentState } from "../services/built-in-agents.js";
const WEEKDAY_LABELS = ["Sun", "Mon", "Tue", "Wed", "Thu", "Fri", "Sat"];
@@ -186,7 +186,7 @@ export function builtInAgentRoutes(db: Db) {
await assertBuiltInAgentsEnabled();
await assertCanProvisionBuiltInAgents(req, companyId);
const actor = getActorInfo(req);
if (req.body.budgetMonthlyCents !== undefined) assertBoard(req);
if (req.body.budgetMonthlyCents !== undefined) assertBoard(req, COMPANY_BUDGETS_PAGE);
const result = await svc.provision(companyId, key, req.body, {
requestedByAgentId: actor.actorType === "agent" ? actor.actorId : null,
requestedByUserId: actor.actorType === "user" ? actor.actorId : null,
server/src/routes/openapi.ts
+1 -1
View File
@@ -11849,7 +11849,7 @@ registerCurrentRoute({
path: "/api/companies/{companyId}/tools/connections",
tags: ["tool-access"],
summary: "Create a tool connection",
description: "Saves a connection as given without checking that its server answers. To connect an MCP server from its URL, use POST /api/companies/{companyId}/tools/apps/connect with link instead: it checks the server first, and a fresh setup saves no connection when the server cannot be reached." + " " + OWN_MCP_SERVER_KEY,
description: "Saves a connection as given without checking that its server answers. To connect an MCP server from its URL, use POST /api/companies/{companyId}/tools/apps/connect with link instead: it checks the server first, and a fresh setup saves no connection when the server cannot be reached." + " " + OWN_MCP_SERVER_KEY + " " + OWN_MCP_SERVER_KEY,
body: createToolConnectionSchema,
responses: {
201: r.ok(),
server/src/routes/secrets.ts
+11 -1
View File
@@ -1122,7 +1122,17 @@ export function secretRoutes(db: Db, deps: SecretRoutesDeps = {}) {
);
if (!existing) return;
const removed = await svc.remove(id);
const resolvedByUserId = req.actor.userId ?? "board";
const reason = "The referenced secret was deleted.";
const removed = await svc.remove(id, {
beforeDelete: async (secret) => {
const rejected = await proposals.rejectPendingBindingsForSecret(secret.companyId, secret.id, { resolvedByUserId, reason });
// Same signal as an ordinary rejection, so the proposing agent continues.
for (const proposal of rejected) {
await notifySecretProposalResolution({ proposal, status: "rejected", userId: resolvedByUserId, reason, issues, heartbeat });
}
},
});
if (!removed) {
res.status(404).json({ error: "Secret not found" });
return;
server/src/services/environment-runtime.ts
+27 -20
View File
@@ -3322,6 +3322,30 @@ function createPluginEnvironmentDriver(
};
}
async function releasePluginRunLease(input: EnvironmentDriverReleaseInput, executionStopObserved = false) {
await db.update(environmentLeases).set({ metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || '{"pluginLeaseCleanupMethod":"release"}'::jsonb` }).where(and(eq(environmentLeases.id, input.lease.id), eq(environmentLeases.companyId, input.lease.companyId)));
const settled = await settleDispatchedEnvironmentTask(db, workerManager, input.lease, { executionStopObserved });
const taskStopped = settled !== false ||
(input.lease.metadata?.runnerTaskLease === true && input.lease.metadata?.runnerTaskDispatched !== true && !input.lease.metadata?.runnerTaskStartup && !input.lease.metadata?.runnerTaskStartupCleanup);
if (settled !== "completed") {
const { plugin, driverKey, driverConfig } = await resolvePluginDriverForRelease(input);
await workerManager.call(plugin.id, "environmentReleaseLease", {
driverKey,
companyId: input.lease.companyId,
environmentId: input.environment.id,
issueId: input.lease.issueId,
config: driverConfig,
providerLeaseId: input.lease.providerLeaseId,
leaseMetadata: input.lease.metadata ?? undefined,
});
}
return await environmentsSvc.releaseLease(input.lease.id, input.status, {
cleanupStatus: "success",
...(taskStopped ? { remoteExecutionTermination: remoteTerminationReceipt(input.lease,
{ providerLeaseId: input.lease.providerLeaseId, state: "stopped" }) } : {}),
});
}
return {
driver: "plugin",
@@ -3363,7 +3387,7 @@ function createPluginEnvironmentDriver(
providerLeaseId: previous.providerLeaseId, leaseMetadata: previous.metadata ?? undefined,
});
if (!resumed?.providerLeaseId) {
await this.releaseRunLease({ environment: input.environment, lease: toEnvironmentLeaseSnapshot(previous), status: "released" });
await releasePluginRunLease({ environment: input.environment, lease: toEnvironmentLeaseSnapshot(previous), status: "released" }, true);
resumed = undefined;
}
}
@@ -3423,25 +3447,8 @@ function createPluginEnvironmentDriver(
});
},
async releaseRunLease(input) {
await db.update(environmentLeases).set({ metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || '{"pluginLeaseCleanupMethod":"release"}'::jsonb` }).where(and(eq(environmentLeases.id, input.lease.id), eq(environmentLeases.companyId, input.lease.companyId)));
const taskStopped = await settleDispatchedEnvironmentTask(db, workerManager, input.lease) ||
(input.lease.metadata?.runnerTaskLease === true && input.lease.metadata?.runnerTaskDispatched !== true && !input.lease.metadata?.runnerTaskStartup && !input.lease.metadata?.runnerTaskStartupCleanup);
const { plugin, driverKey, driverConfig } = await resolvePluginDriverForRelease(input);
await workerManager.call(plugin.id, "environmentReleaseLease", {
driverKey,
companyId: input.lease.companyId,
environmentId: input.environment.id,
issueId: input.lease.issueId,
config: driverConfig,
providerLeaseId: input.lease.providerLeaseId,
leaseMetadata: input.lease.metadata ?? undefined,
});
return await environmentsSvc.releaseLease(input.lease.id, input.status, {
cleanupStatus: "success",
...(taskStopped ? { remoteExecutionTermination: remoteTerminationReceipt(input.lease,
{ providerLeaseId: input.lease.providerLeaseId, state: "stopped" }) } : {}),
});
releaseRunLease(input) {
return releasePluginRunLease(input);
},
async resumeRunLease(input) {
server/src/services/environment-task-runtime.ts
+13 -6
View File
@@ -493,7 +493,8 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage
}
/** Failure leaves the dispatch marker and provider handle available to cleanup retries. */
export async function settleDispatchedEnvironmentTask(db: Db, workers: PluginWorkerManager, lease: { id: string; companyId: string }) {
export async function settleDispatchedEnvironmentTask(db: Db, workers: PluginWorkerManager, lease: { id: string; companyId: string },
options: { executionStopObserved?: boolean } = {}): Promise<false | "stopped" | "completed"> {
const [current] = await db.update(environmentLeases).set({
metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || '{"runnerTaskRetiring":true}'::jsonb`,
}).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, lease.companyId)))
@@ -503,19 +504,25 @@ export async function settleDispatchedEnvironmentTask(db: Db, workers: PluginWor
if (current?.metadata?.runnerTaskDispatched !== true && !startup && !startupCleanup) {
if (current?.metadata) await settlePluginTaskCredentialHandoff(db, { companyId: lease.companyId, leaseId: lease.id }, current.metadata,
async () => { throw new Error("Undispatched task must not request credentials"); });
return current?.metadata?.runnerTaskExecutionStopped === true;
return current?.metadata?.runnerTaskExecutionStopped === true ? "stopped" : false;
}
const scope = { companyId: lease.companyId, leaseId: lease.id };
try {
const contextId = (startupCleanup as { contextId?: unknown } | undefined)?.contextId ?? (startup as { contextId?: unknown } | undefined)?.contextId;
const draining = typeof contextId === "string" ? workers.drainStartup?.(contextId) : undefined;
void draining?.catch(() => undefined);
await executeEnvironmentTask(db, workers, { ...scope, operation: { kind: "stop" } });
const state = await executeEnvironmentTask(db, workers, { ...scope, operation: { kind: "status" } });
if (state.kind !== "status" || state.executionStopped !== true || ((startup || startupCleanup?.requiresPreparationStop) && state.preparationStopped !== true)) throw new Error("Plugin Runner termination is unconfirmed");
const confirmed = (state?: Awaited<ReturnType<typeof executeEnvironmentTask>>) => state?.kind === "status" && state.executionStopped === true &&
(!(startup || startupCleanup?.requiresPreparationStop) || state.preparationStopped === true);
let state = options.executionStopObserved === true && !startup && !startupCleanup
? await executeEnvironmentTask(db, workers, { ...scope, operation: { kind: "status" } }) : undefined;
if (!confirmed(state)) {
await executeEnvironmentTask(db, workers, { ...scope, operation: { kind: "stop" } });
state = await executeEnvironmentTask(db, workers, { ...scope, operation: { kind: "status" } });
}
if (!confirmed(state)) throw new Error("Plugin Runner termination is unconfirmed");
await draining;
await executeEnvironmentTask(db, workers, { ...scope, operation: { kind: "complete" } });
return true;
return "completed";
} catch {
await writeEnvironmentCleanup(db, tx => tx.update(environmentLeases).set({ status: "pending_cleanup", cleanupStatus: "failed",
failureReason: "plugin_runner_task_cleanup_unconfirmed", releasedAt: new Date(), updatedAt: new Date(),
server/src/services/heartbeat/run-completion.test.ts
+24 -1
View File
@@ -16,7 +16,7 @@ import { createHash, randomUUID } from "node:crypto";
import { appendHeartbeatRunEvent } from "../heartbeat-run-events.js";
import { createNativeRunLogCoverageVerifier, formatNativeRunLogEvent } from "../native-runtime/native-run-log-export.js";
import type { Db } from "@paperclipai/db";
import { activityLog, agents, agentWakeupRequests, approvals, issueApprovals, budgetReservations, caseHandoffs, cases, companies, completionContracts, costEvents, createDb, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, heartbeatRuns, heartbeatRunEvents, issueComments, issueQuestionResponseDeliveries, issueRecoveryActions, issues, nativeRunFinalizations } from "@paperclipai/db";
import { activityLog, agents, agentWakeupRequests, approvals, issueApprovals, budgetReservations, caseHandoffs, cases, companies, completionContracts, costEvents, createDb, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, heartbeatRuns, heartbeatRunEvents, issueComments, issueQuestionResponseDeliveries, issueRecoveryActions, issues, issueThreadInteractions, nativeRunFinalizations } from "@paperclipai/db";
import { and, eq, sql } from "drizzle-orm";
import { afterAll, beforeAll, beforeEach, describe, expect, it, onTestFinished, vi } from "vitest";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
@@ -481,6 +481,10 @@ describe.skipIf(!support.supported && !process.env.PAPERCLIP_TEST_DATABASE_URL)(
{ hold: "finished turn", reasonCode: "conversation_turn_finished", replies: true },
{ hold: "own question card", reasonCode: "governed_response_waiting", replies: false },
{ hold: "execution stage and own question card", reasonCode: "governed_response_waiting", replies: false },
// A card the board addressed to someone else holds the task, but this run
// did not create it, so it cannot be the turn's response.
{ hold: "someone else's confirmation card", reasonCode: "governed_response_waiting", replies: true },
{ hold: "earlier turn's question card", reasonCode: "governed_response_waiting", replies: true },
] as const)("posts a waiting conversation turn's accepted answer: $hold", async ({ hold, reasonCode, replies }) => {
const f = await fixture({ runtimeMode: "native" }, `CNV${randomUUID().slice(0, 6)}`);
const [issue] = await db.insert(issues).values({ companyId: f.run.companyId, title: "Chat with CEO",
@@ -501,6 +505,22 @@ describe.skipIf(!support.supported && !process.env.PAPERCLIP_TEST_DATABASE_URL)(
selectionMode: "single", required: true, options: [{ id: "flash", label: "Flash" }] }] } },
{ agentId: f.agent.id, runId: f.run.id });
}
if (hold === "someone else's confirmation card") {
// Saved as the board's create route saves it: no source run.
await db.insert(issueThreadInteractions).values({ companyId: f.run.companyId, issueId: issue.id,
kind: "request_confirmation", status: "pending", continuationPolicy: "none", title: "Riley's website proposal",
requestedResolverPolicy: "human_only", effectiveResolverPolicy: "human_only", resolverPolicyProvenance: "explicit",
addresseeUserId: "riley", createdByUserId: "local-board",
payload: { version: 1, prompt: "Riley: approve creating the website implementation task?" } });
}
if (hold === "earlier turn's question card") {
const [earlier] = await db.insert(heartbeatRuns).values({ companyId: f.run.companyId, agentId: f.agent.id,
status: "succeeded", runtimeMode: "native", responsibleUserId: "local-board" }).returning();
await issueThreadInteractionService(db).create(issue, { kind: "request_confirmation", sourceRunId: earlier.id,
resolverPolicy: "human_only", continuationPolicy: "wake_assignee",
payload: { version: 1, prompt: "Create the website implementation task?" } },
{ agentId: f.agent.id, runId: earlier.id });
}
const answer = "Resubmitted the hire with the cheaper model. [Review the pending approval](/CNV/approvals/a1).";
f.input.issueId = issue.id;
f.input.issueRef = issue;
@@ -528,6 +548,9 @@ describe.skipIf(!support.supported && !process.env.PAPERCLIP_TEST_DATABASE_URL)(
expect((await read(f.run.id)).resultJson).toMatchObject({ presentationDecision: replies
? { chosenSource: "final_agent_message", commentAction: "create", commentId: posted[0]?.id }
: { chosenSource: "none", commentAction: "none" } });
// The posted answer (or the run's own card) is the turn's response: the
// conversation waits for the person.
expect((await db.select().from(issues).where(eq(issues.id, issue.id)))[0].conversationState).toBe("waiting");
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
it("settles the accepted native exact rate-card price without replacing it with zero", async () => {
server/src/services/heartbeat/run-completion.ts
+6 -9
View File
@@ -45,7 +45,7 @@ import { isUnresolvedWorkspaceBaseRefError, readUnresolvedWorkspaceBaseRefDiagno
import { runnerGoalService } from "../runner-goals.js";
import { resolveChatRunPresentationAuthorizationReason } from "../chat-run-publications.js";
import { hasChatRunOwnedProviderInteraction } from "../chat-interaction-arbitration.js";
import { conversationTurnAnswered } from "../native-runtime/native-governance.js";
import { conversationTurnPresentsAnswer } from "../native-runtime/native-governance.js";
import { readAcceptedTaskContinuationSummary } from "../heartbeat-run-summary.js";
import { SANDBOX_PROVIDER_PLUGIN_NOT_READY_REASON } from "../recovery/stranded-notice.js";
import { MAX_TURN_CONTINUATION_RETRY_REASON, isResolvedInteractionContinuationWakeContext } from "../../modules/run-dispatch/index.js";
@@ -847,14 +847,11 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC
interactionOwnsResponse: Boolean(issueId && readAcceptedTaskContinuationSummary(persistedResultJson) &&
await hasChatRunOwnedProviderInteraction(db, { companyId: livenessRun.companyId, issueId, runId: livenessRun.id })),
conversationTurnFinished: Boolean(issueId) && isConversation(issueContext) && livenessRun.status === "succeeded"
&& (persistedResultJson?.finalizationReasonCode === "conversation_turn_finished"
// A turn left waiting on an approval, an execution stage or a
// blocker still answered the person. Only its own question card
// presents it instead.
|| (persistedResultJson?.finalizationPhase === "committed"
&& await measureRunPhase("response.resolve", () => conversationTurnAnswered({
db, companyId: livenessRun.companyId, issueId: issueId!, runId: livenessRun.id,
})))),
&& await measureRunPhase("response.resolve", () => conversationTurnPresentsAnswer({
db, companyId: livenessRun.companyId, issueId: issueId!, runId: livenessRun.id,
finalizationPhase: persistedResultJson?.finalizationPhase,
finalizationReasonCode: persistedResultJson?.finalizationReasonCode,
})),
nativeConversationFinalResponse: livenessRun.runtimeMode === "native" && isConversation(issueContext),
existingComment: existingRunComment,
finalAgentMessage,
server/src/services/native-runtime/agent-chat-response-recovery.test.ts
+96 -3
View File
@@ -1,8 +1,7 @@
import { randomUUID } from "node:crypto";
import { and, eq, sql } from "drizzle-orm";
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
import { agents, authUsers, companies, completionContracts, createDb, heartbeatRunEvents, heartbeatRuns, issueComments, issues,
nativeRunFinalizations, nativeRunResults, issueThreadInteractions, chatTaskHandoffs, chatCompletionDeliveries, agentWakeupRequests, statusDecisions, type Db } from "@paperclipai/db";
import { agents, authUsers, companies, completionContracts, createDb, heartbeatRunEvents, heartbeatRuns, issueComments, issues, nativeRunFinalizations, nativeRunResults, issueThreadInteractions, chatTaskHandoffs, chatCompletionDeliveries, agentWakeupRequests, statusDecisions, type Db, approvals, issueApprovals } from "@paperclipai/db";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
import { resolveExternalTestDatabaseUrl } from "../../__tests__/helpers/external-test-database.js";
import { NativeRunCoordinatorStore } from "./native-run-coordinator-store.js";
@@ -26,7 +25,8 @@ describe.skipIf(!support.supported)("committed Agent Chat reply recovery", () =>
}, 60_000);
afterAll(async () => { await db?.$client.end(); await database?.cleanup(); });
async function fixture(conversation = true, options: { succeed?: boolean; disposition?: "done" | "needs_review" | "yielded"; channel?: "final" | "unknown" } = {}) {
async function fixture(conversation = true, options: { succeed?: boolean; disposition?: "done" | "needs_review" | "yielded"; channel?: "final" | "unknown";
beforeFinalize?: (ids: { companyId: string; issueId: string; runId: string; agentId: string }) => Promise<void> } = {}) {
const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), runId = randomUUID();
const sessionId = randomUUID(), runnerId = randomUUID(), userId = `fixture-${randomUUID()}`, wakeId = randomUUID();
await db.insert(authUsers).values({ id: userId, name: "Conversation owner", email: `${userId}@example.test`,
@@ -55,6 +55,7 @@ describe.skipIf(!support.supported)("committed Agent Chat reply recovery", () =>
if (options.channel) prpEvent.payload.channel = options.channel;
await db.insert(heartbeatRunEvents).values({ companyId, runId, agentId, seq: 170, eventType: "item.completed", payload: { prpEvent } });
const disposition = options.disposition ?? "done";
await options.beforeFinalize?.({ companyId, issueId, runId, agentId });
await port.completeRun({ result: { schema: "paperclip.run_result.v1", reportedWorkDisposition: disposition,
summary: "A completed hire, with its final message preserved separately.",
...(disposition === "yielded" ? { continuation: { kind: "response_wake" as const, summary: "Wait for the next human message", idempotencyKey: "conversation-next-message" } } : {}),
@@ -720,4 +721,96 @@ describe.skipIf(!support.supported)("committed Agent Chat reply recovery", () =>
await expect(recoverCommittedNativeChatResponse(db, f.input)).resolves.toBe(true);
expect(await comments(f.runId)).toHaveLength(1);
});
// A chat turn can end while its task stays held: a resubmitted approval or
// an execution stage keeps the status, but the turn still answered the
// person. A restart between the status commit and the live presentation
// must not lose that answer. The run is saved exactly as the previous
// release's finalizer saves it, with no presentation recorded yet.
type Hold = "pending approval" | "approval sent back for revision" | "execution stage" | "own question card"
| "someone else's confirmation card";
async function heldTurn(hold: Hold, disposition: "yielded" | "done" = "yielded") {
return fixture(true, { disposition, beforeFinalize: async ({ companyId, issueId, runId, agentId }) => {
if (hold.includes("approval")) {
const [approval] = await db.insert(approvals).values({ companyId, type: "hire_agent",
status: hold === "pending approval" ? "pending" : "revision_requested", payload: { name: "Researcher" } }).returning();
await db.insert(issueApprovals).values({ companyId, issueId, approvalId: approval.id });
}
if (hold === "execution stage") await db.update(issues).set({ executionState: { status: "pending" } }).where(eq(issues.id, issueId));
if (hold === "own question card") {
await db.insert(issueThreadInteractions).values({ companyId, issueId, kind: "ask_user_questions", status: "pending",
sourceRunId: runId, createdByAgentId: agentId, payload: { version: 1, questions: [{ id: "model", prompt: "Which model?",
selectionMode: "single", required: true, options: [{ id: "flash", label: "Flash" }] }] } });
}
// The board addressed this card to another person; the run did not create it.
if (hold === "someone else's confirmation card") {
await db.insert(issueThreadInteractions).values({ companyId, issueId, kind: "request_confirmation", status: "pending",
continuationPolicy: "none", title: "Riley's website proposal", requestedResolverPolicy: "human_only",
effectiveResolverPolicy: "human_only", resolverPolicyProvenance: "explicit", addresseeUserId: "riley",
createdByUserId: "local-board", payload: { version: 1, prompt: "Riley: approve creating the website implementation task?" } });
}
} });
}
it.each([
["pending approval", "yielded"], ["approval sent back for revision", "yielded"], ["execution stage", "yielded"],
["pending approval", "done"], ["someone else's confirmation card", "yielded"],
] as const)(
"recovers a held conversation turn's answer exactly once after a restart: %s (%s)", async (hold, disposition) => {
const f = await heldTurn(hold, disposition);
const [decision] = await db.select().from(statusDecisions).where(eq(statusDecisions.runId, f.runId));
expect(decision.reasonCode).not.toBe("conversation_turn_finished");
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0].resultJson)
.not.toHaveProperty("presentationDecision");
expect(await comments(f.runId)).toEqual([]);
const work = nativeResponsePresentationWork(db), entries = await work.load(f.input);
expect(entries).toEqual([expect.objectContaining({ id: f.runId })]);
await expect(work.check(entries[0])).resolves.toBeNull();
expect(await comments(f.runId)).toEqual([expect.objectContaining({ body: f.reply, authorAgentId: f.agentId, createdByRunId: f.runId })]);
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0].resultJson).toMatchObject({
presentationDecision: { chosenSource: "final_agent_message", commentAction: "create", commentId: expect.any(String) } });
expect(await issue(f.issueId)).toMatchObject({ conversationState: "waiting" });
expect(await work.load(f.input)).toEqual([]);
await expect(recoverCommittedNativeChatResponse(db, f.input)).resolves.toBe(false);
expect(await comments(f.runId)).toHaveLength(1);
},
);
it("does not repost a held conversation turn's answer that the live path already presented", async () => {
const f = await heldTurn("pending approval");
const [saved] = await db.insert(issueComments).values({ companyId: f.companyId, issueId: f.issueId, authorAgentId: f.agentId,
createdByRunId: f.runId, body: f.reply }).returning();
await db.update(heartbeatRuns).set({ resultJson: sql`coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) || ${JSON.stringify({
presentationDecision: { chosenSource: "final_agent_message", commentAction: "create", commentId: saved.id, reasonCodes: [] } })}::jsonb` })
.where(eq(heartbeatRuns.id, f.runId));
expect(await nativeResponsePresentationWork(db).load(f.input)).toEqual([]);
await expect(recoverCommittedNativeChatResponse(db, f.input)).resolves.toBe(false);
expect(await comments(f.runId)).toEqual([saved]);
});
it("keeps a held turn's presentation that the previous release's live path already recorded", async () => {
// Before the live answered rule, a held yielded turn was recorded as a
// control-plane wait. That decision stands; recovery never re-presents it.
const f = await heldTurn("pending approval");
await db.update(heartbeatRuns).set({ resultJson: sql`coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) || ${JSON.stringify({
presentationDecision: { chosenSource: "none", commentAction: "none", reasonCodes: ["yielded_control_plane_wait"] } })}::jsonb` })
.where(eq(heartbeatRuns.id, f.runId));
expect(await nativeResponsePresentationWork(db).load(f.input)).toEqual([]);
await expect(recoverCommittedNativeChatResponse(db, f.input)).resolves.toBe(false);
expect(await comments(f.runId)).toEqual([]);
});
it("settles a held conversation turn presented by its own question card without posting its prose", async () => {
const f = await heldTurn("own question card");
const cards = await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.issueId, f.issueId));
expect(cards).toEqual([expect.objectContaining({ kind: "ask_user_questions", status: "pending", sourceRunId: f.runId })]);
const work = nativeResponsePresentationWork(db), entries = await work.load(f.input);
expect(entries).toEqual([expect.objectContaining({ id: f.runId })]);
await expect(work.check(entries[0])).resolves.toBeNull();
expect(await comments(f.runId)).toEqual([]);
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0].resultJson).toMatchObject({
presentationDecision: { chosenSource: "none", commentAction: "none" } });
// Settled like the live path: the card is the turn's response.
expect(await issue(f.issueId)).toMatchObject({ conversationState: "waiting" });
expect(await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.issueId, f.issueId))).toEqual(cards);
expect(await work.load(f.input)).toEqual([]);
await expect(recoverCommittedNativeChatResponse(db, f.input)).resolves.toBe(false);
expect(await comments(f.runId)).toEqual([]);
});
});
server/src/services/native-runtime/native-governance.ts
+28 -4
View File
@@ -31,6 +31,8 @@ async function pendingNativeInteraction(input: {
issueId: string;
runId: string;
ignoredInteractionIds?: string[];
/** Only a card this run created; any other card can hold the task but not present the turn. */
createdByRun?: boolean;
}): Promise<{ id: string } | null> {
const [issue] = await input.db.select({ conversationAgentId: issues.conversationAgentId, conversationUserId: issues.conversationUserId })
.from(issues).where(and(eq(issues.id, input.issueId), eq(issues.companyId, input.companyId)));
@@ -43,6 +45,7 @@ async function pendingNativeInteraction(input: {
eq(issueThreadInteractions.issueId, input.issueId),
eq(issueThreadInteractions.status, "pending"),
...(input.ignoredInteractionIds?.length ? [notInArray(issueThreadInteractions.id, input.ignoredInteractionIds)] : []),
...(input.createdByRun ? [eq(issueThreadInteractions.sourceRunId, input.runId)] : []),
activeIssueInteractionCondition({ runId: input.runId, conversationMode: isConversation(issue) }),
),
)
@@ -89,9 +92,10 @@ export async function pendingNativeGovernance(input: {
/**
* A committed conversation turn has answered the person, whatever status its
* task keeps. An approval or an execution stage can hold the task, but neither
* presents that answer in the conversation. Only a pending thread interaction
* (a question or confirmation card) is the turn's visible response.
* task keeps. An approval, an execution stage or a card the run did not create
* (an earlier turn's, or one the board addressed to someone else) can hold the
* task, but none presents that answer in the conversation. Only a pending
* question or confirmation card this run created is the turn's visible response.
*/
export async function conversationTurnAnswered(input: {
db: Db;
@@ -101,5 +105,25 @@ export async function conversationTurnAnswered(input: {
}): Promise<boolean> {
// Check cards on their own: an execution stage outranks a card as the
// status gate, but it does not present the turn.
return !(await pendingNativeInteraction(input));
return !(await pendingNativeInteraction({ ...input, createdByRun: true }));
}
/**
* The one rule for whether a successful conversation turn's prose is its
* answer. Live run completion and background presentation recovery both use
* it, so a restart between the status commit and the live presentation
* presents the same reply the live path would have.
*/
export async function conversationTurnPresentsAnswer(input: {
db: Db;
companyId: string;
issueId: string;
runId: string;
finalizationPhase: unknown;
finalizationReasonCode: unknown;
}): Promise<boolean> {
if (input.finalizationReasonCode === "conversation_turn_finished") return true;
// A turn left waiting on an approval, an execution stage or a blocker still
// answered the person. Only its own question card presents it instead.
return input.finalizationPhase === "committed" && await conversationTurnAnswered(input);
}
server/src/services/native-runtime/native-inherited-runner-retry.ts
+24 -10
View File
@@ -89,6 +89,28 @@ export function readInheritedRunnerRetainedSandbox(profile: Record<string, unkno
*/
export type InheritedRunnerRetirement = "retired" | "not_applicable" | "transient";
type RunnerIdentityRun = Pick<typeof heartbeatRuns.$inferSelect,
"id" | "companyId" | "agentId" | "nativeSessionId" | "createdAt" | "processPid" | "processGroupId" | "processStartedAt">;
/** The settled predecessor whose exact runner process identity this run carries. */
async function settledPredecessorWithRunnerIdentity(tx: Pick<Db, "select">, run: RunnerIdentityRun) {
if (!run.nativeSessionId || (run.processPid === null && run.processGroupId === null)) return null;
const [predecessor] = await tx.select({ id: heartbeatRuns.id, processStartedAt: heartbeatRuns.processStartedAt })
.from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, run.companyId), eq(heartbeatRuns.agentId, run.agentId),
eq(heartbeatRuns.nativeSessionId, run.nativeSessionId), ne(heartbeatRuns.id, run.id),
lt(heartbeatRuns.createdAt, run.createdAt), inArray(heartbeatRuns.status, TERMINAL_RUN_STATUSES),
run.processPid === null ? isNull(heartbeatRuns.processPid) : eq(heartbeatRuns.processPid, run.processPid),
run.processGroupId === null ? isNull(heartbeatRuns.processGroupId) : eq(heartbeatRuns.processGroupId, run.processGroupId),
)).orderBy(desc(heartbeatRuns.createdAt)).limit(1);
return predecessor && sameTime(predecessor.processStartedAt, run.processStartedAt) ? predecessor : null;
}
/** The run's recorded runner process identity is exactly a settled predecessor's, not its own. */
export async function carriesInheritedRunnerIdentity(tx: Pick<Db, "select">, run: RunnerIdentityRun): Promise<boolean> {
return (await settledPredecessorWithRunnerIdentity(tx, run)) !== null;
}
export async function retireInheritedRemoteRunnerIdentity(db: Db, runId: string): Promise<InheritedRunnerRetirement> {
return db.transaction(async transaction => {
const tx = transaction as unknown as Db;
@@ -127,16 +149,8 @@ export async function retireInheritedRemoteRunnerIdentity(db: Db, runId: string)
if ((await nativeSessionProviderEvidence(tx, [run.id])).has(run.id)) return "not_applicable";
if (await hasNativeRunnerCheckpoint(tx, { companyId: run.companyId, issueId: issue.id, runId: run.id })) return "not_applicable";
// The identity must be exactly the one a settled predecessor recorded.
const [predecessor] = await tx.select({ id: heartbeatRuns.id, processStartedAt: heartbeatRuns.processStartedAt })
.from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, run.companyId), eq(heartbeatRuns.agentId, run.agentId),
eq(heartbeatRuns.nativeSessionId, run.nativeSessionId), ne(heartbeatRuns.id, run.id),
lt(heartbeatRuns.createdAt, run.createdAt), inArray(heartbeatRuns.status, TERMINAL_RUN_STATUSES),
run.processPid === null ? isNull(heartbeatRuns.processPid) : eq(heartbeatRuns.processPid, run.processPid),
run.processGroupId === null ? isNull(heartbeatRuns.processGroupId) : eq(heartbeatRuns.processGroupId, run.processGroupId),
)).orderBy(desc(heartbeatRuns.createdAt)).limit(1);
if (!predecessor || !sameTime(predecessor.processStartedAt, run.processStartedAt)) return "not_applicable";
const predecessor = await settledPredecessorWithRunnerIdentity(tx, run);
if (!predecessor) return "not_applicable";
// Keep the run's own sandbox when its lease is still live.
const [lease] = await tx.select({ id: environmentLeases.id }).from(environmentLeases).where(and(
server/src/services/native-runtime/native-recovered-owner-followup.test.ts
+47 -7
View File
@@ -337,6 +337,16 @@ describe.skipIf(!recoveryPostgresSupport.supported)("follow-up after a recovered
expect(followUp.logs.join("")).not.toContain("starting a new runner for this conversation");
}, 60_000);
it("starts an unclaimed remote follow-up from the previous owner's external marker under a bootstrap claim", async () => {
const fixture = await staleProjection("codex", true);
expect(await fixture.followUp.start("created", { kind: "bootstrap_incomplete", runId: fixture.followUp.runId,
leaseOwner: "successor", controllerGeneration: 1, providerAttempt: 0, restartKind: "hard", recoveryRequestId: null } as never))
.toBe(ADMITTED);
expect(JSON.parse(await readFile(fixture.path, "utf8")).identity).toEqual(fixture.latest.identity);
await fixture.assertBackup();
expect(await quarantined()).toEqual([]);
}, 60_000);
it("rehydrates a same-source external marker before reading the suspended resumed sandbox", async () => {
const fixture = await staleProjection("codex", true), { latest, followUp, root, path } = fixture;
await writeFile(path, JSON.stringify({ schema: "paperclip.runner.external-control-plane-state.v1", identity: latest.identity }));
@@ -670,6 +680,27 @@ describe.skipIf(!recoveryPostgresSupport.supported)("follow-up after a recovered
expect(await archivedOwners(root)).toEqual([prior.runId]);
}, 60_000);
it.each([
{ label: "ended on another sandbox", status: "expired", sameSandbox: false, recorded: true, outcome: ADMITTED },
{ label: "released on another sandbox", status: "released", sameSandbox: false, recorded: true, outcome: ADMITTED },
{ label: "still retained", status: "retained", sameSandbox: false, recorded: true,
outcome: stranded("journal_commands_unsettled(runner.suspend.pending)") },
{ label: "ended on this sandbox", status: "expired", sameSandbox: true, recorded: true,
outcome: stranded("journal_commands_unsettled(runner.suspend.pending)") },
{ label: "not recorded", status: "expired", sameSandbox: false, recorded: false,
outcome: stranded("journal_commands_unsettled(runner.suspend.pending)") },
])("retires a finished remote owner whose suspend never finished when its sandbox lease is $label", async ({ status, sameSandbox, recorded, outcome }) => {
const prior = await seedRecoveredPriorRun("pending-command");
const followUp = await admitFollowUp(prior);
if (recorded) {
await db.insert(environments).values({ id: followUp.environmentId, name: `ended-${followUp.environmentId}`, driver: "sandbox" });
await db.insert(environmentLeases).values({ companyId, environmentId: followUp.environmentId, issueId: prior.issueId,
heartbeatRunId: prior.runId, provider: "daytona", providerLeaseId: sameSandbox ? followUp.providerLeaseId : `sandbox-${prior.runId}`,
status, releasedAt: status === "retained" ? null : new Date(), metadata: { remoteCwd: "/workspace" } });
}
expect(await followUp.start()).toBe(outcome);
}, 60_000);
it("admits again when a retry rebuilds the same projection from the database", async () => {
const prior = await seedRecoveredPriorRun();
const followUp = await admitFollowUp(prior);
@@ -873,10 +904,13 @@ describe.skipIf(!recoveryPostgresSupport.supported)("follow-up after a recovered
}
}, 60_000);
it.each([
{ label: "never delivered to a dead runner", connectionCount: 0, claim: true, admitted: true },
{ label: "possibly delivered", connectionCount: 1, claim: true, admitted: false },
{ label: "without a dead-runner claim", connectionCount: 0, claim: false, admitted: false },
])("resumes from the previous turn when a warm attach rotation was $label", async ({ connectionCount, claim, admitted }) => {
{ label: "never delivered to a dead runner", connectionCount: 0, claim: true, admitted: true, archiveOnly: false },
{ label: "possibly delivered", connectionCount: 1, claim: true, admitted: false, archiveOnly: false },
{ label: "without a dead-runner claim", connectionCount: 0, claim: false, admitted: false, archiveOnly: false },
{ label: "cut off before its new control plane was written", connectionCount: 0, claim: true, admitted: true, archiveOnly: true },
{ label: "cut off before its new control plane, without a dead-runner claim", connectionCount: 0, claim: false, admitted: false,
archiveOnly: true },
])("resumes from the previous turn when a warm attach rotation was $label", async ({ connectionCount, claim, admitted, archiveOnly }) => {
const prior = await seedRecoveredPriorRun();
const followUp = await admitFollowUp(prior);
const root = await restoreHeldRoot(prior, followUp);
@@ -890,22 +924,28 @@ describe.skipIf(!recoveryPostgresSupport.supported)("follow-up after a recovered
await mkdir(join(root, "runner"), { recursive: true, mode: 0o700 });
await writeFile(join(root, "runner", "codex-provider-state.json"), "{}");
const identity = { ...priorIdentity, runId: followUp.runId, turnId: `turn-${followUp.runId}`, itemId: `item-${followUp.runId}` };
await mkdir(join(root, "control-plane"), { mode: 0o700 });
if (!archiveOnly) await mkdir(join(root, "control-plane"), { mode: 0o700 });
const undelivered = JSON.stringify({ schema: "paperclip.runner.durable.control-plane-state.v2", identity,
bootstrapHistory: { schema: "paperclip.runner.bootstrap-history.v1", originIdentity: identity, credentialsIssued: 0, adopted: false },
runAttachTemplate: {}, tickets: {}, leases: {},
commands: [{ commandId: "attach", controllerSeq: 1, type: "run.attach", status: "pending", payload: {} }],
committedEvents: [], ackedSourceSeq: 0, connectionCount, commandDeliveryCounts: {} });
await writeFile(join(root, "control-plane", "control-plane-state.json"), undelivered);
if (!archiveOnly) await writeFile(join(root, "control-plane", "control-plane-state.json"), undelivered);
const outcome = await followUp.startLocal(claim ? { kind: "resume_dead_runner", runId: followUp.runId, leaseOwner: "successor",
controllerGeneration: 2, providerAttempt: 1, restartKind: "hard", recoveryRequestId: null } as never : undefined);
if (!admitted) {
expect(outcome).toBe("runner_state_identity_mismatch: authority_indeterminate");
expect(outcome).toBe(archiveOnly ? "runner_state_identity_mismatch: durable_identity_unreadable"
: "runner_state_identity_mismatch: authority_indeterminate");
return;
}
expect(outcome).toBe(ADMITTED);
expect(JSON.parse(await readFile(join(root, "control-plane", "control-plane-state.json"), "utf8")).identity).toEqual(priorIdentity);
expect(existsSync(join(root, "authority-epochs"))).toBe(false);
if (archiveOnly) {
expect(existsSync(join(root, "continuity-breaks"))).toBe(false);
expect(await quarantined()).toEqual([]);
return;
}
const [archive] = await readdir(join(root, "continuity-breaks"));
expect(await readFile(join(root, "continuity-breaks", archive!, "control-plane", "control-plane-state.json"), "utf8")).toBe(undelivered);
expect(await quarantined()).toEqual([]);
server/src/services/native-runtime/native-response-presentation-query.ts
+8 -3
View File
@@ -1,5 +1,5 @@
import { QueryBuilder } from "drizzle-orm/pg-core";
import { and, asc, eq, exists, gt, inArray, isNotNull, isNull, notExists, or, sql } from "drizzle-orm";
import { and, asc, eq, exists, gt, inArray, isNotNull, isNull, ne, notExists, or, sql } from "drizzle-orm";
import { chatActions, heartbeatRunEvents, heartbeatRuns, issues, nativeRunFinalizations, nativeRunResults, statusDecisions, statusDecisionEffects } from "@paperclipai/db";
import type { NativeResponsePresentationScope } from "../native-response-presentation-notifications.js";
@@ -42,9 +42,11 @@ export function nativeResponsePresentationCandidates(scope?: NativeResponsePrese
or(
and(eq(heartbeatRuns.status, "succeeded"),
or(sql`(coalesce(${issues.conversationAgentId}::text, '') = '' or coalesce(${issues.conversationUserId}, '') = '')`,
and(eq(statusDecisions.reasonCode, "conversation_turn_finished"),
// An unsettled conversation turn. A held turn keeps the status its
// gate committed; recovery applies the live path's answered rule.
and(or(ne(statusDecisions.reasonCode, "conversation_turn_finished"), eq(issues.status, "in_progress")),
eq(issues.conversationAgentId, heartbeatRuns.agentId), eq(issues.conversationUserId, heartbeatRuns.responsibleUserId),
eq(issues.status, "in_progress"), eq(issues.conversationState, "active"),
eq(issues.conversationState, "active"),
or(isNull(issues.executionRunId), eq(issues.executionRunId, heartbeatRuns.id)),
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issues.id}::text`,
sql`jsonb_typeof(${heartbeatRuns.contextSnapshot}->'conversationSessionGeneration') = 'number'`,
@@ -65,6 +67,9 @@ export function nativeResponsePresentationCandidates(scope?: NativeResponsePrese
and(sql`(coalesce(${issues.conversationAgentId}::text, '') = '' or coalesce(${issues.conversationUserId}, '') = '')`,
sql`${nativeRunResults.resultJson} #>> '{result,reportedWorkDisposition}' = 'needs_review'`,
sql`${nativeRunResults.resultJson} #>> '{terminal,reportedWorkDisposition}' = 'needs_review'`),
and(sql`(coalesce(${issues.conversationAgentId}::text, '') <> '' and coalesce(${issues.conversationUserId}, '') <> '')`,
sql`${nativeRunResults.resultJson} #>> '{result,reportedWorkDisposition}' = 'yielded'`,
sql`${nativeRunResults.resultJson} #>> '{terminal,reportedWorkDisposition}' = 'yielded'`),
)),
and(sql`${nativeRunResults.resultJson} #>> '{result,reportedWorkDisposition}' = 'yielded'`,
sql`${nativeRunResults.resultJson} #>> '{terminal,reportedWorkDisposition}' = 'yielded'`,
server/src/services/native-runtime/native-restart-recovery.ts
+15 -10
View File
@@ -1,3 +1,4 @@
import { carriesInheritedRunnerIdentity } from "./native-inherited-runner-retry.js";
import { readPendingNativeRunStop } from "./native-cancellation-request.js";
import { CONTROLLER_LEASE_RENEW_MS } from "../controller-instance-lease.js";
import { currentControllerProcessStamp, UNKNOWN_PROCESS_SPACE } from "../controller-process-identity.js";
@@ -1324,6 +1325,15 @@ export async function claimNativeRestartRecoveries(input: {
const hasProviderEvidence =
hasCheckpointProviderIdentity || providerEvents.length > 0;
const admissionCheckpointOnly = !hasCheckpoint || (checkpointIdentityMatches && checkpointRecord.cursor == null
&& (!Array.isArray(checkpointRecord.terminalTurns) || checkpointRecord.terminalTurns.length === 0)
&& checkpointRecord.activeTurnId == null && checkpointRecord.semanticResult == null && checkpointRecord.terminal == null
&& (checkpointRecord.pendingRuntimeRequests == null
|| (Array.isArray(checkpointRecord.pendingRuntimeRequests) && checkpointRecord.pendingRuntimeRequests.length === 0)));
const inheritedCheckpointOnly = hasCheckpoint && admissionCheckpointOnly && remoteSandbox && !pluginTask
&& !durableAuthority && stagedAttachment === null && providerEvents.length === 0;
const inheritedRunnerIdentity = inheritedCheckpointOnly && Boolean(row.run.processPid || row.run.processGroupId)
&& await carriesInheritedRunnerIdentity(tx, row.run);
let unclaimedBootstrap =
remoteWorkspace?.state === "prepared" &&
row.run.runnerProfileJson?.nativePreparationFencingVersion === NATIVE_PREPARATION_FENCING_VERSION &&
@@ -1338,14 +1348,12 @@ export async function claimNativeRestartRecoveries(input: {
// suspends. Without an owner, an elapsed expiry grants no authority.
(!row.coordinator.leaseExpiresAt || row.coordinator.leaseExpiresAt <= now) &&
!row.coordinator.failureDetail?.successorRunId &&
!row.run.processPid &&
!row.run.processGroupId &&
!row.run.processStartedAt &&
(inheritedRunnerIdentity || (!row.run.processPid && !row.run.processGroupId && !row.run.processStartedAt)) &&
!row.run.sessionIdBefore &&
!row.run.sessionIdAfter &&
Boolean(row.run.nativeSessionId && row.run.runnerInstanceId) &&
!hasCheckpoint &&
!hasProviderEvidence;
(!hasCheckpoint || inheritedCheckpointOnly) &&
(!hasProviderEvidence || inheritedCheckpointOnly);
if (unclaimedBootstrap) {
if (activeNativeExecutionRunIds.has(row.run.id) &&
controller.bootId === (await currentNativeControllerIdentity()).bootId) {
@@ -1430,9 +1438,6 @@ export async function claimNativeRestartRecoveries(input: {
const remoteBootstrapProvenIncomplete = remoteBootstrapProof !== null
&& (!hasProviderEvidence || remoteBootstrapProof === "session_opened_without_turn");
const admissionCheckpointOnly = !hasCheckpoint || (checkpointIdentityMatches && checkpointRecord.cursor == null
&& (!Array.isArray(checkpointRecord.terminalTurns) || checkpointRecord.terminalTurns.length === 0)
&& checkpointRecord.activeTurnId == null && checkpointRecord.semanticResult == null && checkpointRecord.terminal == null);
const remoteRunnerNeverReceivedCommands = !unclaimedBootstrap && !remoteRunnerNeverLaunched
&& !remoteRunnerExitedBeforeProviderWork && !remoteBootstrapProvenIncomplete
&& remoteSandbox && remoteWorkspace !== null && !pluginTask && !durableAuthority && stagedAttachment === null
@@ -1468,12 +1473,12 @@ export async function claimNativeRestartRecoveries(input: {
knownProviderProcessAlive: providerProcesses.livePids.length > 0,
knownProviderProcessIdentityAmbiguous:
providerProcesses.ambiguousLivePids.length > 0,
hasCheckpoint: hasCheckpoint && !remoteRunnerNeverReceivedCommands,
hasCheckpoint: hasCheckpoint && !remoteRunnerNeverReceivedCommands && !(unclaimedBootstrap && inheritedCheckpointOnly),
stagedAttachmentVerified: stagedAttachment !== null,
checkpointIdentityMatches:
checkpointIdentityMatches &&
(!remoteSandbox || remoteWorkspace !== null),
hasProviderEvidence,
hasProviderEvidence: hasProviderEvidence && !(unclaimedBootstrap && inheritedCheckpointOnly),
checkpointFailed:
(checkpointRecord.terminal as Record<string, unknown> | undefined)
?.runTerminalState === "failed" &&
server/src/services/native-runtime/native-run-finalizer.ts
+29 -5
View File
@@ -15,7 +15,7 @@ import { notifyCancelledWorkspacePreservation } from "../cancelled-workspace-pre
import { notifyInstructionCollection } from "../instruction-collection-notifications.js";
import { readRequiredQualityReview } from "../required-quality-review.js";
import { NativeDeliverableEvidenceError, validateNativeDeliverableEvidence } from "./native-deliverable-feedback.js";
import { pendingNativeGovernance } from "./native-governance.js";
import { conversationTurnPresentsAnswer, pendingNativeGovernance } from "./native-governance.js";
import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js";
import { onboardingFirstTaskCompletionAllowed } from "../onboarding-first-task-admission.js";
import { notifyExecutionRunWork } from "../execution-control-notifications.js";
@@ -830,6 +830,7 @@ async function recoverCommittedNativeChatResponseInScope(
let agentId: string | null = null;
let materialized = false;
let conversationReply = false;
let settledWithoutReply = false;
const entered = beginRunPhase("response.transaction.entry");
try {
materialized = await measureRunPhase("response.transaction", () => db.transaction(async (transaction) => {
@@ -996,9 +997,13 @@ async function recoverCommittedNativeChatResponseInScope(
return true;
}
const conversation = isConversation(issue);
const currentConversation = conversation && decision.reasonCode === "conversation_turn_finished"
// The turn is still the conversation's unsettled current turn. A held
// turn keeps whatever status its gate committed; a finished turn
// preserves in_progress.
const currentConversation = conversation
&& issue.conversationAgentId === run.agentId && issue.conversationUserId === run.responsibleUserId
&& issue.status === "in_progress" && issue.conversationState === "active"
&& (decision.reasonCode !== "conversation_turn_finished" || issue.status === "in_progress")
&& issue.conversationState === "active"
&& (!issue.executionRunId || issue.executionRunId === run.id)
&& run.contextSnapshot?.issueId === issue.id
&& typeof run.contextSnapshot.conversationSessionGeneration === "number"
@@ -1014,6 +1019,8 @@ async function recoverCommittedNativeChatResponseInScope(
terminal.turnTerminalState === "completed" &&
((result.reportedWorkDisposition === "done" && terminal.reportedWorkDisposition === "done") ||
(!conversation && result.reportedWorkDisposition === "needs_review" && terminal.reportedWorkDisposition === "needs_review") ||
// A conversation turn can yield while an approval, an execution
// stage or a blocker holds its task; it still answered the person.
(currentConversation && result.reportedWorkDisposition === "yielded" && terminal.reportedWorkDisposition === "yielded"
&& !await hasChatRunOwnedProviderInteraction(tx, input))) &&
(await resolveChatRunPresentationAuthorizationReason(tx, input)) === "internal_agent_write"
@@ -1054,7 +1061,10 @@ async function recoverCommittedNativeChatResponseInScope(
const resolved = resolveHeartbeatRunResponse({
resultJson: { ...record(run.resultJson), nativeResult: result },
nativeConversationFinalResponse: conversation,
conversationTurnFinished: currentConversation,
// The same answered rule as live run completion.
conversationTurnFinished: currentConversation && await conversationTurnPresentsAnswer({
db: tx, ...input, finalizationPhase: coordinator.phase, finalizationReasonCode: decision.reasonCode,
}),
interactionOwnsResponse: conversation && await hasChatRunOwnedProviderInteraction(tx, input),
existingComment: findHeartbeatRunCompletionComment(comments, run.resultJson),
finalAgentMessage,
@@ -1070,6 +1080,19 @@ async function recoverCommittedNativeChatResponseInScope(
}).where(eq(heartbeatRuns.id, run.id));
return false;
}
if (conversation && resolved.decision.commentAction === "none") {
// The turn's own question or confirmation card presents it, as on
// the live path. Record that decision once so the turn settles and
// leaves the pending presentation set.
await tx.update(heartbeatRuns).set({
resultJson: sql`coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) || ${JSON.stringify({
presentationDecision: { ...resolved.decision,
reasonCodes: [...resolved.decision.reasonCodes, "committed_conversation_interaction_presented"] },
})}::jsonb`, updatedAt: new Date(),
}).where(eq(heartbeatRuns.id, run.id));
settledWithoutReply = true;
return false;
}
if (!resolved.text || resolved.decision.commentAction !== "create") return false;
await notifyChatRunMilestones(tx, input);
const comment = await issueService(db).addComment(input.issueId, resolved.text,
@@ -1159,6 +1182,7 @@ async function recoverCommittedNativeChatResponseInScope(
} finally {
entered();
}
if (settledWithoutReply) await settleConversationTurnForRun(db, input.companyId, input.runId);
if (materialized && agentId) {
if (conversationReply) await settleConversationTurnForRun(db, input.companyId, input.runId);
publishChatPublicationCommitSignal({
@@ -1585,7 +1609,7 @@ export async function finalizeNativeRun(input: {
// A reassigned-away run keeps its accepted result, but cannot close the
// new owner's task. The status committer rechecks this under the issue lock.
if (!isConversation(authoritativeIssue)) {
proposedDecision = supersedeStaleOwnerCompletion(proposedDecision, authoritativeIssue, run.agentId);
proposedDecision = supersedeStaleOwnerCompletion(proposedDecision, authoritativeIssue, run.agentId, run.id);
}
if (proposedDecision.statusAction === "done") {
const continuation = record(run.contextSnapshot?.executionContinuation);
server/src/services/native-runtime/native-run-trace.ts
+50 -5
View File
@@ -262,11 +262,16 @@ function safeAttributes(
* seams inherit that context. The persisted run-event copy retains the same
* semantic parent name for developer installs without an OTLP collector.
*/
/** The runner span for the provider loading its saved session on resume. */
const PROVIDER_SESSION_LOAD_SPAN = "provider.session.load";
export function createNativeRunTrace(input: {
runId: string;
startedAtMs?: number;
onEvent?: NativeRunTraceSink;
traceContext?: StartupTraceContextHandle;
/** Called once when the trace finishes. */
onFinish?: () => void;
/** Speed-budget subject (invariant 8). Absent or remote means no check. */
budget?: {
adapter: SpeedBudgetAdapter | null;
@@ -279,6 +284,8 @@ export function createNativeRunTrace(input: {
const tracing =
input.traceContext ?? getStartupTraceContext("paperclip.native-runner");
const traceStartedAtMs = finiteMs(input.startedAtMs ?? Date.now());
/** Start times of the provider's saved-session loads (restart recovery split). */
const providerLoadStartsMs: number[] = [];
const runIdHash = hashedId(input.runId);
let rootSpan: SpanHandle;
try {
@@ -429,6 +436,9 @@ export function createNativeRunTrace(input: {
durationMs,
outcome,
...attributes,
// The span's own start time. The row can be written late (deferred
// diagnostics), so readers must not infer the start from its write time.
wallStartedAtMs: startedAtMs,
},
});
} catch {
@@ -447,6 +457,11 @@ export function createNativeRunTrace(input: {
} = {},
): NativeRunSpanScope => {
const startedAtMs = finiteMs(options.startedAtMs ?? Date.now());
if (name === PROVIDER_SESSION_LOAD_SPAN) {
providerLoadStartsMs.push(startedAtMs);
// The measurement can arrive before its provider span is recorded; judge now.
if (pendingRecovery) judgeRestartRecovery();
}
const attributes = safeAttributes(options.attributes);
const parent = parentFor(options.parentName);
let span: SpanHandle;
@@ -635,8 +650,13 @@ export function createNativeRunTrace(input: {
const finish = async (outcome: NativeRunSpanOutcome): Promise<void> => {
if (ended) return;
try { input.onFinish?.(); } catch { /* Diagnostics cannot change runner control flow. */ }
// Settled diagnostics can defer more (a budget breach after a deferred
// measurement), so drain until none remain.
// measurement), so drain until none remain. Judge restart recovery after
// the first drain, so a deferred provider span is counted, then drain its
// breach report.
while (pendingDiagnostics.size > 0) await Promise.all([...pendingDiagnostics]);
judgeRestartRecovery();
while (pendingDiagnostics.size > 0) await Promise.all([...pendingDiagnostics]);
if (ended) return;
ended = true;
@@ -666,8 +686,28 @@ export function createNativeRunTrace(input: {
};
/** Restart recovery latency: control plane ready to the recovered run's
* next provider event, checked once per run. */
* next provider event, checked once per run. Only Paperclip's share is
* judged: ready to the start of the provider's session load (or to the
* event when nothing reloads). It is judged as soon as both are known: at
* the check when the load span is already recorded or nothing reloads (a
* reattached runner, whose dispositions launch nothing), when the load span is recorded, and otherwise when
* the run finishes. A run that detaches for another restart therefore keeps
* its warning unless the detach lands between the event and a deferred span. */
let recoveryChecked = false;
let pendingRecovery: RestartRecoveryMeasurement | null = null;
const judgeRestartRecovery = (): void => {
const measurement = pendingRecovery;
pendingRecovery = null;
if (!input.budget || !measurement || measurement.readyAtMs === null) return;
const readyAtMs = measurement.readyAtMs;
const loads = providerLoadStartsMs.filter(at => at >= readyAtMs && at <= measurement.committedAtMs);
const loadStartedAtMs = loads.length ? Math.min(...loads) : null;
const totalMs = measurement.committedAtMs - readyAtMs;
reportBudgetBreach(evaluateSpeedBudget({ adapter: input.budget.adapter, environment: input.budget.environment,
metric: "restart_recovery_control_plane", start: null, measuredMs: (loadStartedAtMs ?? measurement.committedAtMs) - readyAtMs,
disposition: measurement.disposition, restartTotalMs: totalMs,
providerLoadMs: loadStartedAtMs === null ? 0 : measurement.committedAtMs - loadStartedAtMs }));
};
const checkRestartRecovery = (measurement: RestartRecoveryMeasurement): void => {
if (!input.budget || recoveryChecked || measurement.readyAtMs === null) return;
// An event emitted before ready (buffered by a surviving runner and
@@ -676,15 +716,20 @@ export function createNativeRunTrace(input: {
if (measurement.emittedAtMs !== null && measurement.emittedAtMs < measurement.readyAtMs) return;
if (measurement.committedAtMs < measurement.readyAtMs) return;
recoveryChecked = true;
reportBudgetBreach(evaluateSpeedBudget({ adapter: input.budget.adapter, environment: input.budget.environment,
metric: "restart_recovery", start: null, measuredMs: measurement.committedAtMs - measurement.readyAtMs,
disposition: measurement.disposition }));
pendingRecovery = measurement;
const readyAtMs = measurement.readyAtMs;
// Only a launched runner reloads a provider session (native-restart-recovery).
if (!["resume_dead_runner", "bootstrap_incomplete"].includes(measurement.disposition)
|| providerLoadStartsMs.some(at => at >= readyAtMs && at <= measurement.committedAtMs)) judgeRestartRecovery();
};
return {
traceStartedAtMs,
setBudgetStart,
checkRestartRecovery,
/** Judge a pending restart-recovery check now (the controller is detaching
* this run for another restart and will not finish the trace). */
flushRestartRecovery: () => judgeRestartRecovery(),
start,
annotate,
activate,
server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
+45
View File
@@ -2757,6 +2757,51 @@ process.on("SIGTERM", () => child.kill("SIGTERM"));
expect(coordinator.recoveryHistory.at(-1)).toMatchObject({ reason: "remote_runner_never_received_commands" });
});
it.each([
{ label: "restarts", providerEvent: false, process: "none" },
{ label: "restarts with its predecessor's runner identity", providerEvent: false, process: "inherited" },
{ label: "keeps its own runner identity held instead of restarting", providerEvent: false, process: "own" },
{ label: "keeps a projected provider event held instead of restarting", providerEvent: true, process: "none" },
{ label: "keeps a pending runtime request held instead of restarting", providerEvent: false, process: "none", pendingRequest: true },
])("$label an unclaimed remote follow-up that carries only its inherited checkpoint", async ({ providerEvent, process, pendingRequest }) => {
const fixture = await seedUnlaunchedRemoteClaim(`UNCLAIMED-INHERITED-${providerEvent}-${process}-${Boolean(pendingRequest)}`);
if (process !== "none") {
const processStartedAt = new Date(Date.now() - 120_000);
await fixture.db.update(heartbeatRuns).set({ processPid: 596, processGroupId: null, processStartedAt, processHostHash: "remote" })
.where(eq(heartbeatRuns.id, fixture.runId));
if (process === "inherited") {
const [current] = await fixture.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId));
await fixture.db.insert(heartbeatRuns).values({ companyId, agentId, status: "succeeded", runtimeMode: "native",
nativeIssueId: fixture.issueId, nativeSessionId: current.nativeSessionId, runnerInstanceId: current.runnerInstanceId,
createdAt: new Date(current.createdAt.getTime() - 60_000), finishedAt: new Date(current.createdAt.getTime() - 5_000),
processPid: 596, processGroupId: null, processStartedAt, processHostHash: "remote",
contextSnapshot: { issueId: fixture.issueId } });
}
}
await fixture.db.update(nativeRunFinalizations).set({ attempt: 0, controllerGeneration: 0, controllerBootId: null,
controllerPid: null, controllerProcessStartedAt: null, leaseOwner: null, leaseExpiresAt: null })
.where(eq(nativeRunFinalizations.runId, fixture.runId));
const [seeded] = await fixture.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId));
await fixture.db.update(heartbeatRuns).set({ runnerProfileJson: { ...seeded.runnerProfileJson, sessionCheckpoint: {
driverKind: "codex_app_server", backendKind: "runner", sessionId: "inherited-thread", cursor: null,
identity: { companyId, issueId: fixture.issueId, runId: fixture.runId, agentId, sessionId: fixture.native.normalizedSessionId },
providerSessionId: "inherited-thread", activeTurnId: null, terminal: null, terminalTurns: [], semanticResult: null,
pendingRuntimeRequests: pendingRequest ? [{ requestId: "input-1", kind: "user_input" }] : [], lineage: [],
} } }).where(eq(heartbeatRuns.id, fixture.runId));
if (providerEvent) await fixture.db.insert(heartbeatRunEvents).values({ companyId, runId: fixture.runId, agentId, seq: 901,
stream: "system", level: "info", eventType: "session.started" });
const [claim] = await claimNativeRestartRecoveries({ db: fixture.db, controller: successor,
restartKind: "hard", runIds: [fixture.runId] });
if (providerEvent || process === "own" || pendingRequest) {
expect(claim).not.toMatchObject({ kind: "bootstrap_incomplete" });
return;
}
expect(claim).toMatchObject({ kind: "bootstrap_incomplete", runId: fixture.runId });
const [coordinator] = await fixture.db.select().from(nativeRunFinalizations)
.where(eq(nativeRunFinalizations.runId, fixture.runId));
expect(coordinator.recoveryHistory.at(-1)).toMatchObject({ reason: "remote_execution_never_claimed" });
});
it("restarts a remote bootstrap retry that shutdown cut before any runner was requested", async () => {
const fixture = await seedUnlaunchedRemoteClaim("BOOTSTRAP-RETRY", { bootstrapRetry: true });
expect(await claimNativeRestartRecoveries({ db: fixture.db, controller: successor,
server/src/services/native-runtime/native-session-executor.test.ts
+95
View File
@@ -13556,6 +13556,101 @@ describe("native session bounded recovery", () => {
expect(state.upsertRecoveryAction).not.toHaveBeenCalled();
});
// H1 (restart chaos): a budget pause's lifecycle Stop interrupted a Codex
// turn, which ended without a semantic result. That Stop must stay a
// cancellation (resume continues it), never an interrupted-run retry the
// pause then strands.
it.each(["Agent lifecycle stop requested", "Cancelled due to budget pause"])("does not turn an acknowledged governance pause Stop (%s) into a retry", async (reason) => {
const updates: Array<{ table: unknown; values: Record<string, unknown> }> = [];
const stop: Record<string, unknown> = {};
state.execute.mockReset().mockImplementationOnce(async () => {
Object.assign(stop, {
cancellation: { reason, source: "control_plane", expected: true, initiator: { type: "system" },
recordedAt: new Date().toISOString() },
...(reason === "Agent lifecycle stop requested" ? { pauseStop: { holds: ["company_paused"], companyPauseReason: "budget" } } : {}),
nativeCancellation: {
schema: "paperclip.native-cancellation.v1", ...execution.binding, scope: "run", reasonCode: "cancellation_run_only",
dispatched: true, dispatchState: "acknowledged", intentAuditId: "intent", acknowledgementAuditId: "ack",
} });
throw new Error("native_finalization_missing: session returned no semantic result");
});
state.upsertRecoveryAction.mockClear();
await expect(executePaperclipNativeSession({
db: leaseDb(execution, {}, stop, updates), execution, runnerInstanceId: "governance-stop",
})).rejects.toThrow("native_cancellation_pending_recovery");
expect(updates.some(update => update.table === heartbeatRuns && update.values.status === "failed")).toBe(false);
expect(updates.some(update => update.table === nativeRunFinalizations && update.values.failureCode === "native_retry_cancelled")).toBe(true);
expect(state.upsertRecoveryAction).not.toHaveBeenCalled();
});
it("waits for an in-flight governance pause Stop to save its acknowledgement before judging a missing result", async () => {
let releaseCleanup!: () => void, endTurn!: () => void, opened!: () => void;
const turnEnded = new Promise<void>(resolve => { endTurn = resolve; });
const sessionOpen = new Promise<void>(resolve => { opened = resolve; });
state.cancel.mockReset().mockReturnValue({ cleanup: new Promise<void>(resolve => { releaseCleanup = resolve; }) });
state.execute.mockReset().mockImplementationOnce(async (options) => {
await options.onSession({ cancel: state.cancel });
opened();
await turnEnded;
throw new Error("native_finalization_missing: session returned no semantic result");
});
state.upsertRecoveryAction.mockClear();
const runResultJson: Record<string, unknown> = {};
const updates: Array<{ table: unknown; values: Record<string, unknown> }> = [];
const outcome = executePaperclipNativeSession({ db: leaseDb(execution, {}, runResultJson, updates), execution, runnerInstanceId: "pause-stop-in-flight" })
.catch(error => error);
// The pause's fence recorded its cancellation before dispatch.
const persistence = cancellationDb({ resultJson: {
cancellation: { reason: "Agent lifecycle stop requested", source: "control_plane", expected: true,
initiator: { type: "system" }, recordedAt: new Date().toISOString() },
pauseStop: { holds: ["company_paused"], companyPauseReason: "budget" } } });
const transaction = persistence.db.transaction.bind(persistence.db);
// The executor reads the same durable run row the Stop writes.
(persistence.db as unknown as { transaction: typeof transaction }).transaction = (async (operation: Parameters<typeof transaction>[0]) => {
const result = await transaction(operation);
Object.assign(runResultJson, persistence.getResultJson());
return result;
}) as typeof transaction;
await sessionOpen;
const stopping = cancelNativeSession(execution.binding.runId, "Agent lifecycle stop requested", { db: persistence.db, scope: "run" });
try {
await vi.waitFor(() => expect(state.cancel).toHaveBeenCalledOnce());
// The interrupted turn ends before the Stop saves its acknowledgement.
endTurn();
await new Promise(resolve => setTimeout(resolve, 20));
expect(updates.some(update => update.table === heartbeatRuns && update.values.status === "failed")).toBe(false);
releaseCleanup();
await stopping;
expect(await outcome).toMatchObject({ name: "NativeCancellationPendingRecoveryError" });
expect(updates.some(update => update.table === heartbeatRuns && update.values.status === "failed")).toBe(false);
expect(updates.some(update => update.table === nativeRunFinalizations && update.values.failureCode === "native_retry_cancelled")).toBe(true);
expect(state.upsertRecoveryAction).not.toHaveBeenCalled();
} finally { releaseCleanup(); endTurn(); await outcome; }
});
it("still retries a missing result after a system Stop that is not a governance pause", async () => {
const updates: Array<{ table: unknown; values: Record<string, unknown> }> = [];
const stop: Record<string, unknown> = {};
state.execute.mockReset().mockImplementationOnce(async () => {
Object.assign(stop, {
cancellation: { reason: "Agent lifecycle stop requested", source: "control_plane", expected: true,
initiator: { type: "system" }, recordedAt: new Date().toISOString() },
nativeCancellation: {
schema: "paperclip.native-cancellation.v1", ...execution.binding, scope: "run", reasonCode: "cancellation_run_only",
dispatched: true, dispatchState: "acknowledged", intentAuditId: "intent", acknowledgementAuditId: "ack",
} });
throw new Error("native_finalization_missing: session returned no semantic result");
});
state.upsertRecoveryAction.mockReset().mockResolvedValue({});
const service = vi.spyOn(issueServiceModule, "issueService").mockReturnValue({ update: vi.fn(async () => null) } as unknown as ReturnType<typeof issueServiceModule.issueService>);
try {
await expect(executePaperclipNativeSession({
db: leaseDb(execution, {}, stop, updates), execution, runnerInstanceId: "termination-stop",
})).rejects.not.toThrow("native_cancellation_pending_recovery");
expect(updates.some(update => update.table === nativeRunFinalizations && update.values.failureCode === "native_retry_cancelled")).toBe(false);
} finally { service.mockRestore(); }
});
it.each(["pending", "acknowledged"])("preserves a %s Stop when cancellation wins before the first turn", async (dispatchState) => {
const updates: Array<{ table: unknown; values: Record<string, unknown> }> = [];
const stop: Record<string, unknown> = {};
server/src/services/native-runtime/native-session-executor.ts
+148 -41
View File
@@ -1,4 +1,5 @@
import { remoteRecoveryLeaseMismatch } from "./remote-recovery-lease-mismatch.js";
import { isGovernancePauseStop } from "../governance-pause-stop.js";
import { nativeLeaseHolder, notifyNativeTokenLeaseClaimed } from "./native-token-lease-settlement.js";
import { captureRemoteNativeProviderArtifacts } from "./native-provider-artifact-capture.js";
import { NativeSessionDiagnostics, nativeSessionDiagnostics, type NativeDiagnosticLease } from "./native-session-diagnostics.js";
@@ -379,6 +380,9 @@ export class NativeControllerDetachedForRestartError extends Error {
const activeNativeSessions = new Map<string, ActiveNativeSession>();
// Shutdown can race provider startup before onSession publishes its handle.
const nativeRunsDetachingForRestart = new Map<string, symbol>();
/** Flushes a run's pending restart-recovery budget check when the controller
* detaches it for another restart (its trace never finishes). */
const nativeRunRecoveryFlushers = new Map<string, () => void>();
// A restart must not cut authority rotation between archiving the previous
// runner state and authenticating its replacement. Keep the startup owner
@@ -464,6 +468,7 @@ export async function detachNativeSessionsForRestart(
continue;
}
nativeRunsDetachingForRestart.set(runId, Symbol("native controller detach"));
try { nativeRunRecoveryFlushers.get(runId)?.(); } catch { /* Diagnostics cannot change detach. */ }
let active = activeNativeSessions.get(runId);
if (!active) {
// Only a startup registered before this fence may complete read-only
@@ -5424,7 +5429,7 @@ type SettledRunnerdProjection = { bytes: string; refused?: never } | { bytes?: n
const LOCAL_RUNNER_LIFECYCLE_COMMANDS = new Set(["runner.suspend", "runner.drain", "runner.shutdown"]);
function readSettledRunnerdProjection(root: string, identity: Record<string, string>,
options: { localRunner?: boolean; controlBytes?: string } = {}): SettledRunnerdProjection {
options: { localRunner?: boolean; controlBytes?: string; remoteRunnerGone?: boolean } = {}): SettledRunnerdProjection {
const refuse = (refused: string): SettledRunnerdProjection => ({ refused });
try {
if (!isSafeNativeStateDirectory(root)) return refuse("root_unsafe");
@@ -5444,7 +5449,7 @@ function readSettledRunnerdProjection(root: string, identity: Record<string, str
const control = record(JSON.parse(bytes));
const commands = (Array.isArray(control.commands) ? control.commands : []).map(record);
// A stopped local Runner can no longer act on its own lifecycle commands.
const settledCommands = options.localRunner && lifecycle === "not_suspended" && Array.isArray(control.commands)
const settledCommands = ((options.localRunner && lifecycle === "not_suspended") || options.remoteRunnerGone) && Array.isArray(control.commands)
? commands.filter(command => !LOCAL_RUNNER_LIFECYCLE_COMMANDS.has(String(command.type)) || command.status === "completed")
: commands;
const settledControl = settledCommands === commands ? control : { ...control, commands: settledCommands };
@@ -5548,11 +5553,16 @@ async function inspectFinishedPerTurnRunnerdOwner(input: {
let originalBytes: string;
try { originalBytes = readBoundedNativeFile(statePath, NATIVE_CONTROL_PLANE_STATE_MAX_BYTES, "runner_durable_identity_too_large").toString("utf8"); }
catch { return refuse("journal_unreadable"); }
const projectionOptions = { localRunner, controlBytes: input.projectedControlBytes };
const { binding } = input.execution;
const remoteRunnerGone = !localRunner && await priorRemoteSandboxLeasesEnded(input.db, {
companyId: binding.companyId, runId: identity.runId,
currentProviderLeaseId: target?.kind === "remote" && target.transport === "sandbox"
? target.sandboxLeaseAcquisition?.providerLeaseId : undefined,
});
const projectionOptions = { localRunner, controlBytes: input.projectedControlBytes, remoteRunnerGone };
const projection = readSettledRunnerdProjection(input.root, identity, projectionOptions);
if (projection.refused !== undefined) return refuse(projection.refused);
const controlBytes = projection.bytes;
const { binding } = input.execution;
let prior: { run: typeof heartbeatRuns.$inferSelect; owner: typeof nativeRunFinalizations.$inferSelect } | undefined;
try {
[prior] = await input.db.select({ run: heartbeatRuns, owner: nativeRunFinalizations }).from(heartbeatRuns)
@@ -5614,7 +5624,7 @@ async function inspectFinishedPerTurnRunnerdOwner(input: {
for (const name of readdirSync(quarantine)) {
if (!name.startsWith(`${basename(input.root)}.`)) continue;
if (quarantinedBootstrapOfOwnerHeldNoProviderWork(resolve(quarantine, name), identity, localRunner)) continue;
const copy = readSettledRunnerdProjection(resolve(quarantine, name), identity, { localRunner });
const copy = readSettledRunnerdProjection(resolve(quarantine, name), identity, { localRunner, remoteRunnerGone });
if (copy.refused !== undefined) return refuse(`quarantine_${copy.refused}`);
}
}
@@ -5638,6 +5648,20 @@ export function quarantinedBootstrapOfOwnerHeldNoProviderWork(root: string, iden
|| (!localRunner && runnerdStateProvesNoRemoteProviderTurn(root));
}
async function priorRemoteSandboxLeasesEnded(db: Db, input: {
companyId: string; runId: string; currentProviderLeaseId?: string;
}): Promise<boolean> {
try {
const leases = await db.select({ status: environmentLeases.status, releasedAt: environmentLeases.releasedAt,
providerLeaseId: environmentLeases.providerLeaseId }).from(environmentLeases)
.where(and(eq(environmentLeases.companyId, input.companyId), eq(environmentLeases.heartbeatRunId, input.runId)));
return leases.length > 0 && leases.every(lease => (lease.status === "released" || lease.status === "expired")
&& lease.releasedAt !== null && lease.providerLeaseId !== input.currentProviderLeaseId);
} catch {
return false;
}
}
async function retireFinishedPerTurnRunnerdOwner(input: Parameters<typeof inspectFinishedPerTurnRunnerdOwner>[0])
: Promise<{ retired: true } | { retired: false; refused: string }> {
const proof = await inspectFinishedPerTurnRunnerdOwner(input);
@@ -5656,8 +5680,34 @@ async function retireFinishedPerTurnRunnerdOwner(input: Parameters<typeof inspec
return { retired: true };
}
function undeliveredAttachRotationSource(root: string, execution: NativeExecutionInput): string | null {
function latestSuspendedArchivedEpoch(root: string, matches: (prior: Record<string, unknown>) => boolean): string | null {
const archives = resolve(root, "authority-epochs");
if (!isSafeNativeStateDirectory(archives)) return null;
const candidates = readdirSync(archives).map(name => resolve(archives, name)).filter(epoch => {
const state = resolve(epoch, "control-plane", "control-plane-state.json");
return isSafeNativeStateDirectory(epoch) && existsSync(state) && existsSync(resolve(epoch, "runner-state.json"));
}).sort((left, right) => lstatSync(resolve(right, "control-plane", "control-plane-state.json")).mtimeMs
- lstatSync(resolve(left, "control-plane", "control-plane-state.json")).mtimeMs);
const latest = candidates[0];
if (!latest) return null;
const prior = record(readRunnerdControlPlaneState(latest).identity);
const runner = record(JSON.parse(readBoundedNativeFile(resolve(latest, "runner-state.json"),
NATIVE_RUNNER_STATE_MAX_BYTES, "runner_state_too_large").toString("utf8")));
const digest = createHash("sha256").update(JSON.stringify(prior)).digest("hex").slice(0, 24);
if (basename(latest) !== `epoch-${digest}` || !matches(prior) ||
runner.schema !== RUNNERD_STATE_SCHEMA || runner.lifecycle !== "suspended" ||
["runnerInstanceId", "environmentLeaseId", "runId", "normalizedSessionId"].some(key => runner[key] !== prior[key])) return null;
return latest;
}
function interruptedRotationSource(root: string, execution: NativeExecutionInput): string | null {
try {
if (existsSync(resolve(root, "runner", "runner-state.json"))) return null;
if (!existsSync(resolve(root, "control-plane", "control-plane-state.json"))) {
if (existsSync(resolve(root, "control-plane")) && readdirSync(resolve(root, "control-plane")).length > 0) return null;
return latestSuspendedArchivedEpoch(root, prior => typeof prior.runId === "string" &&
prior.runId !== execution.binding.runId && prior.normalizedSessionId === nativeSessionKey(execution));
}
const control = readRunnerdControlPlaneState(root);
const identity = record(control.identity);
const commands = Array.isArray(control.commands) ? control.commands.map(record) : [];
@@ -5669,26 +5719,9 @@ function undeliveredAttachRotationSource(root: string, execution: NativeExecutio
Object.keys(record(control.commandDeliveryCounts)).length !== 0 ||
control.warmTransition != null || control.completedWarmTransition != null || control.stagedRunAttachment != null ||
history.adopted === true || history.credentialsIssued !== 0 ||
commands.length !== 1 || commands[0]!.type !== "run.attach" || commands[0]!.status !== "pending" ||
existsSync(resolve(root, "runner", "runner-state.json"))) return null;
const archives = resolve(root, "authority-epochs");
if (!isSafeNativeStateDirectory(archives)) return null;
const candidates = readdirSync(archives).map(name => resolve(archives, name)).filter(epoch => {
const state = resolve(epoch, "control-plane", "control-plane-state.json");
return isSafeNativeStateDirectory(epoch) && existsSync(state) && existsSync(resolve(epoch, "runner-state.json"));
}).sort((left, right) => lstatSync(resolve(right, "control-plane", "control-plane-state.json")).mtimeMs
- lstatSync(resolve(left, "control-plane", "control-plane-state.json")).mtimeMs);
const latest = candidates[0];
if (!latest) return null;
const prior = record(readRunnerdControlPlaneState(latest).identity);
const runner = record(JSON.parse(readBoundedNativeFile(resolve(latest, "runner-state.json"),
NATIVE_RUNNER_STATE_MAX_BYTES, "runner_state_too_large").toString("utf8")));
const digest = createHash("sha256").update(JSON.stringify(prior)).digest("hex").slice(0, 24);
if (basename(latest) !== `epoch-${digest}` || prior.runId === identity.runId ||
["runnerInstanceId", "environmentLeaseId", "normalizedSessionId"].some(key => prior[key] !== identity[key]) ||
runner.schema !== RUNNERD_STATE_SCHEMA || runner.lifecycle !== "suspended" ||
["runnerInstanceId", "environmentLeaseId", "runId", "normalizedSessionId"].some(key => runner[key] !== prior[key])) return null;
return latest;
commands.length !== 1 || commands[0]!.type !== "run.attach" || commands[0]!.status !== "pending") return null;
return latestSuspendedArchivedEpoch(root, prior => prior.runId !== identity.runId &&
["runnerInstanceId", "environmentLeaseId", "normalizedSessionId"].every(key => prior[key] === identity[key]));
} catch {
return null;
}
@@ -5697,13 +5730,15 @@ function undeliveredAttachRotationSource(root: string, execution: NativeExecutio
async function rollBackUndeliveredAttachRotation(root: string, execution: NativeExecutionInput,
onLog?: (stream: "stdout" | "stderr", chunk: string) => Promise<void>): Promise<void> {
if (!existsSync(root) || !isSafeNativeStateDirectory(root)) return;
const epoch = undeliveredAttachRotationSource(root, execution);
const epoch = interruptedRotationSource(root, execution);
if (!epoch) return;
const archivedAt = Date.now();
const breakRoot = resolve(root, "continuity-breaks", `${archivedAt}-${randomUUID()}`);
mkdirSync(breakRoot, { recursive: true, mode: 0o700 });
noteNativeRunnerEvidence(archivedAt);
renameSync(resolve(root, "control-plane"), resolve(breakRoot, "control-plane"));
if (existsSync(resolve(root, "control-plane"))) {
const breakRoot = resolve(root, "continuity-breaks", `${archivedAt}-${randomUUID()}`);
mkdirSync(breakRoot, { recursive: true, mode: 0o700 });
renameSync(resolve(root, "control-plane"), resolve(breakRoot, "control-plane"));
}
mkdirSync(resolve(root, "runner"), { recursive: true, mode: 0o700 });
renameSync(resolve(epoch, "runner-state.json"), resolve(root, "runner", "runner-state.json"));
renameSync(resolve(epoch, "control-plane"), resolve(root, "control-plane"));
@@ -6720,9 +6755,14 @@ async function restorePerTurnRunnerdController(input: {
runnerExecutionTarget?: AdapterExecutionTarget | null;
restartRecovery?: NativeRestartRecoveryClaim;
}): Promise<void> {
if (input.restartRecovery || input.runnerExecutionTarget?.kind !== "remote" ||
if ((input.restartRecovery && input.restartRecovery.kind !== "bootstrap_incomplete") ||
input.runnerExecutionTarget?.kind !== "remote" ||
input.execution.session.lifecyclePolicy.mode !== "per_turn") return;
const root = scopedRunnerdStateRoot(input.execution);
if (input.restartRecovery) {
const prior = readRunnerdDurableIdentity(root) ?? readRunnerdDurableIdentity(root, true);
if (!prior || prior.runId === input.execution.binding.runId) return;
}
await restoreDurableHarnessForExecution(input.db, input.execution, root);
const identity = readRunnerdDurableIdentity(root);
const reference = identity ?? readRunnerdDurableIdentity(root, true);
@@ -9591,6 +9631,24 @@ async function executePaperclipNativeSessionWithinScope(
prepareAfterClaim?: () => Promise<Parameters<typeof executePaperclipNativeSession>[0]>,
cleanupOnly = false,
ownDiagnosticLease?: (lease: NativeDiagnosticLease) => void,
): Promise<AdapterExecutionResult> {
// Drop this attempt's restart-recovery flush however it ends (a failure that
// never finishes its trace included), without touching a newer attempt's.
const runId = input.execution.binding.runId;
const priorFlush = nativeRunRecoveryFlushers.get(runId);
try {
return await executePaperclipNativeSessionAttempt(input, prepareAfterClaim, cleanupOnly, ownDiagnosticLease);
} finally {
const current = nativeRunRecoveryFlushers.get(runId);
if (current !== undefined && current !== priorFlush) nativeRunRecoveryFlushers.delete(runId);
}
}
async function executePaperclipNativeSessionAttempt(
input: Parameters<typeof executePaperclipNativeSession>[0],
prepareAfterClaim?: () => Promise<Parameters<typeof executePaperclipNativeSession>[0]>,
cleanupOnly = false,
ownDiagnosticLease?: (lease: NativeDiagnosticLease) => void,
): Promise<AdapterExecutionResult> {
if (input.execution.provider.kind === "acpx" && input.execution.provider.agent === "antigravity") {
const candidate = resolveAntigravityCandidateAdmission({
@@ -9648,16 +9706,25 @@ async function executePaperclipNativeSessionWithinScope(
preparationSpans,
Date.now(),
);
const traceRunId = input.execution.binding.runId;
const flushRecovery = () => {
if (nativeRunRecoveryFlushers.get(traceRunId) === flushRecovery) nativeRunRecoveryFlushers.delete(traceRunId);
trace.flushRestartRecovery();
};
const trace = createNativeRunTrace({
runId: input.execution.binding.runId,
runId: traceRunId,
startedAtMs: preparationStarts.runStartedAtMs,
onEvent: input.onEvent,
onFinish: () => {
if (nativeRunRecoveryFlushers.get(traceRunId) === flushRecovery) nativeRunRecoveryFlushers.delete(traceRunId);
},
budget: {
adapter: speedBudgetAdapter(input.execution.provider),
environment: !input.runnerExecutionTarget || input.runnerExecutionTarget.kind === "local" ? "local" : "remote",
restartRecovery: Boolean(input.restartRecovery),
},
});
nativeRunRecoveryFlushers.set(traceRunId, flushRecovery);
await emitStartupReuse(input.onEvent, "environment", environmentStartupReuse(input.runnerExecutionTarget));
if (!input.runnerExecutionTarget || input.runnerExecutionTarget.kind === "local") {
await emitStartupReuse(input.onEvent, "workspace", { state: "not_applicable", reason: "not_applicable" });
@@ -10706,6 +10773,7 @@ async function executePaperclipNativeSessionWithinScope(
`native-warm-session:${input.execution.binding.runId}`,
);
let existingWarmSession: NativeSession | undefined;
let reusingReleasedRunner = false;
let pluginTaskDescription: Extract<ReturnType<typeof environmentTaskResultSchema.parse>,{kind:"description"}> | undefined;
let managedCredentialSession: NativeSession | undefined;
let githubAccess: NativeGitHubAccess | undefined;
@@ -10797,6 +10865,7 @@ async function executePaperclipNativeSessionWithinScope(
entry.instructionWorkingCopy = input.instructionWorkingCopy?.checkpointWarm ? undefined : input.instructionWorkingCopy;
entry.busy = true;
entry.ownerToken = warmSessionOwnerToken;
reusingReleasedRunner = entry.released === true;
entry.released = false;
entry.environmentId =
input.runnerExecutionTarget?.environmentId ?? null;
@@ -10959,6 +11028,10 @@ async function executePaperclipNativeSessionWithinScope(
const resolveWake = () => timeController("wake_binding", () => resolveCurrentWakeCommentsBinding(input.db, input.execution.binding));
const expectedCurrentWakeComments = await (executionContext ? executionContext.wait(resolveWake) : resolveWake());
let remoteWarmRecoveryPending = remoteWarmRecovery !== undefined;
const beginRetainedRunAttachment = (session: NativeSession) => () => {
session.beginRunAttachment?.({ identity: { runId: input.execution.binding.runId, sessionId: session.identity().sessionId,
companyId: input.execution.binding.companyId, issueId: input.execution.binding.issueId, agentId: input.execution.binding.agentId } });
};
const createExecutionBackend = () => createRunnerdBackend({
...input,
onDiagnosticLease: ownDiagnosticLease,
@@ -10978,6 +11051,7 @@ async function executePaperclipNativeSessionWithinScope(
runnerInstanceId: effectiveRunnerInstanceId,
pluginTaskDescription,
retainedPluginTaskOwner: existingWarmSession ? pluginTaskSessionOwners.get(existingWarmSession) : undefined,
beginRetainedRunAttachment: existingWarmSession && reusingReleasedRunner ? beginRetainedRunAttachment(existingWarmSession) : undefined,
durableEnvironmentLeaseId: durableRunnerBinding?.environmentLeaseId,
remoteWarmRecovery: remoteWarmRecovery ? {
...remoteWarmRecovery,
@@ -11581,9 +11655,14 @@ async function executePaperclipNativeSessionWithinScope(
}
trace.activate(taskSettleScope);
liveQuestions.close();
const activeAtFailure = activeNativeSessions.get(input.execution.binding.runId);
activeNativeSessions.delete(input.execution.binding.runId);
clearSteeringDeliveries(input.execution.binding.runId);
clearNativeRuntimeRequestResolutions(input.execution.binding.runId);
// A Stop interrupts the provider before it saves its acknowledgement.
// When the interrupted turn ends first, wait for that Stop to settle, so
// the checks below judge its saved receipt, not a failure.
if (activeAtFailure?.cancelRequested) await activeAtFailure.cancellationSettled?.catch(() => undefined);
// Stop before paperclip_finish is normal. Bounded provider teardown has
// finished; a missing result must not overwrite cancellation or trigger
// another turn to perform completion bookkeeping.
@@ -11608,7 +11687,14 @@ async function executePaperclipNativeSessionWithinScope(
stopIntent.scope === "run" &&
typeof stopIntent.intentAuditId === "string" && stopIntent.intentAuditId.length > 0 &&
["pending", "acknowledged"].includes(String(stopIntent.dispatchState));
// A governance pause (budget, company or agent pause) Stop is the same
// kind of intentional stop. The fence records its cancellation before
// dispatch, so judge it whether or not the terminal write has landed;
// resume continues it (budget-resume-continuation.ts).
const governancePauseStop = !!stoppedRun && hasAcknowledgedNativeRunStopReceipt(stoppedRun)
&& isGovernancePauseStop({ status: "cancelled", resultJson: stoppedRun.resultJson });
if (stoppedRun && (pendingStartupStop || hasAcknowledgedNativeStopIntent(stoppedRun) || hasAcknowledgedNativeReassignmentStopIntent(stoppedRun)
|| governancePauseStop
|| (error instanceof NativeCancellationPendingRecoveryError && hasAcknowledgedNativeRunStopReceipt(stoppedRun)))) {
const [settled] = await writeExecutionControl(input.db, [DELIVERY_QUEUES.executionFinalization, DELIVERY_QUEUES.accounting], tx => tx.update(nativeRunFinalizations).set({
phase: "terminal_failure", failureCode: "native_retry_cancelled", nextAttemptAt: null,
@@ -14180,6 +14266,7 @@ export async function createRunnerdBackend(input: {
durableEnvironmentLeaseId?: string;
remoteWarmRecovery?: Omit<RemoteWarmSessionRecovery, "revalidate"> & { revalidate: () => Promise<void> };
retainedPluginTaskOwner?: ReturnType<typeof createPluginRunnerTaskOwner>;
beginRetainedRunAttachment?: () => void;
pluginTaskDescription?: Extract<ReturnType<typeof environmentTaskResultSchema.parse>,{kind:"description"}>;
onSpawn?: (meta: {
pid: number;
@@ -14474,7 +14561,13 @@ async function createRunnerdBackendWithinSessionClaim(
if (relayAssignedMcp && hasAssignedMcp && !assignedGatewayPublicId) {
throw new Error("assigned native MCP gateway path is invalid");
}
const [preparedEnvironment, preparedAssignedMcp] = await Promise.allSettled([
let durableState: ReturnType<typeof readDurableState> | undefined;
let pluginTaskOwner: ReturnType<typeof preparePluginTaskOwner> | undefined;
const earlyPluginTaskOwner = pluginTask && remoteTarget?.transport === "sandbox" && input.beginRetainedRunAttachment
&& input.retainedPluginTaskOwner && !input.remoteWarmRecovery && !input.restartRecovery
? pluginTaskOwner = preparePluginTaskOwner().then(prepared => { input.beginRetainedRunAttachment!(); return prepared; })
: undefined;
const [preparedEnvironment, preparedAssignedMcp, preparedPluginTaskOwner] = await Promise.allSettled([
Promise.resolve().then(() => input.prepareRunnerEnvironment?.() ?? input.runnerEnvironment),
Promise.resolve().then(() => relayAssignedMcp && !nativeReview && assignedMcpUrl && assignedMcpToken
? timeController("assigned_mcp", () => createAssignedMcpTools({
@@ -14485,9 +14578,11 @@ async function createRunnerdBackendWithinSessionClaim(
...(input.execution.provider.kind === "opencode" ? { delivery: "on_demand" as const } : {}),
}))
: undefined),
earlyPluginTaskOwner,
]);
if (preparedEnvironment.status === "rejected") throw preparedEnvironment.reason;
if (preparedAssignedMcp.status === "rejected") throw preparedAssignedMcp.reason;
if (preparedPluginTaskOwner.status === "rejected") throw preparedPluginTaskOwner.reason;
const runnerEnvironment = preparedEnvironment.value;
const assignedMcpTools = preparedAssignedMcp.value;
const dotAttachmentActor = input.execution.schema === "paperclip.native-execution-input.v6"
@@ -14552,13 +14647,7 @@ async function createRunnerdBackendWithinSessionClaim(
authorityEpoch.revoke();
throw error;
}
const root = runnerdStateRoot(input.execution);
const durableIdentity = readRunnerdDurableIdentity(root);
const durableBinding = retainedTransition
? loadRunnerdDurableBinding(input.execution, retainedTransition)
: durableIdentityMatchesSession(durableIdentity, input.execution)
? durableIdentity
: null;
const { root, durableIdentity, durableBinding } = durableState ??= readDurableState();
const stagedRecovery = input.restartRecovery?.kind === "reattach_remote_runner"
? await verifyStagedAttachmentRestart(input) : null;
const retainedSystemInstructions = input.restartRecovery?.kind === "reattach_remote_runner"
@@ -14619,7 +14708,21 @@ async function createRunnerdBackendWithinSessionClaim(
input.durableEnvironmentLeaseId ??
input.execution.binding.executionWorkspaceId;
mkdirSync(root, { recursive: true, mode: 0o700 });
if (pluginTask && remoteTarget?.transport === "sandbox" && input.execution.schema !== "paperclip.native-execution-input.v6" && input.execution.schema !== "paperclip.native-execution-input.v8") {
function readDurableState() {
const root = runnerdStateRoot(input.execution);
const durableIdentity = readRunnerdDurableIdentity(root);
const durableBinding = retainedTransition
? loadRunnerdDurableBinding(input.execution, retainedTransition)
: durableIdentityMatchesSession(durableIdentity, input.execution)
? durableIdentity
: null;
return { root, durableIdentity, durableBinding };
}
async function preparePluginTaskOwner() {
if (!pluginTask || remoteTarget?.transport !== "sandbox" || input.execution.schema === "paperclip.native-execution-input.v6" || input.execution.schema === "paperclip.native-execution-input.v8") {
throw new Error("Plugin Runner tasks require a workspace session");
}
const { root, durableIdentity, durableBinding } = durableState ??= readDurableState();
const provider = input.execution.provider;
if (provider.kind !== "codex" && provider.kind !== "opencode" && provider.kind !== "acpx") throw new Error("Plugin Runner tasks require a workspace harness");
if (remoteTarget.sandboxLeaseAcquisition?.outcome === "resumed" && !durableBinding && !input.retainedPluginTaskOwner && !input.remoteWarmRecovery) {
@@ -14706,6 +14809,10 @@ async function createRunnerdBackendWithinSessionClaim(
if (input.execution.session.lifecyclePolicy.mode === "warm" && (!credentials.accountIdentity || description.warmTaskVersion !== 1)) throw new Error("Plugin warm task capability or account unavailable");
const owner = retainedOwner ?? createPluginRunnerTaskOwner(ownerInput);
if (retainedOwner) retainedOwner.rebind(ownerInput);
return { provider, description, prior, adoptionIdentity, adopting, owner };
}
if (pluginTask && remoteTarget?.transport === "sandbox" && input.execution.schema !== "paperclip.native-execution-input.v6" && input.execution.schema !== "paperclip.native-execution-input.v8") {
const { provider, description, prior, adoptionIdentity, adopting, owner } = await (pluginTaskOwner ??= preparePluginTaskOwner());
if (description.workspaceFileReadVersion === 1) {
authority.binding.readRemoteApiFile = owner.readWorkspaceFile;
authority.binding.readRemoteTaskFile = file => owner.readWorkspaceFile(file.contentRef, file.byteSize);
server/src/services/native-runtime/native-speed-budgets.test.ts
+65 -8
View File
@@ -134,12 +134,12 @@ describe("speed budget checks in the native run trace", () => {
it("checks restart recovery latency once, from ready to the first provider event", async () => {
const { trace, breaches } = traced({ adapter: "opencode", environment: "local" });
const late = LOCAL_SPEED_BUDGETS.opencode.restart_recovery.p95Ms + 1_000;
const late = LOCAL_SPEED_BUDGETS.opencode.restart_recovery_control_plane.p95Ms + 1_000;
trace.checkRestartRecovery({ readyAtMs: 10_000, emittedAtMs: 10_000 + late, committedAtMs: 10_000 + late, disposition: "resume_dead_runner" });
trace.checkRestartRecovery({ readyAtMs: 10_000, emittedAtMs: 10_000 + late * 2, committedAtMs: 10_000 + late * 2, disposition: "resume_dead_runner" });
await trace.finish("ok");
expect(breaches()).toEqual([expect.objectContaining({ metric: "restart_recovery", measuredMs: late,
expect(breaches()).toEqual([expect.objectContaining({ metric: "restart_recovery_control_plane", measuredMs: late,
disposition: "resume_dead_runner", startKind: "unknown" })]);
});
@@ -148,22 +148,79 @@ describe("speed budget checks in the native run trace", () => {
await trace.record({ name: "heartbeat.queue", startedAtMs: 0, endedAtMs: 1 });
await trace.record({ name: "runner.session.resume", startedAtMs: 1, endedAtMs: 120_000 });
await trace.record({ name: "task.time_to_first_agent_activity", startedAtMs: 0, endedAtMs: 120_000 });
const at = LOCAL_SPEED_BUDGETS.codex.restart_recovery.p95Ms + 1;
const at = LOCAL_SPEED_BUDGETS.codex.restart_recovery_control_plane.p95Ms + 1;
trace.checkRestartRecovery({ readyAtMs: 0, emittedAtMs: at, committedAtMs: at, disposition: "resume_dead_runner" });
await trace.finish("ok");
expect(breaches().map(breach => breach.metric)).toEqual(["restart_recovery"]);
expect(breaches().map(breach => breach.metric)).toEqual(["restart_recovery_control_plane"]);
});
it("judges only Paperclip's share of a recovery, not the provider's session load", async () => {
const { trace, breaches } = traced({ adapter: "codex", environment: "local", restartRecovery: true });
const total = LOCAL_SPEED_BUDGETS.codex.restart_recovery.p95Ms + 3_000;
// Ready → claim, prepare and launch in 1.5 s; the provider reloads its long session.
await trace.record({ name: "provider.session.load", startedAtMs: 1_500, endedAtMs: total - 500 });
trace.checkRestartRecovery({ readyAtMs: 0, emittedAtMs: total, committedAtMs: total, disposition: "resume_dead_runner" });
await trace.finish("ok");
expect(breaches()).toEqual([]);
});
it("reports a slow control-plane share with the full latency and the provider's share", async () => {
const { trace, breaches } = traced({ adapter: "codex", environment: "local", restartRecovery: true });
const control = LOCAL_SPEED_BUDGETS.codex.restart_recovery_control_plane.p95Ms + 2_000;
trace.checkRestartRecovery({ readyAtMs: 0, emittedAtMs: control + 3_000, committedAtMs: control + 3_000, disposition: "resume_dead_runner" });
// The provider span is committed after the provider event that ended the recovery.
await trace.record({ name: "provider.session.load", startedAtMs: control, endedAtMs: control + 2_800 });
await trace.finish("ok");
expect(breaches()).toEqual([expect.objectContaining({ metric: "restart_recovery_control_plane", measuredMs: control,
restartTotalMs: control + 3_000, providerLoadMs: 3_000 })]);
});
it("flushes a pending recovery check when the controller detaches the run", async () => {
const { trace, breaches } = traced({ adapter: "codex", environment: "local", restartRecovery: true });
const late = LOCAL_SPEED_BUDGETS.codex.restart_recovery_control_plane.p95Ms + 1_000;
// A launched runner whose load span has not been written yet.
trace.checkRestartRecovery({ readyAtMs: 0, emittedAtMs: late, committedAtMs: late, disposition: "resume_dead_runner" });
await new Promise(resolve => setTimeout(resolve, 0));
expect(breaches()).toEqual([]);
trace.flushRestartRecovery();
await new Promise(resolve => setTimeout(resolve, 0));
expect(breaches()).toEqual([expect.objectContaining({ metric: "restart_recovery_control_plane", measuredMs: late })]);
});
it("logs each span's own start time in its run-log row", async () => {
const events: AdapterRuntimeEvent[] = [];
const trace = createNativeRunTrace({ runId: "run-1", startedAtMs: 1_000, onEvent: async event => { events.push(event); } });
await trace.record({ name: "provider.session.load", startedAtMs: 4_000, endedAtMs: 9_000 });
await trace.finish("ok");
expect(events.find(event => (event.payload as Record<string, unknown>)?.span === "provider.session.load")?.payload)
.toMatchObject({ startOffsetMs: 3_000, durationMs: 5_000, wallStartedAtMs: 4_000 });
});
it("judges a recovery before the run finishes, so a later detach for restart keeps the warning", async () => {
const control = LOCAL_SPEED_BUDGETS.codex.restart_recovery_control_plane.p95Ms + 1_000;
// The load span arrives after the provider event: judged when it is recorded.
const resumed = traced({ adapter: "codex", environment: "local", restartRecovery: true });
resumed.trace.checkRestartRecovery({ readyAtMs: 0, emittedAtMs: control + 2_000, committedAtMs: control + 2_000, disposition: "resume_dead_runner" });
await resumed.trace.record({ name: "provider.session.load", startedAtMs: control, endedAtMs: control + 1_900 });
// A reattached runner reloads nothing: judged at the check.
const reattached = traced({ adapter: "codex", environment: "local", restartRecovery: true });
reattached.trace.checkRestartRecovery({ readyAtMs: 0, emittedAtMs: control, committedAtMs: control, disposition: "reattach_existing_runner" });
// Neither trace finishes (the controller detached for another restart).
await new Promise(resolve => setTimeout(resolve, 0));
expect(resumed.breaches()).toEqual([expect.objectContaining({ metric: "restart_recovery_control_plane", measuredMs: control })]);
expect(reattached.breaches()).toEqual([expect.objectContaining({ metric: "restart_recovery_control_plane", measuredMs: control, providerLoadMs: 0 })]);
});
it("does not spend the restart-recovery check on an event emitted before ready", async () => {
const { trace, breaches } = traced({ adapter: "codex", environment: "local", restartRecovery: true });
const late = LOCAL_SPEED_BUDGETS.codex.restart_recovery.p95Ms + 1_000;
const late = LOCAL_SPEED_BUDGETS.codex.restart_recovery_control_plane.p95Ms + 1_000;
// Buffered during the outage, committed after ready.
trace.checkRestartRecovery(restartRecoveryMeasurementFor({ event: { eventType: "item.delta", emittedAt: new Date(9_000).toISOString() },
startup: { phase: "ready", updatedAt: new Date(10_000).toISOString() }, committedAtMs: 10_500, disposition: "reattach_existing_runner" })!);
trace.checkRestartRecovery(restartRecoveryMeasurementFor({ event: { eventType: "item.delta", emittedAt: new Date(10_000 + late - 200).toISOString() },
startup: { phase: "ready", updatedAt: new Date(10_000).toISOString() }, committedAtMs: 10_000 + late, disposition: "reattach_existing_runner" })!);
await trace.finish("ok");
expect(breaches()).toEqual([expect.objectContaining({ metric: "restart_recovery", measuredMs: late })]);
expect(breaches()).toEqual([expect.objectContaining({ metric: "restart_recovery_control_plane", measuredMs: late })]);
});
it("measures a resumed attempt's boot from its own claim, not the run's first queue span", async () => {
@@ -189,11 +246,11 @@ describe("speed budget checks in the native run trace", () => {
it("measures to the commit time, so delivery delays after emission count", async () => {
const { trace, breaches } = traced({ adapter: "codex", environment: "local", restartRecovery: true });
const budget = LOCAL_SPEED_BUDGETS.codex.restart_recovery.p95Ms;
const budget = LOCAL_SPEED_BUDGETS.codex.restart_recovery_control_plane.p95Ms;
// Emitted 1 s after ready, committed 20 s after ready.
trace.checkRestartRecovery({ readyAtMs: 0, emittedAtMs: 1_000, committedAtMs: budget + 12_000, disposition: "resume_dead_runner" });
await trace.finish("ok");
expect(breaches()).toEqual([expect.objectContaining({ metric: "restart_recovery", measuredMs: budget + 12_000 })]);
expect(breaches()).toEqual([expect.objectContaining({ metric: "restart_recovery_control_plane", measuredMs: budget + 12_000 })]);
});
it("writes nothing for remote runs or without a budget subject", async () => {
server/src/services/native-runtime/native-speed-budgets.ts
+26 -1
View File
@@ -22,7 +22,16 @@ export type SpeedBudgetAdapter = "claude" | "codex" | "opencode";
* a reported metric without a budget: tool-first adapters such as Claude often
* call tools before they write text. The budget numbers are the TTFT baseline's.
*/
export type SpeedBudgetMetric = "time_to_first_activity" | "agent_boot" | "restart_recovery";
/**
* `restart_recovery_control_plane`: control plane ready to the moment the
* recovered run's provider starts loading its saved session
* (`provider.session.load`), or to the run's next provider event when nothing
* reloads (a reattached runner). That is Paperclip's own share of a recovery:
* claim, prepare, runtime stage and runner launch. The provider's session load
* grows with session size and with how many sessions resume at once, so the
* full `restart_recovery` latency is reported without a budget.
*/
export type SpeedBudgetMetric = "time_to_first_activity" | "agent_boot" | "restart_recovery" | "restart_recovery_control_plane";
export type SpeedBudgetStart = "cold" | "warm";
export type SpeedBudget = { p50Ms: number; p95Ms: number };
@@ -43,22 +52,29 @@ const s = (seconds: number) => Math.round(seconds * 1000);
export const LOCAL_SPEED_BUDGETS: Readonly<Record<SpeedBudgetAdapter, {
time_to_first_activity: Record<SpeedBudgetStart, SpeedBudget>;
agent_boot: Record<SpeedBudgetStart, SpeedBudget>;
/** Full recovery (ready → next provider event). Reported, not judged. */
restart_recovery: SpeedBudget;
/** Paperclip's share of a recovery; judged. Measured 1.2–2.0 s per run with six
* Codex runs resuming at once (wave w-20261109T045320-s9807, local). */
restart_recovery_control_plane: SpeedBudget;
}>> = {
claude: {
time_to_first_activity: { cold: { p50Ms: s(16.5), p95Ms: s(30) }, warm: { p50Ms: s(9.5), p95Ms: s(28.5) } },
agent_boot: { cold: { p50Ms: s(9), p95Ms: s(20.5) }, warm: { p50Ms: s(3.5), p95Ms: s(16.5) } },
restart_recovery: { p50Ms: s(15), p95Ms: s(22.5) },
restart_recovery_control_plane: { p50Ms: s(2.5), p95Ms: s(4) },
},
codex: {
time_to_first_activity: { cold: { p50Ms: s(6), p95Ms: s(17) }, warm: { p50Ms: s(10), p95Ms: s(28.5) } },
agent_boot: { cold: { p50Ms: s(2.5), p95Ms: s(7) }, warm: { p50Ms: s(5), p95Ms: s(16.5) } },
restart_recovery: { p50Ms: s(6.5), p95Ms: s(8) },
restart_recovery_control_plane: { p50Ms: s(2.5), p95Ms: s(4) },
},
opencode: {
time_to_first_activity: { cold: { p50Ms: s(17.5), p95Ms: s(49) }, warm: { p50Ms: s(26.5), p95Ms: s(88) } },
agent_boot: { cold: { p50Ms: s(12), p95Ms: s(37) }, warm: { p50Ms: s(22), p95Ms: s(65.5) } },
restart_recovery: { p50Ms: s(25.5), p95Ms: s(38.5) },
restart_recovery_control_plane: { p50Ms: s(2.5), p95Ms: s(4) },
},
};
@@ -77,6 +93,7 @@ export function speedBudgetFor(adapter: SpeedBudgetAdapter, metric: SpeedBudgetM
start: SpeedBudgetStart | null): SpeedBudget {
const budgets = LOCAL_SPEED_BUDGETS[adapter];
if (metric === "restart_recovery") return budgets.restart_recovery;
if (metric === "restart_recovery_control_plane") return budgets.restart_recovery_control_plane;
const byStart = budgets[metric];
if (start) return byStart[start];
return { p50Ms: Math.max(byStart.cold.p50Ms, byStart.warm.p50Ms), p95Ms: Math.max(byStart.cold.p95Ms, byStart.warm.p95Ms) };
@@ -98,6 +115,10 @@ export type SpeedBudgetBreach = {
/** First activity only: time the input waited for its run to exist (for example a
* comment posted while the previous turn was running). Not judged. */
inputWaitMs?: number;
/** Control-plane recovery only: the full ready → next provider event latency
* and the provider's share of it (session load onwards). Not judged. */
restartTotalMs?: number;
providerLoadMs?: number;
};
/** A breach when one measured run exceeds its p95 budget, else null. */
@@ -109,6 +130,8 @@ export function evaluateSpeedBudget(input: {
measuredMs: number;
disposition?: string;
inputWaitMs?: number;
restartTotalMs?: number;
providerLoadMs?: number;
}): SpeedBudgetBreach | null {
if (!input.adapter || input.environment !== "local" || !Number.isFinite(input.measuredMs) || input.measuredMs < 0) return null;
const budget = speedBudgetFor(input.adapter, input.metric, input.start);
@@ -118,6 +141,8 @@ export function evaluateSpeedBudget(input: {
startKind: input.start ?? "unknown", measuredMs: Math.round(input.measuredMs), budgetP50Ms: budget.p50Ms,
budgetP95Ms: budget.p95Ms, percentile: "p95", baseline: SPEED_BUDGET_BASELINE,
...(input.disposition ? { disposition: input.disposition } : {}),
...(input.restartTotalMs !== undefined ? { restartTotalMs: Math.round(input.restartTotalMs) } : {}),
...(input.providerLoadMs !== undefined ? { providerLoadMs: Math.round(input.providerLoadMs) } : {}),
...(input.inputWaitMs ? { inputWaitMs: Math.round(input.inputWaitMs) } : {}),
};
}
server/src/services/native-runtime/native-stale-owner-completion.test.ts
+70
View File
@@ -27,6 +27,12 @@ describe("stale owner completion policy", () => {
expect(supersedeStaleOwnerCompletion(done, issue, "agent-a")).toEqual({ ...done, statusAction: "preserve",
toStatus: "in_progress", reasonCode: "completion_owner_superseded", effects: [] });
});
it("preserves status when another run of the same owner now holds the task", () => {
expect(supersedeStaleOwnerCompletion(done, { ...owner, executionRunId: "run-2" }, "agent-a", "run-1")).toEqual({ ...done,
statusAction: "preserve", toStatus: "in_progress", reasonCode: "completion_run_superseded", effects: [] });
expect(supersedeStaleOwnerCompletion(done, { ...owner, executionRunId: "run-1" }, "agent-a", "run-1")).toBe(done);
expect(supersedeStaleOwnerCompletion(done, { ...owner, executionRunId: null }, "agent-a", "run-1")).toBe(done);
});
it("leaves non-terminal decisions to their existing owner checks", () => {
const blocked: NativeStatusDecision = { ...done, statusAction: "blocked", toStatus: "blocked", reasonCode: "x" };
expect(supersedeStaleOwnerCompletion(blocked, { ...owner, assigneeAgentId: "agent-b" }, "agent-a")).toBe(blocked);
@@ -92,6 +98,46 @@ describe.skipIf(!support.supported)("native completion after reassignment", () =
.where(eq(heartbeatRuns.id, f.runId));
}
/** The board's Stop, then the user's continue comment starts a new run that holds the task. */
async function stopAndContinue(f: Awaited<ReturnType<typeof acceptedDoneResult>>) {
await db.update(heartbeatRuns).set({ status: "cancelled", errorCode: "cancelled", finishedAt: new Date() })
.where(eq(heartbeatRuns.id, f.runId));
await db.update(issues).set({ executionRunId: null, checkoutRunId: null }).where(eq(issues.id, f.issueId));
const continuationRunId = randomUUID();
await db.insert(heartbeatRuns).values({ id: continuationRunId, companyId: f.companyId, agentId: f.agentId,
nativeIssueId: f.issueId, status: "running", runtimeMode: "native",
contextSnapshot: { issueId: f.issueId, source: "issue.comment" } });
await db.update(issues).set({ executionRunId: continuationRunId, checkoutRunId: continuationRunId })
.where(eq(issues.id, f.issueId));
return continuationRunId;
}
function actBeforeStatusCommit(act: () => Promise<unknown>) {
let acted = false;
const racing = new Proxy(db, {
get(target, property, receiver) {
const value = Reflect.get(target, property, receiver);
if (property !== "update") return typeof value === "function" ? value.bind(target) : value;
return (table: unknown) => {
const builder = target.update(table as typeof nativeRunFinalizations);
if (table !== nativeRunFinalizations || acted) return builder;
const set = builder.set.bind(builder);
return Object.assign(builder, { set: (values: Record<string, unknown>) => {
const next = set(values as never);
if (values.phase !== "arbitrating" || !("assessmentId" in values)) return next;
const where = next.where.bind(next);
return Object.assign(next, { where: (...args: Parameters<typeof where>) => {
const query = where(...args);
return { then: (resolve: (value: unknown) => unknown, reject: (error: unknown) => unknown) =>
query.then(async (result) => { acted = true; await act(); return result; }).then(resolve, reject) };
} });
} });
};
},
}) as Db;
return { racing, acted: () => acted };
}
const issue = async (issueId: string) => (await db.select().from(issues).where(eq(issues.id, issueId)))[0];
const decision = async (runId: string) => {
const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId));
@@ -154,6 +200,30 @@ describe.skipIf(!support.supported)("native completion after reassignment", () =
expect(await issue(f.issueId)).toMatchObject({ status: "in_progress", assigneeAgentId: f.successorId });
expect((await decision(f.runId)).row).toMatchObject({ reasonCode: "completion_owner_superseded" });
});
it("does not let a stopped run's saved done result close the task a newer run is continuing", async () => {
const f = await acceptedDoneResult();
const continuationRunId = await stopAndContinue(f);
await finalizeNativeRun({ db, runId: f.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: false });
expect(await issue(f.issueId)).toMatchObject({ status: "in_progress", executionRunId: continuationRunId });
const { coordinator, row } = await decision(f.runId);
expect(coordinator.phase).toBe("committed");
expect(row).toMatchObject({ toStatus: "in_progress", reasonCode: "completion_run_superseded" });
});
it("does not close the task when a newer run takes it after arbitration but before the status commit", async () => {
const f = await acceptedDoneResult();
let continuationRunId: string | null = null;
const { racing, acted } = actBeforeStatusCommit(async () => { continuationRunId = await stopAndContinue(f); });
await finalizeNativeRun({ db: racing, runId: f.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: false });
expect(acted()).toBe(true);
expect(await issue(f.issueId)).toMatchObject({ status: "in_progress", executionRunId: continuationRunId });
expect((await decision(f.runId)).row).toMatchObject({ reasonCode: "completion_run_superseded" });
});
async function approveEvidenceLater(f: Awaited<ReturnType<typeof acceptedDoneResult>>) {
await db.update(issueWorkProducts).set({ reviewState: "approved", updatedAt: new Date(Date.now() + 1_000) })
.where(eq(issueWorkProducts.id, f.workProductId));
server/src/services/native-runtime/native-status-reassessment.ts
+1 -1
View File
@@ -236,7 +236,7 @@ export async function reassessNativeStatusFromSource(
}
// New evidence from a reassigned-away run cannot close the new owner's task.
if (currentIssue && !currentIssue.conversationAgentId) {
decision = supersedeStaleOwnerCompletion(decision, { ...currentIssue, status: row.issueStatus }, row.agentId);
decision = supersedeStaleOwnerCompletion(decision, { ...currentIssue, status: row.issueStatus }, row.agentId, row.runId);
}
let committed: Awaited<ReturnType<typeof commitNativeStatusDecision>>;
try {
server/src/services/native-runtime/plugin-runner-release-resume.integration.test.ts
+30 -4
View File
@@ -6,10 +6,10 @@ import https, { createServer } from "node:https";
import { createServer as createTcpServer } from "node:net";
import { join, resolve } from "node:path";
import { getCACertificates, setDefaultCACertificates } from "node:tls";
import { eq } from "drizzle-orm";
import { and, eq } from "drizzle-orm";
import WebSocket, { WebSocketServer, type RawData } from "ws";
import { expect, it, vi } from "vitest";
import { completionContracts, environmentLeases, environments, executionWorkspaces, heartbeatRuns, issues, nativeRunFinalizations, nativeRunResults } from "@paperclipai/db";
import { completionContracts, environmentLeases, environments, executionWorkspaces, heartbeatRunEvents, heartbeatRuns, issues, nativeRunFinalizations, nativeRunResults } from "@paperclipai/db";
import { environmentTaskOperationSchema, type PluginEnvironmentTaskOperation } from "@paperclipai/plugin-sdk";
import { readRunnerdArtifactBinding, defaultCapabilityRunnerdBinary,
parseNativeExecutionInput, parseNativeRuntimeContext, QUALIFIED_ACPX_PROFILES, type DurableRecoveryIdentity, type RunnerProcessHandle, type RunnerProcessLaunchSpec } from "../../vendor/paperclip-runner/index.js";
@@ -23,6 +23,8 @@ import * as remoteWarmRecovery from "./remote-warm-session-recovery.js";
import { reconcileNativeFinalizations } from "./native-finalization-reconciler.js";
import { buildNativeCompletionContract } from "./completion-contracts.js";
import { nativeRuntimeContextFixture } from "./runtime-context.test-fixture.js";
import { PROCESS_START_REQUESTED } from "../native-local-process-stop.js";
import { PaperclipRunnerToolAuthority } from "./paperclip-runner-tool-authority.js";
const { createPrivateHostControlFixture } = await import(new URL("../../../../packages/paperclip-runner/test/support/private-host-control.ts", import.meta.url).href) as {
createPrivateHostControlFixture(): {
@@ -42,9 +44,11 @@ describeEmbeddedPostgres("plugin Runner release/resume through the Core executio
const environment = nativeRestartFixture();
const nativeIt = process.platform === "win32" ? it.skip : realProcessIt;
nativeIt.for([...(["codex", "claude"] as const).flatMap(harness => (["cached", "adopted"] as const).map(mode => [harness, mode] as const)), ["codex", "adopted cancellation"] as const, ["codex", "adopted legacy cancellation"] as const,
["codex", "cached cancellation"] as const, ["codex", "cached held claim"] as const, ["claude", "cached held claim"] as const,
["codex", "cached managed GitHub"] as const, ["codex", "adopted managed GitHub"] as const])("completes %s runs through a retained %s process", { timeout: 60_000 }, async ([harness, mode], context) => {
if (harness === "claude" && !existsSync(fakeAcpxSidecar)) context.skip();
const adopted = mode.startsWith("adopted"), cancellation = mode.includes("cancellation"), qualifiedCancellation = mode === "adopted cancellation";
const adopted = mode.startsWith("adopted"), cancellation = mode.includes("cancellation"), heldClaimMode = mode === "cached held claim";
const qualifiedCancellation = mode === "adopted cancellation" || mode === "cached cancellation";
const managedGitHub = mode.endsWith("managed GitHub"), brokerUrl = "https://core.example.test/api", brokerToken = (runId: string) => `fixture-broker-${runId}`;
const preparedBrokers: Array<{ runId: string; token?: string }> = [];
const first = await environment.seedRun(`RELEASE-ENTRY-${harness}-${mode}`);
@@ -121,6 +125,18 @@ describeEmbeddedPostgres("plugin Runner release/resume through the Core executio
const host = createPrivateHostControlFixture(), runnerRoot = join(root, "native"), binary = defaultCapabilityRunnerdBinary();
const artifact = readRunnerdArtifactBinding(binary), preparations = new Map<string, { operation: Extract<PluginEnvironmentTaskOperation, { kind: "prepare" }>; launch: { configurationId: string } }>();
const releases: DurableRecoveryIdentity[] = [], claims: DurableRecoveryIdentity[] = [];
const operationsDuringClaim: string[] = [];
let claimHeld = false, holdPreparationFor: string | undefined, claimedDuringPreparation = false;
const toolDefinitions = PaperclipRunnerToolAuthority.prototype.definitions;
const preparation = vi.spyOn(PaperclipRunnerToolAuthority.prototype, "definitions").mockImplementation(function (this: PaperclipRunnerToolAuthority) {
const definitions = toolDefinitions.call(this);
if (holdPreparationFor === undefined || this.binding.runId !== holdPreparationFor) return definitions;
holdPreparationFor = undefined;
return waitForCondition("claim during preparation", () => claims.length === 1)
.then(() => { claimedDuringPreparation = true; return definitions; }) as unknown as typeof definitions;
});
const processStartRequested = async (id: string) => (await db.select({ id: heartbeatRunEvents.id }).from(heartbeatRunEvents)
.where(and(eq(heartbeatRunEvents.runId, id), eq(heartbeatRunEvents.eventType, PROCESS_START_REQUESTED)))).length > 0;
let handle: RunnerProcessHandle | undefined, submissions = 0, stops = 0, completes = 0;
let releaseClaimReply!: () => void, observeHeldClaim!: () => void;
const claimReply = new Promise<void>(done => { releaseClaimReply = done; });
@@ -132,6 +148,7 @@ describeEmbeddedPostgres("plugin Runner release/resume through the Core executio
const sameSource = (a: DurableRecoveryIdentity, b: DurableRecoveryIdentity) => Object.entries(a).every(([key, value]) => b[key as keyof DurableRecoveryIdentity] === value);
const task = async (raw: unknown, authorityRunId: string): Promise<unknown> => {
const operation = environmentTaskOperationSchema.parse(raw);
if (claimHeld) operationsDuringClaim.push(operation.kind);
if (operation.kind === "describe") return { kind: "description", taskId, launchPreparationVersion: 1,
projectAuthCheckVersion: 1, warmTaskVersion: 1, releaseResumeVersion: 1, runner: artifact, ...(qualifiedCancellation ? { executionContextVersion: 1 } : {}),
...(managedGitHub ? { githubCredentialBrokerVersion: 1 } : {}) };
@@ -190,6 +207,13 @@ describeEmbeddedPostgres("plugin Runner release/resume through the Core executio
if (reply.status !== "claimed") throw new Error("Fixture native claim refused");
expect(await providerStarts()).toBe(retainedProviderCount);
claims.push(operation.identity);
if (heldClaimMode) {
expect(await processStartRequested(authorityRunId)).toBe(false);
claimHeld = true;
await waitForCondition("preparation completes while the claim is held", () => processStartRequested(authorityRunId));
await new Promise(done => setTimeout(done, 200));
claimHeld = false;
}
if (cancellation) { observeHeldClaim(); await claimReply; }
return { kind: "resume", taskId, outcome: "resumed" };
}
@@ -247,6 +271,7 @@ describeEmbeddedPostgres("plugin Runner release/resume through the Core executio
await db.insert(environmentLeases).values({ id: leaseB, companyId, environmentId, heartbeatRunId: runB, issueId, executionWorkspaceId: workspaceId,
provider: "plugin:fixture:tasks", providerLeaseId: taskId, status: "active", leasePolicy: "reuse_by_environment", metadata: { remoteCwd: workspace } });
const next = { ...execution, binding: { ...execution.binding, runId: runB } };
if (heldClaimMode) holdPreparationFor = runB;
pendingFollowup = execute(next, leaseB);
if (cancellation) {
void pendingFollowup.catch(() => undefined);
@@ -290,6 +315,7 @@ describeEmbeddedPostgres("plugin Runner release/resume through the Core executio
expect(adoption).toHaveBeenCalledTimes(1);
}
expect(claims).toEqual(releases.slice(0, -1)); expect(releases[1]!.runId).toBe(runB);
if (heldClaimMode) expect({ claimedDuringPreparation, operationsDuringClaim }).toEqual({ claimedDuringPreparation: true, operationsDuringClaim: [] });
expect(releases[1]!.environmentLeaseId).toBe(leaseA); expect(handle!.child.pid).toBe(pid); expect(submissions).toBe(1);
expect([...new Set(preparedBrokers.map(prepared => prepared.runId))]).toEqual(releases.map(release => release.runId));
const methods = (await readFile(calls, "utf8")).trim().split(/\r?\n/);
@@ -316,7 +342,7 @@ describeEmbeddedPostgres("plugin Runner release/resume through the Core executio
const failures = cleanup.filter((result): result is PromiseRejectedResult => result.status === "rejected");
if (failures.length) throw new AggregateError(failures.map(result => result.reason), "Fixture cleanup failed");
} finally {
adoption.mockRestore();
adoption.mockRestore(); preparation.mockRestore();
https.globalAgent.destroy(); https.globalAgent = oldAgent; setDefaultCACertificates(oldCAs);
}
}
server/src/services/native-runtime/status-arbiter.ts
+11 -3
View File
@@ -103,17 +103,25 @@ export function ownsNativeTaskCompletion(
return issue.assigneeAgentId === agentId && !issue.assigneeUserId;
}
export function heldByAnotherRun(issue: { executionRunId?: string | null }, runId: string) {
return Boolean(issue.executionRunId && issue.executionRunId !== runId);
}
export function supersedeStaleOwnerCompletion(
decision: NativeStatusDecision,
issue: { status: string; assigneeAgentId: string | null; assigneeUserId: string | null },
issue: { status: string; assigneeAgentId: string | null; assigneeUserId: string | null;
executionRunId?: string | null },
agentId: string,
runId?: string,
): NativeStatusDecision {
if (decision.statusAction !== "done" || ownsNativeTaskCompletion(issue, agentId)) return decision;
if (decision.statusAction !== "done") return decision;
const ownerSuperseded = !ownsNativeTaskCompletion(issue, agentId);
if (!ownerSuperseded && !(runId && heldByAnotherRun(issue, runId))) return decision;
return {
policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION,
statusAction: "preserve",
toStatus: issue.status as NativeAuthoritativeIssueStatus,
reasonCode: "completion_owner_superseded",
reasonCode: ownerSuperseded ? "completion_owner_superseded" : "completion_run_superseded",
unblockDescriptor: null,
effects: [],
};
server/src/services/native-runtime/status-decision-committer.ts
+4
View File
@@ -46,6 +46,7 @@ import {
workspaceOperations,
} from "@paperclipai/db";
import {
heldByAnotherRun,
NATIVE_STATUS_ARBITER_POLICY_VERSION,
type NativeGovernanceGate,
type NativeStatusDecision,
@@ -1793,6 +1794,9 @@ export async function commitNativeStatusDecision(input: {
if (input.decision.statusAction === "done" && !await onboardingFirstTaskCompletionAllowed(tx as unknown as Db, {
companyId: input.companyId, issueId: input.issueId, runId: input.runId,
})) throw new NativeStatusRaceError();
if (input.decision.statusAction === "done" && !isConversation(issue) && heldByAnotherRun(issue, input.runId)) {
throw new NativeStatusRaceError();
}
if (input.decision.statusAction === "done") {
try {
const [saved] = await tx.select({ context: heartbeatRuns.contextSnapshot, receipts: heartbeatRuns.resultJson,
server/src/services/plugin-agent-runtime-state.ts
+30 -16
View File
@@ -1,4 +1,4 @@
import { and, desc, eq } from "drizzle-orm";
import { and, desc, eq, sql } from "drizzle-orm";
import { agents, environments, plugins, pluginCompanySettings, resourceLifecycleEvents, fileStorageDb, type Db } from "@paperclipai/db";
import type { PluginAgentRuntimeState, PluginAgentRuntimeIdentity } from "@paperclipai/plugin-sdk";
import { agentRecordQueries } from "../lib/agent-records.js";
@@ -13,28 +13,25 @@ import { instanceSettingsService } from "./instance-settings.js";
import { agentInstanceSecretService } from "./agent-instance-secrets.js";
import { environmentSkillConfig } from "./environment-skills.js";
async function readState(db: Db, pluginId: string, companyId: string, agentId: string, identityOnly: true): Promise<PluginAgentRuntimeIdentity | null>;
async function readState(db: Db, pluginId: string, companyId: string, agentId: string, identityOnly?: false): Promise<Omit<PluginAgentRuntimeState, "skills" | "connectionAssignments"> | null>;
async function readState(db: Db, pluginId: string, companyId: string, agentId: string, identityOnly = false): Promise<PluginAgentRuntimeIdentity | Omit<PluginAgentRuntimeState, "skills" | "connectionAssignments"> | null> {
function assertOwningLifecyclePlugin(plugin: Pick<typeof plugins.$inferSelect, "status" | "manifestJson"> | null | undefined,
settings: Pick<typeof pluginCompanySettings.$inferSelect, "enabled"> | null | undefined, requiredPluginIds: string[] | null | undefined, pluginId: string) {
if (plugin?.status !== "ready" || settings?.enabled === false || !plugin?.manifestJson.capabilities?.includes("agents.lifecycle.manage")
|| !plugin.manifestJson.resourceLifecycle?.includes("agent_runtime") || !requiredPluginIds?.includes(pluginId)) {
throw forbidden("Agent runtime state requires the owning lifecycle plugin");
}
}
async function readState(db: Db, pluginId: string, companyId: string, agentId: string): Promise<Omit<PluginAgentRuntimeState, "skills" | "connectionAssignments"> | null> {
return fileStorageDb(db).transaction(async tx => {
const [agent] = await tx.select().from(agents)
.where(and(eq(agents.companyId, companyId), eq(agents.id, agentId))).for("update");
if (!agent) return null;
const [plugin] = await tx.select().from(plugins).where(eq(plugins.id, pluginId));
const [settings] = await tx.select().from(pluginCompanySettings).where(and(eq(pluginCompanySettings.companyId, companyId), eq(pluginCompanySettings.pluginId, pluginId)));
if (plugin?.status !== "ready" || settings?.enabled === false || !plugin?.manifestJson.capabilities?.includes("agents.lifecycle.manage")
|| !plugin.manifestJson.resourceLifecycle?.includes("agent_runtime") || !agent.lifecycleRequiredPluginIds?.includes(pluginId)) {
throw forbidden("Agent runtime state requires the owning lifecycle plugin");
}
assertOwningLifecyclePlugin(plugin, settings, agent.lifecycleRequiredPluginIds, pluginId);
const [latest] = await tx.select({ id: resourceLifecycleEvents.id }).from(resourceLifecycleEvents)
.where(and(eq(resourceLifecycleEvents.companyId, companyId), eq(resourceLifecycleEvents.resourceType, "agent_runtime"), eq(resourceLifecycleEvents.resourceId, agentId)))
.orderBy(desc(resourceLifecycleEvents.id)).limit(1);
if (identityOnly) return {
revision: String(latest?.id ?? 0),
agent: { id: agent.id, companyId: agent.companyId, status: agent.status as PluginAgentRuntimeIdentity["agent"]["status"],
lifecycleState: agent.lifecycleState, lifecycleVersion: agent.lifecycleVersion },
instanceSecrets: await agentInstanceSecretService(tx as unknown as Db).read({ companyId, agentId, pluginId }),
};
const instructions = await agentInstructionsService(tx as unknown as Db).getBundleLocation(agent);
const records = agentRecordQueries(tx as unknown as Db);
const companyAgents = await records.listCompanyAgentEligibilityRows(companyId);
@@ -46,9 +43,26 @@ async function readState(db: Db, pluginId: string, companyId: string, agentId: s
});
}
export async function pluginAgentRuntimeIdentity(db: Db, pluginId: string, companyId: string, agentId: string): Promise<PluginAgentRuntimeIdentity | null> {
return readState(db, pluginId, companyId, agentId, true);
const [row] = await db.select({
id: agents.id, companyId: agents.companyId, status: agents.status, lifecycleState: agents.lifecycleState,
lifecycleVersion: agents.lifecycleVersion, requiredPluginIds: agents.lifecycleRequiredPluginIds,
plugin: { status: plugins.status, manifestJson: plugins.manifestJson }, settings: { enabled: pluginCompanySettings.enabled },
revision: sql`(${db.select({ id: resourceLifecycleEvents.id }).from(resourceLifecycleEvents)
.where(and(eq(resourceLifecycleEvents.companyId, companyId), eq(resourceLifecycleEvents.resourceType, "agent_runtime"), eq(resourceLifecycleEvents.resourceId, agentId)))
.orderBy(desc(resourceLifecycleEvents.id)).limit(1)})`.mapWith(resourceLifecycleEvents.id),
}).from(agents)
.leftJoin(plugins, eq(plugins.id, pluginId))
.leftJoin(pluginCompanySettings, and(eq(pluginCompanySettings.companyId, companyId), eq(pluginCompanySettings.pluginId, pluginId)))
.where(and(eq(agents.companyId, companyId), eq(agents.id, agentId)));
if (!row) return null;
assertOwningLifecyclePlugin(row.plugin, row.settings, row.requiredPluginIds, pluginId);
return {
revision: String(row.revision ?? 0),
agent: { id: row.id, companyId: row.companyId, status: row.status as PluginAgentRuntimeIdentity["agent"]["status"],
lifecycleState: row.lifecycleState, lifecycleVersion: row.lifecycleVersion },
instanceSecrets: await agentInstanceSecretService(db).read({ companyId, agentId, pluginId }),
};
}
export async function pluginAgentRuntimeState(db: Db, pluginId: string, companyId: string, agentId: string, environmentId?: string, installEnvironmentSkills = false): Promise<PluginAgentRuntimeState | null> {
server/src/services/recovery/ownership-invariant-report.ts
+9 -4
View File
@@ -9,6 +9,7 @@ import {
import { taskClosedByAgentSql, TEMPORARY_REFUSAL_TRIGGERS } from "../work-intent-outbox.js";
import { deliveredQueuedCommentIds } from "../queued-comment-delivery.js";
import { logger } from "../../middleware/logger.js";
import { getServerProcessStartedAt } from "../../server-info.js";
import { persistActivity } from "../activity-log.js";
import { currentProcessProbe, proveOwnerGone, type OwnerProof, type ProcessProbe } from "../owner-proof.js";
import { LEGACY_OWNER_REASONS, recordVersionAttestations } from "./owner-gone-attestation.js";
@@ -466,7 +467,7 @@ async function checkStaleDroppedComments(db: Db, now: Date, limit: number): Prom
* finds a user comment on an assigned, non-conversation task with no live run
* or lock, newer than every run of the task, that no wake request carries.
* The reconciliation worker saves it as a deferred receipt and promotes it. */
async function checkUserCommentsWithoutWake(db: Db, now: Date, limit: number): Promise<{ violations: InvariantViolation[]; nextDueAt: Date | null }> {
async function checkUserCommentsWithoutWake(db: Db, now: Date, limit: number, processStartedAt: Date): Promise<{ violations: InvariantViolation[]; nextDueAt: Date | null }> {
const since = new Date(now.getTime() - INTENT_LOOKBACK_MS).toISOString();
const rows = await db.execute<{ id: string; company_id: string; issue_id: string; created_at: string }>(sql`
select c.id, c.company_id, c.issue_id, c.created_at::text as created_at from issue_comments c
@@ -498,8 +499,10 @@ async function checkUserCommentsWithoutWake(db: Db, now: Date, limit: number): P
or coalesce(w.payload#>'{_paperclipWakeContext,wakeCommentIds}', '[]'::jsonb) ? c.id::text))
order by c.created_at limit ${Math.min(limit, MAX_RECORDS_PER_KIND)}`);
// A comment inside the grace may still get its normal wake: check it again
// when the grace ends (a saved deadline), and report only older ones.
const graceEnd = now.getTime() - COMMENT_WAKE_GRACE_MS;
// when the grace ends (a saved deadline), and report only older ones. This
// process writes that wake after the comment commits, so a comment from
// before it started has no writer left and is owed now.
const graceEnd = Math.max(now.getTime() - COMMENT_WAKE_GRACE_MS, processStartedAt.getTime());
const young = [...rows].map(row => new Date(row.created_at).getTime()).filter(at => at >= graceEnd);
return {
violations: [...rows].filter(row => new Date(row.created_at).getTime() < graceEnd)
@@ -594,6 +597,7 @@ export async function runOwnershipReconciliationPass(db: Db, options: {
limitPerKind?: number;
probe?: ProcessProbe;
now?: Date;
processStartedAt?: Date;
/** Write new violations to the activity log. Defaults to true. */
record?: boolean;
/** Settle proven-gone records of kinds that register a `settle`, and surface
@@ -686,7 +690,8 @@ export async function runOwnershipReconciliationPass(db: Db, options: {
() => checkExactlyOnceIntents(db, state, now, limit),
() => checkStaleDroppedComments(db, now, limit),
async () => {
const result = await checkUserCommentsWithoutWake(db, now, limit);
const result = await checkUserCommentsWithoutWake(db, now, limit,
options.processStartedAt ?? new Date(getServerProcessStartedAt()));
if (result.nextDueAt && (!nextDueAt || result.nextDueAt < nextDueAt)) nextDueAt = result.nextDueAt;
return result.violations;
},
server/src/services/secret-proposals.ts
+38 -1
View File
@@ -446,6 +446,13 @@ export function createSecretProposalsService(db: Db) {
return createWithinQuota(
{ companyId: context.companyId, agentId: run.agentId, runId: run.id, issueId: originIssueId },
async (txDb) => {
if (resolvedSecretId) {
// Hold the secret against a concurrent delete: the delete marks it
// deleted first, then settles the proposals that reference it.
const live = await txDb.select({ status: companySecrets.status }).from(companySecrets)
.where(eq(companySecrets.id, resolvedSecretId)).for("share").then((rows) => rows[0] ?? null);
if (!live || live.status === "deleted") throw notFound("Secret not found");
}
if (input.secretProposalId) {
const dependency = await txDb.select().from(companySecretProposals).where(and(
eq(companySecretProposals.id, input.secretProposalId),
@@ -915,5 +922,35 @@ export function createSecretProposalsService(db: Db) {
return row?.at == null ? null : new Date(row.at).getTime();
}
return { getById, view: enrich, createSecret, createBinding, listForAgent, listForBoard, assertBindingSnapshotCurrent, approve, transition, sweepExpired, sweepExpiredForCompany, expiryDeadlines, nextExpiryAt };
/**
* Deleting a secret also deletes the binding proposals that reference it.
* Settle pending ones first, so their review cards close as rejected with
* the reason instead of pointing at a proposal that no longer exists. Call
* it after the secret is marked deleted: createBinding holds the secret row
* and refuses a deleted one, so no new pending proposal can follow.
*/
async function rejectPendingBindingsForSecret(companyId: string, secretId: string, input: { resolvedByUserId?: string | null; reason: string }) {
const pending = await db.select().from(companySecretProposals).where(and(
eq(companySecretProposals.companyId, companyId),
eq(companySecretProposals.secretId, secretId),
eq(companySecretProposals.kind, "binding"),
eq(companySecretProposals.status, "pending"),
));
const rejected: Proposal[] = [];
for (const row of pending) {
try {
await transition(companyId, row.id, "rejected", {
resolvedByUserId: input.resolvedByUserId ?? null,
reason: input.reason,
});
rejected.push(row);
} catch (error) {
// Resolved concurrently; the secret deletion still removes the row.
if (!(error instanceof HttpError) || error.status !== 409) throw error;
}
}
return rejected;
}
return { getById, view: enrich, createSecret, createBinding, listForAgent, listForBoard, assertBindingSnapshotCurrent, approve, transition, rejectPendingBindingsForSecret, sweepExpired, sweepExpiredForCompany, expiryDeadlines, nextExpiryAt };
}
server/src/services/secrets.ts
+23 -11
View File
@@ -970,6 +970,11 @@ function assertSelectableProviderConfig(config: {
}
}
export type RemoveSecretOptions = {
/** Runs after the secret is marked deleted and before its row is removed. */
beforeDelete?: (secret: typeof companySecrets.$inferSelect) => Promise<void>;
};
export function secretService(db: Db | DbTransaction) {
const authorization = authorizationService(db);
@@ -1359,10 +1364,11 @@ export function secretService(db: Db | DbTransaction) {
version: number | "latest",
options?: SecretResolutionOptions,
readOnlyEnvironmentInspection = false,
preloaded?: { secret: typeof companySecrets.$inferSelect; version: typeof companySecretVersions.$inferSelect | null },
): Promise<RuntimeSecretResolution> {
const bindingContext = options?.bindingContext;
const accessContext = options?.accessContext ?? bindingContext;
const secret = await getById(secretId);
const secret = preloaded && preloaded.secret.id === secretId ? preloaded.secret : await getById(secretId);
if (!secret) throw notFound("Secret not found");
if (secret.companyId !== companyId) throw unprocessable("Secret must belong to same company");
const instanceOwned = secret.scope === "instance" && options?.instanceOwner &&
@@ -1384,7 +1390,8 @@ export function secretService(db: Db | DbTransaction) {
throw unprocessable("Secret is not active", { code: "secret_inactive" });
}
const binding = await assertBindingContext(companyId, secret.id, bindingContext);
const versionRow = await getSecretVersion(secret.id, resolvedVersion);
const versionRow = preloaded?.secret === secret && preloaded.version?.version === resolvedVersion
? preloaded.version : await getSecretVersion(secret.id, resolvedVersion);
if (!versionRow) throw new HttpError(404, "Secret version not found", { code: "version_missing" });
if (versionRow.status === "disabled" || versionRow.status === "destroyed" || versionRow.revokedAt) {
throw unprocessable("Secret version is not active", { code: "version_inactive" });
@@ -3399,18 +3406,18 @@ export function secretService(db: Db | DbTransaction) {
// `removeSecretUnlocked` re-reads the secret under the lock, so a delete
// that lost a race between the pre-check and the lock still sees current
// state, not the pre-check's stale read.
async function removeSecretInternal(secretId: string) {
async function removeSecretInternal(secretId: string, options: RemoveSecretOptions = {}) {
const preCheckSecret = await getById(secretId);
if (!preCheckSecret) return null;
return withAccountHomeSecretMutationLock(undefined, preCheckSecret.companyId, () =>
removeSecretUnlocked(secretId),
removeSecretUnlocked(secretId, options),
);
}
// The body of `removeSecretInternal` above, unchanged. Extracted to a named
// function so that wrapper can hold the lock across this whole delete
// sequence without duplicating it.
async function removeSecretUnlocked(secretId: string) {
async function removeSecretUnlocked(secretId: string, options: RemoveSecretOptions = {}) {
const secret = await getById(secretId);
if (!secret) return null;
const versionRow = await getSecretVersion(secret.id, secret.latestVersion);
@@ -3428,6 +3435,9 @@ export function secretService(db: Db | DbTransaction) {
})
.where(eq(companySecrets.id, secretId));
}
// New references now see the deleted status, so the caller can settle
// every reference that exists before the row and its cascades go.
await options.beforeDelete?.(secret);
const providerConfig = secret.providerConfigId
? await getProviderConfigById(secret.providerConfigId)
: null;
@@ -3513,16 +3523,18 @@ export function secretService(db: Db | DbTransaction) {
const instanceSecrets = {
async read(scope: InstanceSecretScope): Promise<Record<string, string>> {
const rows = await db.select().from(companySecrets).where(and(eq(companySecrets.companyId, scope.companyId),
eq(companySecrets.scope, "instance"), scope.agentId !== undefined ? eq(companySecrets.ownerAgentId, scope.agentId) : eq(companySecrets.ownerProjectId, scope.projectId),
eq(companySecrets.ownerPluginId, scope.pluginId), ne(companySecrets.status, "deleted")));
const rows = await db.select({ secret: companySecrets, version: companySecretVersions }).from(companySecrets)
.leftJoin(companySecretVersions, and(eq(companySecretVersions.secretId, companySecrets.id), eq(companySecretVersions.version, companySecrets.latestVersion)))
.where(and(eq(companySecrets.companyId, scope.companyId),
eq(companySecrets.scope, "instance"), scope.agentId !== undefined ? eq(companySecrets.ownerAgentId, scope.agentId) : eq(companySecrets.ownerProjectId, scope.projectId),
eq(companySecrets.ownerPluginId, scope.pluginId), ne(companySecrets.status, "deleted")));
const values: Record<string, string> = Object.create(null);
for (const row of rows) {
const resolved = await resolveSecretValueInternal(scope.companyId, row.id, "latest", {
const resolved = await resolveSecretValueInternal(scope.companyId, row.secret.id, "latest", {
instanceOwner: scope, accessContext: { consumerType: "plugin_worker", consumerId: scope.pluginId,
actorType: "plugin", actorId: scope.pluginId, pluginId: scope.pluginId },
});
values[row.name] = resolved.value;
}, false, row);
values[row.secret.name] = resolved.value;
}
return values;
},
tests/restart-chaos/api.test.ts
+58
View File
@@ -0,0 +1,58 @@
import { createServer, type Server } from "node:http";
import type { AddressInfo } from "node:net";
import { afterEach, describe, expect, it } from "vitest";
import { ChaosApi, ChaosApiError, rateLimitDelayMs } from "./api.js";
let server: Server | null = null;
afterEach(async () => {
await new Promise<void>(resolve => server ? server.close(() => resolve()) : resolve());
server = null;
});
async function serve(handler: (count: number) => { status: number; body: unknown }) {
let count = 0;
const arrivals: number[] = [];
server = createServer((_request, response) => {
count += 1;
arrivals.push(performance.now());
const reply = handler(count);
response.writeHead(reply.status, { "content-type": "application/json" });
response.end(JSON.stringify(reply.body));
});
await new Promise<void>(resolve => server!.listen(0, "127.0.0.1", () => resolve()));
return { url: `http://127.0.0.1:${(server!.address() as AddressInfo).port}`, count: () => count, arrivals };
}
describe("ChaosApi rate limits", () => {
it("retries a GET the server rate-limited, after the delay it asked for", async () => {
const target = await serve(count => count <= 2
? { status: 429, body: { error: "Too many concurrent issue-list requests", retryAfterSeconds: 0.05 } }
: { status: 200, body: [{ id: "c1" }] });
await expect(new ChaosApi(target.url).get("/api/issues")).resolves.toEqual([{ id: "c1" }]);
expect(target.count()).toBe(3);
const [first, second, third] = target.arrivals;
expect(second! - first!).toBeGreaterThanOrEqual(45);
expect(third! - second!).toBeGreaterThanOrEqual(95);
});
it("does not repeat a write the server rate-limited", async () => {
const target = await serve(() => ({ status: 429, body: { retryAfterSeconds: 0.01 } }));
await expect(new ChaosApi(target.url).post("/api/issues/i1/comments", { body: "x" })).rejects.toBeInstanceOf(ChaosApiError);
expect(target.count()).toBe(1);
});
it("gives up after a bounded number of attempts", async () => {
const target = await serve(() => ({ status: 429, body: { retryAfterSeconds: 0.01 } }));
await expect(new ChaosApi(target.url, 10_000, 3).get("/api/issues")).rejects.toThrow(/returned 429/);
expect(target.count()).toBe(3);
});
it("reads the delay from Retry-After or the body, capped, and defaults to one second", () => {
const response = (status: number, retryAfter: string | null = null) => ({ status, headers: { get: () => retryAfter } });
expect(rateLimitDelayMs(response(200), "")).toBeNull();
expect(rateLimitDelayMs(response(429, "2"), "")).toBe(2_000);
expect(rateLimitDelayMs(response(429), JSON.stringify({ retryAfterSeconds: 3 }))).toBe(3_000);
expect(rateLimitDelayMs(response(429, "600"), "")).toBe(10_000);
expect(rateLimitDelayMs(response(429), "not json")).toBe(1_000);
});
});
tests/restart-chaos/api.ts
+27 -3
View File
@@ -8,10 +8,29 @@ export class ChaosApiError extends Error {
constructor(readonly status: number | null, message: string) { super(message); this.name = "ChaosApiError"; }
}
export function rateLimitDelayMs(response: { status: number; headers: { get(name: string): string | null } }, text: string): number | null {
if (response.status !== 429) return null;
let seconds = Number(response.headers.get("retry-after"));
if (!Number.isFinite(seconds) || seconds <= 0) {
try { seconds = Number((JSON.parse(text) as { retryAfterSeconds?: unknown }).retryAfterSeconds); } catch { seconds = NaN; }
}
return Number.isFinite(seconds) && seconds > 0 ? Math.min(10_000, seconds * 1_000) : 1_000;
}
export class ChaosApi {
constructor(readonly baseUrl: string, private readonly timeoutMs = 10_000) {}
constructor(readonly baseUrl: string, private readonly timeoutMs = 10_000, private readonly rateLimitAttempts = 5) {}
private async request<T>(method: string, route: string, body?: unknown, timeoutMs = this.timeoutMs): Promise<T> {
for (let attempt = 1; ; attempt += 1) {
const outcome = await this.attempt<T>(method, route, body, timeoutMs);
if (!("retryAfterMs" in outcome)) return outcome.value;
if (method !== "GET" || attempt >= this.rateLimitAttempts) throw outcome.error;
await new Promise(resolve => setTimeout(resolve, Math.min(10_000, outcome.retryAfterMs * attempt)));
}
}
private async attempt<T>(method: string, route: string, body: unknown, timeoutMs: number):
Promise<{ value: T } | { retryAfterMs: number; error: ChaosApiError }> {
let response: Response;
try {
response = await fetch(`${this.baseUrl}${route}`, {
@@ -25,8 +44,13 @@ export class ChaosApi {
throw new ChaosApiError(null, `${method} ${route} unreachable: ${error instanceof Error ? error.message : String(error)}`);
}
const text = await response.text();
if (!response.ok) throw new ChaosApiError(response.status, `${method} ${route} returned ${response.status}: ${text.slice(0, 500)}`);
return (text ? JSON.parse(text) : null) as T;
if (!response.ok) {
const error = new ChaosApiError(response.status, `${method} ${route} returned ${response.status}: ${text.slice(0, 500)}`);
const retryAfterMs = rateLimitDelayMs(response, text);
if (retryAfterMs !== null) return { retryAfterMs, error };
throw error;
}
return { value: (text ? JSON.parse(text) : null) as T };
}
get<T>(route: string, timeoutMs?: number) { return this.request<T>("GET", route, undefined, timeoutMs); }
tests/restart-chaos/main.ts
+2 -1
View File
@@ -469,7 +469,8 @@ async function main(): Promise<number> {
}
await writeFile(path.join(waveDir, "faults.json"), `${JSON.stringify(faults, null, 2)}\n`, { mode: 0o600 });
const speed = await (async () => {
const childTasks = (await Promise.all(launched.map(observer => observer.childSpeedTasks()))).flat();
const childTasks: Awaited<ReturnType<TaskObserver["childSpeedTasks"]>> = [];
for (const observer of launched) childTasks.push(...await observer.childSpeedTasks());
return collectWaveSpeed(api, plan.options.env, [...launched.flatMap(observer => observer.record.issueId
? [{ issueId: observer.record.issueId, adapter: observer.plan.adapter, samples: observer.record.samples }] : []), ...childTasks], faults);
})().catch(error => { timeline.add("wave_speed_failed", { error: String(error) }); anyFailure = true; return null; });
tests/restart-chaos/ttft-baseline-stats.test.ts
+34 -2
View File
@@ -119,12 +119,44 @@ describe("ttft baseline statistics", () => {
]);
const silent = run("r2", "restart", []);
expect(recoverySamples([recovered, silent], [{ kind: "K1", readyAt, runIds: ["r1", "r2", "missing"] }])).toEqual([
{ runId: "r1", adapter: "codex", kind: "K1", disposition: "resume_dead_runner", readyAt, latencyMs: 3_250 },
{ runId: "r2", adapter: "codex", kind: "K1", disposition: "none", readyAt, latencyMs: null },
{ runId: "r1", adapter: "codex", kind: "K1", disposition: "resume_dead_runner", readyAt, latencyMs: 3_250, controlPlaneMs: 3_250, providerLoadMs: 0 },
{ runId: "r2", adapter: "codex", kind: "K1", disposition: "none", readyAt, latencyMs: null, controlPlaneMs: null, providerLoadMs: null },
]);
});
});
describe("restart recovery split", () => {
// The trace starts at the epoch; the queue span is written promptly and anchors it.
const queue = { eventType: "run.performance.span", createdAt: at(100), payload: { span: "heartbeat.queue", startOffsetMs: 0, durationMs: 100 } };
const load = (writtenAt: number) => ({ eventType: "run.performance.span", createdAt: at(writtenAt),
payload: { span: "provider.session.load", startOffsetMs: 11_500, durationMs: 4_500 } });
it("splits a recovery at the start of the provider's session load", () => {
const readyAt = Date.parse(at(10_000));
const recovered = run("r1", "restart", [queue, load(16_000), { eventType: "session.resumed", createdAt: at(16_400) }]);
expect(recoverySamples([recovered], [{ kind: "K2", readyAt, runIds: ["r1"] }])).toEqual([
{ runId: "r1", adapter: "codex", kind: "K2", disposition: "none", readyAt, latencyMs: 6_400, controlPlaneMs: 1_500, providerLoadMs: 4_900 },
]);
});
it("uses the start the server logged even when every span row was written late", () => {
const readyAt = Date.parse(at(10_000));
// All rows deferred by 8 s, so no write time can anchor the trace.
const late = (span: string, offset: number, duration: number) => ({ eventType: "run.performance.span", createdAt: at(offset + duration + 8_000),
payload: { span, startOffsetMs: offset, durationMs: duration, wallStartedAtMs: Date.parse(at(offset)) } });
const recovered = run("r1", "restart", [late("heartbeat.queue", 0, 100), { eventType: "session.resumed", createdAt: at(16_400) },
late("provider.session.load", 11_500, 4_500)]);
expect(recoverySamples([recovered], [{ kind: "K2", readyAt, runIds: ["r1"] }])[0]).toMatchObject({ controlPlaneMs: 1_500, providerLoadMs: 4_900 });
});
it("takes the load start from the span's offset, not its (possibly deferred) write time", () => {
const readyAt = Date.parse(at(10_000));
// The load span's run-log row is written 5 s late, after the provider event.
const recovered = run("r1", "restart", [queue, { eventType: "session.resumed", createdAt: at(16_400) }, load(21_000)]);
expect(recoverySamples([recovered], [{ kind: "K2", readyAt, runIds: ["r1"] }])[0]).toMatchObject({ controlPlaneMs: 1_500, providerLoadMs: 4_900 });
});
});
describe("restart sample guard", () => {
const tool = (eventType: string, executionId: string, extra: Record<string, unknown> = {}): BaselineEvent => ({
eventType, createdAt: at(0), payload: { prpEvent: { payload: { executionId, ...extra } } },
tests/restart-chaos/ttft-baseline-stats.ts
+45 -2
View File
@@ -132,9 +132,42 @@ export function runMetrics(run: BaselineRun): RunMetrics {
const PROVIDER_EVENT_TYPES = new Set(["turn.submitted", "turn.started", "turn.accepted", "item.started", "item.delta",
"item.completed", "tool.execution.started", "tool.execution.progressed", "tool.execution.completed", "session.resumed"]);
export interface RecoverySample { runId: string; adapter: string; kind: string; disposition: string; readyAt: number; latencyMs: number | null }
export interface RecoverySample { runId: string; adapter: string; kind: string; disposition: string; readyAt: number; latencyMs: number | null;
/** Paperclip's share: ready to the start of the provider's session load (`provider.session.load`), or the whole
* latency when nothing reloads. The judged row. */
controlPlaneMs: number | null;
/** The provider's share: session load start to the next provider event. Reported. */
providerLoadMs: number | null }
/** Control plane ready → the recovered run's next provider event, keyed by runner disposition. */
/** The run's trace start: every span records its offset from it. A span event is
* written when the span ends, possibly through deferred work, so each one gives
* an upper bound (write time minus offset and duration); the earliest is used. */
export function spanTraceStartMs(run: BaselineRun): number | null {
let earliest: number | null = null;
for (const event of run.events) {
if (event.eventType !== "run.performance.span" || !event.payload) continue;
const candidate = Date.parse(event.createdAt) - (Number(event.payload.startOffsetMs) + Number(event.payload.durationMs));
if (Number.isFinite(candidate) && (earliest === null || candidate < earliest)) earliest = candidate;
}
return earliest;
}
/** Stamps each provider session load with its wall-clock start (`wallStartMs`).
* The server writes the span's own start as `wallStartedAtMs`; runs logged before
* that field existed fall back to the trace start plus the span's offset, never to
* its write time. Call on the raw run, before outage removal rewrites offsets. */
export function stampProviderLoadStarts(run: BaselineRun): BaselineRun {
const anchor = spanTraceStartMs(run);
return { ...run, events: run.events.map(event => {
if (event.eventType !== "run.performance.span" || event.payload?.span !== "provider.session.load" || event.payload.wallStartMs !== undefined) return event;
const logged = Number(event.payload.wallStartedAtMs);
const offset = Number(event.payload.startOffsetMs);
const start = Number.isFinite(logged) ? logged : anchor !== null && Number.isFinite(offset) ? anchor + offset : null;
return start === null ? event : { ...event, payload: { ...event.payload, wallStartMs: start } };
}) };
}
export function recoverySamples(runs: readonly BaselineRun[], faults: readonly BaselineFault[]): RecoverySample[] {
const byId = new Map(runs.map(run => [run.runId, run]));
const samples: RecoverySample[] = [];
@@ -146,9 +179,19 @@ export function recoverySamples(runs: readonly BaselineRun[], faults: readonly B
const transition = after.find(event => event.eventType === "native.recovery.transition"
&& Date.parse(event.createdAt) >= fault.readyAt - 60_000);
const next = run.events.find(event => PROVIDER_EVENT_TYPES.has(event.eventType) && Date.parse(event.createdAt) >= fault.readyAt);
const nextAt = next ? Date.parse(next.createdAt) : null;
const stamped = stampProviderLoadStarts(run);
const loadStarts = nextAt === null ? [] : stamped.events.flatMap(event => {
if (event.eventType !== "run.performance.span" || event.payload?.span !== "provider.session.load") return [];
const start = Number(event.payload.wallStartMs);
return Number.isFinite(start) && start >= fault.readyAt && start <= nextAt ? [start] : [];
});
const loadStart = loadStarts.length ? Math.min(...loadStarts) : null;
samples.push({ runId, adapter: run.adapter, kind: fault.kind, readyAt: fault.readyAt,
disposition: String(transition?.payload?.runnerDisposition ?? "none"),
latencyMs: next ? Date.parse(next.createdAt) - fault.readyAt : null });
latencyMs: nextAt === null ? null : nextAt - fault.readyAt,
controlPlaneMs: nextAt === null ? null : (loadStart ?? nextAt) - fault.readyAt,
providerLoadMs: nextAt === null ? null : loadStart === null ? 0 : nextAt - loadStart });
}
}
return samples;
tests/restart-chaos/ttft-baseline.ts
+3 -2
View File
@@ -204,7 +204,7 @@ function report(runs: BaselineRun[], faults: BaselineFault[]): string {
lines.push("", "| Pre-turn step | n | p50 | p95 | Adapters |", "|---|---|---|---|---|");
for (const step of rankPreTurnSteps(runs).slice(0, 25)) lines.push(`| \`${step.step}\` | ${step.n} | ${ms(step.p50)} | ${ms(step.p95)} | ${step.adapters.join(", ")} |`);
const samples = recoverySamples(runs, faults);
lines.push("", "| Adapter | Fault | Disposition | n | Recovery p50 | Recovery p95 | No provider event |", "|---|---|---|---|---|---|---|");
lines.push("", "| Adapter | Fault | Disposition | n | Recovery p50 | Recovery p95 | Paperclip share p50 / p95 | No provider event |", "|---|---|---|---|---|---|---|---|");
const groups = new Map<string, typeof samples>();
for (const sample of samples) {
const key = `${sample.adapter}|${sample.kind}|${sample.disposition}`;
@@ -213,7 +213,8 @@ function report(runs: BaselineRun[], faults: BaselineFault[]): string {
for (const [key, group] of [...groups.entries()].sort()) {
const [adapter, kind, disposition] = key.split("|");
const d = distribution(group.flatMap(sample => sample.latencyMs === null ? [] : [sample.latencyMs]));
lines.push(`| ${adapter} | ${kind} | ${disposition} | ${d.n} | ${ms(d.p50)} | ${ms(d.p95)} | ${group.length - d.n} |`);
const control = distribution(group.flatMap(sample => sample.controlPlaneMs === null ? [] : [sample.controlPlaneMs]));
lines.push(`| ${adapter} | ${kind} | ${disposition} | ${d.n} | ${ms(d.p50)} | ${ms(d.p95)} | ${ms(control.p50)} / ${ms(control.p95)} | ${group.length - d.n} |`);
}
return `${lines.join("\n")}\n`;
}
tests/restart-chaos/wave-speed.test.ts
+18 -2
View File
@@ -45,7 +45,23 @@ describe("wave speed grade", () => {
const recovered: BaselineRun = { runId: "r", adapter: "claude", environment: "local", phase: "cold",
events: [{ eventType: "turn.started", createdAt: new Date(10_000 + 30_000).toISOString() }] };
const speed = gradeWaveSpeed("local", [recovered], [{ kind: "K1", readyAt: 10_000, runIds: ["r"] }], 0);
expect(speed.rows.find(row => row.metric === "restart_recovery")).toMatchObject({ adapter: "claude", breaches: ["p50", "p95"] });
// Nothing reloaded, so the whole latency is Paperclip's share and is judged.
expect(speed.rows.find(row => row.metric === "restart_recovery_control_plane")).toMatchObject({ adapter: "claude", breaches: ["p50", "p95"] });
expect(speed.rows.find(row => row.metric === "restart_recovery")).toMatchObject({ budget: null, breaches: [], measured: { p50: 30_000 } });
});
it("judges Paperclip's share of a recovery and only reports the provider's session load", () => {
const loadEnd = 10_000 + 1_500 + 9_000;
const recovered: BaselineRun = { runId: "r", adapter: "codex", environment: "local", phase: "cold", events: [
{ eventType: "run.performance.span", createdAt: new Date(100).toISOString(), payload: { span: "heartbeat.queue", startOffsetMs: 0, durationMs: 100 } },
{ eventType: "run.performance.span", createdAt: new Date(loadEnd).toISOString(),
payload: { span: "provider.session.load", startOffsetMs: 11_500, durationMs: 9_000 } },
{ eventType: "session.resumed", createdAt: new Date(loadEnd + 300).toISOString() }] };
const speed = gradeWaveSpeed("local", [recovered], [{ kind: "K2", readyAt: 10_000, runIds: ["r"] }], 0);
expect(speed.rows.find(row => row.metric === "restart_recovery_control_plane")).toMatchObject({ measured: { p50: 1_500 }, breaches: [] });
expect(speed.rows.find(row => row.metric === "restart_provider_load")).toMatchObject({ measured: { p50: 9_300 }, budget: null, breaches: [] });
expect(speed.rows.find(row => row.metric === "restart_recovery")).toMatchObject({ measured: { p50: 10_800 }, budget: null });
expect(speed.breaches).toBe(0);
});
it("removes a later outage from restart recovery", () => {
@@ -54,7 +70,7 @@ describe("wave speed grade", () => {
const windows = outageWindows([{ startedAt: 12_000, readyAt: 312_000 } as FaultReceipt]);
const speed = gradeWaveSpeed("local", [recovered], [{ kind: "K1", readyAt: 10_000, runIds: ["r"] }], 0, [],
{ adjusted: 0, removedMs: 0, windows });
expect(speed.rows.find(row => row.metric === "restart_recovery")).toMatchObject({ measured: { p50: 2_000 }, breaches: [] });
expect(speed.rows.find(row => row.metric === "restart_recovery_control_plane")).toMatchObject({ measured: { p50: 2_000 }, breaches: [] });
});
it("has no budget for remote environments or unbudgeted adapters", () => {
tests/restart-chaos/wave-speed.ts
+26 -10
View File
@@ -1,7 +1,7 @@
import { LOCAL_SPEED_BUDGETS, type SpeedBudget, type SpeedBudgetAdapter } from "../../server/src/services/native-runtime/native-speed-budgets.js";
import type { ChaosApi } from "./api.js";
import {
baselineRows, distribution, recoverySamples, runMetrics, runSpans, runWarmth,
baselineRows, distribution, recoverySamples, runMetrics, runSpans, runWarmth, stampProviderLoadStarts,
type BaselineEvent, type BaselineFault, type BaselineRun, type Distribution,
} from "./ttft-baseline-stats.js";
import type { AdapterId, EnvironmentKind, FaultReceipt, RunSample } from "./types.js";
@@ -11,7 +11,8 @@ type Json = Record<string, unknown>;
export interface WaveSpeedTask { issueId: string | null; adapter: AdapterId; samples: Array<{ runs?: RunSample[] }> }
export interface SpeedRow {
adapter: string; metric: "time_to_first_activity" | "ttft" | "agent_boot" | "workspace_provision" | "restart_recovery"; start: "cold" | "warm" | "restart";
adapter: string; metric: "time_to_first_activity" | "ttft" | "agent_boot" | "workspace_provision" | "restart_recovery"
| "restart_recovery_control_plane" | "restart_provider_load"; start: "cold" | "warm" | "restart";
measured: Distribution; budget: SpeedBudget | null; breaches: Array<"p50" | "p95">;
}
@@ -33,9 +34,11 @@ export function budgetFor(environment: EnvironmentKind, adapter: string, metric:
start: SpeedRow["start"]): SpeedBudget | null {
const key = BUDGET_ADAPTER[adapter as AdapterId];
if (environment !== "local" || !key) return null;
if (metric === "ttft" || metric === "workspace_provision") return null;
// Reported without a budget: the full recovery and the provider's session load
// grow with session size and simultaneous resumes; Paperclip's share is judged.
if (metric === "ttft" || metric === "workspace_provision" || metric === "restart_recovery" || metric === "restart_provider_load") return null;
const budgets = LOCAL_SPEED_BUDGETS[key] as unknown as Record<string, Record<"cold" | "warm", SpeedBudget> | SpeedBudget>;
if (metric === "restart_recovery") return budgets.restart_recovery as SpeedBudget;
if (metric === "restart_recovery_control_plane") return budgets.restart_recovery_control_plane as SpeedBudget;
const byStart = (metric === "time_to_first_activity" ? budgets.time_to_first_activity ?? budgets.ttft : budgets[metric]) as
Record<"cold" | "warm", SpeedBudget>;
return byStart[start === "warm" ? "warm" : "cold"];
@@ -180,12 +183,24 @@ export function gradeWaveSpeed(environment: EnvironmentKind, runs: readonly Base
}
}
const clock = outageFreeClock(outage.windows ?? []);
const samples = recoverySamples(runs, faults).map(sample => sample.latencyMs === null ? sample
: { ...sample, latencyMs: Math.max(0, clock(sample.readyAt + sample.latencyMs) - clock(sample.readyAt)) });
const outageFree = (readyAt: number, ms: number | null) => ms === null ? null : Math.max(0, clock(readyAt + ms) - clock(readyAt));
const samples = recoverySamples(runs, faults).map(sample => sample.latencyMs === null ? sample : {
...sample, latencyMs: outageFree(sample.readyAt, sample.latencyMs),
controlPlaneMs: outageFree(sample.readyAt, sample.controlPlaneMs),
providerLoadMs: sample.providerLoadMs === null || sample.controlPlaneMs === null ? sample.providerLoadMs
: Math.max(0, clock(sample.readyAt + sample.controlPlaneMs + sample.providerLoadMs) - clock(sample.readyAt + sample.controlPlaneMs)),
});
for (const adapter of [...new Set(samples.map(sample => sample.adapter))].sort()) {
const measured = distribution(samples.flatMap(sample => sample.adapter === adapter && sample.latencyMs !== null ? [sample.latencyMs] : []));
const budget = budgetFor(environment, adapter, "restart_recovery", "restart");
rows.push({ adapter, metric: "restart_recovery", start: "restart", measured, budget, breaches: breachesOf(measured, budget) });
for (const [metric, pick] of [["restart_recovery", (sample: typeof samples[number]) => sample.latencyMs],
["restart_recovery_control_plane", (sample: typeof samples[number]) => sample.controlPlaneMs],
["restart_provider_load", (sample: typeof samples[number]) => sample.providerLoadMs]] as const) {
const measured = distribution(samples.flatMap(sample => {
const value = sample.adapter === adapter ? pick(sample) : null;
return value === null ? [] : [value];
}));
const budget = budgetFor(environment, adapter, metric, "restart");
rows.push({ adapter, metric, start: "restart", measured, budget, breaches: breachesOf(measured, budget) });
}
}
const singleRunBreaches = runs.filter(run => run.phase !== "restart" && run.phase !== "verify").filter(run => {
const start = runWarmth(run);
@@ -248,7 +263,8 @@ export async function collectWaveSpeed(api: ChaosApi, environment: EnvironmentKi
}
if (run.retryOfRunId && faultHit.has(run.retryOfRunId)) raw.phase = "restart";
if (faultBeforeFirstActivity(raw, windows, Date.parse(run.createdAt!))) raw.phase = "restart";
const trimmed = withoutOutage(raw, windows);
// Stamp provider load starts from raw offsets before outage removal rewrites them.
const trimmed = withoutOutage(stampProviderLoadStarts(raw), windows);
outage.adjusted += trimmed.adjusted; outage.removedMs += trimmed.removedMs;
runs.push(trimmed.run);
timing.push({ id: run.id, createdAt: Date.parse(run.createdAt!), finishedAt: run.finishedAt ? Date.parse(run.finishedAt) : null });