mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-10 12:37:55 +01:00
docs(plans): Phase 0 ownership inventory and TTFT baseline (#19846)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A control-plane restart can land between two durable writes. It leaves an in-progress record owned by a dead process. About 69 restart-chaos PRs fixed that shape one subsystem at a time. > - The durable ownership plan (#19831) replaces those fixes with one owner stamp, one prover, write-ahead intents and one reconciler. Phase 0 must first list every in-progress record kind and measure how fast turns start today. > - Without the inventory, Phase 1 cannot choose one stamp or one proof. Without the baseline, no later phase can show that it kept turns fast (invariant 8). > - This pull request adds the Phase 0 inventory, the TTFT, boot and restart-recovery baseline, and the run-log-only driver that produced the baseline. > - The benefit is one reviewed map of 83 record kinds to plan sections A–G, and measured p50/p95 budgets per adapter. ## Linked Issues or Issue Description Refs #19831. Refs #19838 (the TTFT anchor that the baseline uses). **What is the problem?** Phase 0 of the durable ownership plan has no inventory of in-progress record kinds and no speed baseline. **What is the expected outcome?** Two reviewed documents in `doc/plans/`, plus a repeatable way to measure the baseline again in later phases. **Why does it matter?** Phase 1 cannot start without them, and every later phase's exit criteria compare against the baseline. ## What Changed - `doc/plans/2026-11-07-ownership-inventory.md`: - 83 durable in-progress record kinds (94 reader rows) across native runtime, environments, workspaces, accounting and the wake queue; - each kind with its table and column, writer, states, owner, release rule, proof, obligation, invariants and plan sections A–G; - the overlaps, the merge decisions and the zero-downtime upgrade record kinds. - `doc/plans/2026-11-07-ttft-baseline.md`: - TTFT, agent boot, pre-turn steps and restart recovery per adapter (codex, claude, opencode), local, cold and warm, p50/p95; - restart recovery from the overnight chaos evidence, the slowest steps ranked, and the gaps. - `tests/restart-chaos/ttft-baseline.ts`: an on-demand driver on the chaos supervisor. It reads the run log only, never Telemetry, and always stops its embedded Postgres. - `tests/restart-chaos/ttft-baseline-stats.ts` and its unit tests: pure percentile, boot, pre-turn ranking and recovery statistics. No runtime behavior changes. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/ttft-baseline-stats.test.ts` - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` - The baseline numbers come from the driver run named in the report. To reproduce them, follow the report's "Method" section. - The owner spot-checked the inventory's file:line claims at `f2d8a51027`. One example: `processLossRetryCount` has no writer, and `heartbeat/recovery.ts:2675` is its only reader. ## Risks - Low risk. This adds documentation and an opt-in test driver. Nothing runs in CI except the pure stats unit tests. - The driver spends a small amount of provider credit when someone runs it by hand. - The baseline sample is small (6 per cell). The local host was shared with other worktrees, so the p95 tails include load. The report states both. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use, code execution and five parallel read-only subagents for the inventory. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed
1 parent
54b83b15a3
commit
6cfb69b4b0
5 files changed
+1297
No files matched your search
@@ -0,0 +1,442 @@
|
||||
# Ownership inventory: durable in-progress records (Phase 0)
|
||||
|
||||
Status: Phase 0 deliverable A, 2026-11-07. Needs one review from Dotta and from the owner of the zero-downtime upgrade work before Phase 1.
|
||||
Plan: [`2026-11-07-durable-ownership-and-reconciliation.md`](2026-11-07-durable-ownership-and-reconciliation.md) (#19831). Companion: [`2026-11-07-ttft-baseline.md`](2026-11-07-ttft-baseline.md).
|
||||
Source revision: `origin/master` at `f2d8a51027`. Every row cites file:line at that revision. A row marked "unverified" means that the reader could not confirm the line.
|
||||
|
||||
## Summary
|
||||
|
||||
The control plane has **83 distinct durable in-progress record kinds**, in 94 inventory rows. 11 rows describe storage that another subsystem already owns. They are merged below and kept in the appendix as cross-references.
|
||||
|
||||
| Obligation | Kinds |
|
||||
|---|---|
|
||||
| Ownership (includes work-intent delivery and attempt budgets) | 44 |
|
||||
| Workspace preservation | 23 |
|
||||
| Accounting | 13 |
|
||||
| Accepted result | 3 |
|
||||
|
||||
Every kind has a row in the index below. The full columns (writer, in-progress states, owner, release rule, proof) are in the appendix tables. The lists under "Cross-subsystem overlaps" name the kinds whose release today depends on age, or that nothing releases at all.
|
||||
|
||||
The five findings that matter most for Phase 1:
|
||||
|
||||
1. **"Owner gone" is decided by at least 31 separate signal-0 sites and 3 boot ids, and the proofs disagree.** One process mints three boot UUIDs (`controller-process-identity.ts:13`, `native-restart-recovery.ts:284`, `legacy-controller-lease.ts:8`). The rules contradict each other:
|
||||
- On EPERM: `heartbeat/recovery.ts:308` treats the pid as alive; `local-service-supervisor.ts:245` treats it as dead.
|
||||
- On pid reuse: `native-failed-run-owner-release.ts:97,113` and `native-session-executor.ts:592` count any 1 ms start-time difference as a reused pid. `ps lstart` has 1 s precision, so on macOS they can release a live owner. `native-restart-recovery.ts:348-358` and `controller-process-identity.ts:90` need a gap of at least 1 s.
|
||||
|
||||
Section A must pick one stamp and one prover.
|
||||
2. **Three proofs accept "the pid is dead here" as proof that an owner on another host is gone.** This breaks the plan's rule that a different host is never proof:
|
||||
- `evaluateNativeControllerTakeover` (`native-restart-recovery.ts:360-437`); `native_run_finalizations` has no host column;
|
||||
- the runtime-control sweep (`workspace-operations.ts:126-135`);
|
||||
- the orphan reaper `terminalizeOrphanedRunningRun` (`recovery/service.ts:6096`).
|
||||
|
||||
Three of the ten proofs check the host:
|
||||
- `controllerProcessExited` (`controller-process-identity.ts:83-91`) compares a host, boot and pid-namespace hash. On another host it returns "not exited", and the record waits out a TTL (15 or 30 min).
|
||||
- `proveLocalWorkspaceOwnerStopped` (`native-workspace-owner-stop-proof.ts:62-106`) returns `unsupported` / `foreign_host`. That is the closest existing equivalent of `unprovable`.
|
||||
- `recordedLocalExecutionStopped` (`native-local-process-stop.ts:27-40`) requires the recorded hostname.
|
||||
|
||||
No shared prover returns `unprovable`, and only `controllerProcessExited` also checks the boot id and the pid namespace.
|
||||
3. **Company-wide admission still waits on cleanup and accounting through five gates** (invariant 3, section D):
|
||||
- `waitingRunCount` (`budget-exposure.ts:201-202`);
|
||||
- `recoveringRunCount` (`budgets.ts:858`);
|
||||
- the scope budget hold (`budgets.ts:860`);
|
||||
- the 15 s `budget_evaluation_work` gate (`budgets.ts:863-872`);
|
||||
- the per-issue `hasPendingRunCleanup` successor block (`heartbeat/queue.ts:1129`, `execution-blocker.ts:70-102`), which fails closed when the run row is missing.
|
||||
|
||||
A run with `accountingCaptureFailed=true` is skipped by accounting (`run-cost-accounting.ts:250,389`). It is not "recovering" either (`budget-exposure.ts:140-146`), so it holds `waitingRunCount` until the UTC month rolls over.
|
||||
4. **Several records have no automatic release, so a crash leaves a permanent hold with no next action** (invariants 2 and 7):
|
||||
- `remote_workspace_provision` rows left `running`/`uncertain` (`remote-workspaces.ts:973-990`). They block every later reservation of that workspace.
|
||||
- The runtime skill-cache lock (`runtime-skill-cache.ts:134-140`). It is on the turn path: the next run waits 60 s, then fails.
|
||||
- Unpriced service-invocation reservations (`decision-models.ts:217`, `prompt-injection-prevention.ts:178`, `fast-responses.ts:691`).
|
||||
- Create intents on providers other than Daytona: only Daytona declares `creationOperationVersion: 1` (`daytona/src/manifest.ts:42`), so a crashed create elsewhere ends in operator reconciliation.
|
||||
- The same-boot legacy settlement receipt (`legacy-process-settlement.ts:68,98`). Without it, instance sleep stays blocked (`idle-retry-schedule.ts:100-132`).
|
||||
5. **Restarts still spend the work budget, and the per-restart guard is dead code** (invariant 5, section F):
|
||||
- `processLossRetryCount` has no writer and is read only at `heartbeat/recovery.ts:2675`, so its guard always passes.
|
||||
- Interrupted preparation, where no provider work started, charges the same `failureRetries` counter as real failures (`execution-recovery-attempt.ts:61-67`).
|
||||
- "Provider work started" has five independent proofs (four in accounting, plus the Runner's `startup_attempt.before_spawn`). Section F should collapse them onto the B outcome records.
|
||||
|
||||
## Definitions used by every row
|
||||
|
||||
- **In-progress state:** a non-terminal value that a live process is expected to move forward.
|
||||
- **Owner recorded:** the identity stored on the record that may move it, such as a boot id, pid, lease token, run id or fence token. "none" means the record stores no owner.
|
||||
- **Proof:** the code that decides that the owner is gone or that the record is safe to release. Age-based proofs are marked TTL/age.
|
||||
- **Obligation:** exactly one of ownership, accepted result, workspace preservation, or accounting (plan, "Relationship to the zero-downtime upgrade work", item 3). Readers labelled work-intent delivery as ownership (it moves to E), and the attempt budget as ownership (it moves to F).
|
||||
|
||||
## Merge decisions
|
||||
|
||||
The 11 rows that describe the same storage as another row are merged as follows. Where two readers named different obligations for one record, the owner picked one, and the reason is given.
|
||||
|
||||
| Storage | Rows merged | Obligation chosen |
|
||||
|---|---|---|
|
||||
| `heartbeat_runs.runner_profile_json.nativeWorkspaceFinalizationOwner` | native "Workspace finalization owner receipt"; workspaces "Native copyback owner receipt" | **Ownership.** The receipt decides who may run copyback. Copyback itself is preservation. |
|
||||
| `runner_profile_json.nativeWorkspaceSync` | native "Pending workspace export"; workspaces "Native workspace sync reference" | Workspace preservation |
|
||||
| `runner_profile_json.nativeHarnessFilesystemIndependence` | native "Filesystem-independence (WEB) receipt"; workspaces "WEB evidence receipt" | Workspace preservation. It is a receipt, not an in-progress record. Its only consumer is the backup-migration gate (`native-harness-backup-migration.ts:30`), and it grants no resume authority. |
|
||||
| `environment_leases.metadata.pendingCleanup*` | environments "Teardown claim"; workspaces "Recovery cleanup in-flight claim"; wake queue "Pending-cleanup environment lease claim" | **Ownership.** The claim fences who may destroy the sandbox. Retaining files is a separate obligation, held by `sandboxStopAndRetain` / `nativeWorkspaceExportResume`. |
|
||||
| `environment_leases.metadata.nativeWorkspaceExportResume` | environments "Native export resume"; workspaces "Native terminal-export retention" | Workspace preservation |
|
||||
| `environment_leases.status='active'` | environments "Run lease (active)"; wake queue "Orphaned active environment lease" | Ownership |
|
||||
| `heartbeat_runs.controller_boot_id` / `controller_lease_expires_at` | native "Legacy controller lease"; wake queue "Legacy controller lease (running run)" | Ownership |
|
||||
| `native_run_finalizations.lease_owner` (executor) | native "Executor execution claim"; wake queue "Native run controller lease (read here)"; accounting "Native finalization lease used as a recovery-owner proof"; wake queue "Run-log writer authority" (a read-only gate) | Ownership |
|
||||
|
||||
## Cross-subsystem overlaps
|
||||
|
||||
### One fact, many proofs
|
||||
|
||||
| Fact | Separate proofs (file:line) | Disagreement |
|
||||
|---|---|---|
|
||||
| The recorded controller process is gone | `controllerProcessExited` controller-process-identity.ts:83-91 · `evaluateNativeControllerTakeover` native-restart-recovery.ts:360-437 · `recordedProcessMayBeLive`/`runProcessMayBeLive`/`controllerMayBeLive` native-failed-run-owner-release.ts:93-127 · `verifiedRecoveryProcessIsAlive` native-session-executor.ts:581-598 · `proveLocalWorkspaceOwnerStopped` native-workspace-owner-stop-proof.ts:62-106 · `recordedLocalExecutionStopped` native-local-process-stop.ts:27-40 · `isProcessAlive` heartbeat/recovery.ts:308 · `isPidAlive` local-service-supervisor.ts:245 · `ownerProcessIsAlive` workspace-operations.ts:126-135 · Rust `local_process_definitely_gone` provider_backend.rs:447-458 | They differ on the host check (3 of 10 check it: `controllerProcessExited`, `proveLocalWorkspaceOwnerStopped`, `recordedLocalExecutionStopped`), EPERM handling, and the start-time tolerance (1 ms or 1 s). |
|
||||
| The controller of this process | 3 boot UUIDs: controller-process-identity.ts:13 → `sandbox_creation_operations`, teardown claims · native-restart-recovery.ts:284 → `native_run_finalizations.controller_boot_id` · legacy-controller-lease.ts:8 → `heartbeat_runs.controller_boot_id` | Different values for the same process. Section A must migrate the stored columns. |
|
||||
| Provider work never started | `resultJson.executionRecovery.providerWorkStarted` · `usageJson.accountingProviderWorkStarted` (run-cost-accounting.ts:41-42) · `nativeProviderWorkNeverStarted` (empty fence map + seq 0 + no journal row, run-cost-accounting.ts:108-120) · `noProviderWork` (service invocations) · Runner `startup_attempt.before_spawn` (provider_backend.rs:2190-2210) | The fence-map test reads a recorder rebind with sequence 0 as "work began" (usage-receipts.ts:582, native-usage-delivery.ts:35). |
|
||||
| The native coordinator is settled | `settledNativeCoordinator` run-cost-accounting.ts:85-88 · "recovering" budget-exposure.ts:142-146 · inline re-checks run-cost-accounting.ts:205-206, 258-261, 392-394 | Three encodings |
|
||||
| The block is a hard stop, not temporary | `evaluateAdmissionInTransaction` budgets.ts:852 · `isBoardOwnedBudgetBlock` budget-block-ownership.ts:9-19 | Different snapshots of spend |
|
||||
| The remote writer is gone | `readRemoteWorkspaceWriterHold` remote-workspace-writer-hold.ts:25-79 · 60 min `lastActivityAt` cutoff heartbeat/workspace-preparation.ts:1050 and workspace-busy-wait.ts:71 | Reservation and waiter disagree for up to 60 min |
|
||||
| A provider allocation is absent | Host creation-absence TTL sandbox-creation-operations.ts:49-52 · Daytona `destroyFailedCreation` daytona/src/plugin.ts:1209-1234 | Daytona inspect returns `unknown` on NotFound (:3246), so the host falls back to age |
|
||||
|
||||
### One record, many release paths
|
||||
|
||||
- **`native_run_finalizations.lease_owner`: 8 or more release paths.** Executor settlement (native-session-executor.ts:11378, 11521), `releaseUnstartedNativeExecutionClaim` (native-execution-lease.ts:58), `settleFailedRunOwner` (native-failed-run-owner-release.ts:170), `releaseDeadControllerResultLeases` (native-dead-controller-result-lease.ts:82), the restart takeover (native-restart-recovery.ts:1514), the resume claim (native-finalization-reconciler.ts:299), checkpoint-failure expiry (native-runner-checkpoint.ts:165), and the finalizer (native-run-finalizer.ts:240, 451). Restart recovery and the periodic loop also both call `reconcileNativeFinalizations` (heartbeat/recovery.ts:956, 2836).
|
||||
- **An `active` environment lease: 4 release paths.** `releaseRunLeases` (environment-run-orchestrator.ts:326, 796), native settlement (native-environment-settlement-work.ts:50-75), the reaper orphan flip (heartbeat/recovery.ts:2894), and live-recovery `orphanLease` (heartbeat/recovery.ts:2960).
|
||||
- **A `pending_cleanup` teardown: 3 claimers, 2 renewal loops.** The stop-only claim (environment-runtime.ts:3951) does not renew its 15 min fence.
|
||||
- **A `running` run with a dead process: 2 reapers with different outcomes.** `recoverActiveExecutions` (heartbeat/recovery.ts:2374) sets `failed/process_lost` and can retry. `terminalizeOrphanedRunningRun` (recovery/service.ts:6096) sets `interrupted` and never retries. Whichever runs first decides whether a retry happens.
|
||||
- **A `workspace_finalize` operation: 3 release paths.** #19807 supersession (native-workspace-finalization-supersession.ts:21-91), offline reconciliation (offline-workspace-operation-reconciliation.ts:30-118), and `reconcileOnlineWorkspaceOperations` (:152), which has no production caller.
|
||||
- **The issue execution lock: about 10 clear sites.** These are the wake-queue release (wake-queue postgres.ts:1255), about 8 inline clears in heartbeat/queue.ts, `sweepStaleIssueLocks`, and the checkout release. Each one decides on its own terms that the run is terminal.
|
||||
- **Deferred-wake promotion: 2 loops.** The release drain (use-cases.ts:135) and deferred-receipt work (deferred-receipt-work.ts:16, every 30 s). Only a CAS (postgres.ts:497) reconciles them.
|
||||
- **The legacy shutdown fence.** Both the owner and the sweep release it. After a boot-id mismatch, the sweep still waits for the deadline.
|
||||
|
||||
### Released only by TTL/age (invariant 2 targets)
|
||||
|
||||
- Finalizer lease: 5 min (native-run-finalizer.ts:243).
|
||||
- Cleanup lease: 60 s (native-session-executor.ts:3887).
|
||||
- Legacy controller lease: 60 s (legacy-controller-lease.ts:133).
|
||||
- Dispatched legacy runs with no pid: 5 min `updatedAt` (heartbeat/recovery.ts:2378, 2580; RC-27).
|
||||
- Orphaned active environment lease: the run is terminal plus 5 min.
|
||||
- Teardown claim from another host: 15 min. Create intent from another host: 30 min, then operator action.
|
||||
- Claimed native-status wake: 60 s. Deferred receipt recheck: 30 s.
|
||||
- Remote writer lane: 60 min.
|
||||
- Agent-scoped runtime lease: 30 min (workspace-runtime-leases.ts:116-118).
|
||||
- Accounting closure: 10 min (run-cost-accounting.ts:135).
|
||||
- Service invocations: 120 s, or `expiresAt` plus 30 s.
|
||||
- Pending harness backups and runner checkpoints: retention window.
|
||||
- `waitingRunCount` debt: the UTC month rollover.
|
||||
|
||||
### Write-ahead coverage (section B)
|
||||
|
||||
| Effect | Intent before the effect? |
|
||||
|---|---|
|
||||
| Sandbox create | Yes: `sandbox_creation_operations` (sandbox-creation-operations.ts:98-125) |
|
||||
| Stop and retain | Yes: `sandboxStopAndRetain` (sandbox-stop-and-retain.ts:148-169) |
|
||||
| Claimed destroy | Yes: `pending_cleanup` and the controller (environment-runtime.ts:3704-3733) |
|
||||
| Runner process spawn | Yes: `startup_attempt` with `before_spawn` (provider_backend.rs:2190-2210) |
|
||||
| `turn.start` | Yes: Runner command journal `pending` → `completed`/`indeterminate` (runner/DURABLE_TRANSPORT.md:70-75) |
|
||||
| Local process start | Yes: `native.process_start_requested` (native-session-executor.ts:10843-10855) |
|
||||
| Workspace finalize / copyback | Yes: `workspace_operations` (#19807) |
|
||||
| Ordinary lease release / stop | **No**: `driver.releaseRunLease` (environment-runtime.ts:2243-2294) |
|
||||
| Reusable sandbox destroy | **No**: `destroyReusableSandboxLease` (environment-runtime.ts:3052-3127), also inline before `turn.start` |
|
||||
| Remote provision command | Partly: the row is written, but only success settles it (remote-workspaces.ts:973-990) |
|
||||
| Reservation hold / release | No owner stamp: `budget_reservations` stores only `run_id` |
|
||||
| Orphan cleanup spool | The spool has no production `append` caller. Orphans live only in memory (environment-runtime.ts:1302, 1322-1335). |
|
||||
|
||||
### Local-disk records that conflict with an ephemeral filesystem
|
||||
|
||||
These records live on a local disk. A replacement host loses them, and neither `ephemeral-server-contract.ts` nor `heartbeat/upgrade-readiness.ts` lists them:
|
||||
- the usage-receipt spool (`idle-local-work.ts:18`);
|
||||
- the sandbox orphan-cleanup spool (`idle-local-work.ts:19-20`);
|
||||
- the runtime skill-cache lock (`runtime-skill-cache.ts:116-140`).
|
||||
|
||||
Also, `upgrade-readiness.ts:53-61` reads only `active` runner-task leases. It ignores in-flight `pending_cleanup` claims and `pending` create intents, so a replacement server inherits them on the 15 and 30 min TTLs.
|
||||
|
||||
## Record kinds from the zero-downtime upgrade work
|
||||
|
||||
The plan says to include these record kinds and to keep the four obligations separate. Their owner should review this section.
|
||||
|
||||
| Kind | Obligation | Release today | Plan |
|
||||
|---|---|---|---|
|
||||
| Executor or restart claim on a remote runner with a durable checkpoint (`remoteLeaseFenced`, native-restart-recovery.ts:379-383) | Ownership | Fenced handoff by lease and generation. TTL by design. | A: stays a fence-token handoff and never uses `proveOwnerGone`. |
|
||||
| Runner controller checkpoint (`native_runner_checkpoints`, native-runner-checkpoint.ts:119-152) | Ownership | Fence compare on NRF boot, generation and lease owner. Retirement marker. | A, G |
|
||||
| Source-recovery consumer (`environment_leases.metadata.runnerTaskSourceConsumer`, native-source-recovery-ownership.ts:14-49) | Ownership | Lease expiry plus a boot and generation compare | A, G |
|
||||
| WEB filesystem-independence receipt (native-harness-filesystem-independence.ts:735, 791) | Workspace preservation | A receipt for terminal runs only. It gates ephemeral readiness (ephemeral-server-contract.ts:72-74). | B. It must never settle accepted-result or resume authority. Today it does not. |
|
||||
| Upgrade readiness (`heartbeat/upgrade-readiness.ts:21-94`) | (read-only gate) | Observes NRF `controller_boot_id = current` | C: add the teardown claims and create intents it misses. |
|
||||
| Ephemeral server contract (`ephemeral-server-contract.ts:42-83`) | (startup gate over derived obligations) | — | C: list the local-disk spools above. |
|
||||
|
||||
The two obligations cross in two places. Section C should model these as explicit dependencies, not as shared settle paths:
|
||||
- `settleStoppedObservedWorkspace` writes `native_run_finalizations.phase='terminal_failure'` in the same transaction as the preservation fence (native-observed-workspace-preservation.ts:208-217). Here preservation settles an ownership record.
|
||||
- Releasing the export-retention hold requires a committed accepted result (native-workspace-export-resume.ts:79-82). This dependency is intended.
|
||||
|
||||
## Mapping to plan sections A–G
|
||||
|
||||
A kind can move to more than one section. Phase 2 moves kinds over one PR per kind or small group. Section C is the default home of any kind whose release is age-only today.
|
||||
|
||||
- **A. One ownership primitive** (32): Executor execution claim (fresh run); Restart-recovery claim; Accepted-result executor lease; Finalizer lease; Cleanup lease; Ownership-unverified hold; Local process start journal; Workspace finalization owner receipt; Remote preparation failure preservation; Runner controller checkpoint; Source-recovery consumer; Legacy controller lease; Runtime skill cache publication lock; Sandbox creation intent; Run lease (active); Teardown claim (in-flight cleanup); Retained / reusable hold; Reusable-lease handoff receipt; Runner task dispatch flags; Login/probe lease; Managed runtime-control operation (start/stop/restart/repair/run); Execution workspace runtime lease; Workspace runtime service (local process); Remote workspace writer lane; Legacy graceful-shutdown workspace settlement fence; Copyback-owner hold (RC-50); Observed-run workspace preservation fence; Legacy run process identity; Detached-process marker; Hot-restart adoption marker; Legacy process settlement receipt; Case-handoff source release marker
|
||||
- **B. Write-ahead intents** (33): Session-resume claim; Native startup context; Provider-pack admission state; Local process start journal; Workspace finalization owner receipt; Pending workspace export (sync reference); Runner provider startup fence; Runner command journal (`turn.start`); Native harness backup publication; Provider artifact upload; Filesystem-independence (WEB) receipt; Sandbox creation intent; Teardown claim (in-flight cleanup); Pending-cleanup obligation (not in flight); Stop-and-retain intent; Native export resume (stop-only); Orphan cleanup buffer/spool; Remote-workspace destroy receipt; Native workspace finalize attempt (copyback export); Remote provision command intent; Remote snapshot save (initial/checkpoint/final); Remote workspace migration op; Git cache expiration intent; Legacy restore-recovery preservation hold; Run budget reservation; Receipt-closure wait marker; Receipt-source fence map; Usage-receipt journal row; Usage-receipt spool file (durable file); Service-invocation reservation (decision / prompt-injection / fast-response); Browser-use budget hold; Legacy process settlement receipt; Case-handoff source release marker
|
||||
- **C. One reconciler** (45): Executor execution claim (fresh run); Restart-recovery claim; "Awaiting evidence" wait; Accepted-result executor lease; Finalizer lease; Cleanup lease; Ownership-unverified hold; Native startup context; Pending workspace export (sync reference); Runtime skill cache publication lock; Native harness backup publication; Provider artifact upload; Sandbox creation intent; Run lease (active); Teardown claim (in-flight cleanup); Pending-cleanup obligation (not in flight); Stop-and-retain intent; Native export resume (stop-only); Retained / reusable hold; Orphan cleanup buffer/spool; Login/probe lease; Managed runtime-control operation (start/stop/restart/repair/run); Native workspace finalize attempt (copyback export); Remote provision command intent; Remote snapshot save (initial/checkpoint/final); Remote workspace migration op; Execution workspace runtime lease; Workspace runtime service (local process); Remote workspace unsaved-files flag; Remote snapshot ledger; Git restore point; Git cache expiration intent; Legacy graceful-shutdown workspace settlement fence; Legacy restore-recovery preservation hold; System recovery attempt for retained work; Copyback-owner hold (RC-50); Observed-run workspace preservation fence; Run accounting debt; Usage-receipt spool file (durable file); Wake request (claimed); Detached-process marker; Hot-restart adoption marker; Issue execution lock; Issue checkout lock; Execution reconciliation hold
|
||||
- **D. Admission vs accounting** (14): Manual-admission hold; Run budget reservation; Run accounting debt; Receipt-closure wait marker; Usage-capture failure; Late-receipt debt; Budget-evaluation input; Scope budget hold; Service-invocation reservation (decision / prompt-injection / fast-response); Browser-use budget hold; Wake request (deferred behind issue lock); Queued run; Issue execution lock; Execution reconciliation hold
|
||||
- **E. Work-intent outbox** (10): Retry schedule / system-recovery policy; Manual-admission hold; Remote workspace writer lane; Workspace recovery request; Budget-evaluation input; Wake request (queued); Wake request (deferred behind issue lock); Wake request (claimed); Queued run; Scheduled retry run
|
||||
- **F. Work-based attempt budgets** (7): Session-resume claim; Retry schedule / system-recovery policy; Provider-attempt budget; Runner provider startup fence; System recovery attempt for retained work; Scheduled retry run; Retry/attempt budget counters
|
||||
- **G. Generation stamping** (8): Restart-recovery claim; Runner controller checkpoint; Source-recovery consumer; Runner command journal (`turn.start`); Reusable-lease handoff receipt; Runner task dispatch flags; Observed-run workspace preservation fence; Receipt-source fence map
|
||||
|
||||
## Kind index
|
||||
|
||||
There is one row per distinct kind (83). A row notes the other subsystems that also read the same storage. The full columns are in the appendix.
|
||||
|
||||
| # | Kind | Subsystem | Obligation | Release proof | Invariants | Plan sections |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 1 | Executor execution claim (fresh run) <br>(also read by Accounting / budgets; Wake queue / recovery) | Native runtime / Runner | ownership | identity | 1, 2, 6 | A,C |
|
||||
| 2 | Restart-recovery claim | Native runtime / Runner | ownership | identity | 1, 2, 6 | A,C,G |
|
||||
| 3 | "Awaiting evidence" wait | Native runtime / Runner | ownership | identity | 2, 7 | C |
|
||||
| 4 | Session-resume claim | Native runtime / Runner | ownership | age, operator or none | 1, 2, 5 | B,F |
|
||||
| 5 | Accepted-result executor lease | Native runtime / Runner | accepted-result | identity | 2 | A,C |
|
||||
| 6 | Finalizer lease | Native runtime / Runner | accepted-result | age, operator or none | 1, 2 | A,C |
|
||||
| 7 | Cleanup lease | Native runtime / Runner | workspace-preservation | age, operator or none | 1, 2 | A,C |
|
||||
| 8 | Retry schedule / system-recovery policy | Native runtime / Runner | ownership | age, operator or none | 3, 4, 7 | E,F |
|
||||
| 9 | Provider-attempt budget | Native runtime / Runner | ownership | age, operator or none | 5 | F |
|
||||
| 10 | Manual-admission hold | Native runtime / Runner | ownership | age, operator or none | 3, 7 | D,E |
|
||||
| 11 | Ownership-unverified hold | Native runtime / Runner | ownership | age, operator or none | 2, 7 | A,C |
|
||||
| 12 | Native startup context | Native runtime / Runner | ownership | age, operator or none | 1, 2 | B,C |
|
||||
| 13 | Provider-pack admission state | Native runtime / Runner | ownership | age, operator or none | 1 | B |
|
||||
| 14 | Local process start journal | Native runtime / Runner | ownership | identity | 1, 2 | A,B |
|
||||
| 15 | Workspace finalization owner receipt <br>(also read by Workspaces) | Native runtime / Runner | ownership | identity | 1, 2, 7 | A,B |
|
||||
| 16 | Pending workspace export (sync reference) <br>(also read by Workspaces) | Native runtime / Runner | workspace-preservation | age, operator or none | 2, 7 | B,C |
|
||||
| 17 | Remote preparation failure preservation | Native runtime / Runner | workspace-preservation | age, operator or none | 2 | A |
|
||||
| 18 | Runner controller checkpoint | Native runtime / Runner | ownership | identity | 1, 6 | A,G |
|
||||
| 19 | Source-recovery consumer | Native runtime / Runner | ownership | identity | 1, 6 | A,G |
|
||||
| 20 | Legacy controller lease <br>(also read by Wake queue / recovery) | Native runtime / Runner | ownership | age, operator or none | 1, 2 | A |
|
||||
| 21 | Runner provider startup fence | Native runtime / Runner | ownership | identity | 1, 2, 5 | B,F |
|
||||
| 22 | Runner command journal (`turn.start`) | Native runtime / Runner | accepted-result | age, operator or none | 4, 6 | B,G |
|
||||
| 23 | Runtime skill cache publication lock | Native runtime / Runner | ownership | identity | 2, 7, 8 | A,C |
|
||||
| 24 | Native harness backup publication | Native runtime / Runner | workspace-preservation | age, operator or none | 1, 2 | B,C |
|
||||
| 25 | Provider artifact upload | Native runtime / Runner | workspace-preservation | identity | 1, 2 | B,C |
|
||||
| 26 | Filesystem-independence (WEB) receipt <br>(also read by Workspaces) | Native runtime / Runner | workspace-preservation | age, operator or none | — | B |
|
||||
| 27 | Sandbox creation intent | Environments / sandboxes | ownership | identity | 1,2,3,7 | A,B,C |
|
||||
| 28 | Run lease (active) <br>(also read by Wake queue / recovery) | Environments / sandboxes | ownership | age, operator or none | 1,2 | A,C |
|
||||
| 29 | Teardown claim (in-flight cleanup) <br>(also read by Wake queue / recovery; Workspaces) | Environments / sandboxes | ownership | identity | 1,2 | A,B,C |
|
||||
| 30 | Pending-cleanup obligation (not in flight) | Environments / sandboxes | ownership | age, operator or none | 2,7 | B,C |
|
||||
| 31 | Stop-and-retain intent | Environments / sandboxes | workspace-preservation | age, operator or none | 1,2 | B,C |
|
||||
| 32 | Native export resume (stop-only) <br>(also read by Workspaces) | Environments / sandboxes | workspace-preservation | age, operator or none | 1,2 | B,C |
|
||||
| 33 | Retained / reusable hold | Environments / sandboxes | workspace-preservation | age, operator or none | 1,7 | A,C |
|
||||
| 34 | Reusable-lease handoff receipt | Environments / sandboxes | ownership | age, operator or none | 1,6 | A,G |
|
||||
| 35 | Orphan cleanup buffer/spool | Environments / sandboxes | ownership | age, operator or none | 1,2 | B,C |
|
||||
| 36 | Remote-workspace destroy receipt | Environments / sandboxes | workspace-preservation | age, operator or none | 2 | B |
|
||||
| 37 | Runner task dispatch flags | Environments / sandboxes | ownership | age, operator or none | 1,6 | A,G |
|
||||
| 38 | Login/probe lease | Environments / sandboxes | ownership | age, operator or none | 1,2 | A,C |
|
||||
| 39 | Managed runtime-control operation (start/stop/restart/repair/run) | Workspaces | ownership | identity | 1 (no host/boot), 2 | A,C |
|
||||
| 40 | Native workspace finalize attempt (copyback export) | Workspaces | workspace-preservation | age, operator or none | 1, 2 | B,C |
|
||||
| 41 | Remote provision command intent | Workspaces | workspace-preservation | age, operator or none | 1, 2, 3, 7 | B,C |
|
||||
| 42 | Remote snapshot save (initial/checkpoint/final) | Workspaces | workspace-preservation | age, operator or none | 1, 2 | B,C |
|
||||
| 43 | Remote workspace migration op | Workspaces | workspace-preservation | age, operator or none | 1, 2 | B,C |
|
||||
| 44 | Execution workspace runtime lease | Workspaces | ownership | age, operator or none | 1, 2 | A,C |
|
||||
| 45 | Workspace runtime service (local process) | Workspaces | ownership | identity | 1, 2 | A,C |
|
||||
| 46 | Remote workspace writer lane | Workspaces | ownership | age, operator or none | 1, 2, 3, 7 | A,E |
|
||||
| 47 | Remote workspace unsaved-files flag | Workspaces | workspace-preservation | age, operator or none | 1, 2 | C |
|
||||
| 48 | Remote snapshot ledger | Workspaces | workspace-preservation | age, operator or none | 2 | C |
|
||||
| 49 | Git restore point | Workspaces | workspace-preservation | age, operator or none | 1, 2 | C |
|
||||
| 50 | Git cache expiration intent | Workspaces | workspace-preservation | identity | 1 | B,C |
|
||||
| 51 | Legacy graceful-shutdown workspace settlement fence | Workspaces | workspace-preservation | identity | 2 | A,C |
|
||||
| 52 | Legacy restore-recovery preservation hold | Workspaces | workspace-preservation | age, operator or none | 2, 7 | B,C |
|
||||
| 53 | System recovery attempt for retained work | Workspaces | workspace-preservation | age, operator or none | 2, 4, 5 | C,F |
|
||||
| 54 | Workspace recovery request | Workspaces | workspace-preservation | age, operator or none | 4, 7 | E |
|
||||
| 55 | Copyback-owner hold (RC-50) | Workspaces | ownership | identity | 2, 7 | A,C |
|
||||
| 56 | Observed-run workspace preservation fence | Workspaces | workspace-preservation | identity | 1, 2, 6 | A,C,G |
|
||||
| 57 | Run budget reservation | Accounting / budgets | accounting | age, operator or none | 1, 2, 3, 5 | B,D |
|
||||
| 58 | Run accounting debt | Accounting / budgets | accounting | identity | 1, 2, 3 | D,C |
|
||||
| 59 | Receipt-closure wait marker | Accounting / budgets | accounting | age, operator or none | 2, 3 | B,D |
|
||||
| 60 | Usage-capture failure | Accounting / budgets | accounting | age, operator or none | 2, 3, 7 | D |
|
||||
| 61 | Late-receipt debt | Accounting / budgets | accounting | age, operator or none | 2, 4 | D |
|
||||
| 62 | Receipt-source fence map | Accounting / budgets | accounting | age, operator or none | 1, 6 | B,G |
|
||||
| 63 | Usage-receipt journal row | Accounting / budgets | accounting | age, operator or none | 4 | B |
|
||||
| 64 | Usage-receipt spool file (durable file) | Accounting / budgets | accounting | identity | 2, 4 | B,C |
|
||||
| 65 | Recorder durability probe file | Accounting / budgets | accounting | age, operator or none | 8 | — |
|
||||
| 66 | Budget-evaluation input | Accounting / budgets | accounting | age, operator or none | 3 | D,E |
|
||||
| 67 | Scope budget hold | Accounting / budgets | accounting | age, operator or none | 3 | D |
|
||||
| 68 | Service-invocation reservation (decision / prompt-injection / fast-response) | Accounting / budgets | accounting | age, operator or none | 2, 3 | B,D |
|
||||
| 69 | Browser-use budget hold | Accounting / budgets | accounting | age, operator or none | 1, 2, 3 | B,D |
|
||||
| 70 | Wake request (queued) | Wake queue / recovery | ownership | age, operator or none | 4, 7 | E |
|
||||
| 71 | Wake request (deferred behind issue lock) | Wake queue / recovery | ownership | age, operator or none | 3, 4, 7 | E,D |
|
||||
| 72 | Wake request (claimed) | Wake queue / recovery | ownership | age, operator or none | 2, 4 | E,C |
|
||||
| 73 | Queued run | Wake queue / recovery | ownership | age, operator or none | 3, 4, 8 | D,E |
|
||||
| 74 | Legacy run process identity | Wake queue / recovery | ownership | identity | 1, 2 | A |
|
||||
| 75 | Detached-process marker | Wake queue / recovery | ownership | identity | 2, 7 | A,C |
|
||||
| 76 | Hot-restart adoption marker | Wake queue / recovery | ownership | identity | 2 | A,C |
|
||||
| 77 | Issue execution lock | Wake queue / recovery | ownership | age, operator or none | 2, 3, 7 | C,D |
|
||||
| 78 | Issue checkout lock | Wake queue / recovery | ownership | age, operator or none | 2, 7 | C |
|
||||
| 79 | Scheduled retry run | Wake queue / recovery | ownership | identity | 5, 7 | F,E |
|
||||
| 80 | Retry/attempt budget counters | Wake queue / recovery | ownership | age, operator or none | 5 | F |
|
||||
| 81 | Execution reconciliation hold | Wake queue / recovery | ownership | age, operator or none | 3, 7 | C,D |
|
||||
| 82 | Legacy process settlement receipt | Wake queue / recovery | ownership | identity | 2 | A,B |
|
||||
| 83 | Case-handoff source release marker | Wake queue / recovery | ownership | identity | 2, 3 | A,B |
|
||||
|
||||
## Appendix: subsystem tables
|
||||
|
||||
The subsystem readers wrote these tables, and the owner merged them unchanged. Paths are relative to `server/src/services/` unless a prefix is given.
|
||||
|
||||
### Native runtime and Runner
|
||||
|
||||
Scope read: `server/src/services/native-runtime/*` (files named in the brief), `controller-process-identity.ts`, `legacy-controller-lease.ts`, `cancelled-native-startup.ts`, `heartbeat/upgrade-readiness.ts`, `ephemeral-server-contract.ts`, `packages/paperclip-runner` (Rust `runner-core/src/provider_backend.rs`, `runner/DURABLE_TRANSPORT.md`, `drivers/acpx/command-lease-owner.ts`, `runtime-skill-lease.ts`), schema `native_run_finalizations.ts`, `native_runner_checkpoints.ts`, `native_harness_backups.ts`, `native_provider_artifacts.ts`, `heartbeat_runs.ts`. Paths below are relative to `server/src/services/` unless prefixed. NRF = `native_run_finalizations`; HR = `heartbeat_runs`; RPJ = `heartbeat_runs.runner_profile_json`.
|
||||
|
||||
Note on the coordinator row: NRF is one row per run whose `lease_owner` is reused by four different owner kinds (executor, resume claim, finalizer, cleanup). They are listed as separate kinds because each has a different writer, TTL and proof.
|
||||
|
||||
| Kind | Table.column (state col) | Writer (file:line) | In-progress states | Owner recorded | Current release rule (+ file:line) | Liveness/ownership proof (file:line) | Obligation | Invariants (1–8) | Plan sections (A–G) |
|
||||
|---|---|---|---|---|---|---|---|---|---|
|
||||
| Executor execution claim (fresh run) | NRF.lease_owner / lease_expires_at, `phase` | native-runtime/native-session-executor.ts:9813-9845 (claim tx; TTL 20 min local, 60 s remote: :9639) | phase `observed`, result_id null, lease live | lease_owner token `${runnerInstanceId}:${uuid}` + controller_boot_id, controller_pid, controller_process_started_at, controller_generation, attempt. **No host.** | Renewal every 5 min / 20 s (native-execution-lease.ts:206) else expiry; `releaseUnstartedNativeExecutionClaim` (native-execution-lease.ts:55-66); checkpoint failure forces expiry (native-runner-checkpoint.ts:165); `settleFailedRunOwner` → terminal_failure (native-failed-run-owner-release.ts:170-193); restart takeover (native-restart-recovery.ts:1514-1540) | `evaluateNativeControllerTakeover` pid + start-time (native-restart-recovery.ts:360-437); remote with checkpoint = TTL/age only (`remoteLeaseFenced`, :379-383); failed-run path `controllerMayBeLive` (native-failed-run-owner-release.ts:122-127) gated on TTL/age first (:149) | ownership | 1, 2, 6 | A, C |
|
||||
| Restart-recovery claim | NRF.lease_owner + `recovery_state` + `recovery_history` | native-runtime/native-restart-recovery.ts:1488 (owner `${bootId}:${gen}:${uuid}`), :1516-1540 | recovery_state `awaiting_runner_reattach` / `resuming_session` / `bootstrap_incomplete` | lease_owner + boot/pid/start + generation | Executor adopts; `nativeRecoveryClaimRelinquished` (native-execution-lease.ts:33-48); lease TTL (20 min or 60 s, :1523) | Takeover proof (native-restart-recovery.ts:1093-1099); provider processes `evaluateNativeProviderProcesses` (:444-491, used :1214) | ownership | 1, 2, 6 | A, C, G |
|
||||
| "Awaiting evidence" wait | NRF.recovery_state=`awaiting_evidence` / history disposition | native-runtime/native-restart-recovery.ts:1100-1122 | awaiting_evidence (prior controller "still alive") | none new; reads prior controller pid | Re-evaluated on next recovery pass only | Same takeover proof; `coordinated_controller_still_alive` (:420-431) | ownership | 2, 7 | C |
|
||||
| Session-resume claim | NRF.lease_owner `${runnerInstanceId}:resume:${uuid}`, phase | native-runtime/native-finalization-reconciler.ts:140, :297-313 (TTL 20 min at :297) | phase `retryable_failure`→`observed`, result_id null | lease_owner token + attempt (no controller identity written here) | Executor re-claims; `settleUnstartedNativeSessionResume` (native-failed-run-owner-release.ts:217-225); `releaseAbandonedNativeOwners` after TTL (:240-271) | Attempt unchanged = no provider turn (doc :210-216); `runProcessMayBeLive` (:104-119) | ownership | 1, 2, 5 | B, F |
|
||||
| Accepted-result executor lease | NRF.lease_owner with result_id set | executor claim (above); result attach — exact line unverified | phase ∉ {committed, terminal_failure}, result_id not null | executor token + boot/pid/start | Graceful drain expires it; hard kill → `releaseDeadControllerResultLeases` sets expiry=now (native-dead-controller-result-lease.ts:82-91), called first in restart recovery (heartbeat/recovery.ts:945); else TTL (up to 20 min) | `evaluateNativeControllerTakeover(requireStoppedController)` (native-dead-controller-result-lease.ts:68-73); remote workspaces excluded (:60-61) | accepted-result | 2 | A, C |
|
||||
| Finalizer lease | NRF.lease_owner `native-finalizer:*` | native-runtime/native-run-finalizer.ts:196, :240-260 (5 min TTL :243) | phase `workspace_finalizing` / `arbitrating` / pending commit | token only; **does not restamp controller identity** (comment native-dead-controller-result-lease.ts:49-55) | Cleared on commit (native-run-finalizer.ts:451); otherwise TTL/age | TTL/age: reconciler candidate filter (native-finalization-reconciler.ts:517-521), claim busy check (native-run-finalizer.ts:228-234) | accepted-result | 1, 2 | A, C |
|
||||
| Cleanup lease | NRF.lease_owner `native-cleanup:*` | native-runtime/native-session-executor.ts:3407, :3883-3890 (60 s TTL) | retained-session cleanup staging | token only | TTL/age; deferral `waitFor: "lease_expiry"` (native-session-executor.ts:3541-3542) | TTL/age only | workspace-preservation | 1, 2 | A, C |
|
||||
| Retry schedule / system-recovery policy | NRF.phase=`retryable_failure`, `next_attempt_at`, `failure_detail.systemRecovery` | native-runtime/native-restart-recovery.ts:1006-1022 (`deferRemoteRetry`) | retryable_failure with next_attempt_at, failure_code `NATIVE_RETAINED_OWNER_WAIT` | none (lease null); policy.sourceRunId | Resume dispatch `claimNativeSessionResumptions` (native-finalization-reconciler.ts:95-130); exhausted → terminal_failure (:230-233); timers re-armed at boot (heartbeat/recovery.ts:973-1003) | TTL/age (`next_attempt_at`) | ownership | 3, 4, 7 | E, F |
|
||||
| Provider-attempt budget | NRF.attempt (+ `recoveryHistory[].progressWatermarkSeq`) | native-runtime/native-session-executor.ts:9791-9796, native-restart-recovery.ts:1482-1484, :1510 | attempt < `NATIVE_PROVIDER_RECOVERY_ATTEMPT_LIMIT`=3 (:63) | n/a | `native_session_retry_exhausted` (executor :9796); clean-drain/progress credits (native-restart-recovery.ts:149-176, :246, credit limit 12 at :177) | Watermark heuristic over run events (`restartResumeKeepsAttempt` :246) | ownership | 5 | F |
|
||||
| Manual-admission hold | NRF.phase=`terminal_failure`, failure_code ∈ {native_restart_recovery_blocked, native_retry_cancelled}, systemRecovery.state=`exhausted` | writers in native-restart-recovery.ts (blocked paths ~:1388-1397); exact exhausted writer unverified | held (blocks execution) | none | Board Retry, or `remoteExecutionHasStopped` (native-runner-ownership.ts:24-36) | Remote stop receipt (native-runner-ownership.ts:35) | ownership | 3, 7 | D, E |
|
||||
| Ownership-unverified hold | HR.status=`running` + error_code `native_execution_ownership_unverified` (with native_phase terminal_failure) or `native_adopted_runner_authentication_timeout` | native-runtime/native-session-executor.ts:11321, :11744 | running (held) | none | unverified (not found in scope); SQL guard `nativeRunnerOwnershipNotHeldCondition` (native-runner-ownership.ts:71-81) blocks claims | none — explicitly "not evidence that execution stopped" (native-runner-ownership.ts:43) | ownership | 2, 7 | A, C |
|
||||
| Native startup context | RPJ.nativeStartupContext {contextId, deadlineAtMs, leaseOwner, generation, attempt} | native-runtime/native-session-executor.ts:9851-9861 | deadline not passed / cleanup pending | leaseOwner + generation | `requestNativeStartupCleanup` once deadline and lease expired (native-restart-recovery.ts:1078-1085; impl native-startup-cleanup.ts:84) | TTL/age (deadlineAtMs + lease expiry) | ownership | 1, 2 | B, C |
|
||||
| Provider-pack admission state | RPJ.nativeProviderPackAdmissionState (pending epoch) | native-runtime/native-session-executor.ts:9854 | pending | epoch uuid | unverified | unverified | ownership | 1 | B |
|
||||
| Local process start journal | HR events `native.process_start_requested` → `native.process_identity_recorded`; HR.process_pid/process_group_id/process_started_at | native-runtime/native-session-executor.ts:10843-10855 (event payload {hostname}); identity writer native-local-process-stop.ts (unverified line) | start requested / identity recorded, process may be live | hostname + pid + pgid + start time | `recordedLocalExecutionStopped` (native-local-process-stop.ts:27-40) | Host match + `absent(pid)` + `absent(-pgid)` (native-local-process-stop.ts:17-24, :36-40) | ownership | 1, 2 | A, B |
|
||||
| Workspace finalization owner receipt | RPJ.nativeWorkspaceFinalizationOwner {token, bootId, hostname, pid, processStartedAt, processGroupId, worker} + advisory xact lock | native-runtime/native-workspace-finalization-ownership.ts:107-111, :258-263 | receipt present | full stamp incl. host + pgid | Removed in finally (:336-346); stale prior → takeover CAS (:260-262) | same-boot token (:208-212); platform worker proof (:213-217); `evaluateNativeControllerTakeover` (:224-228) + completed-copyback barrier (:232-238) + `proveLocalWorkspaceOwnerStopped` (:243; native-workspace-owner-stop-proof.ts:62-106); else board action (:195-203) | workspace-preservation | 1, 2, 7 | A, B |
|
||||
| Pending workspace export (sync reference) | RPJ.nativeWorkspaceSync.state=`prepared`; NRF.phase=`workspace_finalizing`; workspace_operations(phase=`workspace_finalize`).status | writer unverified (native-workspace-sync.ts); readers native-workspace-finalization-state.ts:70-91, :123-141 | prepared / workspace_finalizing / op not succeeded | via owner receipt above | Reconciler `resumeNativeWorkspaceFinalization` (native-finalization-reconciler.ts:610-640) | owner receipt proof | workspace-preservation | 2, 7 | B, C |
|
||||
| Remote preparation failure preservation | legacy HR.controller_boot_id/lease + environment lease | native-runtime/native-workspace-finalization-state.ts:11-44 | prepared remote workspace, run legacy | legacy boot id + lease | `recordRequiredWorkspaceRestoreFailure` (:41) | TTL/age (controller_lease_expires_at, :26-28, :38-39) | workspace-preservation | 2 | A |
|
||||
| Runner controller checkpoint | native_runner_checkpoints (part `header` + parts) | native-runtime/native-runner-checkpoint.ts:119-152 (writer), :25-39 (migration) | live checkpoint for an active run | fenced by NRF.controller_boot_id + controller_generation + lease_owner (:56-66, :83-112) | Retirement marker in header (native-runner-checkpoint-retirement.ts:12, :46); rows deleted by harness-backup expiration (native-harness-backup-expiration.ts:445) | Fence compare; inactive owner allowed only with result + successor check (native-runner-checkpoint.ts:89-111) — TTL/age for lease part | ownership | 1, 6 | A, G |
|
||||
| Source-recovery consumer | environment_leases.metadata.runnerTaskSourceConsumer / runnerTaskSourceRecovery | writer unverified (environment code); checks native-source-recovery-ownership.ts:14-49 | consumer bound, source not retired | consumer lease id + controllerBootId + controllerGeneration + identity | Retirement header (native-source-recovery-ownership.ts:15-18) | Lease status/expiry (TTL/age, :41-42) + boot/generation compare (:46) | ownership | 1, 6 | A, G |
|
||||
| Legacy controller lease | HR.controller_boot_id / controller_lease_expires_at / execution_stage | legacy-controller-lease.ts:30-37 (claim), :46-54 (renew every 10 s, TTL 60 s) | execution_stage `preparing`/`dispatching`, lease live | legacy boot uuid (no pid, no host) | `revokeExpiredLegacyControllerWithState` (legacy-controller-lease.ts:116-141), called heartbeat/recovery.ts:2649 | TTL/age only (:133) | ownership | 1, 2 | A |
|
||||
| Runner provider startup fence | Runner state file `codex-provider-state.json` `.startup_attempt` (provider_backend.rs:40) | packages/paperclip-runner/runner/crates/runner-core/src/provider_backend.rs:2190-2210 (intent, `before_spawn: true`), :2232-2237 (spawned) | phase `intent` / `spawned` / `initialization_failed` | launch_id + attempted_process_generation + pid + pgid | `commit_startup_admission` clears (:2313-2323); `retire_orphaned_startup_attempt` (:2380-2420) | `before_spawn` flag; `local_process_definitely_gone` pid + group via signal 0 (:447-458) | ownership | 1, 2, 5 | B, F |
|
||||
| Runner command journal (`turn.start`) | Runner durable journal (runner state; exact file unverified), exposed as `snapshot.control.commands[]` | Runner (DURABLE_TRANSPORT.md:70-75); consumed native-runtime/native-session-executor.ts:2700-2706, :3322-3325 | command `pending` → `completed` / `indeterminate` | controller sequence + runId | Result persisted after effect; indeterminate refuses re-execution (DURABLE_TRANSPORT.md:72-75); adoption of accepted turn (:77-83) | Authenticated command result `providerTurnId` match | accepted-result | 4, 6 | B, G |
|
||||
| Runtime skill cache publication lock | file `<cache>/<fingerprint>.lock` (hard link to `{pid, host}`) | runtime-skill-cache.ts:116-140 | lock file present | pid + hostname | unlink in finally (:139); **crashed owner → error "remove its stale publication lock"; no automatic release** (:134-138) | same-host `process.kill(pid,0)` (:135) used only to word the error | ownership | 2, 7, 8 | A, C |
|
||||
| Native harness backup publication | native_harness_backups.status | native-runtime/native-harness-backup-storage.ts:238 (insert `pending`) | `pending` | run_id (fenced via `fence()` :237, :248) | → `current`/`superseded` (:264-271); abandoned `pending` retired by retention (native-harness-backup-retention.ts:12-23) | TTL/age (`updatedAt < closedBefore`, retention.ts:12) | workspace-preservation | 1, 2 | B, C |
|
||||
| Provider artifact upload | native_provider_artifacts.status + upload_token | native-runtime/native-provider-artifacts.ts:203 (insert pending), :281 | `pending` (upload token), `retiring` | upload_token (no process identity) | `ready` (:344-358) or delete (:395); `retiring`→delete (:736, :794); abandoned pending via `releaseExpiredNativeProviderArtifactRun` (:800, rule unverified) | token compare; expiry rule unverified | workspace-preservation | 1, 2 | B, C |
|
||||
| Filesystem-independence (WEB) receipt | RPJ.nativeHarnessFilesystemIndependence | native-runtime/native-harness-filesystem-independence.ts:735, :791 | n/a — written only for terminal runs (:103-106); gates ephemeral readiness (ephemeral-server-contract.ts:72-74) | none | n/a (receipt, not in-progress) | Settled-owner checks (:103-106) | workspace-preservation | — | B |
|
||||
|
||||
Not record kinds (checked): `heartbeat/upgrade-readiness.ts:21-94` is read-only (`accessMode: "read only"`, :64) and only *observes* NRF ownership by `controller_boot_id = current` (:43); `ephemeral-server-contract.ts` is a startup gate over derived obligations (:42-83); `native-warm-ingress-ownership.ts:41-99` is a proof over NRF + environment_leases, no own row; `native-cleanup-release-wake.ts:11-17` is an in-memory hint; `cancelled-native-startup.ts` is admission proofs (see overlaps); Runner `command-lease-owner.ts` and `runtime-skill-lease.ts` are in-memory/tmpdir per launch (crash leaks a tmpdir, no durable obligation).
|
||||
|
||||
### Environments and sandboxes
|
||||
|
||||
Scope read: `server/src/services/{environment-runtime,sandbox-creation-operations,environment-lease-handoff,environment-lease-release-state,environment-cleanup-state,environment-cleanup-work,sandbox-stop-and-retain,remote-workspace-lifecycle,controller-process-identity,adapter-login-lease,sandbox-orphan-cleanup-spool,environments}.ts`, the environment-lease parts of `heartbeat/recovery.ts`, `packages/db/src/schema/{environment_leases,sandbox_creation_operations}.ts`, Daytona `plugin.ts` / `lease-tracing.ts`, Kubernetes `lease-lifecycle.ts`. Paths below are relative to `server/src/services/` unless prefixed.
|
||||
|
||||
`controller-process-identity.ts` (#19696) is used in two places in this subsystem:
|
||||
- the create intent (`sandbox-creation-operations.ts:4,32,37-38,97`);
|
||||
- the teardown claim. It is written at `environment-runtime.ts:3704,3954` (#19749) and `heartbeat/recovery.ts:1512-1520`, and proved at `heartbeat/recovery.ts:1442-1490`.
|
||||
|
||||
No other environment record carries a process owner.
|
||||
|
||||
**Write-ahead coverage:**
|
||||
- **Create** has a full intent. The `sandbox_creation_operations` row is inserted before the provider call (`sandbox-creation-operations.ts:98-125`).
|
||||
- **Stop-and-retain** has an intent. `sandboxStopAndRetain` is written before dispatch (`sandbox-stop-and-retain.ts:148-169`).
|
||||
- **Destroy through `destroyClaimedSandboxLease`** has a claim intent. It writes `pending_cleanup` and the controller before the provider call (`environment-runtime.ts:3704-3733`).
|
||||
- **No intent** covers:
|
||||
- ordinary release or stop through `driver.releaseRunLease` (`environment-runtime.ts:2243-2294`): the lease stays `active` during the provider call;
|
||||
- `destroyReusableSandboxLease` (`environment-runtime.ts:3052-3127`): the lease stays `released`/`retained`/`active` during the provider delete;
|
||||
- the plugin-backed release path.
|
||||
|
||||
| Kind | Table.column (state col) | Writer (file:line) | In-progress states | Owner recorded | Current release rule (+ file:line) | Liveness/ownership proof (file:line) | Obligation | Invariants (1–8) | Plan sections (A–G) |
|
||||
|---|---|---|---|---|---|---|---|---|---|
|
||||
| Sandbox creation intent | `sandbox_creation_operations.state` | `sandbox-creation-operations.ts:98-108` (inside `createOwnedSandbox` :78) | `pending`; `needs_reconciliation` (recoverable only for runner-task variant, `environment-cleanup-state.ts:7-17`) → terminal `attached` / `cleanup_pending` (hands off to a lease row) / `resolved` | `metadata.creationController` {bootId,pid,hostHash,processStartedAt} + unique (company, originalEnvironmentId, runId, requestKey) (`packages/db/src/schema/sandbox_creation_operations.ts:26`) | success: `attachSandboxCreation` → `attached` (`environments.ts:60-83`); recovery → `cleanup_pending`/`needs_reconciliation` (`sandbox-creation-operations.ts:195-278`); lease cleanup success → `resolved` (`environments.ts:1776-1779`) | Promotion: `promoteAbandonedSandboxCreations` (:22-47) using `controllerProcessExited` (`controller-process-identity.ts:83-91`). But absence stays inconclusive until `creationSettleAt = max(30 min, timeout+60 s)` (:16, :49-52, :96, :236-238) **TTL/age**. Provider proof: Daytona label/name lookup `paperclip-create-<id>` (`packages/plugins/sandbox-providers/daytona/src/plugin.ts:3222-3245`); NotFound → `unknown` | ownership | 1,2,3,7 | A,B,C |
|
||||
| Run lease (active) | `environment_leases.status='active'` | `environments.ts:1508` (`acquireLease`), insert :1704-1713; reacquire :1657-1700 | `active` | `heartbeat_run_id` only (no process/boot stamp on the lease) | normal: `releaseRunLeases` (`environment-runtime.ts:3896`) from `environment-run-orchestrator.ts:326,796`; native settlement (`native-runtime/native-environment-settlement-work.ts:50-75`); orphan flip → `pending_cleanup` (`heartbeat/recovery.ts:1625-1720`) | Run status terminal or run id null, **plus** `updatedAt` older than the backoff (`heartbeat/recovery.ts:1641-1650`). The backoff is `staleThresholdMs` (default 5 min, :2378), applied at :2894 and as a literal 5 min at :2960. **TTL/age** + run-status | ownership | 1,2 | A,C |
|
||||
| Teardown claim (in-flight cleanup) | `environment_leases.status='pending_cleanup'` + `metadata.pendingCleanupInFlight/AttemptId/LeaseExpiresAtMs/Controller` | `environment-runtime.ts:3704-3733` (destroy), :3951-3970 (stop-only), `heartbeat/recovery.ts:1505-1530` (sweep retry) | `pendingCleanupInFlight=true` | `pendingCleanupController` {bootId,pid,hostHash,processStartedAt,attemptId,countedAttempt?} | finally clears in-flight (`environment-runtime.ts:3770-3777`); `releaseLease(..., expectedPendingCleanupAttemptId)` (`environments.ts:1720-1800`); `settleStopOnlyCleanup` (`sandbox-stop-and-retain.ts:171-198`) | `releaseLostPendingCleanupClaims` (`heartbeat/recovery.ts:1442-1490`) with `controllerProcessExited`; otherwise a 15-min fence (`pendingCleanupDeadlineSql`, `environment-cleanup-state.ts:19-29`) **TTL/age**. Renewal every 30 s at `environment-runtime.ts:3739-3752` and `heartbeat/recovery.ts:1869-1872` | ownership | 1,2 | A,B,C |
|
||||
| Pending-cleanup obligation (not in flight) | `environment_leases.status='pending_cleanup'`, `cleanup_status='failed'`, `metadata.pendingCleanupRetryAttempts/RetryAfterMs/CapWarned` | `environments.ts:1827-1870` (`insertPendingCleanupLease`); `heartbeat/recovery.ts:1704-1716` (orphan flip); `environment-runtime.ts:3115-3127` (failed reusable destroy); `sandbox-creation-operations.ts:179,255` | `pending_cleanup` | none (an owner exists only while claimed, row above) | `sweepPendingCleanupLeases` (`heartbeat/recovery.ts:1730+`) → `releaseLease(expired)` (:1903) or back to `pending_cleanup` (:1928). Scheduled by `environment-cleanup-work.ts:47-50` (30 s spacing, 5 min retry `ENVIRONMENT_CLEANUP_RETRY_MS`, `environment-cleanup-state.ts:5`); attempt cap 5 escalates, never abandons (`heartbeat/recovery.ts:198`) | Provider termination receipt (`remoteExecutionTermination`). Daytona: missing sandbox ⇒ `destroyed` (`daytona/src/plugin.ts:3155,3291`); delete confirmed (:3302-3303) | ownership | 2,7 | B,C |
|
||||
| Stop-and-retain intent | `environment_leases.metadata.sandboxStopAndRetain` (lease `pending_cleanup`, `failure_reason='sandbox_stop_pending'`) | `sandbox-stop-and-retain.ts:148-169`, called from `environment-runtime.ts:3946` | `pending_cleanup` with intent present | `requestId` (= `pendingCleanupAttemptId`) + run/lease/provider binding. Controller only once claimed (`environment-runtime.ts:3954`) | `settleStopOnlyCleanup` → `released` + `sandboxStopAndRetainReceipt` (`sandbox-stop-and-retain.ts:171-198`); `restoreSandboxStopAfterFailedRestart` (:113-133) | Provider stop receipt, validated by `hasConfirmedSandboxStopAndRetain` (:17-46) | workspace-preservation | 1,2 | B,C |
|
||||
| Native export resume (stop-only) | `environment_leases.metadata.nativeWorkspaceExportResume` | `native-runtime/native-workspace-export-resume.ts:45` (cross-ref: native inventory) | `pending_cleanup` with key present | run/result binding (unverified detail) | `settleNativeWorkspaceExportResume` (`native-runtime/native-workspace-export-resume.ts:103`) via `settleStopOnlyCleanup` (`sandbox-stop-and-retain.ts:172`) | stop receipt (as above) | workspace-preservation | 1,2 | B,C |
|
||||
| Retained / reusable hold | `environment_leases.status IN ('retained','released')`, `lease_policy` reuse/retain_on_failure | `environments.ts:1784` (`retained` keeps `releasedAt=null`); keep_running path `environment-runtime.ts:~3995-4010`; `retain_on_failure` :2288-2290 | `retained` (hold, not strictly in-progress) | none (issue/agent/env/provider binding only) | next acquire reuses it (`environments.ts:1657-1700`) or destroys it on fingerprint mismatch (`environment-runtime.ts:1545-1559`, inline on acquire); environment delete retry | Provider-side autoStop/autoArchive/autoDelete (`daytona/src/plugin.ts:398-400,448-450`) **TTL/age**. Lease `expiresAt` (`environment-runtime.ts:2244-2249`) **TTL/age** | workspace-preservation | 1,7 | A,C |
|
||||
| Reusable-lease handoff receipt | `environment_leases.metadata.reusableLeasePredecessor/Successor` | `environments.ts:1645` (`createReusableLeaseHandoff`, `environment-lease-handoff.ts:36-38`) | n/a (receipt on retire→successor) | previous/successor run ids + `handoffId` | consumed by `retiredReusableLeasePredecessors` (`environment-lease-handoff.ts:40-74`) in `workspace-recovery-source.ts:182`, `remote-execution-termination.ts:71`, `native-runtime/native-retired-backup-source.ts:49` | exact binding compare (same provider lease, `releasedAt == successor.acquiredAt`) :51-68 | ownership | 1,6 | A,G |
|
||||
| Orphan cleanup buffer/spool | in-process array + local files `<instance>/data/sandbox-orphan-cleanup/*.json` (`idle-local-work.ts:19-20`) | buffer `environment-runtime.ts:1302,1322-1335`; spool `append` `sandbox-orphan-cleanup-spool.ts:172` (no non-test caller found by grep) | record present | none | `flushDeferredOrphanCleanupsOnce` inserts a `pending_cleanup` row, then removes the file (`environment-runtime.ts:1338-1390`); runs at the start of each sweep (`heartbeat/recovery.ts:1752-1770`) | none (record carries provider id) | ownership | 1,2 | B,C |
|
||||
| Remote-workspace destroy receipt | `environment_leases.metadata.remoteExecutionTermination.state='destroyed'` | `environments.ts:1732-1734` (written by `releaseLease`) | — (receipt) | lease id + provider lease id | `cleanupRemoteWorkspaceAllocation` requires it (`remote-workspace-lifecycle.ts:79-98`) via `destroyRemoteWorkspaceLease` (`environment-runtime.ts:3785-3800`) | `hasRemoteTerminationReceipt` (`remote-workspace-lifecycle.ts:92-94`) | workspace-preservation | 2 | B |
|
||||
| Runner task dispatch flags | `environment_leases.metadata.runnerTaskDispatched/Retiring/ExecutionStopped` | `environment-task-runtime.ts:242,275,430` (cross-ref: runner inventory) | `active` and dispatched and not stopped | run id + `runnerTaskCreationContextId` | retire/stop writes (:275,:430); upgrade gate reads it (`heartbeat/upgrade-readiness.ts:53-61`) | lease `expires_at` **TTL/age** (:102) + run status | ownership | 1,6 | A,G |
|
||||
| Login/probe lease | `environment_leases.status='active'`, `heartbeat_run_id IS NULL` | `adapter-login-lease.ts:53-64` → `acquireRunLease`; `copilot-connection-probe.ts:42` (`requestedExpiresAt` +90 s) | `active` | none (no run) | `device-login-service.ts:1706,1720`; `device-login-reaper.ts:289` on session expiry | session expiry **TTL/age**; crash fallback is the orphan flip on `heartbeatRunId IS NULL` + age (`heartbeat/recovery.ts:1641-1650`) | ownership | 1,2 | A,C |
|
||||
|
||||
### Workspaces
|
||||
|
||||
Scope: workspace operations (#19807 intents), runtime leases, runtime services, remote workspace authority/snapshots/restore points, git expiration, legacy shutdown settlement, workspace recovery (system recovery / preservation holds / export receipts), observed-workspace preservation, workspace-owner recovery (record release only), WEB evidence receipts. Paths are relative to `server/src/services/` unless prefixed. Native finalization ownership internals (advisory lock, `native_run_finalizations` coordinator) are the native reader's; they appear here only where workspace records depend on them.
|
||||
|
||||
| Kind | Table.column (state col) | Writer (file:line) | In-progress states | Owner recorded | Current release rule (+ file:line) | Liveness/ownership proof (file:line) | Obligation | Invariants (1–8) | Plan sections (A–G) |
|
||||
|---|---|---|---|---|---|---|---|---|---|
|
||||
| Managed runtime-control operation (start/stop/restart/repair/run) | `workspace_operations.status` (+ `metadata.runtimeControlOwner`) | workspace-operations.ts:559-578 (stamp :539-550, 10 s heartbeat :586-612) | `running` | `{ownerId: random UUID per process, pid, heartbeatAt}`. No host, boot id, or process start time (:25-29, :69-74) | Owner settle :671-690, owner failure :701-728, own deadline 30 min / 4 h (:46, :53, :656-668). Sweep `sweepRuntimeControlOperations` :297-415, run at startup (index.ts:1087) and before each control (:448-454, routes/execution-workspaces.ts:484, :570) | Same ownerId and not in the in-process set → abandoned (:334-335). Otherwise `process.kill(pid,0)` (:126-135) or **TTL/age**: 60 s without a heartbeat (:38, :346). CAS on `updated_at` (:142-147, :379-383) | ownership | 1 (no host/boot), 2 | A, C |
|
||||
| Native workspace finalize attempt (copyback export) | `workspace_operations.status`, phase `workspace_finalize`, `metadata.owningService='native_workspace_finalizer'` | native-runtime/native-workspace-finalizer.ts:170 → workspace-operations.ts:559 | `running` (`pending` is also read as unresolved) | none on the row. Run id only. Physical owner is the `nativeWorkspaceFinalizationOwner` receipt (row below) | (a) Normal settle workspace-operations.ts:671-728. (b) #19807 supersession native-workspace-finalization-supersession.ts:21-91, called from native-workspace-finalizer.ts:94. (c) Offline maintenance offline-workspace-operation-reconciliation.ts:30-118, called from maintenance/ephemeral-migration-data.ts:52. (d) Online variant :152-157 (**no production caller**) | (b) Caller holds finalization ownership, plus the in-process registry `isNativeWorkspaceFinalizationOperationActive` (workspace-operations.ts:114-119) and a later terminal attempt. (c) Advisory lock :41, committed coordinator with no lease :52-57, finalized sync hash :58-60, accepted result :65-70, later succeeded receipt :83-92. Never age | workspace-preservation | 1, 2 | B, C |
|
||||
| Remote provision command intent | `workspace_operations.status` + `metadata.uncertain`, phase `remote_workspace_provision` | remote-workspaces.ts:973-975 (`running`, `uncertain:true`) | `running` / `uncertain=true` | `heartbeatRunId` + `metadata.epoch` (fenced by authority epoch/writerRunId :964) | Success only: remote-workspaces.ts:989-990. On a non-zero exit, a timeout (:980-982), or a crash, **no write settles the row** (grep finds no other writer) | none. Reservation for any other run is refused with `remote_workspace_provision_unverified` (:220-225). Offline migration counts it as a blocker (remote-workspace-migration-tree.ts:160, remote-workspace-migration.ts:237) | workspace-preservation | 1, 2, 3, 7 | B, C |
|
||||
| Remote snapshot save (initial/checkpoint/final) | `workspace_operations.status`, phase `remote_workspace_snapshot` | remote-workspaces.ts:729 | `running`, `pending` | `heartbeatRunId` + `metadata.epoch`. Fenced by `assertOwned` and the authority writerRunId/epoch (:776-779) | Owner settle in `saveRemoteWorkspace`. Earlier saves are retired only after a final save commits: `retireQuiescedSaves` :592-630 | Save-work quiescence (`readSaveWork` activeIds/quiescedAt :616-618) plus a committed final snapshot (:597-608). No owner-death proof | workspace-preservation | 1, 2 | B, C |
|
||||
| Remote workspace migration op | `workspace_operations.status`, phase `remote_workspace_migration` | remote-workspace-migration.ts:375 | `running` | none (workspace id) | Explicit publication or repair by migration (unverified line). The runtime-control sweep skips it on purpose (workspace-operations.ts:319-323) | none | workspace-preservation | 1, 2 | B, C |
|
||||
| Execution workspace runtime lease | `execution_workspace_runtime_leases` (row existence) | workspace-runtime-leases.ts:194-209 (create), :235-247 (renew), :255-273 (reclaim). Called only from routes/execution-workspaces.ts:567 | row present | `owner_key` `issue:`/`run:`/`agent:` (+ FK columns). No process identity | `release` :291-313. In production it is called only with `force` on archive (routes/execution-workspaces.ts:1270). Otherwise the lease is reclaimed lazily at the next claim (:250-283) | `evaluateStaleOwner` :85-119: issue hidden or terminal, run terminal, or **TTL/age** 30 min (:39, :90). An agent-scoped owner has TTL only (:116-118) | ownership | 1, 2 | A, C |
|
||||
| Workspace runtime service (local process) | `workspace_runtime_services.status` | workspace-runtime.ts (insert line unverified) | `provisioning`, `starting`, `running` | `startedByRunId`, `ownerAgentId`. The pid lives in a local service registry file, not on the row | Startup `reconcilePersistedRuntimeServicesOnStartup` workspace-runtime.ts:8636, which adopts or stops each service (index.ts:1101, not awaited) | Registry record match (workspace-runtime.ts:8781-8797) plus `isPidAlive` local-service-supervisor.ts:218, :245-250. Same host only | ownership | 1, 2 | A, C |
|
||||
| Remote workspace writer lane | `execution_workspaces.metadata.remoteWorkspace.writerRunId` (+ `epoch`) | remote-workspaces.ts:230 (epoch+1), :243 (new) | non-null writerRunId | writer run id + epoch | Final save sets it to null: remote-workspaces.ts:780-781. Migration: remote-workspace-migration.ts:516 | `readRemoteWorkspaceWriterHold` remote-workspace-writer-hold.ts:25-79: run status (:42), lease termination/stop receipts (:53-56), recovery-save progress (:63-76). The waiter gives up after **TTL/age** 60 min of writer inactivity (heartbeat/workspace-preparation.ts:1050-1051, run-dispatch/domain/wake-context.ts:36) | ownership | 1, 2, 3, 7 | A, E |
|
||||
| Remote workspace unsaved-files flag | `execution_workspaces.metadata.remoteWorkspace.dirty` | remote-workspaces.ts:780-781 (`dirty: boundary !== "final"`) | `dirty=true` | writerRunId (same object) | Final save :781. Failed-run recovery native-runtime/native-failed-workspace-recovery.ts:52-60 (pins/saves) | Writer-hold rule (above). Lease termination receipt | workspace-preservation | 1, 2 | C |
|
||||
| Remote snapshot ledger | `execution_workspace_snapshots.status` | remote-workspace-snapshot-storage.ts:73-83 (`pending`, deleteAfter = now + 7 d) | `pending`, `uploaded` | `source_operation_id`, `source_run_id` | Commit remote-workspaces.ts:782-784. Orphans become `delete_requested` by **TTL/age** 7 d from createdAt (remote-workspace-snapshot-retention.ts:9, :113-122), deferred while ops are running (:36) | none (age) | workspace-preservation | 2 | C |
|
||||
| Git restore point | `execution_workspace_restore_points.status` | git-restore-point-storage.ts:146-155 | `pending` (and `expiring`) | none (workspace + generation) | → `active` :187-188. → `superseded` :141-145, :182-185, :208-216 | none found | workspace-preservation | 1, 2 | C |
|
||||
| Git cache expiration intent | `execution_workspaces.metadata.gitCacheExpiration.completed` | git-workspace-expiration.ts:384-390, :401-406 | `completed=false` | `lockToken` (random, :384, :405). File locks git-workspace-expiration-locks.ts:72-76 | Deleting the key on completion git-workspace-expiration.ts:454. Restore/rehome deletes it (execution-workspaces.ts:4516, isolated-repository-rehome.ts:485) | Transaction advisory lock `execution_workspace_lifecycle:` (:302-303). Resumable by any worker using the same token (recover: git-workspace-expiration-locks.ts:57-63) | workspace-preservation | 1 | B, C |
|
||||
| Legacy graceful-shutdown workspace settlement fence | `environment_leases.metadata.legacyShutdownWorkspaceSettlement.state` | legacy-shutdown-workspace-settlement.ts:90-99, called at heartbeat/recovery.ts:1275 (deadline now+30 s, :1183) | `pending` | `controllerBootId` (legacyControllerBootId) + runId | Owner `settled` :103-114 (recovery.ts:1323). Expiry `expired` :190-194 via sweep :78-87 (recovery.ts:2893, :2961) | Boot id compare plus in-process control (:144-146) **and** deadline **TTL/age** 30 s, renewed every 10 s (:12-13, :32-46, :147-148). A boot-id mismatch alone does not release | workspace-preservation | 2 | A, C |
|
||||
| Legacy restore-recovery preservation hold | `environment_leases.lease_policy='retain_on_failure'` + `metadata.workspaceRestoreRecovery` | legacy-shutdown-workspace-settlement.ts:178-182. legacy-execution-recovery.ts:506 (`recordRequiredWorkspaceRestoreFailure`) | hold present | runId in the marker | `scheduleRecoveredWorkspaceCleanup` workspace-recovery.ts:488-508, only with `hasVerifiedWorkspaceRepair` (workspace-recovery-receipt.ts:90-166) | Saved-file receipt (succeeded `workspace_recovery` op plus `workspaceRecovery.status=recovered`) | workspace-preservation | 2, 7 | B, C |
|
||||
| Native terminal-export retention | `environment_leases.metadata.nativeWorkspaceExportResume` (+ status `pending_cleanup`) | native-runtime/native-workspace-export-resume.ts:45-69 | marker present | `requestId` (UUID) + runId/resultId | `releaseCompletedNativeWorkspaceExportRetention` :74-100 (needs a **committed** coordinator plus a finalized sync hash plus destroy disposition). `settleNativeWorkspaceExportResume` :103-125 on a stopped receipt. sandbox-stop-and-retain.ts:212 | Provider stop receipt, or the finalized host sha | workspace-preservation | 2 | B, C |
|
||||
| Recovery cleanup in-flight claim | `environment_leases.metadata.pendingCleanupInFlight` / `pendingCleanupAttemptId` / `pendingCleanupLeaseExpiresAtMs` | workspace-recovery.ts:252-253, :283 (keepalive :288-291) | `pendingCleanupInFlight=true` | attempt id = finalization ownership token | Cleared at workspace-recovery.ts:508. Other paths belong to the environment reader | **TTL/age** 15 min (`expiresAt > now`, legacy-shutdown-workspace-settlement.ts:165-169) | workspace-preservation | 1, 2 | A, C |
|
||||
| System recovery attempt for retained work | `issue_recovery_actions.evidence.systemRecovery.state` | workspace-recovery.ts:184-191 (`reserveSystemRecoveryAttempt` system-recovery-state.ts:91-101, CAS `updateSystemRecovery` system-recovery.ts:65) | `scheduled`, `running`, `awaiting_evidence` | `attemptId` (UUID) + `generationId`. The attempt runs inside `withNativeWorkspaceFinalizationOwnership` (workspace-recovery.ts:443-444) | `complete` workspace-recovery.ts:392. `exhausted` by **TTL/age** window 30 min (system-recovery-state.ts:8, :97, :110; workspace-recovery.ts:187-189). Liveness stamp `systemRecoveryActiveAt` every 15 s (:199-208) | Finalization ownership lock or receipt (native reader). A `running` attempt is re-entered without any liveness check (system-recovery-state.ts:96) | workspace-preservation | 2, 4, 5 | C, F |
|
||||
| Workspace recovery request | `issue_recovery_actions.evidence.workspaceRecovery.status` | workspace-recovery.ts:119-153 (`queued`, stage `verifying_stop`) | `queued`, `recovering` | runId + requestIds | → `recovered` / `blocked` (workspace-recovery-receipt.ts:9-22). Completed condition :28-50 | Same as system recovery | workspace-preservation | 4, 7 | E |
|
||||
| Copyback-owner hold (RC-50) | `issue_recovery_actions.evidence.workspaceOwnerRecovery.state` | native-runtime/native-workspace-finalization-ownership.ts:184, :200. workspace-recovery.ts:466-480 | `operator_required` | recorded `owner` receipt (token, hostname, pid, processStartedAt, controllerBootId) | Board confirmation native-runtime/native-workspace-owner-recovery.ts:45-105, :114-164 (→ `resuming` :153-157). Ownership path ownership.ts:308 | Same host: `evaluateNativeControllerTakeover` (owner-recovery.ts:33-39). Other host: board-attested `stopEvidence` only | ownership | 2, 7 | A, C |
|
||||
| Native copyback owner receipt | `heartbeat_runs.runner_profile_json.nativeWorkspaceFinalizationOwner` | native-runtime/native-workspace-finalization-ownership.ts:108, :259 | present | token, hostname, pid, controllerBootId, processStartedAt | Owner removes it at ownership.ts:344. Board removes it at owner-recovery.ts:86-89, :148-152 | Native reader (advisory lock ownership.ts:31-32, :121; boot/pid :208-220; platform proof) | ownership | 1, 2 | A |
|
||||
| Native workspace sync reference (sync-out / copyback) | `heartbeat_runs.runner_profile_json.nativeWorkspaceSync.state` (+ lease stamp `environment_leases.metadata.nativeWorkspaceSync`) | native-runtime/native-workspace-sync.ts:1112 (`prepared`, via `persistRunReference` :612-637). Lease stamp :640-663 | `prepared` | none (run id; binding includes leaseId/providerLeaseId) | → `finalized` by `finalizePreparedRuntime` :848-886 (under finalization ownership). Local files removed by `cleanupNativeWorkspaceSync` :1254 | Relies on finalization ownership (native reader) | workspace-preservation | 1, 2 | B, G |
|
||||
| Observed-run workspace preservation fence | `issue_recovery_actions.evidence.observedWorkspacePreservation.state` | native-runtime/native-observed-workspace-preservation.ts:182-187 (called from native-failed-workspace-recovery.ts:195) | `stop_pending` | coordinator `leaseOwner`, `controllerGeneration`, `providerAttempt`, source/coordinator sha256 | → `stopped` :218, called from native-environment-settlement-work.ts:105. **The same transaction sets `native_run_finalizations.phase='terminal_failure'` (:208-214)** | Fence compare (:144-157), remote termination receipt plus confirmed stop-and-retain (:205). The expired coordinator lease (:86) is **TTL/age**, gated by a 1 s lock_timeout (:32) | workspace-preservation | 1, 2, 6 | A, C, G |
|
||||
| WEB evidence receipt | `heartbeat_runs.runner_profile_json.nativeHarnessFilesystemIndependence` (schema `paperclip.native-harness-web-evidence.v1`) | native-runtime/native-harness-filesystem-independence.ts:703-743 (write :735) | terminal receipt (no in-progress state) | `lastRunId`, `sourceRunId`, `scopeDigest`, factsDigest | Never released. Revalidated by `hasValidNativeHarnessFilesystemIndependence` :203-218 | Digest/fact compare (:191-201). Consumed only by the backup-migration gate (native-harness-backup-migration.ts:20-30) | workspace-preservation | — | B |
|
||||
|
||||
Non-durable, excluded: transaction advisory locks (execution-workspace-lifecycle-lock.ts:15, :63; workspace-source-lock.ts:50-51) end with the connection. The in-memory maps `activeRuntimeControls` (workspace-operations.ts:170-198), `liveRuntimeControlOperationIds` (:102) and `liveNativeFinalizationOperations` (:112) are also excluded. `notify*Work` calls (e.g. terminal-workspace-work.ts:6) are in-process hints, not outbox rows. `isRuntimeOwnedGitBranch` (execution-workspace-branch-ownership.ts:10-14) is a static cleanup-authority flag (`metadata.createdByRuntime` + version) with no in-progress state.
|
||||
|
||||
### Accounting and budgets
|
||||
|
||||
Scope: `server/src/services/{budgets,budget-exposure,run-cost-accounting,budget-reservations,budget-block-ownership,budget-evaluation-work,usage-receipts,accounting-transaction}.ts`, `native-runtime/native-usage-delivery.ts`, schema `packages/db/src/schema/{accounting,budget_evaluation_work,heartbeat_runs}.ts`. Paths below are relative to `server/src/services/` unless prefixed. Line numbers are from origin/master at f2d8a51027.
|
||||
|
||||
| Kind | Table.column (state col) | Writer (file:line) | In-progress states | Owner recorded | Current release rule (+ file:line) | Liveness/ownership proof (file:line) | Obligation | Invariants (1–8) | Plan sections (A–G) |
|
||||
|---|---|---|---|---|---|---|---|---|---|
|
||||
| Run budget reservation | `budget_reservations.state` (run_id set; schema `packages/db/src/schema/accounting.ts:52-75`) | `budget-reservations.ts:138` (`reserveRunBudget`, called from `heartbeat.ts:2938`) | `held` | none on the row. Uses `run_id` only. Recovery reuse is gated by `native_run_finalizations.lease_owner` plus `lease_expires_at > clock_timestamp()` (`budget-reservations.ts:99-103`) | → `settled`/`released` only inside `accountRunCost` (`run-cost-accounting.ts:278`). Undispatched release: the `heartbeat.ts:2953-2961` finally block sets `accountingProviderWorkStarted:false`, then calls `accountRunCost` | The reservation has no proof of its own. It settles when run accounting settles (see row 2). The recovery-owner check is a **TTL lease** (`budget-reservations.ts:22`, `:52`, `:103`) | accounting | 1, 2, 3, 5 | B, D |
|
||||
| Run accounting debt | `heartbeat_runs.cost_accounting_pending` (+ `cost_accounted_at`) | `budget-reservations.ts:140`; `heartbeat/run-completion.ts:665`; `heartbeat/runtime-execution.ts:968`; `heartbeat.ts:2955`; `native-runtime/native-session-executor.ts:11729`; `usage-receipts.ts:162,187,214,624` | `true` while the run is terminal | none. The run id is the identity, and the receipt source is `usageJson.accountingReceiptSourceId` | → `false` at `run-cost-accounting.ts:277` (settle), `:235` (already accounted) and `:355` (stays `true` when late debt exists). Driven by `reconcileRunCosts` (`:435`) over `actionableRunAccounting` (`:382-399`) | Closure proofs (`run-cost-accounting.ts`): `closedNativeReceiptSource` `:90-99` (local: pid/pgid null **and** last process event = `native.local_process_stopped`; remote: termination receipt on every lease `:57-70`); `nativeProviderWorkNeverStarted` `:108-120`; `preProviderFailure` flags `:41-42`, `:253-254`; **TTL/age**: `closureEvidenceTimedOut` `:138-144` (`ACCOUNTING_CLOSURE_TIMEOUT_MS` = 7 min + 3 min = 10 min, `:135`) | accounting | 1, 2, 3 | D, C |
|
||||
| Receipt-closure wait marker | `heartbeat_runs.usage_json->>'accountingReceiptReady'` (`false`) | `budget-reservations.ts:141`; `native-runtime/native-usage-delivery.ts:37`; `usage-receipts.ts:584`, `:188`, `:624` | `false` | the source id (`accountingReceiptSourceId`), a recorder uuid (`usage-receipts.ts:553`) or `uuid(runId:runnerInstanceId)` (`native-usage-delivery.ts:26`) | Cleared when a complete receipt is persisted (`usage-receipts.ts` ~`:214-223`, unverified exact line). Otherwise settled as unpriced or a lower bound by `accountRunCost` once closure is proven or the closure timeout passes (`run-cost-accounting.ts:268-271`) | Same closure proofs as row 2, plus the 10-min **TTL/age** fallback | accounting | 2, 3 | B, D |
|
||||
| Usage-capture failure | `usage_json->>'accountingCaptureFailed'` (`true`) | `usage-receipts.ts:624` (`persistFailure`); `:188` (receipt after source handoff) | `true` | source id (guarded at `usage-receipts.ts:618`, `:627`) | **No automatic release.** `accountRunCost` returns false (`run-cost-accounting.ts:250`), and `actionableRunAccounting` excludes the run (`:389`). The run stays `cost_accounting_pending=true` until an operator correction (path unverified) | none | accounting | 2, 3, 7 | D |
|
||||
| Late-receipt debt | `usage_json->>'accountingLateReceiptPending'` + `accountingLateReceiptDebts` | `usage-receipts.ts:162-165` | `true` | none (debt map keyed by receipt id) | `reconcileLateRunUsageReceiptInTransaction` (`usage-receipts.ts:295`, update `:342`), called from `run-cost-accounting.ts:231` | the receipt id compare `accountingLateReceiptId` vs `AttemptedId` (`run-cost-accounting.ts:387`) | accounting | 2, 4 | D |
|
||||
| Receipt-source fence map | `usage_json->'accountingReceiptSourceFences'` (map sourceId→last sequence) | `usage-receipts.ts:581-584` (`createRunUsageRecorder`: **always writes the key, `{}` on first bind**); `native-runtime/native-usage-delivery.ts:32-35` (adds an entry only when a prior source exists) | n/a. This is a fence, not a state. Read as "provider work began / restart-derived" when non-empty | prior source ids | never released; it is history | Read as evidence at `run-cost-accounting.ts:112` (empty ⇒ provider never started) and `budget-exposure.ts:168` (non-empty ⇒ restart-derived ⇒ exempt from the unpriced block). Fencing of late producer receipts happens at `usage-receipts.ts:118-119`, `:175-187` | accounting | 1, 6 | B, G |
|
||||
| Usage-receipt journal row | `run_usage_receipts` (`complete` bool) (`schema/accounting.ts:23-38`) | `usage-receipts.ts:110-113` | `complete=false` (partial checkpoint) | `source_id` + `sequence` (unique `:35`) | immutable. Its existence is read as "provider started" (`run-cost-accounting.ts:113-114`) | none (append-only) | accounting | 4 | B |
|
||||
| Usage-receipt spool file (durable file) | `<instanceRoot>/accounting-receipts/<id>.json` (`idle-local-work.ts:18`), with `retry-*.json` on failure | `usage-receipts.ts:63-75` (`spoolUsageReceipt`) | file present | none (the envelope carries company/run/source) | `replayUsageReceipts` deletes the file after the DB persist (`usage-receipts.ts:454-475`). Recorder bind imports and removes it (`:567-586`). `accountRunCost` removes it after commit (`run-cost-accounting.ts:375-378`) | none. A local file on an ephemeral filesystem is lost on host replacement (unverified against `ephemeral-server-contract.ts`) | accounting | 2, 4 | B, C |
|
||||
| Recorder durability probe file | `<spool>/<sourceId>.probe` | `usage-receipts.ts:557-558` | transient | none | removed in a `finally` at `:558`. Leaks on SIGKILL | none | accounting | 8 | — |
|
||||
| Budget-evaluation input | `budget_evaluation_work` row (`schema/budget_evaluation_work.ts:5`) | `budget-evaluation-work.ts:10` (called on every reservation, settlement and receipt) | row exists | none | consumed by `flushBudgetEvaluationWorkPage` (`budgets.ts:1283-`), from the budgetEnforcement worker (`budget-enforcement-worker.ts:48`; retry 1 s, coalesce 1 s `:35-36`) | none. Admission blocks when the oldest row is **> 15 s old** (TTL/age, `budgets.ts:866-870`, `budget-evaluation-work.ts:23-29`) | accounting | 3 | D, E |
|
||||
| Scope budget hold | `companies.pause_reason='budget'`, `projects.pause_reason='budget'`, `agents.lifecycle_holds @> ['budget']` (`budgets.ts:126,148,168`) | `budgets.ts:403`, `:427` (`pauseAndCancelScopeForBudget` `:434`) | held | none | `resumeScopeFromBudget` (`budgets.ts:448`), called when no policy reaches a hard stop (`:826-833`, `:752`). Admission refuses while held (`:860-861`) | `hardStopReached` (`budgets.ts:272-275`) | accounting | 3 | D |
|
||||
| Service-invocation reservation (decision / prompt-injection / fast-response) | `budget_reservations.state` (one of the `*_invocation_id` / `fast_response_request_id` set) | `decision-models.ts:284`; `prompt-injection-prevention.ts:243`; `fast-responses.ts:945` | `held` | none (invocation id) | settle/release on completion only when `costStatus !== 'unpriced'` (`decision-models.ts:217`, `prompt-injection-prevention.ts:178`, `fast-responses.ts:691-693`). **Unpriced ⇒ stays held** until a manual cost correction (`billing-reconciliation.ts:151,159,165`) | **TTL/age**: stale `running` invocations are settled as interrupted after 120 s (`decision-models.ts:302`, `prompt-injection-prevention.ts:269`); fast-responses use `expiresAt` + 30 s (`fast-responses.ts:1120-1126`). Wake deadline 120 s (`run-cost-accounting.ts:406-413`) | accounting | 2, 3 | B, D |
|
||||
| Browser-use budget hold | `browser_use_runs.result->'paperclipBudgetHold'->>'state'` | `browser-use.ts:586-588`; `dispatchCommittedAt` `:638`; re-held `:1127-1128` | `held` (+ `billingUnknown`) | `invocationId` (checked at `budget-exposure.ts:94`); lease fence `runLeaseFence` (`browser-use.ts:1250`) | `released` (`browser-use.ts:682`, `:1250`), `settled` (`:1147-1150`) | provider run id / `dispatchCommittedAt` (`browser-use.ts:634`, `:1240-1244`) | accounting | 1, 2, 3 | B, D |
|
||||
| Native finalization lease, used here as a recovery-owner proof | `native_run_finalizations.lease_owner/lease_expires_at` (owned by the native-runtime reader) | not written here | n/a | `lease_owner` | n/a | **TTL lease** read at `budget-reservations.ts:20-23`, `:49-52`, `:101-103`. `resultId`/`phase` read as "recovering" in `budget-exposure.ts:142-146` and as "settled coordinator" in `run-cost-accounting.ts:85-88` | ownership | 1, 2 | A |
|
||||
|
||||
### Wake queue and recovery
|
||||
|
||||
Scope: `server/src/services/heartbeat/queue.ts` (q), `server/src/modules/wake-queue/**` (wq), `server/src/services/recovery/service.ts` (rs), `heartbeat/recovery.ts` (hr), `heartbeat/run-state.ts`, `idle-retry-schedule.ts`, `legacy-process-settlement.ts` (lps), `case-handoff-lifecycle.ts` (chl), `run-log-authority.ts`, `execution-blocker.ts` (eb), `pre-dispatch-*.ts`, `explicit-native-continuation.ts` (enc), `legacy-run-task-ownership.ts`, the claim paths in `heartbeat.ts` (hb), and `legacy-controller-lease.ts` (lcl), the lease these paths use.
|
||||
All paths are relative to `server/src/services/` unless noted. Native-run finalization leases (`native_run_finalizations`) appear here only where this subsystem reads them. They belong to the native/upgrade inventory.
|
||||
|
||||
| Kind | Table.column (state col) | Writer (file:line) | In-progress states | Owner recorded | Current release rule (+ file:line) | Liveness/ownership proof (file:line) | Obligation | Invariants (1–8) | Plan sections (A–G) |
|
||||
|---|---|---|---|---|---|---|---|---|---|
|
||||
| Wake request (queued) | agent_wakeup_requests.status (db schema agent_wakeup_requests.ts:29) | q:4322/4331/4421 (enqueue); q:1643 (re-queue after suppression race) | `queued` | none (runId link only, set at run creation) | → `claimed` at run claim q:1412, q:4885; → `coalesced` q:4464/4473, wq/adapters/postgres.ts:1060; → `skipped`/`cancelled` by gates (§1b) | none. It is valid while its run row is queued. | ownership (work-intent delivery) | 4, 7 | E |
|
||||
| Wake request (deferred behind issue lock) | agent_wakeup_requests.status = `deferred_issue_execution`, payload.executionWait, payload._paperclipWakeContext | q:3384 (deferBlockedExecution); q:4039; deferred-execution-wait.ts:13 (updates condition) | `deferred_issue_execution` (runId null) | none. It implicitly waits on issues.executionRunId. | Release drain wq/application/use-cases.ts:135. Promotion is a CAS in wq/adapters/postgres.ts:497 → new queued run. Terminal endings: `cancelled` (postgres.ts:353), `failed` (postgres.ts:387, when the agent is not invokable). Backstop: deferred-receipt-work.ts:90/111 rechecks updatedAt+30 s. | The issue lock is released: run terminal + `hasPendingRunCleanup` false (eb:70) | ownership (work-intent delivery) | 3, 4, 7 | E, D |
|
||||
| Wake request (claimed) | agent_wakeup_requests.status = `claimed`, claimedAt, runId | q:1412, q:4887 | `claimed` | runId | Its run settles it: `setWakeupStatus(...,"failed")` hr:2724, q:743 (`skipped`/`failed` on a chat-control stop). Native-status intents: re-dispatch at claimedAt + 60 s (deferred-receipt-work.ts:88/109) | Run status. For native-status intents, **TTL/age 60 s** | ownership | 2, 4 | E, C |
|
||||
| Queued run | heartbeat_runs.status = `queued` | wq/adapters/postgres.ts (finalizePromotedWake), q enqueue | `queued` | none | claimQueuedRun q:808 → `running`. The claim cancels the run when: agent gone/invalid org chain (q:880-898), budget hard stop (q:917), daily cap (q:933 → q:437), pause hold (q:970-1007), blocked dependencies, stale gate (run-dispatch adapters/postgres.ts:1137). Startup: q:1897 fast path, q:1863 full pass | n/a | ownership | 3, 4, 8 | D, E |
|
||||
| Legacy controller lease (running run) | heartbeat_runs.controllerBootId, controllerLeaseExpiresAt, executionStage (`preparing`→`dispatching`→`settled`) | lcl:30 `legacyControllerClaim`, committed with the claim UPDATE at q:1321/q:1428/q:1495. Renewed every 10 s: lcl:41/lcl:242 | status `running` + lease not expired | controllerBootId = `legacyControllerBootId` (lcl:8, random UUID per process) | Normal: clear lease at hb:4044-4050. Crash: `recoverActiveExecutions` hr:2374 → `revokeExpiredLegacyControllerWithState` lcl:116 (swaps in a random boot id) → `failed/process_lost` hr:2696 → `releaseIssueExecutionAndPromote` hr:2763 | **TTL/age** lease expiry 60 s (lcl:9, lcl:132). Then **TTL/age** `updatedAt` ≥ 5 min (hr:2378, hr:2580), skipped only when stage=`preparing`, the boot id differs and there is no pid (hr:2572). Boot-id compare lcl:51. Local child pid/group check `isProcessAlive` hr:308 (no host check for legacy) | ownership | 1, 2, 8 | A, C, G |
|
||||
| Legacy run process identity | heartbeat_runs.processPid, processGroupId, processStartedAt | adapter spawn (via run-state/heartbeat; writer not traced here, **unverified**) | `running` with pid recorded | pid + pgid (no host stamp) | hr:2600-2630: if the pid is alive → mark `process_detached` and recheck every 30 s. rs:6096 `terminalizeOrphanedRunningRun` → `interrupted/orphaned_running_run` | hr:308 `isProcessAlive` (EPERM=alive) vs rs:6186 `isPidAlive` local-service-supervisor.ts:245 (EPERM=dead). Neither checks host/pid namespace for legacy. | ownership | 1, 2 | A |
|
||||
| Detached-process marker | heartbeat_runs.errorCode=`process_detached` while `running` | hr:2618-2621 | `running` | pid/pgid | Cleared only when the pid dies, then the next sweep reaps the run (hr:843-845 adoption path). Rechecked every 30 s (hr:2636) | pid liveness hr:2598-2605 | ownership | 2, 7 | A, C |
|
||||
| Hot-restart adoption marker | heartbeat_runs.resultJson.hotRestartAdoption | hr:826 (`mergeHotRestartAdoptionResultJson` hr:351) during `reconcileHotRestartAdoption` hr:639 | `running` | pid/pgid + intent | Run stays running while the pid lives. The reaper skips it and rechecks every 30 s (hr:2607-2610) | `isProcessAlive`/`isProcessGroupAlive` hr:815-816 | ownership | 2 | A, C |
|
||||
| Native run controller lease (read here) | native_run_finalizations.controllerBootId/controllerPid/controllerProcessStartedAt/leaseExpiresAt/leaseOwner/controllerGeneration | native runtime (out of scope) | phase `observed`, `retryable_failure`, recoveryState `awaiting_*` | boot+pid+start time (+ generation) | hr:941 `recoverNativeRunsAfterRestart`. Unprovable cases are held with `markNativeOwnershipUnverified` hr:2019, called at hr:2542 | Exact identity match hr:2491-2498. Locality `nativeProcessIsLocal` hr:1995 (hostname + process-identity event). **TTL/age** lease expiry hr:2410-2412, hr:2503 | ownership | 1, 2, 6 | A, C, G |
|
||||
| Issue execution lock | issues.executionRunId, executionAgentNameKey, executionLockedAt | q:1139 `bindClaimedIssueExecution`; q:1569; q:3729; retries.ts:1248 (handed to the scheduled retry); wq/adapters/postgres.ts:563/718/784/1403; run-lifecycle.ts:2553 | non-null | run id | Release: wq use-cases.ts:983 → postgres.ts:1255. Cancellations clear it at q:762/1276/1391/1653/1703/3536/3687/3919. Backstop: `sweepStaleIssueLocks` rs:6351 (clears at rs:6519, but not while the run is live or a `retryable_failure` resume is pending, rs:6462-6536) | The successor claim requires the predecessor run to be terminal, not in `liveRunExecutions` (process-local), and `hasPendingRunCleanup`=false (q:1126-1129, eb:70). `hasPendingRunCleanup` fails closed: `state?.pending !== false` (eb:102) | ownership | 2, 3, 7 | C, D |
|
||||
| Issue checkout lock | issues.checkoutRunId | issues.ts:7853/7915 (agent checkout), plugin-host-services.ts:2305 | non-null | run id | issues.ts:7976/8054/12823; retries.ts:1249; rs:6519 | Referenced run is terminal (rs:6452 `isCleanable`). No process proof. | ownership | 2, 7 | C |
|
||||
| Scheduled retry run | heartbeat_runs.status=`scheduled_retry`, scheduledRetryAt/Attempt/Reason | retries.ts:1120 (keeps the issue lock, retries.ts:1248) | `scheduled_retry` | none (time-owned) | Due time → `promoteDueScheduledRetries` retries.ts:1757 → run-dispatch use-cases.ts:32. Cancellation sweep scheduled-retry-cancellation.ts:13 (hr:2804) | Due time (schedule). Idle sleep allowed only with a same-boot settlement receipt (idle-retry-schedule.ts:100/123/132) | ownership | 5, 7 | F, E |
|
||||
| Retry/attempt budget counters | heartbeat_runs.contextSnapshot.executionRetryAccounting.failureRetries; scheduledRetryAttempt; processLossRetryCount (db heartbeat_runs.ts:107) | retries.ts:535/589/1080/1129 via `accountingForScheduledRetry` execution-recovery-attempt.ts:69 | counter < max (2: retries.ts:135-152) | n/a | Exhaustion → not_scheduled (retries.ts:530-560) | n/a. See §5 | accounting (attempt budget) | 5 | F |
|
||||
| Execution reconciliation hold | issue_recovery_actions.status ∈ {active, escalated} with cause ∈ EXECUTION_RECONCILIATION_CAUSES (packages/shared execution-projection.ts:59) | rs (stranded recovery, e.g. rs:1958, rs:3110); enc:785 | `active`, `escalated` | ownerType/ownerAgentId/ownerUserId (a person or agent, not a process) | Operator resolution; `reconcileActiveRecoveryActions` rs:3461; enc:627 (resolved on explicit continuation) | Inferred evidence: `executionBlockerPredicate` eb:106, `getExecutionBlocker` eb:224, `noRetainedExecutionPredicate` pre-dispatch-execution-evidence.ts (about 12 absent-column checks) | ownership | 3, 7 | C, D |
|
||||
| Legacy process settlement receipt | heartbeat_runs.legacyProcessSettlement (json receipt) | lps:66 `recordLegacyProcessSettlement`, lps:96 preparation variant (called at hb:4082/4091) | null while not settled | controllerBootId (must equal **current** boot, lps:68/98) | Written once by the owning boot after cleanup | Exact identity match lps:47. Only the same boot can write it. | ownership | 2 | A, B |
|
||||
| Case-handoff source release marker | heartbeat_runs.resultJson.caseHandoffExecutionReleased; case_handoffs.state (`awaiting_release`, `pending_dispatch`, `delivered`) | chl:232; cleanup intent event chl:29 (written before provider dispatch, hb:3009/3543) | `awaiting_release` | controllerBootId + host + pid in the intent payload (chl:40-42) | `isCaseHandoffSourceReleased` chl:249. Blocks the successor claim (q:1080-1092) | `localProcessAbsent` chl:23 (ESRCH only), hostname match chl:89, remote stop receipts chl:265 | ownership | 2, 3 | A, B |
|
||||
| Run-log writer authority (read-only gate) | heartbeat_runs.controllerBootId/leaseExpiresAt; native_run_finalizations.leaseOwner/controllerBootId/leaseExpiresAt | n/a | `running` | boot id | Throws `run_log_writer_authority_lost` (run-log-authority.ts:30-60) | Boot-id compare and **TTL/age** lease expiry (run-log-authority.ts:31-35, 52-58) | ownership | 6 | A, G |
|
||||
| Pending-cleanup environment lease claim (read here) | environment_leases.metadata.pendingCleanupController/InFlight/LeaseExpiresAtMs | env runtime | `pending_cleanup` + in-flight | `ControllerProcess` (bootId, pid, hostHash) | `releaseLostPendingCleanupClaims` hr:1442 | `controllerProcessExited` controller-process-identity.ts:83 (the shared primitive) | workspace-preservation | 1, 2 | A, C |
|
||||
| Orphaned active environment lease (read here) | environment_leases.status=`active` for a terminal run | env runtime | `active` | heartbeatRunId | `sweepOrphanedActiveLeases` hr:1625 | **TTL/age**: updatedAt ≤ now − backoff (5 min in live work hr:2960; staleThreshold=0 at startup) hr:1631/1651 | workspace-preservation | 2 | C |
|
||||
|
||||
#### 1b. Terminal wake outcomes and temporary refusals
|
||||
|
||||
"T" means the refusal can be temporary. Each of these writes a terminal row (`skipped`, `cancelled` or `failed`) with no promotion path. A durable caller receipt (`executionWaitRequestId`) is the exception: it is kept as `deferred_issue_execution` (q:2452-2459).
|
||||
|
||||
- `heartbeat.scheduling_suppressed` (q:2527). **T**: a DB restore or worktree mode ends. Task drain is exempt (q:2524).
|
||||
- `company.inactive` (q:2550). **T** if the company is paused. A user request gets a 409 instead.
|
||||
- `budget.blocked` (q:2732). The skip row is written at admission for any budget block. The claim path treats only a board-owned block as terminal (q:917). Admission does not make that distinction. **T** for accounting-settling holds.
|
||||
- `agent.not_invokable` (q:2747). **T** for paused or `pending_approval`. Setup is exempt (q:2744).
|
||||
- `heartbeat.disabled` / `heartbeat.wakeOnDemand.disabled` (q:2761/2765). Configuration.
|
||||
- `heartbeat.timer.no_actionable_work` (q:2774). Not work.
|
||||
- `heartbeat.worktree_execution_cutoff` (q:2649, q:3465). Configuration.
|
||||
- `issue_tree_hold_active` (q:2799). **T**. The tree-resume path only revisits *cancelled runs* (issue-tree-resume.ts:171-199), not skipped admission wakes. Another re-wake path is **unverified**.
|
||||
- `issue_dependencies_blocked` (q:3817 via execution-wait.ts:14). **T**. Covered by the separate dependency backstop rs:5782, which re-wakes rather than promoting this row.
|
||||
- `execution_reconciliation_required` (q:3384 else branch, recordExecutionWait). **T**.
|
||||
- `issue_rewake_throttled` (q:4224). **T**.
|
||||
- Daily cap (q:4265, q:4674). Claim-time path q:437 → run `cancelled`, wake `skipped`. **T**: resets daily.
|
||||
- Chat-control recovery `unresolved` (q:3001 admission; q:695-760 dispatch → run `failed` + wake `failed`). **T**: the code is `CHAT_CONTROL_RECOVERY_UNRESOLVED_CODE`, meaning the stop proof is not yet available. At claim it returns null and stays queued (q:695). At admission and dispatch it is terminal.
|
||||
- Issue missing or state-guard mismatch (q:3149, q:3211), native-status target gone/reassigned (q:4965), worktree needs project code (q:3949). Task changed, so these are legitimately terminal.
|
||||
- Release drain (wq use-cases.ts):
|
||||
- `cancel_pause_hold` → `cancelled` (use-cases.ts:311, policy.ts:192). **T**: a pause hold is temporary.
|
||||
- `fail_not_invokable` → `failed` (postgres.ts:387). **T** for `pending_approval` and for manager_missing/reporting_cycle. Only `paused` defers (policy.ts:196, postgres.ts:230).
|
||||
- Queued-comment discard, terminal task, or reassignment cancels: legitimate.
|
||||
- Claim (q:880-898): `invalidOrgChain`/`terminated` cancels queued runs (agent-invokability.ts:162). Org chain is **T**: a manager can be re-added.
|
||||
@@ -0,0 +1,195 @@
|
||||
# TTFT, boot and restart-recovery baseline (Phase 0)
|
||||
|
||||
Status: Phase 0 deliverable B, 2026-11-07. Phase 1 sets p50/p95 budgets from these numbers (plan, "Performance constraint", rule 4).
|
||||
Plan: [`2026-11-07-durable-ownership-and-reconciliation.md`](2026-11-07-durable-ownership-and-reconciliation.md) (#19831). Companion: [`2026-11-07-ownership-inventory.md`](2026-11-07-ownership-inventory.md).
|
||||
|
||||
## Summary
|
||||
|
||||
- **Runner launch and provider session start dominate TTFT.** Core work before dispatch (queue plus prepare) takes about 0.6 s at p50 and up to 4 s at p95. Starting the runner and the provider session takes 1.6–9.4 s at p50 (cold) and up to 50 s at p95. The cause: `lifecycleMode: per_turn` is the product default (`server/src/services/agent-runner-selection.ts:101`), so every turn, even a follow-up, launches a new runner and then bootstraps or resumes the provider session. The run log has 101 `native.process_start_requested` rows across the 96 runs in the data set: 78 measured turns plus 18 restart runs. Every run launched at least one runner process.
|
||||
- **A follow-up turn is not faster than a new task.**
|
||||
- codex: 8.1 s against 5.0 s (TTFT p50);
|
||||
- opencode: 21.8 s against 14.3 s;
|
||||
- claude: faster (7.7 s against 13.6 s), but it waits 3.7 s between `turn.submitted` and `turn.started`.
|
||||
|
||||
Session resume (`runner.session.resume`) has the worst p95 of any pre-turn step: 48.7 s. Keeping a live runner and provider session between turns (plan rule 2) is the largest TTFT lever this baseline found.
|
||||
- **Control-plane restart: 13–30 s to ready, then 5–21 s to the next provider event.** Of the restart, `recovering` → `ready` takes 0.3–1.7 s. Process boot takes the rest (a dev-mode `tsx` server with UI middleware; production boot was not measured). After ready, the recovered run's next provider event arrives in 5–6.5 s for codex, 8–18.5 s for claude and 17.6–32 s for opencode (p50–p95, run 4). The overnight chaos evidence shows longer tails, 40–150 s at p95. On Daytona, `awaiting_evidence` recoveries take 33–43 s, which is consistent with the 60 s remote controller lease that only age releases (inventory: `remoteLeaseFenced`, "Record kinds from the zero-downtime upgrade work").
|
||||
- **Remote paths were not measured.** Code reading found larger waits before `turn.start` on remote paths:
|
||||
- a full workspace capture and upload on a first remote run;
|
||||
- a sandbox resume loop of up to 60 s;
|
||||
- a remote runner identity poll of up to 20 s.
|
||||
|
||||
They are in the table below and need a Daytona baseline in Phase 1.
|
||||
|
||||
## Method
|
||||
|
||||
- **Source of truth:** the run log only. The data is the `run.performance.span`, `run.startup.reuse` and `native.recovery.transition` rows in `heartbeat_run_events`, read through `GET /api/heartbeat-runs/:id/events`. Nothing was read from Telemetry (AGENTS.md §5.7).
|
||||
- **Driver:** [`tests/restart-chaos/ttft-baseline.ts`](../../tests/restart-chaos/ttft-baseline.ts) starts one isolated control plane with the chaos supervisor: a temporary `PAPERCLIP_HOME`, an embedded Postgres owned by the wrapper, and the debug native runner. It seeds the Chaos Lab and then, per adapter:
|
||||
- **cold:** a new task (a fresh provider session);
|
||||
- **warm:** a follow-up comment on the same finished task (the provider session is reused);
|
||||
- **restart:** a task inside a `python3 -c "import time; time.sleep(75)"` shell call when the control plane is killed with `K1` (SIGKILL) or stopped with `K2` (SIGTERM). The fault fires only while each tracked run has that call open, and every run is rechecked just before the fault. A tool that exposes its command (codex, claude) must show `time.sleep(75)`. OpenCode's `bash` tool exposes no command text, so for opencode any open shell call counts, and the sample is logged as `baseline_restart_command_unverified`.
|
||||
|
||||
[`ttft-baseline-stats.ts`](../../tests/restart-chaos/ttft-baseline-stats.ts) computes every number below. Its unit tests are in `ttft-baseline-stats.test.ts`. To reproduce:
|
||||
|
||||
```sh
|
||||
set -a; source ~/.secrets; set +a
|
||||
node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/ttft-baseline.ts --adapters codex,claude,opencode --samples 6 --restarts 3
|
||||
node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/ttft-baseline.ts --report tests/restart-chaos/results/<wave>/baseline-data.json
|
||||
```
|
||||
- **Revision:** `origin/master` `f2d8a51027`, plus #19838 (the TTFT anchor, since merged as `e2d490c61b`). Runs 1 and 3 had #19838 applied, so TTFT carries its `anchor` attribute. Run 4 measured restart recovery only.
|
||||
- **Host:** Apple M5 Max, 18 cores, 128 GB, macOS. The adapters run in parallel, one task per adapter at a time. The host also ran other agent worktrees, so the p95 tails include machine load.
|
||||
- **Models:** the runner-e2e profiles. These are `runner-codex` (`gpt-6.1-sol`), `runner-acpx-claude` (`claude-sonnet-5`) and `runner-opencode` (`openrouter/deepseek/deepseek-v4-flash-0731`). Every task is a one-line reply, so TTFT is dominated by startup, not by generation.
|
||||
- **Environment:** local only. Daytona was not measured fresh. The Daytona credential is present, but a Daytona wave needs an immutable runner image whose content matches this controller (`assertControllerMatchesImage` in `tests/restart-chaos/daytona-image.ts`). No such image exists for this revision, and building and qualifying one costs more than Phase 0 allows. The driver also seeds only `--env local` today. Daytona restart recovery comes from the overnight chaos evidence (below), with that evidence's bias.
|
||||
|
||||
### Metric definitions
|
||||
|
||||
| Metric | Definition (run-log spans) |
|
||||
|---|---|
|
||||
| **TTFT** | `task.time_to_first_token`. It runs from the accepted input to the first deliverable text committed by Core. The accepted input is the wake comment, the answered question, or the durable wake request when that is earlier (#19838). Otherwise it is run creation. `anchor` names which one. |
|
||||
| **Agent boot** | Run claimed (end of `heartbeat.queue`) → provider session accepted `turn.start` (end of `runner.turn.submit`, or of `runner.session.resume` for a resumed session) |
|
||||
| **Pre-turn steps** | Every span that starts before the turn boundary above, without cumulative wrappers (`task.prepare`, `runner.session.startup`, `environment.startup`) |
|
||||
| **Restart recovery latency** | Fault `readyAt` (the harness saw `/api/ready` = `ready` and a healthy `/api/liveness`) → the recovered run's next provider event, grouped by `native.recovery.transition.runnerDisposition` |
|
||||
| **Control-plane restart** | Fault start → `readyAt`, from the supervisor's fault receipt |
|
||||
| Cold / warm | From `run.startup.reuse` (`layer=provider_session`): `fresh` is cold, `reused` is warm |
|
||||
|
||||
## Time to first token and agent boot (local)
|
||||
|
||||
Two driver runs combined: run 1 (codex and claude) and run 3 (all three adapters).
|
||||
|
||||
| Adapter | Env | Start | n | TTFT p50 | TTFT p95 | Boot p50 | Boot p95 | TTFT anchors |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| claude | local | cold | 12/12 | 13.61 s | 24.82 s | 7.20 s | 16.98 s | run_created×12 |
|
||||
| claude | local | warm | 12/12 | 7.67 s | 23.61 s | 2.88 s | 13.58 s | comment×12 |
|
||||
| codex | local | cold | 12/12 | 4.97 s | 14.13 s | 2.07 s | 5.63 s | run_created×12 |
|
||||
| codex | local | warm | 12/12 | 8.10 s | 23.69 s | 3.84 s | 13.60 s | comment×12 |
|
||||
| opencode | local | cold | 6/6 | 14.28 s | 40.45 s | 9.83 s | 30.77 s | run_created×6 |
|
||||
| opencode | local | warm | 6/6 | 21.79 s | 73.22 s | 18.02 s | 54.33 s | comment×6 |
|
||||
|
||||
### TTFT by stage (p50 / p95)
|
||||
|
||||
Each cell is p50 / p95 in seconds. The stages overlap, so they do not sum to TTFT. TTFT also includes the time from the first provider event to the first deliverable text.
|
||||
|
||||
| Adapter | Start | Queue | Core prepare | Runner start + session | Submitted → started | Started → first provider event | TTFT |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| claude | cold | 0.1 / 11.1 | 0.5 / 1.9 | 6.4 / 14.8 | 0.1 / 0.2 | 2.3 / 4.0 | 13.6 / 24.8 |
|
||||
| claude | warm | 0.1 / 1.3 | 0.5 / 2.9 | 2.4 / 10.5 | 3.7 / 8.0 | 1.0 / 1.7 | 7.7 / 23.6 |
|
||||
| codex | cold | 0.1 / 8.6 | 0.5 / 2.5 | 1.6 / 3.0 | 1.2 / 5.2 | 1.5 / 2.6 | 5.0 / 14.1 |
|
||||
| codex | warm | 0.1 / 1.1 | 0.5 / 4.1 | 3.3 / 11.8 | 1.0 / 8.6 | 3.1 / 4.9 | 8.1 / 23.7 |
|
||||
| opencode | cold | 0.1 / 0.7 | 0.5 / 2.1 | 9.4 / 28.5 | 0.9 / 1.2 | 3.3 / 7.7 | 14.3 / 40.4 |
|
||||
| opencode | warm | 0.1 / 1.1 | 0.6 / 3.9 | 17.4 / 50.2 | 0.6 / 0.8 | 2.9 / 6.1 | 21.8 / 73.2 |
|
||||
|
||||
The column spans are `heartbeat.queue`, `task.prepare`, `runner.session.startup` (bootstrap or resume), `runner.turn.submitted_to_started`, `provider.time_to_first_agent_event` and `task.time_to_first_token`. The cold-queue p95 values for claude (11.1 s) and codex (8.6 s) come from one task each: the first task after the server started, which waited for the startup queued-run dispatch pass.
|
||||
|
||||
## Slowest steps before `turn.start`
|
||||
|
||||
### Measured, ranked by p95
|
||||
|
||||
| Pre-turn step | n | p50 | p95 | Adapters |
|
||||
|---|---|---|---|---|
|
||||
| `runner.session.resume` | 30 | 3.35 s | 48.66 s | claude, codex, opencode |
|
||||
| `runner.session.bootstrap` | 48 | 5.79 s | 15.04 s | claude, codex, opencode |
|
||||
| `heartbeat.prepare_before_environment` | 78 | 0.20 s | 1.40 s | claude, codex, opencode |
|
||||
| `heartbeat.queue` | 78 | 0.13 s | 1.23 s | claude, codex, opencode |
|
||||
| `heartbeat.prepare_after_environment` | 78 | 0.28 s | 0.98 s | claude, codex, opencode |
|
||||
| `comment.to_run_created` | 30 | 0.08 s | 0.65 s | claude, codex, opencode |
|
||||
| `runner.transport.connect` | 78 | 0.01 s | 0.13 s | claude, codex, opencode |
|
||||
| `environment.acquire` | 78 | 0.01 s | 0.12 s | claude, codex, opencode |
|
||||
| `runner.turn.submit` | 48 | 0.03 s | 0.11 s | claude, codex, opencode |
|
||||
| `native.coordinator.claim` | 78 | 0.01 s | 0.10 s | claude, codex, opencode |
|
||||
| `environment.workspace.realize` | 78 | 0.01 s | 0.07 s | claude, codex, opencode |
|
||||
| `skills.prepare` | 78 | 0.00 s | 0.01 s | claude, codex, opencode |
|
||||
| `runner.prp.route.register` | 78 | 0.00 s | 0.00 s | claude, codex, opencode |
|
||||
|
||||
### Found by code reading (from the inventory)
|
||||
|
||||
The inventory readers found these waits and writes on the path before `turn.start`. They are not all visible as spans yet.
|
||||
|
||||
| Step | Where | Fixed wait or cost | Measured above? |
|
||||
|---|---|---|---|
|
||||
| Remote first-run workspace capture and upload | `remote-workspaces.ts:570-572` (`saveRemoteWorkspace(boundary:"initial")`) | A full archive upload before the model starts. This breaks rule 2. | No (remote only) |
|
||||
| Remote provision commands re-run on each new writer epoch | `remote-workspaces.ts:230, 981` | Up to 300 s per command | No (remote only) |
|
||||
| Sandbox resume retry loop | `environment-runtime.ts:1814-1851` | Up to 60 s (RPC timeout), backoff 250 ms → 5 s; Kubernetes adds 30 s readiness | No (remote only) |
|
||||
| Plugin worker-ready poll | `environment-runtime.ts:764-765, 1486-1497` | Up to 5 s at 100 ms | No |
|
||||
| Obsolete reusable sandbox deletes, inline and serial | `environment-runtime.ts:1766-1771 → 3052` | One provider delete per obsolete lease | No |
|
||||
| Remote runner identity poll | `native-session-executor.ts:13282-13283` | Up to 20 s at 1 s | No (remote only) |
|
||||
| Stale remote runner stop | `native-session-executor.ts:6228, 6269-6276` | Up to 10 s at 500 ms | No (remote only) |
|
||||
| Runtime skill-cache lock | `runtime-skill-cache.ts:121, 134-140` | 20 ms poll up to 60 s, then a failure that needs a manual fix | Inside `skills.prepare` |
|
||||
| Budget admission | `budgets.ts:1415`, `heartbeat.ts:2937`, wake-queue `postgres.ts:1420/1503` | 3 full exposure evaluations and a locking reservation transaction per dispatch | Inside `heartbeat.prepare_*` |
|
||||
| Usage recorder start (local adapters) | `usage-receipts.ts:556-560` | 2 directory fsyncs and a probe fsync | Inside `heartbeat.prepare_after_environment` |
|
||||
| Controller identity | `native-restart-recovery.ts:326-335` → `hot-restart.ts:192` | Uncached. Runs `ps -o lstart` on macOS at every claim, preparation check and renewal. | Inside `native.coordinator.claim` / prepare |
|
||||
| Legacy controller lease renewal | `legacy-controller-lease.ts:68-73` | One DB round trip per preparation step. A slow DB stalls up to 60 s. | Legacy runs only |
|
||||
| Successor waits for predecessor cleanup | `heartbeat/queue.ts:1129`, `execution-blocker.ts:70-102` | Unbounded: until the previous run's lease cleanup and finalization finish | In `heartbeat.queue` |
|
||||
|
||||
## Restart recovery latency
|
||||
|
||||
### Measured by the driver (local)
|
||||
|
||||
Run 4 is the source. In it, the driver injected each fault only while every tracked run was inside an open shell tool call, and it checked that again just before the fault. Run 3 had used `sleep 75`, which Claude Code's Bash tool refuses, so run 3's claude samples did not reliably interrupt a tool call and are not used. Run 3's codex and opencode samples agree with run 4 (codex 3.2–5.1 s, opencode 12.5–16.5 s). A replay of run 4's run log against the stricter command check found all 8 codex and claude samples inside `time.sleep(75)` at the fault. Each of the 4 opencode samples had exactly one open shell call, which exposes no command text.
|
||||
|
||||
| Adapter | Fault | Disposition | n | Recovery p50 | Recovery p95 | No provider event |
|
||||
|---|---|---|---|---|---|---|
|
||||
| claude | K1 | resume_dead_runner | 2 | 8.30 s | 18.55 s | 0 |
|
||||
| claude | K2 | awaiting_runner_reattach | 2 | 12.37 s | 14.55 s | 0 |
|
||||
| codex | K1 | resume_dead_runner | 2 | 5.06 s | 6.49 s | 0 |
|
||||
| codex | K2 | awaiting_runner_reattach | 2 | 5.23 s | 5.26 s | 0 |
|
||||
| opencode | K1 | resume_dead_runner | 2 | 17.64 s | 31.95 s | 0 |
|
||||
| opencode | K2 | awaiting_runner_reattach | 2 | 21.21 s | 22.38 s | 0 |
|
||||
|
||||
Fault → ready:
|
||||
- run 4 (4 faults): 24.3, 27.6, 29.9 and 27.2 s, at a 1-minute load average of 17–30;
|
||||
- run 3 (6 faults): 13.2–20.0 s, at a load average of 12–22.
|
||||
|
||||
In every fault, `recovering` → `ready` took 0.27–1.7 s. The rest is process boot.
|
||||
|
||||
### From the overnight chaos campaign (2026-11-06/07)
|
||||
|
||||
These numbers come from the evidence bundles of failed chaos tasks only: 150 bundles and 577 run × fault pairs, at revisions from the campaign. They are biased toward slow and broken recoveries, so use them for tails, not for budgets. K4 includes a 3–5 minute outage hold, and K6 is a crash loop. Rows without a recovery transition or a later provider event are left out.
|
||||
|
||||
| Adapter | Env | Fault | Disposition | n (with next event) | Recovery p50 | Recovery p95 | Fault → ready p50 |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| claude | daytona | K1 | `awaiting_evidence` | 1/1 | 33.1 s | 33.1 s | 11.2 s |
|
||||
| claude | local | K1 | `resume_dead_runner` | 15/21 | 8.8 s | 40.8 s | 22.6 s |
|
||||
| claude | local | K2 | `awaiting_runner_reattach` | 12/20 | 5.9 s | 78.0 s | 15.2 s |
|
||||
| claude | local | K4 | `resume_dead_runner` | 2/2 | 3.6 s | 25.9 s | 257.7 s |
|
||||
| claude | local | K6 | `resume_dead_runner` | 10/10 | 6.2 s | 87.7 s | 42.6 s |
|
||||
| codex | daytona | K1 | `awaiting_evidence` | 5/6 | 37.1 s | 49.4 s | 11.2 s |
|
||||
| codex | daytona | K2 | `awaiting_runner_reattach` | 1/1 | 5.1 s | 5.1 s | 14.5 s |
|
||||
| codex | local | K1 | `resume_dead_runner` | 24/29 | 4.2 s | 113.2 s | 13.6 s |
|
||||
| codex | local | K2 | `awaiting_runner_reattach` | 19/25 | 4.3 s | 8.1 s | 12.7 s |
|
||||
| codex | local | K4 | `resume_dead_runner` | 5/6 | 4.2 s | 150.8 s | 230.3 s |
|
||||
| codex | local | K6 | `resume_dead_runner` | 8/9 | 3.7 s | 13.1 s | 22.0 s |
|
||||
| opencode | daytona | K1 | `awaiting_evidence` | 1/1 | 42.7 s | 42.7 s | 6.0 s |
|
||||
| opencode | daytona | K4 | `reattach_remote_runner` | 1/2 | 3.3 s | 3.3 s | 216.5 s |
|
||||
| opencode | local | K1 | `resume_dead_runner` | 14/15 | 10.9 s | 25.7 s | 10.0 s |
|
||||
| opencode | local | K2 | `awaiting_runner_reattach` | 16/20 | 12.1 s | 78.0 s | 11.7 s |
|
||||
| opencode | local | K4 | `resume_dead_runner` | 4/4 | 11.2 s | 15.8 s | 169.8 s |
|
||||
| opencode | local | K6 | `resume_dead_runner` | 3/5 | 14.4 s | 19.0 s | 21.6 s |
|
||||
|
||||
## Turn tail
|
||||
|
||||
`task.settle` (provider turn completed → run terminal, including result finalization) takes 0.3–1.8 s at p50 and up to 4.3 s at p95:
|
||||
- claude: 1.8 s cold, 1.6 s warm (p50);
|
||||
- codex: 0.7 s cold, 0.9 s warm;
|
||||
- opencode: 0.5 s cold, 0.3 s warm.
|
||||
|
||||
Copyback and sync-out do not appear here, because local runs use the workspace in place.
|
||||
|
||||
## Findings, ranked
|
||||
|
||||
1. **Every local turn launches a runner and starts or resumes a provider session.** This costs 1.6–17.4 s at p50 and up to 50 s at p95, and it makes follow-up turns slower than new tasks for codex and opencode. Plan rule 2 ("a follow-up turn on a still-valid lease uses what is already there") is not met locally. `lifecycleMode: warm` exists and keeps the runner alive between turns, but it is not the default and this baseline did not measure it. Phase 1 target: measure `warm`, then make the follow-up path reuse a live runner and provider session by default, and add sub-spans for `runner.process.launch`, `runner.runtime.stage` and provider session load, which the local path does not emit today.
|
||||
2. **Control-plane restart time is almost all process boot.** It takes 13–30 s to ready, with recovery logic at 0.3–1.7 s. A recovered run then waits 5–21 s more for its next provider event (p50, by adapter). Measure production boot, and keep recovery work off the path to `ready`. The inventory found that restart recovery runs a full finalization reconcile before it reattaches runners (`heartbeat/recovery.ts:945-1006`).
|
||||
3. **Remote recovery waits out a TTL.** On Daytona, `awaiting_evidence` takes 33–43 s, because the remote controller lease (60 s) is released by age, not by a fenced handoff (inventory: section A, `remoteLeaseFenced`).
|
||||
4. **Claude spends 3.7 s (p50) between `turn.submitted` and `turn.started` on a resumed session**, against 0.1 s on a fresh one. This is provider-side session load, and it is a separate target from runner launch.
|
||||
5. **Core preparation is not the bottleneck today, but it holds hidden fixed waits.** Queue plus prepare is about 0.6 s at p50. Its p95 tails (1.4 s `heartbeat.prepare_before_environment`, plus 8–11 s for the first task after server start) and the code-reading table show where restart-safety work must not add more:
|
||||
- budget admission evaluates exposure 3 times per dispatch;
|
||||
- local adapters fsync twice for the usage recorder;
|
||||
- an uncached `ps` runs on every claim.
|
||||
|
||||
## Gaps in this baseline
|
||||
|
||||
- **Daytona was not measured fresh.** See "Method". Phase 1 needs a Daytona baseline before it sets remote budgets.
|
||||
- **The samples are small.** There are 12 cold and 12 warm samples per adapter for codex and claude (two runs), 6 for opencode, and 2 per adapter and fault kind for restart recovery (run 4). The host was shared, with a 1-minute load average of 9–22, so the p95 values carry machine load.
|
||||
- **The `wake_request` TTFT anchor never fired here.** The driver's wakes were never deferred or coalesced, so every anchor was `run_created` (new task) or `comment` (follow-up). Deferred-wake latency needs a workload that queues behind a busy issue.
|
||||
- **No sub-spans exist inside local runner startup.** `runner.session.bootstrap` and `runner.session.resume` cover process launch, staging and the provider handshake together. Finding 1 needs them split.
|
||||
- **The server ran in dev mode** (`tsx`, Vite middleware). This inflates control-plane boot and probably Core preparation.
|
||||
- **`lifecycleMode: warm` was not measured.** Every sample uses the product default, `per_turn`.
|
||||
- **Legacy (non-native) adapters have no TTFT span.** This baseline covers the native runner path only.
|
||||
@@ -0,0 +1,127 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import {
|
||||
baselineRows, distribution, openShellExecutions, percentile, rankPreTurnSteps, recoverySamples, runMetrics, runWarmth,
|
||||
type BaselineEvent, type BaselineRun,
|
||||
} from "./ttft-baseline-stats.js";
|
||||
|
||||
const at = (ms: number) => new Date(1_800_000_000_000 + ms).toISOString();
|
||||
const span = (name: string, startOffsetMs: number, durationMs: number, extra: Record<string, unknown> = {}): BaselineEvent => ({
|
||||
eventType: "run.performance.span", createdAt: at(startOffsetMs + durationMs),
|
||||
payload: { span: name, parentSpan: "task.provider_session", startOffsetMs, durationMs, ...extra },
|
||||
});
|
||||
const reuse = (state: string): BaselineEvent => ({ eventType: "run.startup.reuse", createdAt: at(0),
|
||||
payload: { layer: "provider_session", state } });
|
||||
|
||||
function run(id: string, phase: BaselineRun["phase"], events: BaselineEvent[], adapter = "codex"): BaselineRun {
|
||||
return { runId: id, adapter, environment: "local", phase, events };
|
||||
}
|
||||
|
||||
const coldRun = (id: string, scale = 1) => run(id, "cold", [
|
||||
reuse("fresh"),
|
||||
span("heartbeat.queue", 0, 100 * scale),
|
||||
span("heartbeat.prepare_before_environment", 100 * scale, 200 * scale),
|
||||
span("task.prepare", 100 * scale, 900 * scale),
|
||||
span("runner.process.launch", 400 * scale, 1_500 * scale),
|
||||
span("runner.turn.submit", 2_000 * scale, 500 * scale),
|
||||
span("provider.time_to_first_agent_event", 2_600 * scale, 3_000 * scale),
|
||||
span("task.time_to_first_token", 0, 6_000 * scale, { anchor: "wake_request" }),
|
||||
]);
|
||||
|
||||
describe("ttft baseline statistics", () => {
|
||||
it("uses nearest-rank percentiles and ignores non-finite values", () => {
|
||||
expect(percentile([5, 1, 3, 2, 4], 50)).toBe(3);
|
||||
expect(percentile([5, 1, 3, 2, 4], 95)).toBe(5);
|
||||
expect(percentile([], 50)).toBeNull();
|
||||
expect(distribution([1, Number.NaN, 3])).toEqual({ n: 2, p50: 1, p95: 3, max: 3 });
|
||||
});
|
||||
|
||||
it("measures boot from claim to the durable turn.submitted and keeps only pre-turn steps", () => {
|
||||
const metrics = runMetrics(coldRun("a"));
|
||||
expect(metrics).toMatchObject({ ttftMs: 6_000, ttftAnchor: "wake_request", bootMs: 2_400 });
|
||||
expect(Object.keys(metrics.preTurnSteps).sort()).toEqual([
|
||||
"heartbeat.prepare_before_environment", "heartbeat.queue", "runner.process.launch", "runner.turn.submit",
|
||||
]);
|
||||
});
|
||||
|
||||
it("uses runner.session.resume as the turn boundary for a resumed session", () => {
|
||||
const resumed = run("r", "warm", [reuse("reused"), span("heartbeat.queue", 0, 50), span("runner.session.resume", 60, 940)]);
|
||||
expect(runWarmth(resumed)).toBe("warm");
|
||||
expect(runMetrics(resumed).bootMs).toBe(950);
|
||||
});
|
||||
|
||||
it("classifies warmth from the run log before the driver phase", () => {
|
||||
expect(runWarmth(run("x", "warm", [reuse("fresh")]))).toBe("cold");
|
||||
expect(runWarmth(run("y", "cold", []))).toBe("cold");
|
||||
expect(runWarmth(run("z", "warm", []))).toBe("warm");
|
||||
});
|
||||
|
||||
it("groups rows by adapter and warmth and leaves restart runs out", () => {
|
||||
const rows = baselineRows([coldRun("a"), coldRun("b", 2), run("c", "restart", [span("task.time_to_first_token", 0, 99_000)])]);
|
||||
expect(rows).toHaveLength(1);
|
||||
expect(rows[0]).toMatchObject({ adapter: "codex", warmth: "cold", anchors: { wake_request: 2 },
|
||||
ttft: { n: 2, p50: 6_000, p95: 12_000 } });
|
||||
});
|
||||
|
||||
it("ranks pre-turn steps by p95 without cumulative wrappers", () => {
|
||||
const ranked = rankPreTurnSteps([coldRun("a"), coldRun("b", 2)]);
|
||||
expect(ranked[0]).toMatchObject({ step: "runner.process.launch", n: 2, p95: 3_000 });
|
||||
expect(ranked.map(step => step.step)).not.toContain("task.prepare");
|
||||
});
|
||||
|
||||
it("measures restart recovery from ready to the next provider event, by disposition", () => {
|
||||
const readyAt = Date.parse(at(10_000));
|
||||
const recovered = run("r1", "restart", [
|
||||
{ eventType: "item.delta", createdAt: at(9_000) },
|
||||
{ eventType: "native.recovery.transition", createdAt: at(9_500), payload: { runnerDisposition: "resume_dead_runner" } },
|
||||
{ eventType: "tool.execution.completed", createdAt: at(13_250) },
|
||||
]);
|
||||
const silent = run("r2", "restart", []);
|
||||
expect(recoverySamples([recovered, silent], [{ kind: "K1", readyAt, runIds: ["r1", "r2", "missing"] }])).toEqual([
|
||||
{ runId: "r1", adapter: "codex", kind: "K1", disposition: "resume_dead_runner", latencyMs: 3_250 },
|
||||
{ runId: "r2", adapter: "codex", kind: "K1", disposition: "none", latencyMs: null },
|
||||
]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("restart sample guard", () => {
|
||||
const tool = (eventType: string, executionId: string, extra: Record<string, unknown> = {}): BaselineEvent => ({
|
||||
eventType, createdAt: at(0), payload: { prpEvent: { payload: { executionId, ...extra } } },
|
||||
});
|
||||
const ids = (events: BaselineEvent[], marker?: string) => openShellExecutions(events, marker);
|
||||
|
||||
it("keeps a shell call open until it ends, and ignores other tools", () => {
|
||||
expect(ids([
|
||||
tool("tool.execution.started", "mcp-1", { name: "paperclip.issue.comment", operation: "unknown" }),
|
||||
tool("tool.execution.started", "sh-1", { name: "Bash", operation: "execute" }),
|
||||
tool("tool.execution.progressed", "sh-1"),
|
||||
]).map(call => call.executionId)).toEqual(["sh-1"]);
|
||||
});
|
||||
|
||||
it("closes a shell call on completion or failure", () => {
|
||||
expect(ids([
|
||||
tool("tool.execution.started", "sh-1", { name: "/bin/zsh -lc 'python3 -c ...'" }),
|
||||
tool("tool.execution.completed", "sh-1"),
|
||||
tool("tool.execution.started", "sh-2", { name: "bash" }),
|
||||
tool("tool.execution.failed", "sh-2"),
|
||||
])).toEqual([]);
|
||||
});
|
||||
|
||||
it("matches the intended command from the name, subject or a later progress update", () => {
|
||||
const marker = "time.sleep(75)";
|
||||
expect(ids([
|
||||
tool("tool.execution.started", "codex", { name: "/bin/zsh -lc 'python3 -c \"import time; time.sleep(75)\"'", operation: "execute" }),
|
||||
tool("tool.execution.started", "claude", { name: "Bash", operation: "execute" }),
|
||||
tool("tool.execution.progressed", "claude", { subject: "python3 -c \"import time; time.sleep(75)\"" }),
|
||||
tool("tool.execution.started", "other", { name: "Bash", operation: "execute", subject: "ls -la" }),
|
||||
], marker)).toEqual([
|
||||
{ executionId: "codex", matched: true, unlabelled: false },
|
||||
{ executionId: "claude", matched: true, unlabelled: false },
|
||||
{ executionId: "other", matched: false, unlabelled: false },
|
||||
]);
|
||||
});
|
||||
|
||||
it("marks a shell tool that exposes no command text as unlabelled", () => {
|
||||
expect(ids([tool("tool.execution.started", "oc", { name: "bash", operation: "unknown" })], "time.sleep(75)"))
|
||||
.toEqual([{ executionId: "oc", matched: false, unlabelled: true }]);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,218 @@
|
||||
/**
|
||||
* Pure statistics for the Phase 0 time-to-first-token baseline
|
||||
* (doc/plans/2026-11-07-ttft-baseline.md). Input is the run log only:
|
||||
* `run.performance.span`, `run.startup.reuse` and `native.recovery.transition`
|
||||
* rows from `heartbeat_run_events`. Never Telemetry (AGENTS.md §5.7).
|
||||
*/
|
||||
|
||||
export interface BaselineEvent {
|
||||
eventType: string;
|
||||
createdAt: string;
|
||||
payload?: Record<string, unknown> | null;
|
||||
}
|
||||
|
||||
export interface BaselineRun {
|
||||
runId: string;
|
||||
adapter: string;
|
||||
environment: string;
|
||||
/** Driver intent: a new task, a follow-up comment on the same task, or a run crossed by a restart. */
|
||||
phase: "cold" | "warm" | "restart";
|
||||
events: BaselineEvent[];
|
||||
}
|
||||
|
||||
export interface BaselineFault {
|
||||
kind: string;
|
||||
readyAt: number;
|
||||
/** Runs that were in flight when the fault started. */
|
||||
runIds: string[];
|
||||
}
|
||||
|
||||
export interface Distribution { n: number; p50: number | null; p95: number | null; max: number | null }
|
||||
|
||||
/** Nearest-rank percentile over a finite sample. */
|
||||
export function percentile(values: readonly number[], p: number): number | null {
|
||||
const sorted = values.filter(Number.isFinite).slice().sort((a, b) => a - b);
|
||||
if (sorted.length === 0) return null;
|
||||
const rank = Math.max(1, Math.ceil((p / 100) * sorted.length));
|
||||
return sorted[Math.min(sorted.length, rank) - 1]!;
|
||||
}
|
||||
|
||||
export function distribution(values: readonly number[]): Distribution {
|
||||
const finite = values.filter(Number.isFinite);
|
||||
return { n: finite.length, p50: percentile(finite, 50), p95: percentile(finite, 95),
|
||||
max: finite.length ? Math.max(...finite) : null };
|
||||
}
|
||||
|
||||
export interface RunSpan { name: string; parent: string; startOffsetMs: number; durationMs: number; attributes: Record<string, unknown> }
|
||||
|
||||
export function runSpans(run: BaselineRun): RunSpan[] {
|
||||
return run.events.filter(event => event.eventType === "run.performance.span" && event.payload)
|
||||
.map(event => {
|
||||
const payload = event.payload!;
|
||||
return { name: String(payload.span), parent: String(payload.parentSpan ?? ""),
|
||||
startOffsetMs: Number(payload.startOffsetMs), durationMs: Number(payload.durationMs), attributes: payload };
|
||||
})
|
||||
.filter(span => Number.isFinite(span.startOffsetMs) && Number.isFinite(span.durationMs));
|
||||
}
|
||||
|
||||
/** cold = fresh provider session; warm = verified reuse. Falls back to the driver phase. */
|
||||
export function runWarmth(run: BaselineRun): "cold" | "warm" {
|
||||
const reuse = run.events.find(event => event.eventType === "run.startup.reuse" && event.payload?.layer === "provider_session");
|
||||
if (reuse?.payload?.state === "reused") return "warm";
|
||||
if (reuse?.payload?.state === "fresh") return "cold";
|
||||
return run.phase === "warm" ? "warm" : "cold";
|
||||
}
|
||||
|
||||
export interface RunMetrics {
|
||||
ttftMs: number | null;
|
||||
ttftAnchor: string | null;
|
||||
/** Run claimed (end of heartbeat.queue) → durable `turn.submitted` (provider session accepted turn.start). */
|
||||
bootMs: number | null;
|
||||
/** Durations of every span that starts before the first turn submission. */
|
||||
preTurnSteps: Record<string, number>;
|
||||
}
|
||||
|
||||
/** Spans that wrap others (or the whole run); ranking them would double count. */
|
||||
const CUMULATIVE_SPANS = new Set(["task.time_to_first_token", "task.provider_session.measured", "task.prepare",
|
||||
"native.session.execute", "runner.session.startup", "agent.turn", "environment.startup"]);
|
||||
|
||||
const FIRST_SPAN = (spans: RunSpan[], name: string) => spans.find(span => span.name === name) ?? null;
|
||||
|
||||
export function runMetrics(run: BaselineRun): RunMetrics {
|
||||
const spans = runSpans(run);
|
||||
const ttft = FIRST_SPAN(spans, "task.time_to_first_token");
|
||||
const queue = FIRST_SPAN(spans, "heartbeat.queue");
|
||||
// runner.turn.submit = session.started → turn.submitted; a resumed session has
|
||||
// runner.session.resume ending at turn.submitted instead.
|
||||
const submitted = FIRST_SPAN(spans, "runner.turn.submit") ?? FIRST_SPAN(spans, "runner.session.resume");
|
||||
const claimedAt = queue ? queue.startOffsetMs + queue.durationMs : null;
|
||||
const turnStartAt = submitted ? submitted.startOffsetMs + submitted.durationMs : null;
|
||||
const preTurnSteps: Record<string, number> = {};
|
||||
if (turnStartAt !== null) {
|
||||
for (const span of spans) {
|
||||
// Cumulative wrappers overlap their children; keep them out of the ranking.
|
||||
if (CUMULATIVE_SPANS.has(span.name)) continue;
|
||||
if (span.startOffsetMs < turnStartAt && !(span.name in preTurnSteps)) preTurnSteps[span.name] = span.durationMs;
|
||||
}
|
||||
}
|
||||
return {
|
||||
ttftMs: ttft ? ttft.durationMs : null,
|
||||
ttftAnchor: ttft && typeof ttft.attributes.anchor === "string" ? ttft.attributes.anchor : null,
|
||||
bootMs: claimedAt !== null && turnStartAt !== null ? Math.max(0, turnStartAt - claimedAt) : null,
|
||||
preTurnSteps,
|
||||
};
|
||||
}
|
||||
|
||||
const PROVIDER_EVENT_TYPES = new Set(["turn.submitted", "turn.started", "turn.accepted", "item.started", "item.delta",
|
||||
"item.completed", "tool.execution.started", "tool.execution.progressed", "tool.execution.completed", "session.resumed"]);
|
||||
|
||||
export interface RecoverySample { runId: string; adapter: string; kind: string; disposition: string; latencyMs: number | null }
|
||||
|
||||
/** Control plane ready → the recovered run's next provider event, keyed by runner disposition. */
|
||||
export function recoverySamples(runs: readonly BaselineRun[], faults: readonly BaselineFault[]): RecoverySample[] {
|
||||
const byId = new Map(runs.map(run => [run.runId, run]));
|
||||
const samples: RecoverySample[] = [];
|
||||
for (const fault of faults) {
|
||||
for (const runId of fault.runIds) {
|
||||
const run = byId.get(runId);
|
||||
if (!run) continue;
|
||||
const after = run.events.filter(event => Date.parse(event.createdAt) >= fault.readyAt - 60_000);
|
||||
const transition = after.find(event => event.eventType === "native.recovery.transition"
|
||||
&& Date.parse(event.createdAt) >= fault.readyAt - 60_000);
|
||||
const next = run.events.find(event => PROVIDER_EVENT_TYPES.has(event.eventType) && Date.parse(event.createdAt) >= fault.readyAt);
|
||||
samples.push({ runId, adapter: run.adapter, kind: fault.kind,
|
||||
disposition: String(transition?.payload?.runnerDisposition ?? "none"),
|
||||
latencyMs: next ? Date.parse(next.createdAt) - fault.readyAt : null });
|
||||
}
|
||||
}
|
||||
return samples;
|
||||
}
|
||||
|
||||
export interface BaselineRow { adapter: string; environment: string; warmth: "cold" | "warm"; ttft: Distribution; boot: Distribution; anchors: Record<string, number> }
|
||||
|
||||
export function baselineRows(runs: readonly BaselineRun[]): BaselineRow[] {
|
||||
const groups = new Map<string, { adapter: string; environment: string; warmth: "cold" | "warm"; metrics: RunMetrics[] }>();
|
||||
for (const run of runs) {
|
||||
if (run.phase === "restart") continue;
|
||||
const warmth = runWarmth(run);
|
||||
const key = `${run.adapter}|${run.environment}|${warmth}`;
|
||||
const group = groups.get(key) ?? { adapter: run.adapter, environment: run.environment, warmth, metrics: [] };
|
||||
group.metrics.push(runMetrics(run));
|
||||
groups.set(key, group);
|
||||
}
|
||||
return [...groups.values()].sort((a, b) => `${a.adapter}${a.warmth}`.localeCompare(`${b.adapter}${b.warmth}`)).map(group => {
|
||||
const anchors: Record<string, number> = {};
|
||||
for (const metric of group.metrics) if (metric.ttftAnchor) anchors[metric.ttftAnchor] = (anchors[metric.ttftAnchor] ?? 0) + 1;
|
||||
return { adapter: group.adapter, environment: group.environment, warmth: group.warmth, anchors,
|
||||
ttft: distribution(group.metrics.flatMap(metric => metric.ttftMs === null ? [] : [metric.ttftMs])),
|
||||
boot: distribution(group.metrics.flatMap(metric => metric.bootMs === null ? [] : [metric.bootMs])) };
|
||||
});
|
||||
}
|
||||
|
||||
export interface StepRank { step: string; n: number; p50: number | null; p95: number | null; adapters: string[] }
|
||||
|
||||
/** Pre-turn steps ranked by p95, then p50. Steps under 1 ms at p95 are dropped. */
|
||||
export function rankPreTurnSteps(runs: readonly BaselineRun[]): StepRank[] {
|
||||
const steps = new Map<string, { values: number[]; adapters: Set<string> }>();
|
||||
for (const run of runs) {
|
||||
for (const [step, value] of Object.entries(runMetrics(run).preTurnSteps)) {
|
||||
const entry = steps.get(step) ?? { values: [], adapters: new Set<string>() };
|
||||
entry.values.push(value);
|
||||
entry.adapters.add(run.adapter);
|
||||
steps.set(step, entry);
|
||||
}
|
||||
}
|
||||
return [...steps.entries()].map(([step, entry]) => {
|
||||
const d = distribution(entry.values);
|
||||
return { step, n: d.n, p50: d.p50, p95: d.p95, adapters: [...entry.adapters].sort() };
|
||||
}).filter(rank => (rank.p95 ?? 0) >= 1)
|
||||
.sort((a, b) => (b.p95 ?? 0) - (a.p95 ?? 0) || (b.p50 ?? 0) - (a.p50 ?? 0));
|
||||
}
|
||||
|
||||
/** The tool payload of a run-log `tool.execution.*` row (PRP envelope or bare). */
|
||||
function toolPayload(event: BaselineEvent): Record<string, unknown> {
|
||||
const outer = event.payload ?? {};
|
||||
const prp = outer.prpEvent as Record<string, unknown> | undefined;
|
||||
return ((prp?.payload ?? outer) as Record<string, unknown>) ?? {};
|
||||
}
|
||||
|
||||
const SHELL_TOOL = /\b(bash|sh|zsh|shell|terminal|python3?)\b/i;
|
||||
|
||||
export interface OpenShellExecution {
|
||||
executionId: string;
|
||||
/** The tool exposed command text (name, subject or progress) that contains the marker. */
|
||||
matched: boolean;
|
||||
/** The tool exposed no command text at all, so the marker cannot be checked. */
|
||||
unlabelled: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
* Executions of a shell tool that started and have not ended. A restart sample
|
||||
* is valid only while one is open, so the fault lands inside the tool call.
|
||||
* With `marker`, an execution is `matched` when its exposed command text
|
||||
* contains the marker. Some tools (OpenCode's `bash`) expose no command text;
|
||||
* those are `unlabelled`.
|
||||
*/
|
||||
export function openShellExecutions(events: readonly BaselineEvent[], marker?: string): OpenShellExecution[] {
|
||||
const open = new Map<string, { matched: boolean; labelled: boolean }>();
|
||||
for (const event of events) {
|
||||
if (!event.eventType.startsWith("tool.execution.")) continue;
|
||||
const payload = toolPayload(event);
|
||||
const id = typeof payload.executionId === "string" ? payload.executionId : null;
|
||||
if (!id) continue;
|
||||
const commandText = [payload.subject, payload.progress].filter(value => typeof value === "string" && value).join(" ");
|
||||
const label = `${String(payload.name ?? "")} ${commandText}`;
|
||||
if (event.eventType === "tool.execution.started") {
|
||||
if (payload.operation === "execute" || SHELL_TOOL.test(label)) open.set(id, { matched: false, labelled: false });
|
||||
} else if (event.eventType !== "tool.execution.progressed") {
|
||||
open.delete(id);
|
||||
continue;
|
||||
}
|
||||
const entry = open.get(id);
|
||||
if (!entry) continue;
|
||||
// A shell name alone (\`bash\`) is not command text; a full command line in \`name\` is.
|
||||
if (commandText || /\s/.test(String(payload.name ?? "").trim())) entry.labelled = true;
|
||||
if (marker && label.includes(marker)) entry.matched = true;
|
||||
}
|
||||
return [...open.entries()].map(([executionId, entry]) => ({ executionId, matched: entry.matched, unlabelled: !entry.labelled }));
|
||||
}
|
||||
@@ -0,0 +1,315 @@
|
||||
/**
|
||||
* Phase 0 speed baseline driver (doc/plans/2026-11-07-ttft-baseline.md).
|
||||
*
|
||||
* Starts one isolated control plane through the restart chaos supervisor,
|
||||
* seeds the Chaos Lab, and per adapter runs:
|
||||
* - cold: a new task (fresh provider session);
|
||||
* - warm: a follow-up comment on the same task (provider session reuse);
|
||||
* - restart: a task in a provider tool call when the control plane is killed
|
||||
* (K1, SIGKILL) or stopped gracefully (K2, SIGTERM).
|
||||
* It then writes every run's run log to `<out>/baseline-data.json`. All timing
|
||||
* comes from the run log, never from Telemetry (AGENTS.md §5.7).
|
||||
*
|
||||
* set -a; source ~/.secrets; set +a
|
||||
* node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/ttft-baseline.ts \
|
||||
* --adapters codex,claude,opencode --samples 5 --restarts 2
|
||||
* node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/ttft-baseline.ts --report <out>/baseline-data.json
|
||||
*/
|
||||
import { mkdir, readFile, writeFile } from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { parseArgs } from "node:util";
|
||||
import { setTimeout as delay } from "node:timers/promises";
|
||||
import { ChaosApi } from "./api.js";
|
||||
import { CREDENTIAL_FOR_ADAPTER } from "./config.js";
|
||||
import { ACTIVE_RUN_STATUSES } from "./grader.js";
|
||||
import { seedChaosLab, type ChaosLab } from "./seed.js";
|
||||
import { ChaosSupervisor } from "./supervisor.js";
|
||||
import { Timeline } from "./timeline.js";
|
||||
import type { AdapterId, FaultKind } from "./types.js";
|
||||
import {
|
||||
baselineRows, rankPreTurnSteps, recoverySamples, distribution, openShellExecutions,
|
||||
type BaselineEvent, type BaselineFault, type BaselineRun,
|
||||
} from "./ttft-baseline-stats.js";
|
||||
|
||||
type Json = Record<string, unknown>;
|
||||
const repositoryRoot = path.resolve(import.meta.dirname, "../..");
|
||||
const TASK_TIMEOUT_MS = 8 * 60_000;
|
||||
|
||||
const coldTask = (nonce: string) => [
|
||||
`Baseline ${nonce}. Reply on this task with a one-line comment that says \`READY ${nonce}\`.`,
|
||||
"Do not create files, documents or other tasks. Then mark this task done.",
|
||||
].join("\n");
|
||||
const warmFollowUp = (nonce: string) =>
|
||||
`Follow-up ${nonce}: reply with one more one-line comment that says \`AGAIN ${nonce}\`, then mark the task done again.`;
|
||||
/** The long call a restart sample must interrupt. */
|
||||
const RESTART_SLEEP_MARKER = "time.sleep(75)";
|
||||
const restartTask = (nonce: string) => [
|
||||
// Claude Code's Bash tool refuses a bare long \`sleep\`; a Python sleep is one ordinary shell call everywhere.
|
||||
`Baseline restart ${nonce}. First run the shell command \`python3 -c "import time; time.sleep(75)"\` and wait for it to finish.`,
|
||||
`Then reply with a one-line comment that says \`DONE ${nonce}\` and mark this task done.`,
|
||||
"Do not create files, documents or other tasks.",
|
||||
].join("\n");
|
||||
|
||||
async function createTask(api: ChaosApi, lab: ChaosLab, adapter: AdapterId, title: string, description: string) {
|
||||
const issue = await api.post<Json>(`/api/companies/${lab.companyId}/issues`, {
|
||||
title, description, status: "todo", priority: "medium", assigneeAgentId: lab.agents[adapter].id,
|
||||
});
|
||||
return String(issue.id);
|
||||
}
|
||||
|
||||
/** `/api/issues/:id/runs` rows carry `runId`, not `id`. */
|
||||
const runIdOf = (run: Json) => String(run.runId ?? run.id);
|
||||
|
||||
async function runsFor(api: ChaosApi, issueId: string): Promise<Json[]> {
|
||||
return await api.get<Json[]>(`/api/issues/${issueId}/runs`, 10_000).catch(() => []);
|
||||
}
|
||||
|
||||
/**
|
||||
* Wait until the task has at least `minRuns` runs and none is active. Returns
|
||||
* `timedOut: true` at the deadline, so the caller never treats an active run
|
||||
* as a finished sample.
|
||||
*/
|
||||
async function waitForRest(api: ChaosApi, supervisor: ChaosSupervisor, issueId: string, minRuns: number): Promise<{ runs: Json[]; timedOut: boolean }> {
|
||||
const deadline = Date.now() + TASK_TIMEOUT_MS;
|
||||
let restSince: number | null = null;
|
||||
while (Date.now() < deadline) {
|
||||
if (supervisor.serverReady) {
|
||||
const runs = await runsFor(api, issueId);
|
||||
const resting = runs.length >= minRuns && runs.every(run => !ACTIVE_RUN_STATUSES.has(String(run.status)));
|
||||
restSince = resting ? (restSince ?? Date.now()) : null;
|
||||
// A finished run can queue a continuation; confirm the rest for a few seconds.
|
||||
if (restSince !== null && Date.now() - restSince >= 6_000) return { runs, timedOut: false };
|
||||
}
|
||||
await delay(2_000);
|
||||
}
|
||||
return { runs: await runsFor(api, issueId), timedOut: true };
|
||||
}
|
||||
|
||||
async function allEvents(api: ChaosApi, runId: string): Promise<BaselineEvent[]> {
|
||||
const events: BaselineEvent[] = [];
|
||||
let afterSeq = 0;
|
||||
for (;;) {
|
||||
const page = await api.get<Json[]>(`/api/heartbeat-runs/${runId}/events?afterSeq=${afterSeq}&limit=500`, 15_000);
|
||||
for (const row of page) {
|
||||
events.push({ eventType: String(row.eventType), createdAt: String(row.createdAt),
|
||||
payload: (row.payload ?? null) as Json | null });
|
||||
}
|
||||
if (page.length < 500) return events;
|
||||
afterSeq = Number(page[page.length - 1]!.seq);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* The running run of the task, when it is inside the long sleep call. A tool
|
||||
* that exposes its command must show the sleep marker. A tool that exposes no
|
||||
* command text (OpenCode's `bash`) counts as inside an open shell call, and the
|
||||
* sample is marked `commandUnverified`.
|
||||
*/
|
||||
async function runInShellCall(api: ChaosApi, issueId: string): Promise<{ runId: string; commandUnverified: boolean } | null> {
|
||||
const active = (await runsFor(api, issueId)).find(run => String(run.status) === "running");
|
||||
if (!active) return null;
|
||||
const events = await allEvents(api, runIdOf(active)).catch(() => []);
|
||||
const open = openShellExecutions(events, RESTART_SLEEP_MARKER);
|
||||
if (open.some(call => call.matched)) return { runId: runIdOf(active), commandUnverified: false };
|
||||
if (open.length && open.every(call => call.unlabelled)) return { runId: runIdOf(active), commandUnverified: true };
|
||||
return null;
|
||||
}
|
||||
|
||||
/** Wait until the run is inside its shell tool call (started, not ended). */
|
||||
async function waitForShellCall(api: ChaosApi, issueId: string): Promise<{ runId: string; commandUnverified: boolean } | null> {
|
||||
const deadline = Date.now() + 4 * 60_000;
|
||||
while (Date.now() < deadline) {
|
||||
const inside = await runInShellCall(api, issueId);
|
||||
if (inside) return inside;
|
||||
await delay(1_500);
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Wait for each agent to pass lifecycle verification. An agent that never
|
||||
* reaches `ready` is reported as not measured, never waited on per task.
|
||||
*/
|
||||
async function readyAdapters(api: ChaosApi, lab: ChaosLab, adapters: AdapterId[], timeline: Timeline): Promise<AdapterId[]> {
|
||||
const deadline = Date.now() + 4 * 60_000;
|
||||
const pending = new Set(adapters);
|
||||
const ready: AdapterId[] = [];
|
||||
while (pending.size && Date.now() < deadline) {
|
||||
for (const adapter of [...pending]) {
|
||||
const agent = await api.get<Json>(`/api/agents/${lab.agents[adapter].id}`).catch(() => null);
|
||||
const state = String(agent?.lifecycleState ?? "unknown");
|
||||
if (state === "ready") { ready.push(adapter); pending.delete(adapter); }
|
||||
else if (agent?.status === "paused" && agent?.lifecycleError) {
|
||||
timeline.add("baseline_adapter_not_ready", { adapter, state, error: String(agent.lifecycleError).slice(0, 300) });
|
||||
pending.delete(adapter);
|
||||
}
|
||||
}
|
||||
if (pending.size) await delay(3_000);
|
||||
}
|
||||
for (const adapter of pending) timeline.add("baseline_adapter_not_ready", { adapter, state: "timeout" });
|
||||
return ready;
|
||||
}
|
||||
|
||||
function report(runs: BaselineRun[], faults: BaselineFault[]): string {
|
||||
const ms = (value: number | null) => value === null ? "—" : `${(value / 1000).toFixed(2)} s`;
|
||||
const lines = ["| Adapter | Env | Start | n | TTFT p50 | TTFT p95 | Boot p50 | Boot p95 | TTFT anchors |", "|---|---|---|---|---|---|---|---|---|"];
|
||||
for (const row of baselineRows(runs)) {
|
||||
lines.push(`| ${row.adapter} | ${row.environment} | ${row.warmth} | ${row.ttft.n}/${row.boot.n} | ${ms(row.ttft.p50)} | ${ms(row.ttft.p95)} | ${ms(row.boot.p50)} | ${ms(row.boot.p95)} | ${Object.entries(row.anchors).map(([k, v]) => `${k}×${v}`).join(", ") || "—"} |`);
|
||||
}
|
||||
lines.push("", "| Pre-turn step | n | p50 | p95 | Adapters |", "|---|---|---|---|---|");
|
||||
for (const step of rankPreTurnSteps(runs).slice(0, 25)) lines.push(`| \`${step.step}\` | ${step.n} | ${ms(step.p50)} | ${ms(step.p95)} | ${step.adapters.join(", ")} |`);
|
||||
const samples = recoverySamples(runs, faults);
|
||||
lines.push("", "| Adapter | Fault | Disposition | n | Recovery p50 | Recovery p95 | No provider event |", "|---|---|---|---|---|---|---|");
|
||||
const groups = new Map<string, typeof samples>();
|
||||
for (const sample of samples) {
|
||||
const key = `${sample.adapter}|${sample.kind}|${sample.disposition}`;
|
||||
groups.set(key, [...(groups.get(key) ?? []), sample]);
|
||||
}
|
||||
for (const [key, group] of [...groups.entries()].sort()) {
|
||||
const [adapter, kind, disposition] = key.split("|");
|
||||
const d = distribution(group.flatMap(sample => sample.latencyMs === null ? [] : [sample.latencyMs]));
|
||||
lines.push(`| ${adapter} | ${kind} | ${disposition} | ${d.n} | ${ms(d.p50)} | ${ms(d.p95)} | ${group.length - d.n} |`);
|
||||
}
|
||||
return `${lines.join("\n")}\n`;
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const { values } = parseArgs({ options: {
|
||||
adapters: { type: "string", default: "codex,claude,opencode" },
|
||||
samples: { type: "string", default: "5" },
|
||||
restarts: { type: "string", default: "2" },
|
||||
phases: { type: "string", default: "coldwarm,restart" },
|
||||
out: { type: "string" },
|
||||
report: { type: "string" },
|
||||
} });
|
||||
if (values.report) {
|
||||
const data = JSON.parse(await readFile(values.report, "utf8")) as { runs: BaselineRun[]; faults: BaselineFault[] };
|
||||
process.stdout.write(report(data.runs, data.faults));
|
||||
return;
|
||||
}
|
||||
const phases = new Set(values.phases!.split(",").map(value => value.trim()));
|
||||
const requested = values.adapters!.split(",").map(value => value.trim()) as AdapterId[];
|
||||
const samples = Math.max(1, Number(values.samples));
|
||||
const restarts = Math.max(0, Number(values.restarts));
|
||||
const missing = requested.map(adapter => CREDENTIAL_FOR_ADAPTER[adapter]).filter(key => !process.env[key]);
|
||||
if (missing.length) throw new Error(`Missing ${missing.join(", ")}; run: set -a; source ~/.secrets; set +a`);
|
||||
const waveId = `ttft-${new Date().toISOString().replace(/[-:.]/g, "").slice(0, 15)}`;
|
||||
const outDir = path.resolve(values.out ?? path.join(import.meta.dirname, "results", waveId));
|
||||
await mkdir(outDir, { recursive: true, mode: 0o700 });
|
||||
const timeline = new Timeline(path.join(outDir, "timeline.jsonl"));
|
||||
const supervisor = await ChaosSupervisor.create({ repositoryRoot, waveDir: outDir, timeline });
|
||||
const tasks: Array<{ issueId: string; adapter: AdapterId; phase: BaselineRun["phase"] }> = [];
|
||||
const faults: BaselineFault[] = [];
|
||||
const runPhase = new Map<string, BaselineRun["phase"]>();
|
||||
const timedOutIssues = new Set<string>();
|
||||
// One shutdown path for normal completion and for SIGINT/SIGTERM: the server
|
||||
// runs in a detached process group and the embedded Postgres has no exit hook.
|
||||
let stopping: Promise<void> | null = null;
|
||||
const shutdown = () => (stopping ??= supervisor.stop().catch(error => {
|
||||
process.stderr.write(`supervisor stop: ${String(error)}\n`);
|
||||
}));
|
||||
const onSignal = (signal: NodeJS.Signals) => {
|
||||
timeline.add("baseline_interrupted", { signal });
|
||||
void shutdown().finally(() => process.exit(signal === "SIGINT" ? 130 : 143));
|
||||
};
|
||||
process.once("SIGINT", onSignal);
|
||||
process.once("SIGTERM", onSignal);
|
||||
try {
|
||||
await supervisor.start();
|
||||
const api = new ChaosApi(supervisor.baseUrl);
|
||||
const lab = await seedChaosLab({ api, env: "local", adapters: requested, concurrency: requested.length, budgetCents: 3000,
|
||||
masks: ["rc06"], fixtureRoot: supervisor.paths.temporaryRoot, waveId, timeline });
|
||||
const adapters = await readyAdapters(api, lab, requested, timeline);
|
||||
if (!adapters.length) throw new Error("No agent passed lifecycle verification");
|
||||
|
||||
// Cold then warm, one task per adapter at a time, adapters in parallel.
|
||||
if (phases.has("coldwarm")) await Promise.all(adapters.map(async adapter => {
|
||||
for (let index = 0; index < samples; index += 1) {
|
||||
const nonce = `${adapter}-${index}-${Date.now().toString(36)}`;
|
||||
const issueId = await createTask(api, lab, adapter, `TTFT baseline ${nonce}`, coldTask(nonce));
|
||||
tasks.push({ issueId, adapter, phase: "cold" });
|
||||
const cold = await waitForRest(api, supervisor, issueId, 1);
|
||||
for (const run of cold.runs) runPhase.set(runIdOf(run), "cold");
|
||||
if (cold.timedOut) {
|
||||
// Never send the follow-up into an active turn; drop the whole sample.
|
||||
timedOutIssues.add(issueId);
|
||||
timeline.add("baseline_sample_timeout", { adapter, issueId, phase: "cold" });
|
||||
continue;
|
||||
}
|
||||
timeline.add("baseline_cold_done", { adapter, issueId, runs: cold.runs.length });
|
||||
await api.post(`/api/issues/${issueId}/comments`, { body: warmFollowUp(nonce), reopen: true });
|
||||
const warm = await waitForRest(api, supervisor, issueId, cold.runs.length + 1);
|
||||
for (const run of warm.runs) if (!runPhase.has(runIdOf(run))) runPhase.set(runIdOf(run), "warm");
|
||||
if (warm.timedOut) {
|
||||
timedOutIssues.add(issueId);
|
||||
timeline.add("baseline_sample_timeout", { adapter, issueId, phase: "warm" });
|
||||
continue;
|
||||
}
|
||||
timeline.add("baseline_warm_done", { adapter, issueId, runs: warm.runs.length - cold.runs.length });
|
||||
}
|
||||
}));
|
||||
|
||||
// Restart recovery: one in-flight tool call per adapter, then one fault crosses all of them.
|
||||
const kinds: FaultKind[] = ["K1", "K2"];
|
||||
for (let index = 0; phases.has("restart") && index < restarts * kinds.length; index += 1) {
|
||||
const kind = kinds[index % kinds.length]!;
|
||||
const inFlight = await Promise.all(adapters.map(async adapter => {
|
||||
const nonce = `${adapter}-r${index}-${Date.now().toString(36)}`;
|
||||
const issueId = await createTask(api, lab, adapter, `TTFT restart ${nonce}`, restartTask(nonce));
|
||||
tasks.push({ issueId, adapter, phase: "restart" });
|
||||
const inside = await waitForShellCall(api, issueId);
|
||||
return { issueId, adapter, runId: inside?.runId ?? null };
|
||||
}));
|
||||
// Recheck right before the fault: a fast adapter may have finished its call
|
||||
// while a slower one was still starting. Only runs still inside it count.
|
||||
const stillInside = await Promise.all(inFlight.map(async entry => {
|
||||
if (!entry.runId) return null;
|
||||
const inside = await runInShellCall(api, entry.issueId);
|
||||
return inside?.runId === entry.runId ? inside : null;
|
||||
}));
|
||||
for (const [position, entry] of inFlight.entries()) {
|
||||
const inside = stillInside[position];
|
||||
if (!inside) timeline.add("baseline_restart_sample_skipped", { adapter: entry.adapter, issueId: entry.issueId });
|
||||
else if (inside.commandUnverified) {
|
||||
timeline.add("baseline_restart_command_unverified", { adapter: entry.adapter, issueId: entry.issueId });
|
||||
}
|
||||
}
|
||||
const runIds = stillInside.flatMap(inside => inside ? [inside.runId] : []);
|
||||
for (const runId of runIds) runPhase.set(runId, "restart");
|
||||
const receipt = await supervisor.fault(kind, 0, `ttft-baseline-${index}`);
|
||||
faults.push({ kind, readyAt: receipt.readyAt ?? Date.now(), runIds });
|
||||
const settled = await Promise.all(inFlight.map(entry => waitForRest(api, supervisor, entry.issueId, 1)));
|
||||
for (const [position, entry] of inFlight.entries()) {
|
||||
if (settled[position]!.timedOut) {
|
||||
timedOutIssues.add(entry.issueId);
|
||||
timeline.add("baseline_sample_timeout", { adapter: entry.adapter, issueId: entry.issueId, phase: "restart" });
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const runs: BaselineRun[] = [];
|
||||
for (const task of tasks) {
|
||||
// A sample that never came to rest is reported, not measured.
|
||||
if (timedOutIssues.has(task.issueId)) continue;
|
||||
for (const run of await runsFor(api, task.issueId)) {
|
||||
const runId = runIdOf(run);
|
||||
runs.push({ runId, adapter: task.adapter, environment: "local",
|
||||
phase: runPhase.get(runId) ?? task.phase, events: await allEvents(api, runId) });
|
||||
}
|
||||
}
|
||||
const data = { waveId, adapters, notReady: requested.filter(adapter => !adapters.includes(adapter)),
|
||||
timedOutTasks: timedOutIssues.size, samples, restarts, gitSha: process.env.GIT_SHA ?? null, runs, faults };
|
||||
await writeFile(path.join(outDir, "baseline-data.json"), `${JSON.stringify(data)}\n`, { mode: 0o600 });
|
||||
await writeFile(path.join(outDir, "baseline-report.md"), report(runs, faults), { mode: 0o600 });
|
||||
process.stdout.write(report(runs, faults));
|
||||
if (timedOutIssues.size) process.stdout.write(`\n${timedOutIssues.size} sample(s) timed out and were left out.\n`);
|
||||
process.stdout.write(`\nWrote ${outDir}\n`);
|
||||
} finally {
|
||||
// Always stop the wrapper-owned embedded Postgres (SysV shared memory is scarce).
|
||||
await shutdown();
|
||||
process.off("SIGINT", onSignal);
|
||||
process.off("SIGTERM", onSignal);
|
||||
}
|
||||
}
|
||||
|
||||
main().catch(error => { process.stderr.write(`${error instanceof Error ? error.stack : String(error)}\n`); process.exitCode = 1; });
|
||||
Reference in new issue
Block a user