mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-01 23:43:16 +01:00
Merge qualified backup expiration parent into GitHub connection work
Co-Authored-By: Paperclip <noreply@paperclip.ing> * commit 'aae32f2f493848199f6cb8c0a9996af3ee0061b1': (30 commits) Scope status-card refresh tasks to their claim revision test(chat): retain completion identity before link metadata loads test(chat): locate completion through its rendered task link fix(chat): recover committed native Agent Chat replies (#18465) perf(heartbeat): drive retry promotion from durable deadlines (#18387) fix(ui): make the Coordinator composer footer opaque (#18455) fix(testops): verify unexpanded result job runs (#18462) test(server): observe restored Retry before draining unrelated work test(runner): own listener cleanup across setup failures test(runner): buffer upgrade frames until the wire subscribes fix(runner): preserve OpenCode written-answer defaults (#18456) fix(native): fence retries when replaying an audited Stop (#18457) test: await async outcomes and drive saved deadlines deterministically (#18416) fix(ui): retain startup autosave through session recovery (#18412) perf(server): recover independent native replies concurrently (#18421) perf(chat): scope task questions and batch inbox queue checks (#18422) fix(native): recover controllers across webserver replacement (#18343) fix(ui): restore task document authoring and add persistence journey (#18356) fix(tools): stabilize tied profile entry order (#18427) fix(adapter-utils): tolerate entries removed during workspace snapshots (#18426) ...
This commit is contained in:
commit
f522ab07e2
484 files changed
+33834
-1620
No files matched your search
@@ -1,5 +1,20 @@
|
||||
# Database
|
||||
|
||||
### Proactive agent chat messages
|
||||
|
||||
Migration `0419_slippery_impossible_man.sql` adds an agent-scoped unique constraint
|
||||
on `issue_comments(issue_id, author_agent_id, client_request_id)` alongside the
|
||||
existing human request constraint. Proactive delivery stores a source-task-scoped
|
||||
retry key and server-authored `metadata.chatMessage` provenance/content fingerprint.
|
||||
The comment and audit record commit together; a retry returns the original receipt.
|
||||
Conversation creation reuses the unique company/agent/human identity and existing
|
||||
advisory-lock order. No message outbox is added.
|
||||
|
||||
Chat list/detail unread projections reuse `issue_read_states`, including chats
|
||||
with no human participation. Read cutoffs use the exact PostgreSQL comment
|
||||
timestamp; upserts take the greatest previous/new cutoff so delayed requests
|
||||
cannot move a read receipt backward or clear a later unseen arrival.
|
||||
|
||||
Paperclip uses PostgreSQL via [Drizzle ORM](https://orm.drizzle.team/). There are three ways to run the database, from simplest to most production-ready.
|
||||
|
||||
### Persistent case handoffs
|
||||
|
||||
+89
-5
@@ -207,11 +207,20 @@ of exempt prose, update this policy as well as the task inputs.
|
||||
For ordinary source PRs (including the audited prose inputs above), a separate `Select Vitest files` job refines the
|
||||
general and serialized test lanes before their runners are allocated. It runs
|
||||
on the gate-approved test fleet, loads the real Vitest projects, and follows
|
||||
imports across workspace packages. Only projects with their own source changes
|
||||
are narrowed; unchanged consumer projects retain their complete coarse lane
|
||||
coverage without paying for another graph traversal. Filesystem, subprocess, network, CommonJS,
|
||||
imports across workspace packages. Every project in the coarse plan is assessed,
|
||||
including unchanged consumers. Projects without affected tests drop out of the
|
||||
matrix; an unchanged directory alone does not prove independence. Filesystem, subprocess, network, CommonJS,
|
||||
dynamic-loader, unresolved, and unapproved external-package consumers are
|
||||
retained conservatively, including indirect consumers through local helpers.
|
||||
Module-relative filesystem reads add explicit dependency edges when the path and
|
||||
read API can be resolved without running code. This includes directory listings
|
||||
and transitive helper reads. Dynamic paths, writes, escaped filesystem APIs,
|
||||
mutable or shadowed bindings, generated inputs and paths outside the checkout
|
||||
remain opaque. Setup-file reads invalidate the whole project when their input
|
||||
changes. Object-valued constants, including URL objects, remain opaque even without
|
||||
an observed mutation; inline URL construction and immutable string constants can
|
||||
be resolved. Symlink reads track both the checked-in link and its physical target.
|
||||
Missing inputs retain coverage.
|
||||
The selector explicitly tracks setup/configuration dependencies as well as test
|
||||
imports. Ordinary source names such as `session-config-metadata.test.ts` or
|
||||
`setup-state.ts` remain eligible; the words `config` and `setup` alone do not
|
||||
@@ -220,6 +229,15 @@ inputs. Other resolved setup/config dependencies invalidate their entire project
|
||||
The small pure-dependency allowlist lives in
|
||||
`scripts/ci/vitest-selection-graph.mjs`; new external packages remain opaque
|
||||
until reviewed. This is deliberately broader than `vitest related` alone.
|
||||
The reviewed Drizzle SQL/schema entry points have no ambient repository inputs.
|
||||
Their full import closures are content-hashed in
|
||||
`scripts/ci/reviewed-test-dependencies.mjs`; changed installed contents revoke the
|
||||
exception. CommonJS and dynamic loaders reject a fingerprint rather than leave
|
||||
unhashed children outside the closure. Database drivers and migration readers remain opaque. OS information,
|
||||
in-memory streams, timers and compression do not read repository files themselves;
|
||||
filesystem or process operations using their outputs retain their normal guards.
|
||||
The syntax audit distinguishes data keys and erased types from runtime globals.
|
||||
It examines the called member, not string arguments earlier in a chained call.
|
||||
For conservative retention, explicit complete local mock factories cut the
|
||||
mocked implementation's I/O edges. Partial mocks, spies, automocks, mock bypasses,
|
||||
and dynamic or escaped mocking APIs do not. Edits to a mocked module still use
|
||||
@@ -238,7 +256,8 @@ jobs; empty lanes allocate none. Serialized files remain serialized, and chat's
|
||||
line shards, native Runner lanes, Rust, builds, typechecking and E2E retain their
|
||||
existing selection/execution rules. The job summary reports files and jobs
|
||||
selected per lane, and the `vitest-selection` artifact records the manifest and
|
||||
dependency audit. Selected jobs execute exact Vitest test specifications, so
|
||||
dependency audit, including file-read edges and the first opaque dependency that
|
||||
retained each test. Selected jobs execute exact Vitest test specifications, so
|
||||
similarly named `.test.ts` and `.test.tsx` files cannot overlap between shards.
|
||||
The wrapper consumes and validates the selection before launching Vitest, then
|
||||
removes selection environment variables from the test process. Nested runner
|
||||
@@ -972,7 +991,16 @@ Use the cheap local default unless you are specifically working on browser flows
|
||||
pnpm test
|
||||
```
|
||||
|
||||
`pnpm test` runs the Vitest suite only. For interactive Vitest watch mode use:
|
||||
`pnpm test` runs the Vitest suite only. Its default chat integration coverage runs
|
||||
first in six sequential, exact-cover child processes, each with its own home,
|
||||
config and embedded PostgreSQL. The remaining server invocation excludes that
|
||||
chat file once and retains the native Runner suites. This bounds shared fixture
|
||||
and process lifetime without increasing concurrency or changing case assertions
|
||||
or timeouts. An explicit `PAPERCLIP_TEST_DATABASE_URL` retains the original
|
||||
single, serial database path. Explicit `--group` invocations and CI shard routing
|
||||
are unchanged.
|
||||
|
||||
For interactive Vitest watch mode use:
|
||||
|
||||
```sh
|
||||
pnpm test:watch
|
||||
@@ -2621,6 +2649,33 @@ and `packages/adapter-utils/src/sandbox-callback-bridge.test.ts` for these contr
|
||||
|
||||
### Native runner restart recovery
|
||||
|
||||
For remote controller disk-loss work, use the existing control-plane snapshot
|
||||
transaction and `native-controller-recovery.ts`; do not add a second snapshot
|
||||
writer. The private versioned metadata is limited to 4 KiB. Its importer handles
|
||||
one verified retained source while the source disk is available. Workspace and
|
||||
provider-backup migrations remain with their storage owners. See
|
||||
[the continuity verification guide](testing-native-remote-continuity.md) for
|
||||
the direct/prepared transport and three-controller continuation fixtures.
|
||||
An upgrade readiness report contains preliminary native evidence, not permission
|
||||
to remove a deployment volume. Keep the checkpoint pool bounded and measure all
|
||||
pools when qualifying database cost; the current size of two is a default.
|
||||
|
||||
Committed native Agent Chat turns recover their unanswered final reply through
|
||||
the existing response presentation worker. Recovery verifies the accepted result
|
||||
and final turn, agent, responsible user, native session and conversation generation.
|
||||
It stores the exact final message before the normal conversation settlement moves
|
||||
the chat to waiting. Later user input remains active. Recovery does not run the
|
||||
agent again, replace a selected reply or restore a deleted reply.
|
||||
|
||||
For retained Codex event tails, keep source drain/accounting and target admission
|
||||
separate. `native-staged-attachment-recovery.ts` validates private source/target
|
||||
header evidence; `native-recovery-usage-delivery.ts` binds the source's accounting
|
||||
receiver. The transport persists its staged target before source bootstrap, then
|
||||
uses the existing authenticated attachment protocol after durable drain. Do not
|
||||
rotate the identity first, acknowledge before receipt persistence, or weaken the
|
||||
Runner's settled-provider check. A staged target does not yet certify a resumable
|
||||
session. Its eventual snapshot still uses the existing checkpoint transaction.
|
||||
|
||||
Custom source-mode HTTP launchers should create their listener with
|
||||
`createPaperclipHttpServer` from `server/src/http/server.ts`, passing the selected
|
||||
API origin as `apiUrl`. This registers the native Runner WebSocket transport
|
||||
@@ -2763,6 +2818,35 @@ and hard restarts with live and dead runners, the result-finalization race,
|
||||
incomplete bootstrap, repeated crashes with steering, and fail-closed process
|
||||
identity mismatches.
|
||||
|
||||
Native task-session publication is part of the terminal checkpoint transaction,
|
||||
not resource cleanup. `native-task-session-publication.ts` owns its accepted-result,
|
||||
owner, task-epoch and reset checks; heartbeat supplies prepared session metadata.
|
||||
The goal-less regression in `native-session-resumption.test.ts` holds provider
|
||||
cleanup open and then exercises a normal follow-up and an explicit session reset.
|
||||
Control-port tests also cover terminal snapshots before accepted results and
|
||||
result-replay contention with the status writer's coordinator-first lock order.
|
||||
Remote adoption tests must invoke actual external-state preparation. Inspecting
|
||||
transport options alone does not exercise the resumed-lease harness checks.
|
||||
Include a healthy warm handoff before the crash so recovery exercises the
|
||||
handoff's turn and item attachment IDs as well as first-run identities.
|
||||
Retirement tests must exercise the preceding `turn.stop` as well as drain and
|
||||
suspend after result acceptance. Use the actual native-observed, unclaimed
|
||||
admission phase and the production claim before retiring the predecessor;
|
||||
a synthetic legacy preparation does not cover that boundary. Verify failed
|
||||
claims and Stop leave the predecessor and shared attachment staging untouched.
|
||||
Configuration tests compare fresh and recovered
|
||||
home-mode fingerprints while preserving rejection of real mode changes.
|
||||
Include multiple skill versions read back from Postgres JSONB: version-map key
|
||||
order must not change the generated preference or session fingerprint. Exercise
|
||||
the real environment-selection and configuration services with a retained shared
|
||||
workspace, and keep actual model or skill-version changes incompatible.
|
||||
Run it against an isolated database with `PAPERCLIP_TEST_DATABASE_URL` when the
|
||||
host cannot start embedded Postgres:
|
||||
|
||||
```sh
|
||||
pnpm exec vitest run server/src/__tests__/native-session-resumption.test.ts -t 'publishes the completed task session before provider cleanup'
|
||||
```
|
||||
|
||||
## Version-based skill authoring and delivery
|
||||
|
||||
Managed content uses `managed-skill-content.ts`: prepare and verify outside the
|
||||
|
||||
@@ -93,6 +93,10 @@ Tasks have parentage. Every task exists in service of a parent task, all the way
|
||||
|
||||
The current issue model includes stable issue identifiers, parent/sub-issues, blockers, a single assignee, comments, issue documents, attachments and work products, and review/approval handoffs. That structure keeps work inspectable by both the board and agents while still allowing agents to decompose work into smaller tasks.
|
||||
|
||||
When an agent creates a subtask or follow-up task, its source task's feed shows
|
||||
a linked card at creation time with the current title, task number, and status.
|
||||
The Tasks tab keeps the full relationship list.
|
||||
|
||||
Automatic monitor checks pause when a task is blocked. A person can use Check now
|
||||
to request one check of a retained monitor through normal execution admission.
|
||||
Dependencies, recovery, pause, budgets and ownership still apply. Provider-usage
|
||||
@@ -358,6 +362,14 @@ See [iMessage Photon](connections/IMESSAGE-PHOTON.md) for the implementation
|
||||
contract, setup, recovery, boundaries, and qualification status.
|
||||
### Experimental persistent agent conversations
|
||||
|
||||
Agents can deliver proactive task and routine results into their own chat with
|
||||
the task's responsible human. A morning PR digest arrives as an agent message
|
||||
with a source-task link and unread indicators, even before the first human
|
||||
message. The person can reply using the digest as context. Publishing a message
|
||||
and completing work are separate actions; sending never starts or interrupts a
|
||||
chat turn. Delivery obeys membership, run authority and task privacy, and stays
|
||||
behind the Agent Chat setting. This version delivers in-app only.
|
||||
|
||||
Agent Chat is an opt-in core task presentation (`enableAgentChat`, off by default). Each person has one persistent task-backed conversation per agent and company, with ordinary company task visibility. The shared task composer, transcript, tools, files, and document panel remain the interaction surface. Agents clarify goals and hand substantial execution to linked, assigned tasks; a reply ends a turn without completing the conversation. `/new` starts fresh provider context in the same conversation while preserving visible history and artifacts. Healthy idle conversations wait for a message and do not count as unfinished execution work. See `doc/plans/2026-09-10-agent-chat.md` for the implementation contract.
|
||||
|
||||
### Agent chat project handoff (2026-09-11)
|
||||
|
||||
@@ -376,6 +376,13 @@ Include both agent configuration and task-level model selection where supported.
|
||||
These checks establish correct routing and authority. They do not turn a discovery
|
||||
list into a model allowlist: explicit compatible models still follow section 6.1.
|
||||
|
||||
For Kimi Code, when `PAPERCLIP_RUNNER_REMOTE_PROVIDER_PACK_PATH` selects a
|
||||
controller pack, remote Test prepares and verifies that current pack through the
|
||||
normal installer before its 35-second metadata/catalog budget starts.
|
||||
Installation adds to total Test time. Without a selected pack, remote targets
|
||||
retain manual setup and preinstalled-runtime verification. Neither path sends a
|
||||
model prompt.
|
||||
|
||||
## 7. Deliver instructions, tools, and credentials
|
||||
|
||||
### 7.1 Prove instruction delivery at the native boundary
|
||||
|
||||
@@ -2117,6 +2117,40 @@ placements may use a lazy live character. See [agent-personas.md](agent-personas
|
||||
for persistence, migration, rendering, and integration contracts.
|
||||
### Experimental task-backed agent chat (2026-09-10)
|
||||
|
||||
Ordinary task and routine runs in standard mode may publish with
|
||||
`send_chat_message({body,idempotencyKey})` or `POST /api/agents/me/chat/messages`.
|
||||
The authenticated company, agent and run responsible human determine the sole
|
||||
destination. Recheck execution ownership, membership, task/destination access
|
||||
and canonical source privacy at commit. Reject missing recipients, restricted
|
||||
source tasks, Ask/planning/review/watchdog runs and stopped runs. The shared
|
||||
comment-write boundary enforces the fixed conversation agent.
|
||||
Skill-test and task-bridge credentials cannot publish into standing chats.
|
||||
Persisted skill-test run boundaries remain in force after a task-mode change.
|
||||
Preserve the credential and external-user access intersection and recheck key
|
||||
revocation, scope and responsible-person identity before commit.
|
||||
Each new delivery charges the shared per-run cross-issue influence cap in the
|
||||
same transaction. Retries, conflicting content and receipt-only recovery do not
|
||||
spend another slot. Cap exhaustion returns the canonical 429 response and leaves
|
||||
no message or newly created conversation behind.
|
||||
|
||||
The chat messaging service atomically creates a waiting conversation if needed
|
||||
and commits an attributed issue comment and audit record. Agent request IDs are
|
||||
unique per conversation and author; keys incorporate the source task for retries
|
||||
across runs, and changed content conflicts. Server-authored provenance identifies
|
||||
the source task and run. Existing events refresh discovery, messages and unread
|
||||
projections. Source execution transcripts remain outside the chat's linked-run
|
||||
history. There is no delivery worker, message outbox or extra model turn.
|
||||
|
||||
Sending does not change task disposition, cancel/settle a chat turn or suppress
|
||||
automatic handoff completion notifications. `/new` permits future background
|
||||
messages in the current history. Ordinary chat replies retain session checks.
|
||||
Chat list/detail responses use per-user read state before human participation;
|
||||
the foreground chat marks rendered messages read through an optional comment
|
||||
cutoff, retaining unseen concurrent arrivals. Existing navigation ordering is
|
||||
preserved with accessible unread dots and an aggregate Chat indicator.
|
||||
`enableAgentChat` gates delivery; external channels, push/email and private-chat
|
||||
redesign are outside this contract.
|
||||
|
||||
`enableAgentChat` is an instance experimental flag, default false. Conversation containers remain issues, unique by `(company_id, conversation_agent_id, conversation_user_id)`. The authenticated board actor supplies ownership; local trusted mode uses `local-board`. Ordinary company task access applies. A conversation's agent assignment and identity are immutable through ordinary updates; terminal status mutations are rejected.
|
||||
|
||||
`GET /api/companies/:companyId/chats/:agentRef` reads an existing conversation or null. `POST` atomically resolves its issue when adding a chat or on first send/upload. `GET /api/companies/:companyId/chats` lists only the current board user’s conversations in that company, subject to ordinary issue read access. The Chat navigation opens a searchable secondary sidebar with agent avatars and a picker for starting or reopening the same per-agent conversation. Existing issue comment, attachment, document, interaction, and run APIs apply thereafter. User chat comments require an idempotent UUID `clientRequestId`. Conversation delivery preserves comment order through the existing issue execution queue; the durable comment outbox repairs the commit-to-enqueue crash window.
|
||||
|
||||
@@ -2,6 +2,15 @@
|
||||
|
||||
Target specification for the Paperclip control plane. Living document — updated incrementally during spec interviews.
|
||||
|
||||
Proactive in-app updates let an active standard task or routine publish to its
|
||||
agent's persistent conversation with the run's responsible human. The server
|
||||
derives the destination, enforces current authority and company-visible chat
|
||||
privacy, and persists an attributed, idempotent comment with source-task provenance.
|
||||
Unread state belongs to the viewing human. Sending is independent of task
|
||||
completion and chat execution; future messages remain possible after `/new`.
|
||||
The V1 contract is specified under experimental task-backed agent chat in
|
||||
`doc/SPEC-implementation.md`.
|
||||
|
||||
Experimental standalone cases provide persistent structured work records that can
|
||||
exist independently of execution. Prompt-driven handoffs move their primary
|
||||
ordinary task between agents while retaining case data and history. Existing task
|
||||
|
||||
@@ -14,6 +14,25 @@ permissions; discovery skips services without grants and excludes unsupported
|
||||
actions. The gateway checks actual grant scopes again before each call, including
|
||||
after refresh. Requested scopes alone never authorize an action.
|
||||
|
||||
Before managed or customer-owned OAuth opens, Paperclip saves the validated
|
||||
service choice as non-secret setup intent on the draft. It is bound to the
|
||||
company, connection, initiating actor, grant kind, and credential subject.
|
||||
Declined or expired authorization and a cold **Finish setup** return restore
|
||||
that choice, not the new-account write defaults. Historical drafts without a
|
||||
matching saved intent require an explicit choice. Reconnecting an established
|
||||
account without a new choice keeps its exact grant request.
|
||||
|
||||
Overlapping customer-owned starts preserve the latest saved draft choice through
|
||||
client binding and authorization metadata updates. Each OAuth state and consent
|
||||
URL keeps that attempt's own requested scopes. These updates preserve unrelated
|
||||
current settings and other users' pending proposals in both configuration records.
|
||||
|
||||
An agent can hand setup to its responsible human without choosing services or
|
||||
creating provider OAuth state. If the agent proposes services, the human can
|
||||
review and change them in the picker. Saving the human's choice consumes only
|
||||
that person's pending proposal, so an old proposal cannot replace their choice
|
||||
after a declined sign-in. Neither draft intent nor a proposal is consent.
|
||||
|
||||
Discovery isolates service outages. Healthy services remain usable while the
|
||||
connection's health message identifies unavailable services and recommends a
|
||||
catalog refresh. Failed services' cached actions are disabled until rediscovered;
|
||||
|
||||
+12
-3
@@ -1,5 +1,11 @@
|
||||
# Paperclip evaluation guide
|
||||
|
||||
The explicit-only [Kimi Code qualification suite](../tests/runner-e2e/README.md#kimi-code-qualification)
|
||||
registers twenty candidate cells across local and Daytona. It reuses the
|
||||
browser, semantic interaction, file/artifact, Stop and continuation oracles with
|
||||
scoped candidate admission and an encrypted Code credential. Registration does
|
||||
not establish a live pass, native question/plan support or usable cost accounting.
|
||||
|
||||
The public directory calls full-product testing **E2E tests**. Suites describe
|
||||
what is tested; deployment targets describe which app instance was tested.
|
||||
QA smoke is a named selection of existing journeys against QA, and appears in
|
||||
@@ -49,8 +55,9 @@ original bounds. Its source registration is not an AWS or TestOps pass. The
|
||||
broader YAML catalog remains planning inventory.
|
||||
|
||||
The explicit-only [Daytona restart continuity journey](../tests/runner-e2e/README.md)
|
||||
extends the existing native restart oracle to a 125-second controller outage,
|
||||
using the recorded Claude and Codex models in disposable development fixtures.
|
||||
extends the existing native restart oracle to a 300-second controller outage
|
||||
and another replacement after completion, using the recorded Claude and Codex
|
||||
models in disposable development fixtures.
|
||||
It requires same-process adoption, saved files and finalization without replay;
|
||||
local regression checks and catalog discovery are separate from live acceptance.
|
||||
|
||||
@@ -84,7 +91,9 @@ successor. Registration and calibration are separate from a live staging pass.
|
||||
|
||||
[Reusable Product journeys](../tests/runner-e2e/PRODUCT-JOURNEYS.md) add explicit
|
||||
deployment targets and starting-state recipes. The shared task-properties
|
||||
journey uses no model; Cloud onboarding supplies reusable verified state. Cloud
|
||||
journey and task-document create/edit/reopen journey use no model; Cloud
|
||||
onboarding supplies reusable verified state. The document journey checks exact
|
||||
saved content and both revision records through independent API reads. Cloud
|
||||
fleet operations and trusted CI remain private. Local eval gateway qualification
|
||||
does not qualify Cloud routes, and these selections stay outside `--all`.
|
||||
|
||||
|
||||
+165
-2
@@ -330,6 +330,10 @@ persisted cancellations of parked retries through the normal guarded cancellatio
|
||||
path if the original owner stops or its final database write fails. They preserve
|
||||
the original cancellation request and never start a provider turn for that work.
|
||||
|
||||
Replaying an audited native Stop also fences a later same-run retry. It preserves
|
||||
the original intent and acknowledgement. That retry fence does not prove that
|
||||
the provider stopped or that workspace cleanup finished.
|
||||
|
||||
Recovery instructions in a wake describe only an active or escalated recovery
|
||||
action. Resolving or cancelling that action removes its repair instructions from
|
||||
subsequent wakes, so the original worker can resume the task. This prompt change
|
||||
@@ -2167,8 +2171,11 @@ digest. Earlier live qualification results keep their original profile identity.
|
||||
|
||||
Healthy native warm turns retain their runner and provider processes. After a
|
||||
controller crash, continuation can recover a completed remote session from its
|
||||
saved checkpoint and exact sandbox process identity. The prior controller must
|
||||
be stopped, its result committed, and the successor must hold current task and
|
||||
saved checkpoint and exact sandbox process identity. For an externally persisted
|
||||
controller with validated recovery metadata, the prior lease must be released
|
||||
and its result committed. A PID in the replacement container does not establish
|
||||
whether the prior remote controller is alive. Legacy file-backed recovery retains
|
||||
its stopped-process check. The successor must hold current task and
|
||||
coordinator authority. Recovery reconnects through the prior run's route before
|
||||
the authenticated run attachment changes authority. Unavailable identity probes
|
||||
have a shared 30-second retry budget; an identity conflict never permits replay.
|
||||
@@ -2647,3 +2654,159 @@ The existing prior-owner, suspended-state and backup checks still decide whether
|
||||
the next turn may start. Missing checkpoints and ambiguous identity remain
|
||||
blocked; restoration does not authorize a new provider session or change the
|
||||
configured lifecycle policy.
|
||||
|
||||
Remote admission also handles the absence of both native controller projections.
|
||||
`native-controller-recovery.ts` selects the existing bounded task/session source
|
||||
and restores its encrypted PRP records before admission reads local identity.
|
||||
It reconstructs the supervisor envelope from `runnerProfileJson.sessionCheckpoint`
|
||||
and the private `nativeControllerRecovery` metadata. Missing, incompatible, or
|
||||
mixed evidence holds the continuation; it does not authorize fresh execution.
|
||||
The 32-candidate boundary and newer-progress barriers remain in force. Exhausted
|
||||
history and current-run progress cannot establish fresh-bootstrap proof.
|
||||
|
||||
A settled warm journal may contain an authenticated idle goal-read snapshot
|
||||
after `run.terminal`. Recovery accepts only the supported PRP event v1/v2
|
||||
envelopes with the same complete attachment identity, a null goal and
|
||||
`workingNow: false`. All commands must already be complete. Active goals, new
|
||||
work, unknown versions and mixed bindings remain holds. This check adds no
|
||||
streaming persistence or database work.
|
||||
|
||||
If admission finds a live retained controller owner, it preserves the unstarted
|
||||
business run and records the owner's actual deadline through the existing native
|
||||
resumption path. This does not consume a provider attempt or stop the retained
|
||||
sandbox. A locked run/coordinator check rejects Stop, changed ownership or any
|
||||
provider progress. Startup restores even an overdue wait to the same scheduler;
|
||||
ordinary resumption rechecks current permissions and task authority. No idle
|
||||
scan is added. Per-turn continuation reconstructs a missing suspended-state
|
||||
receipt from the admitted sandbox or prepared-task owner, including after a
|
||||
previous read failed with only a PRP projection written.
|
||||
|
||||
The version-one metadata contains a digest of the exact redacted stored snapshot,
|
||||
configuration digests and the PRP checkpoint binding. It is bounded to 4 KiB,
|
||||
excluded from public run projections, and saved by `checkpointSession` in its
|
||||
existing snapshot transaction. No second snapshot writer, per-text-chunk query,
|
||||
pool, or recurring scan is added. A null checkpoint binding records an incomplete
|
||||
publication and prevents expiration. Snapshot and result writes recheck the
|
||||
coordinator generation and lease after lock waits, including the existing limited
|
||||
post-result maintenance exception.
|
||||
|
||||
Goal-less native turns publish their task-session pointer with the first accepted,
|
||||
successful terminal checkpoint, before provider or instruction cleanup can delay
|
||||
the heartbeat completion callback. The pointer uses the native normalized UUID
|
||||
and the checkpoint's provider display identity. Its prepared configuration metadata
|
||||
comes from normal admission; neither streaming events nor ordinary checkpoints
|
||||
add task-session writes. Goal checkpoints retain their existing publication path.
|
||||
|
||||
The snapshot transaction validates the exact accepted result and current owner,
|
||||
then updates only the authenticated run's existing task-session projection. Its
|
||||
private `nativeTaskSessionPublication` receipt shares the 4 KiB recovery-metadata
|
||||
budget. Coordinator, issue and run locks precede the ownership clock check; the
|
||||
issue's `NO KEY UPDATE` lock remains compatible with result insertion's parent
|
||||
`KEY SHARE`. Reset/deletion, a newer task-session epoch or a competing execution
|
||||
prevents publication. Later maintenance and heartbeat completion honor the receipt,
|
||||
so they cannot restore a cleared pointer or overwrite its successor. Unsupported
|
||||
legacy bindings keep the existing completion path. The receipt is excluded from
|
||||
public run projections and contains no session parameters or credentials.
|
||||
|
||||
A terminal snapshot can arrive before result acceptance. That snapshot publishes
|
||||
neither a continuation pointer nor a handled publication receipt. The existing
|
||||
post-acceptance checkpoint completes publication in its normal transaction.
|
||||
The first production release containing recovery readers, complete attachment
|
||||
identity handling and redaction for both private recovery fields is the
|
||||
disk-independent rollback floor. Earlier
|
||||
development-reader probes do not establish a published rollback target.
|
||||
|
||||
Active remote adoption and compatible idle warm reuse carry an existing verified
|
||||
Runner proof. Preparation revalidates that proof and the exact process's liveness,
|
||||
then preserves its in-place harness state. The later protocol attachment still
|
||||
authenticates the durable PRP authority. A provider lease acquisition described as
|
||||
`resumed` does not establish that this already-running harness was suspended.
|
||||
Ordinary resumed leases retain their verified suspension and backup checks.
|
||||
|
||||
Reattachment keeps the complete durable PRP binding, including its turn and item
|
||||
attachment IDs. A healthy warm handoff can create new attachment IDs within the
|
||||
retained Runner. Replacing that run's controller must preserve those IDs; deriving
|
||||
them again from the business run ID would request an invalid authority rotation.
|
||||
|
||||
Cold Codex continuation can restore a completed provider with a retained event
|
||||
suffix. Native remote controllers first persist a private staged attachment under
|
||||
the target run's existing checkpoint owner, then restore and drain the exact
|
||||
source authority before attaching the target. The source's run, turn and item
|
||||
identity stays on every old event. Receipt persistence precedes acknowledgement;
|
||||
unknown semantic work holds the handoff and never reaches the target's tool
|
||||
authority. The provider's settled-session attachment guard remains mandatory.
|
||||
|
||||
The staged controller format is
|
||||
`paperclip.runner.durable.control-plane-state.staged-attachment.v1`. It contains
|
||||
the source and target attachment tuples and one bound target attachment template
|
||||
inside the existing encrypted PRP authority, subject to its existing size limits.
|
||||
The header stores a bounded descriptor and digest. Before command delivery, the
|
||||
template is a separate changed-record part; after queueing, recovery uses the
|
||||
existing command or receipt. Control checkpoints do not rewrite another copy of
|
||||
instructions or tool context in the header.
|
||||
It is separate from the 4 KiB run-profile recovery envelope. Startup discovers
|
||||
this target obligation through bounded encrypted-header reads before a target
|
||||
session snapshot exists. It verifies both frozen runs, the accepted and released
|
||||
source, its original Runner artifact and the exact source header. Remote process
|
||||
verification then chooses authenticated source adoption or verified suspended
|
||||
source restoration. An in-memory rebound snapshot supplies normal admission;
|
||||
an existing target snapshot always takes precedence over that source baseline.
|
||||
The intent remains through command delivery and authenticated handoff, including
|
||||
an uncertain commit response. Source and target receipt phases require their
|
||||
exact physical authority; receipt replay cannot start a second Runner or attach
|
||||
again under a different endpoint. After the ordinary target snapshot commits,
|
||||
an in-memory notification permits intent retirement in the next existing command
|
||||
write. There is no separate metadata/snapshot transaction for that retirement.
|
||||
|
||||
Source accounting uses the source's immutable producer receipts and billing
|
||||
classification. Opening this receiver does not reset accounting ownership or
|
||||
reopen a finalized run. Equal finalized usage retains the existing audit policy;
|
||||
changed, incomplete or conflicting usage remains unacknowledged for
|
||||
reconciliation. The target's usage receiver never receives source events. Staging
|
||||
adds recovery control work only: text streaming and settled idle operation add no
|
||||
queries, snapshot transactions, pools or recurring scans. The first rollback
|
||||
release supporting this continuation must also read the staged controller format;
|
||||
older readers fail closed. Prepared-launch and other provider paths retain their
|
||||
existing admission and recovery contracts.
|
||||
|
||||
Accepted-result retirement still stops the old provider turn before draining and
|
||||
suspending its Runner. The checkpoint codec classifies `turn.stop` as lifecycle
|
||||
maintenance. Existing exact ownership, generation and accepted-result fences
|
||||
govern this cleanup; starting provider work still requires an active owner.
|
||||
A successor claims its existing native coordinator and starts renewal before
|
||||
retiring a superseded warm session. An unclaimed preparation cannot authorize
|
||||
the predecessor's checkpoint writes. Shared attachment staging follows verified
|
||||
retirement; a failed claim or Stop cannot close that predecessor or stage files.
|
||||
Recovery also preserves the implicit isolated-home default when binding admitted
|
||||
configuration, so an unchanged home does not invalidate the retained session.
|
||||
Explicit home-mode changes retain their normal configuration-change checks.
|
||||
Restoring pinned skill versions also preserves the fresh selection's existing
|
||||
name/key order. JSONB map ordering must not change the generated skill preference
|
||||
and invalidate the session when the selected keys, versions and bytes are unchanged.
|
||||
|
||||
For per-turn direct sandbox recovery, the executor reads the retained Runner's
|
||||
suspended lifecycle through the recorded sandbox lease and validates its complete
|
||||
identity and lack of pending delivery/cleanup. Only a lifecycle projection is
|
||||
cached locally; provider files remain on the execution host. Prepared tasks use
|
||||
their existing `read_state` contract. The normal snapshot rebinder, tool-contract
|
||||
checks and current admission authority still govern the next turn.
|
||||
|
||||
An active remote owner causes startup recovery to retain the obligation at the
|
||||
actual lease deadline. The existing recovery dispatch timer is rearmed on drain
|
||||
resume; successful reconciliation leaves no recurring recovery work.
|
||||
Follow-up waits retain their original dispatch mode and owner across a drain,
|
||||
including when another heartbeat service facade resumes the work. A failed
|
||||
dispatch retains a bounded retry obligation rather than losing its deadline.
|
||||
Upgrade diagnostics retain their existing `ready` meaning and add bounded retained-pointer
|
||||
header evidence. That inventory is preliminary, reports partial coverage, and
|
||||
does not certify workspace durability or volume removal.
|
||||
|
||||
`importNativeControllerRecovery` is an internal one-source operation for the
|
||||
migration owner while old controller files remain available. It requires the
|
||||
admitted configuration digest and exact supervisor snapshot, locks the issue,
|
||||
coordinator and run in order, rejects live or unsettled ownership, and publishes
|
||||
only missing metadata. Repeating the same import is a no-op. Normal lifecycle
|
||||
checkpoints publish the metadata without a separate migration write. Disk-free
|
||||
rollback requires a server that reads this envelope and the supported settled
|
||||
warm journal, including its PRP v2 idle goal suffix. Earlier local-only readers
|
||||
and readers lacking that suffix handling remain outside that rollback contract.
|
||||
@@ -81,6 +81,17 @@ Independent SQL writers and additional processes need an explicit wake protocol;
|
||||
process-local notifications do not provide cross-replica delivery. Worker quiet
|
||||
alone is not evidence that the host or database actually suspended.
|
||||
|
||||
## Native reply recovery
|
||||
|
||||
Committed native replies use the shared delivery coordinator and its persisted
|
||||
retry deadlines. One pass checks at most four independent tasks at a time.
|
||||
Checks for the same company and task run in candidate order within the batch.
|
||||
Other scoped deadline workers remain serial unless their owner enables concurrency.
|
||||
Drain retains unstarted checks, and shutdown waits for every admitted check before
|
||||
releasing tracked work. Completed history creates no recurring discovery reads;
|
||||
new committed work reactivates the existing notification path. Pending reply
|
||||
obligations keep their existing sleep blockers.
|
||||
|
||||
## Scheduled routine wake protocol
|
||||
|
||||
Historical failed ephemeral leases and terminal custom-image setup sessions can
|
||||
|
||||
@@ -37,3 +37,21 @@ validation and token gates pass.
|
||||
The production setup components were checked in a background browser using
|
||||
simulated Storybook data: one profile, personal access and OAuth method switching.
|
||||
Full regression checks and PR review are in progress. No live Google proof yet.
|
||||
|
||||
## Draft resume follow-up — 2026-10-28
|
||||
|
||||
The setup draft now retains its validated service request before either managed
|
||||
or customer-owned OAuth handoff. This non-secret intent is bound to the company,
|
||||
connection, initiating actor, grant kind, and credential subject. Denial, expired
|
||||
OAuth state, and a cold Finish setup return restore that exact request rather
|
||||
than the new-account write defaults. Historical or differently owned drafts
|
||||
require an explicit service choice; existing reconnects keep their grant request.
|
||||
|
||||
Agent handoffs still open the addressed human's picker without issuing provider
|
||||
state or inventing service access. A human's reviewed choice consumes only their
|
||||
own pending service proposal, so an older proposal cannot override that choice
|
||||
on the next draft resume. Other users' proposals remain unchanged.
|
||||
|
||||
This follow-up is local to a fresh checkout of master at `820c458465fa`.
|
||||
Provider-fixture/database and production-component tests cover the changed
|
||||
journey; they do not establish live Google consent, merge, or deployment.
|
||||
@@ -0,0 +1,955 @@
|
||||
# Kimi Code native Runner integration — 2026-10-28
|
||||
|
||||
Owner: this chat, with separate provider, distribution, and product owners.
|
||||
Baseline: `dce004259394486f6361d0c0dd5a33a5272acc0a`.
|
||||
Branch: `codex/kimi-code-runner` in the existing managed worktree.
|
||||
Delivery: tested, reviewable PR stack: Runner foundation, shared Core Stop
|
||||
recovery, product setup and acceptance, then qualified pack delivery. Each slice
|
||||
must compile and group cohesive behavior for review.
|
||||
Merge, publication, and production deployment
|
||||
are separate from this handoff. The user confirmed local plus Daytona acceptance.
|
||||
|
||||
## Merge integration — 2026-10-30
|
||||
|
||||
The master integration keeps the existing released Kimi revision 17 and adopts
|
||||
master's upstream-only release attestation. The old source-bound revision 17
|
||||
fixture remains immutable historical evidence with no current launch authority.
|
||||
The pinned Kimi 2.3.3 distribution and platform closure pins are unchanged.
|
||||
Generated profiles, current digest admission, and shared remote installation
|
||||
must pass again on the integrated source. Earlier native acceptance and public
|
||||
packages retain their original source and digest; they do not establish fresh
|
||||
live or ordinary UI acceptance for this integrated identity.
|
||||
|
||||
## Release contract
|
||||
|
||||
An operator selects Kimi Code in native Runner, saves and reopens its exact
|
||||
model/environment configuration, tests the runtime, assigns ordinary work, sees
|
||||
streamed tools and output, receives independently verified files/artifacts and
|
||||
durable task completion, and can Stop owned work. Existing legacy Kimi agents
|
||||
and their history retain their current execution path.
|
||||
|
||||
- First environment: isolated local macOS arm64; remote: Daytona Linux x64.
|
||||
- Runtime under investigation: official `@moonshot-ai/kimi-code@2.3.3`, ACP over
|
||||
stdio, behind the existing ACPX driver. The ambient Mac binary is 0.41.0 and
|
||||
cannot establish the new runtime's compatibility.
|
||||
- Model: explicit caller-selected identifier, acknowledged before the first
|
||||
prompt. Qualification uses authenticated `kimi-for-coding`;
|
||||
discovery is never a product allowlist.
|
||||
- Authentication: explicit company-secret `KIMI_API_KEY` environment binding.
|
||||
Qualification uses only the active global login's short-lived access token;
|
||||
the provider accepted a real model/tool turn. Refresh tokens and login homes
|
||||
are never transferred. Managed Kimi account connections are deferred.
|
||||
- Instructions: exact composed content must reach native model instructions on
|
||||
create and load, with correlated acknowledgment; missing proof rejects prompt
|
||||
admission. Current official ACP does not consume this field, so a pinned,
|
||||
provider-owned runtime patch is required unless a richer interface proves better.
|
||||
- Required: model/instruction admission, scoped authenticated tools, streaming,
|
||||
semantic questions, permission decisions, cancellation, session continuation,
|
||||
terminal settlement, cleanup, clean installation, and the ordinary product path.
|
||||
Native question/plan controls are inventoried separately before any claim.
|
||||
- Deferred unless separately qualified: managed browser login, active steering,
|
||||
inbound media, native question/plan controls, native child/background agents,
|
||||
first-class child projections, additional OS/architectures, and usage accounting.
|
||||
Native ACP reports context occupancy, not usable per-run token/cost totals.
|
||||
Dollar budget enforcement is therefore not a qualified capability.
|
||||
- Spending: proportionate goal-authorized test/provider/infrastructure work;
|
||||
record material costs and unknowns, diagnose first divergence before paid retry,
|
||||
stop uninformative attempts. No recurring infrastructure commitment.
|
||||
|
||||
## Architecture and ownership
|
||||
|
||||
Provider translation and native patch remain Kimi-owned. Shared Runner supervision,
|
||||
PRP transport, durable decisions, authorization, and task finalization retain
|
||||
their existing contracts. Distribution is a pinned installed closure with no
|
||||
fallback to ambient Kimi. Runtime admission remains pending until required live
|
||||
qualification passes. Provider registration does not migrate legacy agents.
|
||||
|
||||
## Current release state — 2026-10-30
|
||||
|
||||
The pending11 campaign is historical baseline evidence. Its final Daytona
|
||||
Stop/new/resume attempt failed because canonical sandbox reuse left an earlier
|
||||
lease for the same allocation without an explicit ownership handoff. The run
|
||||
created all three provider turns and completed the authored reset, but source inspection correctly
|
||||
rejected ambiguous ownership before file preservation. The original failure and
|
||||
cleanup evidence remain unchanged; the campaign has nineteen passing distinct
|
||||
cases, not twenty.
|
||||
|
||||
The repair records an atomic server-owned handoff on both lease rows. Recovery
|
||||
accepts only the complete backward chain of inactive, finalized native owners
|
||||
with matching company, task, agent, workspace, provider and allocation. Missing,
|
||||
foreign, active, conflicting or additional owners remain blocked. The exact
|
||||
producer regression fails before the repair and passes with real file copyback;
|
||||
all 114 owning recovery tests pass. This is deterministic recovery evidence,
|
||||
not a replacement for the live Daytona case.
|
||||
|
||||
The foundation was reconciled with master
|
||||
`a8e07500db68f187b19de83823eb7fabf1196021`. Shared permission, turn attribution
|
||||
and runtime activity changes required new native artifacts and fresh pending14
|
||||
qualification. Its immutable declaration and Mac/Linux artifacts remain recorded
|
||||
at their actual source identities. The pending11 nineteen passes remain historical
|
||||
baseline evidence with no pending14 or later runtime qualification credit.
|
||||
|
||||
CI rejected a stale Pi fixture in a shared Rust unit test after candidate13 was
|
||||
sealed. The corrected test advanced Kimi to pending14 before its shipping build
|
||||
or live cases. Candidate13 and the original failure remain unchanged.
|
||||
|
||||
Pending14 has two accepted local cases: Stop/new and message-marker. Both retain
|
||||
their original grade, scoped native14 audit and full physical host closure. The
|
||||
next local Plan-revise-accept attempt failed before its revised model turn at
|
||||
PRP `run.attach`: `ACPX durable launch profile digest does not match runner startup`.
|
||||
The sequence stopped. Seven other local cases and all ten pending14 Daytona task
|
||||
cases remain unstarted. The original failed Plan and earlier zero-capture diagnostic
|
||||
stay unchanged; the diagnostic has no acceptance credit.
|
||||
|
||||
A credential-free owner regression reproduces the cold restore error. New-run
|
||||
startup authority validates, but the saved launch digest still binds the prior-run
|
||||
grant. The reviewed native repair proves that exact prior digest by changing only
|
||||
the candidate run binding, retains full company/model/profile/closure/expiry/session
|
||||
and artifact checks, and separates bare cold attach from authenticated warm rotation.
|
||||
Its exact-final baseline fails; five focused tests and all thirty-five owning backend
|
||||
tests pass. The initial sandbox TCP-bind failures remain recorded separately from
|
||||
the successful loopback-authorized owning run. These controls are not model acceptance.
|
||||
|
||||
The cold repair shipped as pending15 at source
|
||||
`9e82c00ad8e2eba628a33a2e1857379769ad7128`. All ten local task cases passed
|
||||
with original grades, scoped native15 audits and physical host closure. The
|
||||
causal Plan-revise-accept case reused the same native/provider session across its
|
||||
first and revised runs. No pending15 Daytona model case ran. Its immutable Linux
|
||||
image was built, independently checked, loaded once and published once at
|
||||
`sha256:61b097ec5e1c581a0868c174881497fab556c9197d92261595ddc75469d1787f`.
|
||||
The earlier public-push review rejection and later exact human authorization stay
|
||||
recorded. This is an original-source artifact, not acceptance of a later runtime.
|
||||
|
||||
Master advanced to `0c70cedce9a803bdc201324528abb35e423e7cd0` before remote
|
||||
qualification. The real reconciliation preserves upstream runtime-context and
|
||||
sidecar contract generation, confirmation replay and provider-pack contract digest
|
||||
verification. The critical Kimi inventory now has 58 source inputs. Maintained
|
||||
identity generation derives pending16, Copilot91 and Grok18; Pi36 is unchanged.
|
||||
Colliding Copilot88 and Grok15 historical fixtures retain the upstream canonical
|
||||
filenames while the unmerged Foundation variants remain byte-exact under
|
||||
branch-qualified names with explicit lineage and no execution authority.
|
||||
|
||||
All earlier pending15 passes and artifacts remain historical, source-bound
|
||||
evidence. They supply zero pending16 acceptance credit. The earlier held qualified16
|
||||
proposal is obsolete and remains immutable. The reconciled source requires new
|
||||
native artifacts on both targets, exact installed pack verification and a fresh
|
||||
20-case local/Daytona matrix with full resource, key and host closure before a new
|
||||
ordinary qualification proposal. The existing immutable image may be a physical
|
||||
base only when the normal installer verifies the current full Linux pack and
|
||||
native daemon; its old pack is not relabelled as current. No new artifact build,
|
||||
model test or promotion is authorized by this source proposal.
|
||||
|
||||
The pending14 zero-model Daytona preflight passed native create/load, HTTPS checks,
|
||||
raw sandbox absence, complete scoped inventory and child-key revocation/list absence.
|
||||
It made no model request and supplies zero task-qualification credit. The retained pending15
|
||||
Foundation, Core and Product exact-head CI and reviews are green; the new private
|
||||
master reconciliation has no published-head CI claim. Original runner-shutdown attempts and the single successful
|
||||
failed-jobs-only retry request per Core/Product run remain recorded.
|
||||
|
||||
At pending16 source `603756c22b2e39de9b5b5c3c26a0eb23c67604a1`, the
|
||||
causal local Plan and message cases passed. The next local ask-question case
|
||||
failed with `native_finalization_missing: session returned no semantic result`.
|
||||
Its native terminal reported completed/yielded without a semantic result. The
|
||||
sequence stopped, and original/physical cleanup passed. The original failure is
|
||||
retained for owner diagnosis. Current acceptance is two passed, one failed and
|
||||
seventeen unstarted; no paid retry or source repair is claimed here.
|
||||
|
||||
A proposed ordinary qualified17 transition is prepared separately and remains
|
||||
HELD. It must not be applied, built for release or deployed until Root accepts all
|
||||
twenty pending16 original cases and their full closure at the frozen execution
|
||||
source. Earlier pending15 local passes remain historical and supply no pending16
|
||||
credit. This ledger does not claim a pending16 pass count, a qualified17 artifact,
|
||||
a hosted result or permission to promote.
|
||||
|
||||
The held transition changes the top Product source only. Foundation and Core stay
|
||||
at their reviewed pending16 prerequisite identities; no extra merge or mirrored
|
||||
generation exists only to make branch metadata equal. The final qualified build
|
||||
must bind its actual top revision, complete native58 inventory, full pack contract
|
||||
and exact target artifacts. Ordinary startup and Test must work without a candidate
|
||||
grant, and retained pending16 identities must remain historical-only for execution.
|
||||
The actual accepted pending16 evidence and final qualified limitations must be
|
||||
recorded before the eventual qualified-source commit and artifact build.
|
||||
|
||||
Pending qualification uses the existing trusted startup-admission channel with
|
||||
exact company, model, profile, platform closure, expiry, run and session bindings.
|
||||
A warm handoff may rotate only the authenticated run identity through the normal
|
||||
Runner authority transition. Saved agent configuration cannot grant candidate
|
||||
authority. Ordinary installed and hosted qualified setup must work without that
|
||||
test authority.
|
||||
|
||||
| Required gate | Current evidence | Remaining proof |
|
||||
| --- | --- | --- |
|
||||
| Merged native behavior | Cold repair RED/GREEN retained; current master union and pending16 deterministic checks reviewed separately | Freeze reconciled source; build and verify both native targets and current pack contract |
|
||||
| Local and Daytona acceptance | Pending15 ten local passes; pending14 Plan failure and all original records unchanged | Fresh pending16 twenty-case matrix and full closure; zero carried acceptance credit |
|
||||
| Credential isolation | Protected startup and private-stdin SSH checks pass; pending14 zero-model preflight closes | Verify normal qualified setup and Test without candidate authority |
|
||||
| Normal distribution | Maintained closure has eighteen public packages and four daemon targets | Exact qualified-source build, full checks and clean installed consumption |
|
||||
| Ordinary product setup | Isolated hosted TEST workspace reserved | Installed Mac and hosted Linux setup/Test/save/reopen, assigned skill, download and reconnect |
|
||||
|
||||
Delivery remains a tested, reviewable PR stack and the isolated hosted TEST
|
||||
workspace. Production deployment, fleet defaults, stable publication and GitHub
|
||||
merge remain outside this handoff.
|
||||
|
||||
## Historical pending11 acceptance ledger
|
||||
|
||||
| Required gate | Environment | Observation | Verdict |
|
||||
| --- | --- | --- | --- |
|
||||
| Exact model and instructions | local and Daytona | Pending11 pinned native create/load/model/instruction controls passed on both platforms; the current authenticated product matrix remains incomplete | partial |
|
||||
| Scoped tools and credentials | local and Daytona | Native v6 credential fences passed; the first pending11 Daytona task received its exact full scoped context result and reached the intended unanswered Write | partial |
|
||||
| Ordinary setup | installed local and isolated hosted test | Ordinary connect/save/reopen/Test/run/reconnect and assigned-skill behavior remain required | unattempted |
|
||||
| Useful task | local and Daytona | Pending11 local and Daytona file editing passed; ordinary installed and hosted downloads remain | partial |
|
||||
| Human input and Stop | local and Daytona | All pending11 local human-input and Stop journeys passed; Daytona question, Plan and Write Stop passed; remote Stop/new/resume remains | partial |
|
||||
| Continuation and child loss | local and Daytona | All pending11 local continuation journeys and authenticated Daytona native worker loss passed; remote Stop/new/resume remains | partial |
|
||||
| Installed distribution | macOS arm64 and Linux x64 | Pending11 native closures, Mac binaries and immutable Linux qualification image verified; final normal public-package installation remains | partial |
|
||||
| Regression and repository checks | candidate revision | Prior foundation/product heads passed CI and review; shared recovery and admission controls passed real PostgreSQL owning suites and focused tests; final integrated full checks remain | partial |
|
||||
|
||||
Current pending11 totals remain 0 fully passed, 7 partial, 1 unattempted of 8
|
||||
gates. The two-environment matrix has nineteen distinct passing case records:
|
||||
all ten local journeys and nine Daytona journeys. Daytona Stop/new/resume remains
|
||||
required. Four original cases retain Core/source
|
||||
`0ca0d6b87d6d69b8a900621fb834ed74cb1ae446` and suite V4. Nine later cases retain
|
||||
source `d721f2f9052e19370985e53a7493e8792e319313` and suite V5. Six Daytona
|
||||
cases retain source `e972a46e5aedfcb48d7cb61328e81fb70ce7dccb` and suite V5.
|
||||
A fresh local Plan workflow at that last revision is representative handoff
|
||||
evidence and adds no distinct case. Each record retains its original expected
|
||||
terminal state; conversation continuity can finish in review. None is relabelled
|
||||
as a later source or suite. Original material failures remain failed.
|
||||
|
||||
The final local Stop/new/resume result and cleanup passed. Its original strict
|
||||
audit rejected the server-only `/new` record because that row retains the default
|
||||
legacy field. A separate reviewed classification verifies the saved human command,
|
||||
company/task/session generation, the complete two-event terminal record, and the
|
||||
unchanged server branch that returns before provider selection or execution. It
|
||||
also checks all three actual provider runs against the exact native Kimi11 model,
|
||||
profile and scope. The original audit remains failed; the derived classification
|
||||
has positive and negative controls, including conflicting duplicate rows in both
|
||||
orders. It does not treat the reset operation as inference.
|
||||
|
||||
The next Daytona plan-revision attempt failed during shared agent-directory
|
||||
adoption before the second provider turn started. Its original failed result,
|
||||
cleanup failure and raw package remain unchanged. A real PostgreSQL regression
|
||||
reproduced a metadata-only race: the collection worker changes `nextAttemptAt`
|
||||
and `updatedAt` between the final predecessor read and the adoption claim. The
|
||||
repair in code commit `759aad8f4fa8262b8af6e797617c0d89061681be` locks and rereads
|
||||
the predecessor under the existing agent and lease locks, compares its complete
|
||||
authority except those two scheduling fields, and retains the original claim
|
||||
predicates with the locked timestamp. Eighteen real-database controls and server
|
||||
no-emit validation pass, including competing owners, same-baseline receipt or
|
||||
checkpoint changes, retirement, canonical changes and foreign scope. The original
|
||||
incident did not retain its exact failed predicate, so the reproduced race is not
|
||||
claimed as conclusive incident causality. Closed-field conflict diagnostics will
|
||||
make a further failure discriminating.
|
||||
|
||||
The failed remote campaign's key is revoked. Additional cleanup uses raw sandbox
|
||||
resource APIs with an explicit verified organization: a toolbox lookup returning
|
||||
404 cannot establish sandbox deletion. Exact filtered inventory exposed two
|
||||
archived verification probes omitted by the earlier summary. All eight known
|
||||
owned allocations, including preflight, now return authoritative resource 404.
|
||||
The provider list retains stale entries; every returned owned entry was separately
|
||||
checked through the resource API. The earlier summary and failed cleanup grade
|
||||
remain preserved. Future campaigns require the corrected raw-resource cleanup
|
||||
and its real-SDK mocked-HTTP regression before allocation.
|
||||
|
||||
A separate host prerequisite failed before any test or model request because
|
||||
PostgreSQL 18.1 exhausted macOS System V semaphores. Qualification uses a verified
|
||||
official PostgreSQL 18.6 build with named POSIX semaphores through an isolated,
|
||||
recorded worktree dependency binding. A fresh migrated control plane reached
|
||||
health 200 at the actual source revision; a same-cluster SQL check confirmed its
|
||||
version, database and empty company schema. Owned processes and ports closed.
|
||||
The binding must be restored before normal package validation. Earlier PostgreSQL
|
||||
18.1 evidence retains its original environment identity.
|
||||
|
||||
The source-mode Product E2E control plane serves the checked-out source. Its
|
||||
source revision, production-byte baseline and separately compiled Core artifact
|
||||
are distinct roles. Native Mac binaries and the immutable Linux image retain
|
||||
actual build source `3a2dc2d78df0da51ef56dd6d898a2993496d3567`; neither is
|
||||
relabelled or rebuilt for a Core-only ownership repair.
|
||||
|
||||
The shared directory repair passed a fresh local and Daytona Plan workflow,
|
||||
including three native turns, the exact human revision decisions and resource
|
||||
cleanup. Five subsequent Daytona cases passed: ordinary and structured questions,
|
||||
structured-question restart, file editing, and restart continuity. The last
|
||||
Stop/new/resume case failed before reset: the cancelled native turn retained a
|
||||
workspace preservation obligation, and the saved `/new` message did not cross
|
||||
that hold. The original failed result and its original cleanup grade are retained.
|
||||
Independent raw resource checks found one stopped allocation despite an empty
|
||||
filtered inventory; exact run, organization and ownership evidence authorized
|
||||
its separate deletion, followed by an authoritative resource 404.
|
||||
|
||||
Native admission renewed the real preparation lease and changed the run to native
|
||||
mode without retiring that earlier ownership. The recorded future expiry can
|
||||
block workspace recovery after the provider has stopped. The original inspector
|
||||
log does not identify whether its coordinator, active-control or preparation-lease
|
||||
predicate fired first. A real PostgreSQL regression reproduces the stale fields
|
||||
on both local and remote admission. The repair clears `controllerBootId`,
|
||||
`controllerLeaseExpiresAt` and `executionStage` in the existing row-locked native
|
||||
admission transaction. The current preparation owner is still validated before
|
||||
the native coordinator commits. All twenty-eight controls pass, including foreign
|
||||
and expired ownership, cancellation before admission, transaction rollback, live
|
||||
preparation recovery and a later workspace synchronization failure. Workspace
|
||||
preservation, native ownership and physical Stop guards stay unchanged.
|
||||
|
||||
The preparation handoff repair passed a fresh local Stop/new/resume workflow at
|
||||
`a44ebc8abe8ebc766a207a9e68460db9829d6129`: three native runs, one authored
|
||||
server reset, generation one and a fresh final native session. Original result,
|
||||
raw evidence, native/control classification and physical cleanup pass. It is a
|
||||
representative regression check and adds no distinct case.
|
||||
|
||||
A second real PostgreSQL regression found that a valid Stop can arrive after a
|
||||
workspace check has saved a twenty-four to thirty-six second retry delay. The
|
||||
existing post-cleanup callback did not advance that check when physical Stop
|
||||
became confirmed. The shared repair records one marker bound to the same run,
|
||||
recovery generation and closed native/physical receipt digest. Only the normal
|
||||
post-executor cleanup producer records that evidence. The recovery service can
|
||||
advance its matching Stop recheck once, preserving retries, window and provider
|
||||
cooldown. A marker arriving while the worker runs stays pending until its failure
|
||||
commit, so the notification cannot be lost. File preservation and model admission
|
||||
remain under their existing owners.
|
||||
|
||||
The exact final positive fails against the prior implementation and passes with
|
||||
this repair. Nineteen final focused controls and typecheck pass. The earlier owning
|
||||
recovery and run-control suites passed one hundred thirty-two tests; those results
|
||||
retain their earlier test-module identity. The positive uses the actual resolved
|
||||
preserve-without-replay writer, saved human `/new` intent, real archive copy-back
|
||||
and normal explicit continuation admission. It proves timely reset eligibility;
|
||||
actual browser generation and a fresh native turn still require the final Daytona
|
||||
case. No blocker, timeout, grading, schema or polling change is included.
|
||||
|
||||
The six shared Core source/test changes are a separate prerequisite,
|
||||
[#18245](https://github.com/paperclipai/paperclip/pull/18245), based on the frozen
|
||||
Runner foundation. Product retains its Kimi plan, UI, fixtures and provider files.
|
||||
Product contains the real prerequisite ancestry, including queued-input repair
|
||||
`5c964f5d3cfd65e493b92c53cfe0c80b0d21f54d`. The final compiled Core must
|
||||
identify the integrated product source; earlier Core, native and image identities
|
||||
are retained separately.
|
||||
|
||||
A further real PostgreSQL control reproduced an already queued `/new` request
|
||||
being cancelled before provider admission and its saved wake marked skipped.
|
||||
The shared dispatch owner now keeps only that exact current human wake deferred
|
||||
inside the cancellation transaction. It requires the canonical no-execution proof,
|
||||
matching native preservation action and unfinalized native baseline. The existing
|
||||
queued-writer guard orders copyback after that commit. Real copyback and the
|
||||
normal callback then adopt the original input once, including concurrent callback
|
||||
delivery. Foreign, terminal and unverified requests remain fenced. The unchanged
|
||||
production patch passed 164 owning tests; final focused controls, the actual
|
||||
stranded-recovery retry discriminator and no-emit validation passed. A saved prior
|
||||
session reference remains a hint. The deferred wake prevents a competing automatic
|
||||
retry; no new retry policy, queue hook or polling was added.
|
||||
|
||||
The extra cancelled admission receipt remains in the raw Product E2E evidence.
|
||||
Kimi suite V6 classifies it separately from the three native provider turns and
|
||||
the single authored reset. It requires the complete server event receipt and final
|
||||
public proof of the human command, stopped source, preservation, authorization,
|
||||
reset and fresh native session. Provider counts, deadlines and earlier grades stay
|
||||
unchanged. Forty-nine focused controls and narrow typecheck pass. Controlled
|
||||
Chromium exercised the prior count failure and corrected full flow; that is
|
||||
fixture calibration, not live model acceptance. The nineteen accepted originals
|
||||
retain V4/V5. Only the remaining fresh case uses V6, fingerprint
|
||||
`f7883fff6c2a5fda14571ef150849e3d2514be036fed941e257276e6b956e5d7`.
|
||||
|
||||
Next, complete the one remaining Daytona Stop/new/resume case. The local
|
||||
representative remains valid because the additional event behavior requires a
|
||||
remote physical termination receipt and preserves local Stop policy. The
|
||||
controller must check every exact sandbox ID from authoritative run records,
|
||||
including failed attempts, even when filtered inventory is empty. Retain the
|
||||
nineteen passing cases only under the explicit source impact record. Qualification
|
||||
promotion remains held until twenty distinct cases and physical resource and key
|
||||
closure are verified.
|
||||
Afterward build the normal four-platform daemon and eighteen-package distribution,
|
||||
verify ordinary installed local setup, and deploy the exact qualified revision
|
||||
only to `kimi.test.paperclip.app` for normal configuration, assigned-skill, CSV
|
||||
download and persistence checks. Those installation and hosted journeys remain
|
||||
required. Production, fleet defaults, stable publication and GitHub merge remain
|
||||
outside this handoff.
|
||||
|
||||
## Earlier pending11 recovery observations
|
||||
|
||||
The following observations retain their original source, artifacts and next-step
|
||||
context. The current state above supersedes their campaign instructions.
|
||||
|
||||
At this earlier snapshot, pending11 totals: 0 fully passed, 7 partial, 1 unattempted of 8 gates.
|
||||
The current twenty-cell matrix has four passing cells on frozen Core/source
|
||||
`0ca0d6b87d6d69b8a900621fb834ed74cb1ae446`, pending Kimi11 and suite V4:
|
||||
local native Write Stop, local native worker loss, local message-marker, and
|
||||
Daytona native Write Stop. Their original reports retain that identity; sixteen
|
||||
cells remain. Original failed attempts remain failed. The fixture-only V5 successor
|
||||
clarifies Plan marker formatting and expands detailed run evidence without
|
||||
changing their task prompts, graders, Core, native closures, or immutable image.
|
||||
The remote observer source hash changes with its new diagnostics and remains an
|
||||
explicit test-identity derivative in the scoped context; full model-facing byte
|
||||
equality is not claimed. Original Stop did not invoke the fault branch, common
|
||||
observer functions remain unchanged, and fresh authenticated worker-loss coverage
|
||||
through the new observer is required before final acceptance. Profile9 results remain
|
||||
historical support. These grouped gates do not replace per-cell evidence.
|
||||
|
||||
The explicit-only `kimi-qualification` Product E2E suite has 20 required cells:
|
||||
message-marker, plan-revise-accept, ask-question, structured-question-resume,
|
||||
structured-question-restart-resume, file-edit-validate, continuity-restart, and
|
||||
stop-new-resume, native pending-permission Stop, and native pending-permission worker loss, each on local and Daytona. Each has one required repetition and
|
||||
zero automatic paid retries. Historically, all ten profile9 local cells passed
|
||||
with report packaging and cleanup. Stop retains source `821256d6`, campaign
|
||||
`local-2026-10-29T14-46-04-979Z`, run `91557abb-b9c0-422d-af1e-f0e82e948e96`.
|
||||
The other nine retain source `f7ac439e` and their individual campaigns between
|
||||
15:10 and 15:33 UTC. The qualification-only observer correction leaves all native
|
||||
runtime/profile/image inputs unchanged. It reads argc-bounded kernel argv while
|
||||
retaining exact compiled-loader hashes, sealed native bytes, inode, ancestry and
|
||||
run identity; a real compiled native/ACPX peer proved title mutation, excluded
|
||||
an environment canary, rejected foreign/ambiguous targets and retired the child.
|
||||
All ten original result hashes were independently checked. The four earlier local
|
||||
failures remain unchanged alongside their later diagnoses and cleanup receipts.
|
||||
|
||||
Historical profile9 Daytona qualification stopped at its first lifecycle case.
|
||||
A later independent message-marker control passed all six matchers, durable Done,
|
||||
native Succeeded and cleanup on source `50736cb6`, run
|
||||
`0a809b5e-9a82-4590-9283-cbaa359f676f`. That is one case, not the full matrix.
|
||||
The first launch stopped at
|
||||
admission because two executable digests were missing; no task or sandbox was
|
||||
created. With the exact image Node/runnerd digests supplied, the first actual
|
||||
Stop cell failed before provider startup: setup Test could not reach the global
|
||||
Kimi catalog. Its failed/cleanup-failed grade is preserved. Six archived Test
|
||||
sandboxes were subsequently ownership-verified, deleted and checked absent404;
|
||||
that post-failure recovery does not change the original grade.
|
||||
|
||||
Assigned-skill behavior and ordinary connect/save/reopen/Test/run/reconnect remain
|
||||
separate requirements, including the requested hosted test workspace.
|
||||
|
||||
### Earlier state before fixture V5
|
||||
|
||||
Four current cells passed with independently checked report hashes, native profile/model identity, terminal state, and cleanup. Local Plan acceptance then stopped because the model omitted the required draft marker in its actual `write_document` input; saved bytes, API reads, and the Plan UI matched that input. The strict marker and revision/approval checks remain. A fixture-only successor makes the literal first-line format explicit and increments the Kimi definition to V5; a later passing response does not establish reliability or alter the original failure.
|
||||
|
||||
The first current Daytona worker-loss attempt failed inside its one-shot fault observer. Its original diagnostic erased the failing guard and cannot establish whether a signal occurred. A separate no-network Linux x64 control used the immutable image, actual Kimi and compiled ACPX, and a simulated model. Python 3.13.5 process-handle signaling and all fourteen Linux selector controls passed. The real pending Write was reached, then inspection failed with a missing-file error before any fault signal. Normal Stop retired all seven owned processes, and both controllers independently verified the sandbox absent before the temporary key was revoked. This is diagnostic evidence, not authenticated task acceptance. A second inspect-only control located the exact read: the selector resolved the authenticated parent executable to a deleted Linux memfd alias, then tried to reopen that name instead of its held descriptor. It failed before signaling. Its normal Stop retired all seven processes, and sandbox deletion and key revocation were independently verified. The descriptor correction then passed a same-process Linux comparison: the old selector reproduced that exact error, the corrected selector verified the held executable and its kernel seals, and the unchanged process-handle operation killed only the owned native Kimi child. The unanswered permission expired with `provider_process_lost` and replay disabled; a correctly scoped stale approval failed, no file effect or second model request occurred, and all seven owned processes retired. Sandbox deletion and temporary-key revocation passed independently. All model responses in these controls were simulated. Fresh authenticated worker-loss acceptance remains required. The existing TypeScript wrapper still expected fourteen Python tests after the suite grew to twenty-five. It reproduced that count failure, then passed all four checks after its expected count was updated. This correction changes test accounting only.
|
||||
|
||||
Pending Kimi11 and Copilot83 are frozen after the actual lock-refresh prerequisite
|
||||
was merged. The prerequisite's squash commit has the same complete tree as the
|
||||
reviewed source. Genuine merges preserved all published identity fixtures and
|
||||
the managed lockfile; no history-only reconciliation or policy bypass was used.
|
||||
|
||||
Fresh pending11 Mac native controls passed. The Linux qualification image was
|
||||
built, published under an immutable digest, and independently checked against all
|
||||
42 exported layers, native executable hashes, and installed metadata readers.
|
||||
These are delivery prerequisites, not the twenty authenticated Product E2E cells.
|
||||
|
||||
The first pending11 Daytona Stop attempt reached real inference, one normal UI
|
||||
Allow once for the scoped empty-input context read, and the exact intended native
|
||||
Write permission. It failed before submitting Stop. The fixture incorrectly
|
||||
equated the native turn with the controller envelope turn and tried to parse a
|
||||
bounded display event as the full tool result. The official encrypted checkpoint
|
||||
retained the full matching result and its completed delivery acknowledgment.
|
||||
|
||||
The run then reached its configured 120-second timeout. Recovery later overwrote
|
||||
that error after mistaking the remote numeric PID for an unrelated live process
|
||||
on the controller host. Commit `a24ebd5e7e3fa0d086e91cd3824efd52c8eaee0b`
|
||||
requires exact current host-authored local process identity before local probes
|
||||
or clearing. Remote or unknown identities stay under native recovery authority.
|
||||
Real PostgreSQL tests reproduce both remote collision states, preserve verified
|
||||
local live/dead behavior, and retain authenticated remote termination handling.
|
||||
|
||||
The corrected fixture passed 62 deterministic controls, targeted typecheck,
|
||||
and the real MCP/sidecar/Core-to-official-PostgreSQL boundary with seven
|
||||
negative scope/session controls. It positively bridges native and controller
|
||||
turns through the exact authenticated request and delivered command. It binds
|
||||
the logical workspace separately from the physical observer lease and retains
|
||||
the original calibration fixture as version1 beside the new version2 fixture.
|
||||
The current suite is version4 and remains explicitly pending.
|
||||
|
||||
The next controlled attempt, on `23958e5ad07e0b93b8eef098a9d0038c215bad15`,
|
||||
passed the full scoped context checkpoint proof and reached the native Write.
|
||||
It then failed before Stop because the UI check counted all visible request
|
||||
cards: the resolved context request and the pending Write. Its original
|
||||
failed/cleanup-failure grade, 49 raw files and 65 executable snapshots remain
|
||||
unchanged. Separate cleanup again verified all owned cloud resources absent and
|
||||
revoked the temporary key. This is a second failed attempt, not Stop acceptance.
|
||||
|
||||
The corrected selector binds the exact target card to the independently verified
|
||||
native pending callback before and after lookup. It retains resolved history and
|
||||
the same card after cancellation or expiry. Ten Chromium controls reproduce the
|
||||
old failure and reject wrong targets, duplicates, changed callback identity and
|
||||
disabled decisions. The existing native/context controls and targeted typecheck
|
||||
pass; suite version4 and its promised behavior are unchanged.
|
||||
|
||||
Fresh CI also exposed intended-local test fixtures that inserted bare PIDs without
|
||||
using the real process-metadata writer. Those fixtures now use that writer and
|
||||
retain their original recovery, contract and concurrency assertions. The separate
|
||||
unknown-owner test kept its original assertion and exposed a real guard-order
|
||||
regression: an observed run could lose its ownership warning. The minimal repair
|
||||
restores the existing observed-owner warning before the unknown-ID hold, without
|
||||
probing host PIDs or granting clearing, signaling or retry authority. Both unknown
|
||||
and remote observed-owner controls pass, as do all 65 recovery-module controls and
|
||||
the original assembled recovery cases. Real remote timeouts retain their errors.
|
||||
|
||||
The original failed/cleanup-failure verdict and raw reports remain unchanged.
|
||||
Separate cleanup verified every owned sandbox absent and revoked the temporary
|
||||
test key. The fixture's later close error did not establish its retirement seal;
|
||||
no cleanup check is relaxed. The generic ACPX display clipping metadata defect
|
||||
is pre-existing and remains a documented limitation; full model-facing delivery
|
||||
uses a separate acknowledged path.
|
||||
|
||||
The repair changes controller recovery and qualification fixtures. It does not
|
||||
change the Kimi native patch, either closure, the tested Mac binaries, or the
|
||||
Linux image inputs. A new matrix revision must record the exact reviewed delta
|
||||
and rebuilt server identity while retaining the original artifact build SHAs.
|
||||
The next paid action is one corrected Daytona Stop cell after the rebuilt controller and
|
||||
strict checkpoint/card admission controls pass. Its original grade, raw files and physical cleanup must
|
||||
pass independent audit before the remaining nine remote and ten local cells.
|
||||
|
||||
After all twenty cells pass, promote ordinary admission in a separate reviewable
|
||||
qualification PR, build the normal four-platform daemon/public-package set, and
|
||||
verify the installed local UI. Then deploy the exact qualified revision only to
|
||||
the user-approved isolated `kimi.test.paperclip.app` test workspace and drive the
|
||||
ordinary hosted configuration, assigned-skill, artifact and persistence journey.
|
||||
Production, fleet defaults, stable publication and GitHub merge remain outside
|
||||
this handoff.
|
||||
|
||||
## Reconciliation and earlier qualification history
|
||||
|
||||
The following records describe earlier candidate states and their original
|
||||
evidence. The current state above supersedes their next-step and blocker text.
|
||||
|
||||
The next shipping prerequisite is now frozen at the existing lock-refresh PR
|
||||
[#18185](https://github.com/paperclipai/paperclip/pull/18185), head
|
||||
`9600623783d118a79f8008deb9ab05d80715a15a`. Reconciliation is in progress;
|
||||
the earlier master freeze below remains historical. The trusted repository
|
||||
policy permits lock changes only through its maintained refresh branch or
|
||||
Dependabot. This existing prerequisite contains the exact thirteen-line lock
|
||||
correction already used by the candidate, with matching reviewed report proof.
|
||||
It also inherits three newer master changes, including native finalization and
|
||||
the Antigravity builder. Actual foundation merge conflicts make a genuine
|
||||
reconciliation necessary. The new behavior and published identity bytes must
|
||||
survive; no policy exception, mixed-branch rename, or history-only merge is used.
|
||||
|
||||
The builder is one of Kimi's 31 attested source files, so that reconciliation
|
||||
requires a new pending identity, expected version11, and a later qualified
|
||||
identity, expected version12. No full paid pending10 cell was started. Its
|
||||
successful Linux build, zero-prompt native create/load proof, complete exported
|
||||
image integrity checks, and controlled Mac native fence remain evidence for
|
||||
pending10 only. The exported image is retained without ECR publication. A fresh
|
||||
build and all twenty live cells will bind the reconciled candidate. Later
|
||||
unrelated master activity alone does not reopen this prerequisite freeze.
|
||||
|
||||
The revised remote fixture also passed the actual maintained MCP-to-checkpoint
|
||||
boundary and an isolated real PostgreSQL observer check. That check used the
|
||||
official encrypted checkpoint loader in a read-only transaction, verified the
|
||||
saved checkpoint remained unchanged, and closed the observer before stopping
|
||||
its owned database. The integrated observer's bytes match the calibrated source.
|
||||
These controls are prerequisites for the first live native Stop case; they do
|
||||
not establish authenticated model behavior or full recovery acceptance.
|
||||
|
||||
The shipping baseline is now frozen at master `9f3fc2094f61e311b2c5adb4d5beccefc0199943`.
|
||||
The reconciled foundation is `7c6cffafa26c70e50c241c71dd536aa79406041d` and
|
||||
product code is `60f0bb7b3171e773f21fe80fb524d9130e36500f`, with 97 and 77
|
||||
review paths respectively. Both existing PR branches were updated by fast-forward.
|
||||
Published master identity bytes win over conflicting unmerged variants; original
|
||||
candidate variants remain in preserved Git history and their evidence receipts.
|
||||
|
||||
Current Kimi10 is pending and hidden from ordinary selection, command digest
|
||||
`sha256:c8a39af9d580cdb5fe20a3333a566e2f97a1b4ec3a0025cb647d2144df358604`.
|
||||
The native2.3.3 patch v6 and vendor closures are unchanged. Current dependent
|
||||
identities are Pi35, Copilot82 and Grok14. Earlier prepared qualified10 and
|
||||
Copilot82 digests and the `9ba72696` draft image identity are obsolete and must
|
||||
not be reused. Qualification will promote a new identity only after live gates.
|
||||
|
||||
Master adds reachable notification context/helper environment and Unix Runner
|
||||
FIFO/listener/thread behavior, plus connection/restart ownership fixes. This
|
||||
changes actual Kimi and native Bash execution, beyond identity text. All twenty
|
||||
local/Daytona cells will therefore refresh against pending10, preceded by actual
|
||||
native credential/context controls. Old9 grades and native binaries are not
|
||||
relabeled as pending10 or as the final release.
|
||||
|
||||
The pending10 image build is pinned to product `60f0bb7b`, canonical candidate
|
||||
content `af2d4e8d8ea38381eb265dfdf8141d23ca31369bcb7ae12db8c6daa1b34a4e15`.
|
||||
Its source was verified through an anonymous public checkout; no operator or
|
||||
provider credential is sent to the build. Later test-only changes require a
|
||||
canonical image-input comparison, not a rewritten image provenance claim.
|
||||
|
||||
The foundation PR #18028 at `08a5516b` and product PR #18029 at `6123a67a`
|
||||
had green CI at those heads. The product review was Greptile 5/5 with one
|
||||
actionable report configuration load-order comment. That comment is addressed
|
||||
in the subsequent harness repair; fresh-head CI/review remain required. Both
|
||||
PRs remain draft because real acceptance is incomplete. Full typecheck and build passed. The
|
||||
local full test command failed three AI-connection cases after a first 15-second
|
||||
timeout, then accumulated an idle-transaction/TRUNCATE wait chain in the unchanged
|
||||
Git-expiration suite. The three connection cases passed separately without
|
||||
source changes. The full command was interrupted after recording the lock chain;
|
||||
all owned processes and its database port were independently verified stopped.
|
||||
This is not a full local test pass or an established explanation of every failure.
|
||||
|
||||
Recovery and re-evidence on 2026-10-29:
|
||||
|
||||
- The previous checkout and its worktree registration disappeared. The cause is
|
||||
unknown. Pushed source was restored into a newly attached managed worktree;
|
||||
frozen dependency installation and the full build passed again. Historical
|
||||
local pass logs and all ten audited result hashes survived, but the original
|
||||
raw report packages did not. Those passes are not reconstructed from logs or
|
||||
synthetic calibration evidence.
|
||||
- Commit `6123a67a` adds an explicit absolute report-root override with unchanged
|
||||
default storage, physical output containment, and traversal/symlink rejection.
|
||||
Actual final publication was checked across checkout removal, early failure,
|
||||
redaction, and missing mandatory evidence. The repair passed 123 focused
|
||||
evidence tests, seven sentinel tests, harness typecheck, actual CLI admission
|
||||
checks, and independent outside-sentinel preservation. These are harness
|
||||
controls, not provider acceptance.
|
||||
- All ten fresh local profile9 cells passed at exact `6123a67a`, including final
|
||||
packaging and cleanup. Their original packages now live outside the checkout.
|
||||
Independent collection rehashed all 221 files without mismatch; each owned
|
||||
server and embedded database stopped and each temporary data root was absent.
|
||||
The 19 retained run records are not a model-call count.
|
||||
- An independent temporary Cloud TEST credential reached the global Kimi endpoint
|
||||
and the exact installed Linux9 runtime passed native startup. The first
|
||||
controller attempt then failed before any test body: a Promise resolver
|
||||
shadowed the path resolver during event validation. The original failed and
|
||||
cleanup-failed grade remains. Credential-free actual-event replay reproduced
|
||||
the error and verified the correction. Empty JUnit suites, retained Playwright
|
||||
events, and source readiness gates establish that product fixture allocation
|
||||
and provider admission were never reached. Owned processes/ports were absent;
|
||||
the independent preflight sandbox was deleted and checked404, and the temporary
|
||||
credential was revoked and checked absent.
|
||||
- The corrected controller reached the actual first Daytona cell. That attempt
|
||||
failed closed at `remote_native_fixture:lease_metadata`: its fixture selected a
|
||||
reusable warm environment, while the native lifecycle observer requires an
|
||||
isolated non-reusable lease. The observer guard is retained. Both observed
|
||||
owned sandboxes were separately deleted and checked404; its temporary credential
|
||||
was revoked and checked absent. The original failed and cleanup-failed grade
|
||||
remains; model usage is unavailable. A narrow fixture-producer correction and
|
||||
fresh Daytona acceptance are required before qualification.
|
||||
|
||||
The follow-up harness repair at `d5fc5c42` changes only the two remote native
|
||||
lifecycle fixture configurations to isolated non-reusable leases. The eight
|
||||
other Daytona cases remain warm and all ten local execution inputs are
|
||||
unchanged. This per-case override is included in the suite fingerprint. The real
|
||||
fixture-producer regression first failed both lifecycle cases, then passed with
|
||||
three unchanged controls. The strict remote observer is unchanged. The same
|
||||
repair loads the normal campaign report setting from its local environment file
|
||||
after admission, with a credential-free actual-launch red/green regression;
|
||||
listing and installed-startup admission retain their prior behavior. Focused
|
||||
catalog/producer/observer/storage checks passed 227 tests with one existing skip,
|
||||
and harness typecheck passed. Native profile9, vendor closures and image inputs
|
||||
are unchanged; retain the fresh local evidence at `6123a67a` and pin a fresh
|
||||
Daytona campaign to the repaired source rather than relabeling old attempts.
|
||||
|
||||
Additional profile9 Daytona first divergences and shipping reconciliation:
|
||||
|
||||
- Restoring source did not restore its ignored Linux companion manifest/binary.
|
||||
The next run failed closed before launch. Exact image9 assets were recovered
|
||||
under a Linux-only release descriptor with their original source `5f4c9edc`;
|
||||
no other platform binary was invented or relabeled. Actual source, compiled
|
||||
Runner and compiled server-vendor readers passed strict manifest/profile/
|
||||
closure/binary checks, while the Mac daemon remained unchanged. This proves
|
||||
the scoped controller asset recovery, not a complete four-platform release.
|
||||
- After those assets were restored, a real Kimi turn requested native Read for
|
||||
the bootstrap file. Its canonical request had no proven file target and was
|
||||
never approved. The lifecycle case timed out and failed session recovery;
|
||||
the original failed/cleanup-failed verdict is retained. All observed owned
|
||||
sandboxes and temporary credentials were separately retired and verified.
|
||||
- A bounded real pinned-native control established that approve-reads delegates
|
||||
these provider requests; it does not automatically approve native Read.
|
||||
The proposed replacement uses server-owned get_task_context, whose empty-input
|
||||
schema and active-task authority are independently checked, with one explicit
|
||||
fixture-human approval after observer baseline and continuation publication.
|
||||
Unknown native Read is still unapproved and unqualified. Native Write must
|
||||
remain genuinely pending before Stop/loss. Synthetic proof rows alone do not
|
||||
establish delivery: production boundary calibration identified semantic input
|
||||
in the durable checkpoint and a delivered result command, rather than a
|
||||
semantic result event in the public run log. The checkpoint observer and full
|
||||
fresh runtime acceptance remain gates before using this replacement.
|
||||
- The master merge preserved sealed private-handoff custody ahead of ordinary
|
||||
report-root configuration, including after dotenv. Actual private/public
|
||||
launch regressions passed 57 tests and a wrong-root mutation failed all four
|
||||
relevant cases. A missing upstream tar-stream lock importer was repaired by
|
||||
the canonical pinned package manager: exactly thirteen added lines, with no
|
||||
unrelated resolution churn. Frozen installation and Runner TypeScript build
|
||||
passed. Combined harness checks passed 325 tests with one existing skip, plus
|
||||
137 Node tests and six Rust notification controls with one existing opt-in
|
||||
ignore. Full repository checks and current-head review remain release gates.
|
||||
- All ten fresh local9 report packages remain outside the checkout, and the
|
||||
explicitly approved 14.8 MB private backup was verified by full checksum,
|
||||
byte count and encryption read-back. Future raw payloads have no implied
|
||||
upload authorization from that one archive approval.
|
||||
|
||||
Material local attempts remain distinct:
|
||||
|
||||
- The first profile9 Stop attempt failed its outdated final UI oracle. Inspection
|
||||
also found duplicated partial assistant text; the provider-neutral consumer now
|
||||
respects the existing nested progress marker. Its retained event regression
|
||||
demonstrated red then green.
|
||||
- A later Stop passed all behavior/cleanup checks but failed report packaging:
|
||||
custom screenshot filenames did not satisfy the existing `final-state.png`
|
||||
contract. Two capture names were corrected, with positive and missing-final
|
||||
packager checks. The original grade remains failed.
|
||||
- One replacement failed before provider admission because two of its three
|
||||
credential-home quorum ports belonged to unrelated listeners dating from
|
||||
October 4 and 21. The guard correctly refused ownership. Foreign processes
|
||||
were untouched, and separate scoped cleanup verified our processes retired.
|
||||
- The next local Stop passed behavior, report evidence and physical cleanup in
|
||||
33.8 seconds. The following provider-loss attempt reached an unanswered Write
|
||||
but refused fault injection: macOS `ps` renders loader bytes differently from
|
||||
the pinned argv. An unpaid real guarded-native probe reproduced that exact
|
||||
hash divergence. The corrected observer reads argc-bounded argv with those exact byte and
|
||||
ownership checks retained; the subsequent real loss cell passed.
|
||||
|
||||
Profile9 distribution proof now covers eighteen actual public-layout tarballs,
|
||||
a clean scripts-disabled consumer, fresh installed CLI setup through the admitted
|
||||
Node/npm proxy, network-denied reuse, unchanged lockfile and installed native ACP
|
||||
create/load/model/EOF. No developer login home, refresh token, private database
|
||||
override or model prompt was used for that packaging proof.
|
||||
|
||||
The native Linux image was built from frozen source `5f4c9edc`; later source
|
||||
changes affect UI and test observers, with image inputs unchanged. An initial
|
||||
export timed out after image construction and saved no smoke proof. The corrected
|
||||
job `14af59a4-93be-4f83-990e-e308625ab1cc` completed in 9m40s with timestamped
|
||||
stages, independently saved native smoke proof and a streaming compressed export.
|
||||
Image ID is `sha256:139869415cdd7c7cd2a59f21626b8bdbbe0088f17209abf8cdda4f6a80b99718`;
|
||||
content identity is `4560ebf62a9640e43aff893b1a6dd497746eeb2787732818b62acd911cdb0b7b`.
|
||||
Create/load instruction acknowledgments and bounded EOF passed without prompts or
|
||||
credentials. Local collection verified the full archive, source, profile,
|
||||
closure, pack and daemon hashes. Its published immutable reference is
|
||||
`public.ecr.aws/q2k8x0v0/paperclip/kimi-qualification-eed3@sha256:bad9994dadf1a55743dabed3cd3f785210131938d4407c8b0ca92a30a922c177`.
|
||||
A real credential-free Daytona import/native smoke passed in 122.4 seconds on
|
||||
Linux x64 with 4 CPU, 4 GiB memory and 10 GiB disk, then its exact sandbox was
|
||||
deleted and checked404. It verified native create/load/model/instruction/EOF
|
||||
boundaries without a model prompt. No authenticated Daytona task has passed. All
|
||||
build failures are retained; the rejected IAM expansion
|
||||
was never applied and no operator/provider token was sent to CodeBuild.
|
||||
|
||||
The user additionally requested hosted verification and confirmed an isolated
|
||||
workspace at `https://kimi.test.paperclip.app`. It was created through the normal
|
||||
Cloud UI under the existing authorized owner; the shared QA identity lease was
|
||||
left untouched. It is active and empty on verified Core
|
||||
`f8484a9f8d1f68e9534fee4f29e00e716dc680f4`, distribution `cloud-40777791731-1`,
|
||||
not the Kimi build. A singleton TEST campaign enabled Native Runner on only this
|
||||
workspace with matching applied/desired hashes; fleet defaults stayed unchanged.
|
||||
Use the maintained Core preview deployment for this unmerged branch, record its
|
||||
exact serving SHA/image, and qualify the Kimi user journey there. This does not
|
||||
certify the private Cloud UI composition. No production or fleet-default change
|
||||
is in scope.
|
||||
|
||||
The active remote blocker is now narrowed by independent controls. The same fresh
|
||||
selected credential verifies through the host catalog path, while the actual
|
||||
remote metadata helper returns `KIMI_PROVIDER_ERROR`. Installed helper/Node bytes,
|
||||
profile admission and key/model forwarding match the expected boundary. A fixed
|
||||
remote GET failed with `ECONNRESET` before HTTP, with sandbox blocking false and
|
||||
no allowlists. Credential-free controls also reset for example.com, including
|
||||
direct IPv4 curl, with no proxy variables present. All three diagnostic sandboxes
|
||||
were destroyed and checked404. No TLS protection was weakened, endpoint switched
|
||||
or inference attempted during those diagnostics.
|
||||
|
||||
[Daytona's current network policy](https://www.daytona.io/docs/en/network-limits/)
|
||||
says Tier1/2 organization restrictions override sandbox settings. Its essential
|
||||
service list includes `*.kimi.com` and `*.moonshot.ai`, but not the selected
|
||||
account's official global `api.kimi.ai` endpoint. This is consistent with the
|
||||
observations, not a confirmed cause: this key cannot read its organization policy
|
||||
(HTTP401). It advertises only the public US target. The user was asked to confirm
|
||||
full egress or another suitable test credential, while the existing TEST Cloud
|
||||
provider setup is being checked. The global account must not be routed to the
|
||||
separate CN endpoint as a workaround.
|
||||
|
||||
Critical path: establish legitimate Daytona egress to the global Kimi endpoint,
|
||||
then run ten authenticated Daytona cells. Only after native qualification should
|
||||
a new immutable profile promote ordinary admission/default packaging, with final
|
||||
installed artifact/image checks and ordinary setup/assigned skills. Deploy the
|
||||
exact candidate to the isolated test workspace, complete its hosted journey, and
|
||||
finish the PR stack. A promotion patch has been prepared and tested only in TEMP;
|
||||
it is not applied and does not count as qualification.
|
||||
|
||||
## Evidence and first divergences
|
||||
|
||||
- Frozen root dependency installation passed with the normal network-enabled
|
||||
package manager. Restricted-network signature fetch failures were retained;
|
||||
no signature gate was disabled.
|
||||
- Runner TypeScript/Rust typecheck passed before final sidecar registration edits.
|
||||
Focused Runner contracts: 297 tests passed. Product setup/admission, token,
|
||||
catalog, and route checks have separate passing results; these are not live proof.
|
||||
- Both locked runtime closures built in 8.4 seconds with 33 required npm packages.
|
||||
macOS arm64 installed native metadata and Linux x64 metadata under emulation
|
||||
passed create/model/load/EOF; Linux emulation is not Daytona qualification.
|
||||
- Real bundled ACPX manager admitted exact `kimi-for-coding` and current bound
|
||||
instructions with physical cleanup. The first two attempted turns failed before
|
||||
semantic tool delivery. The second retained the native 403: account subscription
|
||||
lacked Kimi Code entitlement. Native CLI login succeeded, and the user then
|
||||
subscribed. Those failures remain failures; no cost counters were available.
|
||||
- The first entitled turn completed native execution and called the exact MCP
|
||||
echo with the instruction-only nonce. Its smoke observer incorrectly expected
|
||||
`paperclip_echo` at a handler that receives canonical `echo`. Native arguments,
|
||||
MCP receiver error, and terminal completion establish a grading/observer failure,
|
||||
not a successful tool round trip. The corrected observer then passed a real
|
||||
authenticated semantic echo, native terminal completion, and physical cleanup.
|
||||
Its profile2 evidence is retained separately from the final candidate.
|
||||
- Review found and corrected a real normal-path gap: the packaged sidecar's
|
||||
handwritten agent parser omitted Kimi even though direct driver metadata passed.
|
||||
Final sidecar checks and real Rust/sidecar product acceptance remain required.
|
||||
- Native controlled-model probes use the actual pinned Kimi and bundled ACPX.
|
||||
They found upstream approval construction omitted `toolInput`; the owned patch
|
||||
now carries exact nested and UTF-8 arguments, including empty objects. Denied
|
||||
Write caused no file effect, and native HTTP418 remained a typed failed turn.
|
||||
- The next probe found provider credentials inherited into native Bash. The
|
||||
patch filters `KIMI_API_KEY` and private instruction transport from native
|
||||
subprocesses while preserving explicitly assigned task variables. Actual ACPX
|
||||
always advertises terminal capability, which routed Bash through its own host;
|
||||
the final provider patch forces the owned filtered native process path instead.
|
||||
Final verification must observe zero `terminal/create` requests and no secret
|
||||
in the actual child, rather than merely inspecting a filtered object.
|
||||
- Bundling exposed a `Function.toString()` source difference that changed patched
|
||||
native bytes. The patch now embeds one canonical static helper source; raw and
|
||||
archived CJS setup produce the same pinned output. Final source comment removal
|
||||
changes its byte identity again before profile4 freeze; old evidence keeps its
|
||||
original closure and profile identity.
|
||||
- Required repository checks remaining: final full typecheck, supported full test
|
||||
run with the original infrastructure failure retained, full build, and the
|
||||
ordinary setup/save/reopen/Test/run/reconnect path. Token gates and map structure
|
||||
currently pass. Setup Test now uses the official authenticated model catalog;
|
||||
it does not claim inference, quota, or task execution from a metadata response.
|
||||
|
||||
The full repository build passed on candidate profile5. Its first Product E2E
|
||||
message-marker attempt reached real UI/server/database setup, then stayed paused:
|
||||
the installed-runtime-only Test warning is rejected by the ordinary agent lifecycle.
|
||||
No model turn started. This candidate failure is preserved separately from the
|
||||
retry. The correction performs a fixed-endpoint authenticated model-catalog probe
|
||||
before reporting a successful Test. A real catalog request accepted the credential
|
||||
and returned exact `kimi-for-coding`; the official `KimiCLI/2.3.3` client header
|
||||
is required (a generic client was rejected with HTTP403). The selected ID is never
|
||||
substituted. The first release also disables native question and plan tools so
|
||||
only the promised semantic interaction path remains available.
|
||||
|
||||
Historical profile8 command digest:
|
||||
`sha256:53c63af8722894e14d48ed55526f99fcd79e06c86ae1c7aa384ef7ee8c56efd4`.
|
||||
Native runtime patch v6 SHA256:
|
||||
`79980aa95c54f325a05cbdf883f74443705db28f3e412a1b5ae22929d939eb66`.
|
||||
Closure pins: macOS arm64 `6038325abd0431c30097ff1f572a9b29816e1b4da372d8cb86b2ca5d18ea34e7`,
|
||||
Linux x64 `105d8394adbbc76a953a12cc51b5288eb940ad693291e584f6cdc6407213164d`.
|
||||
The actual native Kimi plus bundled ACPX controlled-service probes passed seven
|
||||
requests for instructions/arguments/auth fencing/denial/failure, then nine for
|
||||
pending-permission Stop and spontaneous owned native-worker loss. Both pending
|
||||
callbacks aborted, stale approval caused no response/write/new model request,
|
||||
and workers physically retired within five seconds. The model service was simulated;
|
||||
these results do not establish authenticated product acceptance.
|
||||
|
||||
To retain native macOS Runner acceptance without changing shared IPC state, an
|
||||
official checksummed PostgreSQL 18.6 build with named POSIX semaphores passed fresh
|
||||
initialization, normal migrations, SQL empty-database checks, and cleanup. Only
|
||||
this worktree's unshared native dependency entry was temporarily pointed at that
|
||||
private build. It was restored before canonical public-package staging. The
|
||||
accepted database process/version must be disclosed with each affected result;
|
||||
no database fixture contract or tracked dependency file was changed.
|
||||
|
||||
Early campaign cost record: subscription model calls have no reliable native
|
||||
usage counters; cost is unknown. At that point one cloud job was stopped before
|
||||
building and no image or Daytona sandbox had been created. Original
|
||||
403s, the grading defect, credential-boundary failures, and database setup failure
|
||||
remain separate failed or blocked attempts; none are overwritten by retries.
|
||||
|
||||
The Code endpoint for this finite release is the global coding API
|
||||
`https://api.kimi.ai/coding/v1`, matching the successfully restored account.
|
||||
Mainland-region routing and managed OAuth refresh are outside this release.
|
||||
Authenticated qualification binds only the existing login's access token to the
|
||||
isolated run; refresh tokens and the ambient login home are not transferred.
|
||||
|
||||
The second local message-marker attempt reached the real model and tools. Run
|
||||
`4d394aef-0e6a-49a8-9fb0-873651d0241f` finished in 24 seconds, accepted the finish
|
||||
contract, saved task Done, and passed physical cleanup. Its original verdict is
|
||||
failed: progress text and final text shared one provider item, so the visible
|
||||
final bubble contained both. Native patch v5 now forwards actual turn-step UUID
|
||||
boundaries. The Kimi projection validates their provenance and order before
|
||||
passing generic assistant boundaries to Runner. It does not infer boundaries
|
||||
from text, tools, or HTTP response identifiers. The actual native Kimi/ACPX
|
||||
controlled peer passed this framing case alongside instructions, tool arguments,
|
||||
credential isolation, denial, typed failure, Stop, and native-worker loss.
|
||||
|
||||
Historical profile7 verification: 162 sidecar/projection/metadata/instruction tests,
|
||||
144 driver/projection tests, and 18 Rust provider-state tests passed. TypeScript,
|
||||
Rust daemon staging, bundled entrypoints, and all three affected immutable
|
||||
profile checks passed. The archived installer passed fresh public downloads,
|
||||
offline reuse, tamper rejection and cancellation on profile7. Final installed
|
||||
CLI/server tarballs remain to be rebuilt and checked. The full repository test
|
||||
run continues against the owned migrated PostgreSQL 18.6 Docker database; the
|
||||
earlier host semaphore and unmigrated-database setup failures remain retained.
|
||||
|
||||
The third local message-marker attempt passed all six independent matchers,
|
||||
accepted finish, durable Done, and cleanup on profile7/source
|
||||
`5fe7d496e1603626ceb5838f37c4fecce23580ee`. Campaign
|
||||
`local-2026-10-28T14-24-09-928Z`, run
|
||||
`68705f91-d195-4532-9fd4-e8b1826d6ed9`, completed in 45.1 seconds including
|
||||
cleanup and evidence collection. Its rendered final reply is the exact marker.
|
||||
The profile7 installed CLI/server tarballs also passed fresh public setup,
|
||||
offline reuse, exact native create/model/load/instruction receipts and EOF.
|
||||
|
||||
Lifecycle authoring then found that native Kimi approval omitted its tool kind.
|
||||
Patch v6 carries the native tool's existing canonical kind. The actual native
|
||||
callback now passes shared normalization: exact Write target with Allow once,
|
||||
Deny and Cancel; missing/foreign workspace contexts offer only Deny/Cancel and
|
||||
reject acceptance. The complete nine-request controlled native peer passed.
|
||||
The engine metadata correction and v6 change advance the installed closures.
|
||||
Profile7 evidence retains its original identity; the affected permission and
|
||||
installed-runtime paths require profile8 evidence before qualification.
|
||||
|
||||
CI found stale dependent provider attestations and fixture expectations. Pi31
|
||||
attests the changed shared environment/sandbox sources without changing its native
|
||||
closure. Copilot73 attests historical-profile decoding changes; Grok11 remains
|
||||
current. Old fixtures stay immutable. Ninety-five Pi/Copilot installation tests
|
||||
passed with one existing opt-in skip, plus 18 image contract tests and four Grok
|
||||
identity-generator tests. Kimi lifecycle authoring passed 44 deterministic cases;
|
||||
the exact Linux pidfd selector passed 14 real cases on native ARM64 Linux and 110
|
||||
TypeScript observer/contract tests with one existing skip. Linux x64 emulation
|
||||
returned ENOSYS for pidfd and is retained as an unsuccessful infrastructure
|
||||
attempt; only actual Daytona Linux x64 can qualify that target.
|
||||
|
||||
The rebased profile7 full typecheck/build passed and profile8 full build passed
|
||||
before the final dependent-provider identity refresh. The local full-test run
|
||||
was interrupted after unrelated shared-database cleanup failures and stale
|
||||
migration module errors. A fresh current-build isolated PostgreSQL migration
|
||||
probe passed. A replacement full run and final checks remain required. Later
|
||||
focused HTTP tests hit import/hook timeouts while host load was 63/120 with 92%
|
||||
CPU use; their original failures remain retained, without committed deadline
|
||||
increases. CI for the corrected source will provide an independent Linux check.
|
||||
|
||||
Review branches are published as draft PRs [Runner foundation #18028](https://github.com/paperclipai/paperclip/pull/18028)
|
||||
and [product setup #18029](https://github.com/paperclipai/paperclip/pull/18029).
|
||||
The first foundation review was 5/5 with no comments on its exact prior head;
|
||||
the corrected final head requires a fresh review. Neither PR is ready for merge
|
||||
until required product acceptance and checks pass.
|
||||
|
||||
### 2026-10-30: native behavior and cleanup recovery accepted
|
||||
|
||||
The profile 16 candidate (`e027eb4e…`, SDK 2.3.3, native artifact source
|
||||
`603756c22b2e39de9b5b5c3c26a0eb23c67604a1`) has evidence for ten distinct
|
||||
local behaviors and ten distinct Daytona behaviors. These are functional
|
||||
acceptances, not twenty originally green attempts: local Ask retains its first
|
||||
failure; the Daytona permission case retains an accounting-observer failure;
|
||||
the final provider-loss case retains its original cleanup-failure grade.
|
||||
Fixture/Core sources for the remote evidence are `8bc5ce3b…` and `e7f65045…`;
|
||||
the fixture-only rebuild changed only build-info among 10,993 compiled files.
|
||||
|
||||
The final Daytona case delivered loss to the exact owned native child while
|
||||
Write approval was pending. Its six matchers passed, including no file effect,
|
||||
expiry of the original callback, rejection of stale Allow, one failed native
|
||||
run, blocked task state, and complete native process retirement. Its workspace
|
||||
recovery completed and confirmed stop, then deliberately queued sandbox
|
||||
destruction for five minutes later. The test's 120-second environment-delete
|
||||
wait ended before that ordinary cleanup deadline.
|
||||
|
||||
A separate recovery restored the same captured control-plane state to an
|
||||
isolated working copy and used the normal server, company-secret rotation,
|
||||
cleanup scheduler, and environment DELETE route. Cleanup became successful,
|
||||
DELETE returned its documented HTTP 200, and the environment was absent from
|
||||
the database. No model turn, new run, new lease, or direct domain SQL update was
|
||||
used for recovery. Both the original and recovery processes, ports, provider
|
||||
resources, and temporary keys were closed. Root's combined acceptance record
|
||||
is SHA256 `25d0eae952eab91c6d54abb6850d4abf1b69f1045e7d7bb05333f8b87c184c0a`;
|
||||
it preserves the original failed grades and distinguishes native proof from
|
||||
later normal cleanup recovery.
|
||||
|
||||
The readiness correction waits for the same agent before changing permission
|
||||
policy, avoiding invalidation of an in-flight verification. Setup and native
|
||||
work now have separate finite windows, and remote deadline diagnostics identify
|
||||
the operation without retaining SDK text. Canonical focused verification passed
|
||||
223 tests with one existing Linux-only skip on Mac, plus the full fixture
|
||||
TypeScript check. Native runtime bytes and model/callback limits were unchanged.
|
||||
Profile 17 public-package installation and the actual hosted walkthrough on
|
||||
`kimi.test.paperclip.app` remain separate delivery gates; this section does not
|
||||
claim either has passed.
|
||||
|
||||
The cleanup fixture now uses one absolute Kimi deadline at the original test
|
||||
start plus 660 seconds, within the unchanged 690-second total limit. It leaves
|
||||
30 seconds for final evidence. Each DELETE request and retry sleep uses only
|
||||
the remaining time. Other flows retain their 120-second default. The actual
|
||||
canonical fixture tests passed 73 assertions across three files, including
|
||||
ordinary deferred cleanup beyond 120 seconds, deadline exhaustion, request
|
||||
timeout forwarding, and HTTP success versus pending cleanup. This source
|
||||
correction does not regrade the preserved failed live attempt.
|
||||
@@ -0,0 +1,583 @@
|
||||
# Native-controller recovery during webserver upgrades
|
||||
|
||||
Date: 2026-10-30. Status: implementation delivered; final qualification is tracked
|
||||
in [PR #18343](https://github.com/paperclipai/paperclip/pull/18343).
|
||||
Source inspected: `0c70cedce9a803bdc201324528abb35e423e7cd0` in the current task worktree.
|
||||
|
||||
## Goal and finish line
|
||||
|
||||
Make native controller state independent of the main Paperclip webserver's disk,
|
||||
as one prerequisite for zero-downtime webserver deployments. A person starts a
|
||||
task, the webserver is replaced, and the same remote execution continues. After
|
||||
completion, another replacement can continue the retained session without the
|
||||
previous webserver's files. Recovery must preserve Stop, current authorization,
|
||||
accepted results, pending tool decisions, and single-owner execution.
|
||||
|
||||
The implementation endpoint is a tested, reviewable PR for the native-controller
|
||||
portion only. General HTTP readiness, traffic switching, request draining,
|
||||
browser delivery, storage migration and Railway rollout belong to other chats.
|
||||
Native shutdown integrates with the existing server lifecycle coordinator.
|
||||
|
||||
Initial supported environment: native remote Runner execution, external Postgres,
|
||||
a stable externally supplied instance encryption key, surviving remote sandbox
|
||||
storage, and mutually compatible server/Runner releases. Qualify the existing
|
||||
Daytona path, including its prepared-task transport. Preserve local/self-hosted
|
||||
behavior. No new database, queue service, or shared filesystem is needed.
|
||||
|
||||
Keep the existing outage contract: a 300-second controller outage, with recovery
|
||||
completed within the Runner's existing 420-second reconnect grace. The remote
|
||||
controller lease is currently 60 seconds. The 300-second fault-tolerance test is
|
||||
not the permitted downtime of the webserver deployment. Do not shorten it or renew the Runner's
|
||||
disconnected lifetime on each retry to make a test pass. This contract permits
|
||||
server-dependent tools to wait; it does not promise uninterrupted HTTP service.
|
||||
|
||||
## What already exists
|
||||
|
||||
The earlier [Explore zero-downtime Railway deploy](codex://threads/01a1836e-8c8e-7b83-b09c-5db49a5e89de)
|
||||
plan included remotely persisted native controller checkpoints. Much of that work
|
||||
has since landed. Build on it instead of introducing another checkpoint store.
|
||||
|
||||
| Existing capability | Owner / evidence inspected |
|
||||
| --- | --- |
|
||||
| Encrypted, company/task/run-bound PRP checkpoints in Postgres | [`native-runner-checkpoint.ts`](../../server/src/services/native-runtime/native-runner-checkpoint.ts), [`native_runner_checkpoints`](../../packages/db/src/schema/native_runner_checkpoints.ts); landed in `09ad9adce2` / PR #17439 |
|
||||
| Opaque checkpoint codec, restore, authenticated replay cursor and persistence-before-effect barriers | [`durable-prp-control-plane.ts`](../../packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts) |
|
||||
| Expiring ownership, controller generations, stale-writer fencing and isolated renewal/checkpoint pools | [`native-restart-recovery.ts`](../../server/src/services/native-runtime/native-restart-recovery.ts), [`native-execution-lease.ts`](../../server/src/services/native-runtime/native-execution-lease.ts) |
|
||||
| Active remote reattachment from DB state and the admitted Runner artifact | [`native-session-executor.ts`](../../server/src/services/native-runtime/native-session-executor.ts), `executePaperclipNativeSessionWithinScope` |
|
||||
| Bounded database discovery of prior task/session checkpoints | [`native-session-resume.ts`](../../server/src/services/native-runtime/native-session-resume.ts) |
|
||||
| Empty-controller-directory, receipt-recovery, concurrency, retention and SQL-cost tests | [`native-runner-restart-recovery.integration.test.ts`](../../server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts), [continuity runbook](../testing-native-remote-continuity.md) |
|
||||
| Read-only active-run upgrade eligibility | [`heartbeat/upgrade-readiness.ts`](../../server/src/services/heartbeat/upgrade-readiness.ts); landed in `567c043b17` / PR #18037 |
|
||||
|
||||
The related [Enable uninterrupted agent upgrades](codex://threads/01a15e84-545f-7023-a854-9a642cbc76e1)
|
||||
chat reports actual PAP staging replacements preserving active executions. It is
|
||||
also working on broader default enablement. Those are reported results from that
|
||||
chat, not a new qualification of this checkout or proof that the entire webserver
|
||||
filesystem can be discarded. Reconcile that work before implementing overlapping
|
||||
changes; do not duplicate its rollout or configuration work.
|
||||
|
||||
## Remaining dependencies and evidence gaps
|
||||
|
||||
1. **Ordinary continuation still discovers PRP state through local identity.**
|
||||
`restoreWarmRunnerdController` and `restorePerTurnRunnerdController` first read
|
||||
the local full state or external marker. They return when neither supplies a
|
||||
matching identity. Active restart recovery instead starts from a DB run
|
||||
binding. Extend that approach to ordinary continuation without skipping the
|
||||
existing prior-owner, session, configuration, and suspended-state checks.
|
||||
2. **Warm supervisor state has its own local persistence path.**
|
||||
`persistWarmNativeCheckpoint` writes `paperclip.native-session-supervisor.v1`
|
||||
with the snapshot, configuration digest, and process-configuration digest;
|
||||
`loadWarmNativeCheckpoint` and warm process reuse read that file. The separate
|
||||
Postgres PRP restore writes only `control-plane-state.json`. Existing saved
|
||||
`runnerProfileJson.sessionCheckpoint` and admitted execution input provide
|
||||
reusable data, but the complete warm reuse decision must become reconstructible.
|
||||
3. **Current focused tests do not establish the whole application contract.**
|
||||
The empty-directory transport fixture restores explicitly and sets the old
|
||||
lease expired. It is valuable protocol evidence, but does not prove normal
|
||||
admission with no markers, natural ownership expiry, HTTP routing during
|
||||
overlap, or a second fresh-server continuation. Existing supervisor/Product
|
||||
E2E coverage may prove some of these; reconcile it before adding new cells.
|
||||
4. **Upgrade eligibility is narrower than disk independence.**
|
||||
The current report checks active runs and bounded PRP headers. It does not
|
||||
certify completed retained sessions or dereference workspace recovery files.
|
||||
An empty active-run list must not imply that all controller history is safe
|
||||
to discard.
|
||||
|
||||
These are inspected dependencies and missing proof, not newly reproduced
|
||||
production failures. Begin with characterization tests to determine which paths
|
||||
already reconstruct sufficient state through their callers.
|
||||
|
||||
## Design
|
||||
|
||||
### One durable source for each kind of state
|
||||
|
||||
| State | Durable owner | Local treatment |
|
||||
| --- | --- | --- |
|
||||
| PRP authentication, command receipts, control events and replay state | Existing encrypted `native_runner_checkpoints` | Reconstructed cache/marker |
|
||||
| Run authority and finalization | Existing `native_run_finalizations`, run and issue locks | No local substitute |
|
||||
| Provider/session snapshot and native reuse compatibility | Existing run profile plus a versioned native recovery envelope where fields are missing | Reconstructed supervisor projection |
|
||||
| Session lookup | Existing task-session pointer and bounded prior-run selection | Never depends on directory enumeration or a marker |
|
||||
| Provider process, provider history and unacknowledged outbox | Surviving remote Runner/sandbox | Never copied from an older controller cache over live state |
|
||||
| Workspace descriptors, manifests, archives and provider backup blobs | Their existing workspace/backup owners, with remote durability supplied by that workstream | Logical references; independent eligibility requirement |
|
||||
|
||||
Keep Runner's codec and opaque record contract in the Runner package. Native
|
||||
runtime owns session selection, compatibility, and the server-side recovery
|
||||
envelope. The database adapter owns encryption and transactional fencing.
|
||||
Heartbeat only coordinates these contracts. Extract the affected persistence and
|
||||
lookup responsibility from the large executor into a cohesive native-runtime
|
||||
module rather than expanding the executor or creating a generic recovery system.
|
||||
|
||||
### Reconstruct from durable identity
|
||||
|
||||
Use the exact active run for restart adoption. For a subsequent turn, use the
|
||||
existing task/session pointer and `native-session-resume.ts` selection policy to
|
||||
locate the eligible prior run. Validate company, task, agent, normalized session,
|
||||
workspace scope, admitted execution, provider configuration and current authority.
|
||||
Retain the existing progress barriers: a newer incompatible or incomplete record
|
||||
must not make recovery select an older convenient checkpoint.
|
||||
|
||||
Load and validate the PRP checkpoint and native snapshot before materializing
|
||||
local projections. The destination is computed from the replacement's local
|
||||
root; absolute paths recorded by the old controller never identify durable data.
|
||||
An existing local projection must agree with the selected committed state or be
|
||||
replaced under the same authority. Missing, corrupt, undecodable or mismatched
|
||||
state remains a specific recovery hold, without a fresh provider turn.
|
||||
|
||||
Default persistence design: reuse `runnerProfileJson.sessionCheckpoint` and add
|
||||
only a private, versioned native recovery envelope for missing compatibility and
|
||||
checkpoint-binding metadata. Persist the snapshot and its metadata together
|
||||
under the current run's fenced ownership. Do not duplicate the PRP journal or
|
||||
store server-specific fields as arbitrary Runner checkpoint parts. If existing
|
||||
profile metadata already supplies a field, reference it instead of copying it.
|
||||
Use a schema migration only if characterization demonstrates that the existing
|
||||
run/session records cannot express a required durable lookup or concurrency check.
|
||||
|
||||
The private `nativeControllerRecovery` envelope is limited to 4 KiB serialized.
|
||||
It contains a format version, digest of the exact redacted stored snapshot,
|
||||
native configuration digest, process-configuration digest or explicit absence,
|
||||
and PRP attachment/checkpoint binding. Run/company/task/agent/session identity is
|
||||
validated against existing records and the snapshot, not duplicated execution
|
||||
inputs. Keep credentials in the
|
||||
existing encrypted authority; persist logical resource identities, not secret
|
||||
values or host paths. Any newly sensitive recovery material uses the existing
|
||||
encryption provider and remains excluded from public run projections.
|
||||
|
||||
`PaperclipControlPlanePort.checkpointSession` already persists the snapshot before
|
||||
the local supervisor callback. Add the envelope to that same transaction and
|
||||
update; do not introduce a second snapshot writer. Compute stable compatibility
|
||||
inputs once per admission/attachment, preserve unrelated profile fields, and
|
||||
recheck durable ownership after lock waits with the established lock ordering.
|
||||
Keep legitimate post-result suspension/cleanup checkpoints permitted. The generic
|
||||
Runner persistence port must not learn Paperclip database fields. Remote local
|
||||
files become projections; local-only consumers retain authoritative files.
|
||||
|
||||
The recovery owner resolves, validates, materializes and attaches through existing
|
||||
owners. Outcomes distinguish recoverable, waiting for an owner with its actual
|
||||
deadline, missing/inconsistent evidence, unsupported format/capability, dependency
|
||||
unavailable and proven no-prior-execution. Missing evidence is not fresh bootstrap.
|
||||
Keep the 32-candidate selection window and newer-progress barriers; fetch the full
|
||||
payload only for the selected source. PRP attachment identity and current business
|
||||
run write authority differ during warm handoff and must remain explicit.
|
||||
|
||||
Publish a resumable snapshot only after its referenced PRP boundary is durable.
|
||||
Across separate commits, interruption may leave a pending obligation, never a
|
||||
false ready state. Recovery reconciles that exact obligation without rerunning
|
||||
the provider. Completion acceptance, workspace settlement, session reuse, and
|
||||
cleanup remain separate facts.
|
||||
|
||||
### Preserve ownership during replacement
|
||||
|
||||
The new server may load recovery data while the old server serves, but it cannot
|
||||
send commands, acknowledge events, or settle results until it owns the execution.
|
||||
Reuse the existing boot identity, generation and lease-owner fences. Cross-host
|
||||
takeover uses DB lease authority, never a local check of another container's PID.
|
||||
|
||||
Graceful shutdown fences new local controller work, completes outstanding durable
|
||||
barriers, detaches supported sessions, and releases only its own authority. A
|
||||
crash uses natural expiry. The successor takes the existing fenced claim,
|
||||
restores state, revalidates permissions and attaches to the exact remote process.
|
||||
Lost authority detaches the stale controller without killing the surviving Runner.
|
||||
|
||||
Retain wake/recovery obligations while the previous lease is live. Reconcile at
|
||||
startup and the actual lease/retry deadline through the shared scheduler. Cover
|
||||
a lost local release notification with durable state and the deadline. Add no
|
||||
permanent checkpoint scans or independent polling timers. A future shared
|
||||
cross-instance notification mechanism can accelerate this path without becoming
|
||||
its source of truth.
|
||||
|
||||
Exercise Stop and tool decisions arriving at either instance. Use existing durable
|
||||
intents and receipts; fix a demonstrated missing native delivery path in its
|
||||
owner. Never retransmit an uncertain external mutation just because its HTTP
|
||||
response was lost. Current access, pause, cancellation and budget controls still
|
||||
apply; restoring an admitted run is not a new grant of authority.
|
||||
|
||||
### Compatibility, adoption and retention
|
||||
|
||||
Use additive readers/writers. Before an ephemeral deployment is eligible, its
|
||||
rollback target must understand the recovery envelope, sparse checkpoint policy
|
||||
and authenticated cursor. A v2 decoder alone is insufficient. Preserve the
|
||||
original Runner artifact identity; an old running binary does not gain new
|
||||
capabilities merely because the webserver upgraded.
|
||||
|
||||
For existing file-backed sessions, allow the current legitimate owner to publish
|
||||
a validated durable snapshot at a normal checkpoint boundary. A bounded explicit
|
||||
backfill may import retained supervisor state while the disk is still available,
|
||||
after rechecking the same bindings and ownership. Never fabricate missing
|
||||
credentials, checkpoints or remote process evidence. Unprovable sessions remain
|
||||
ineligible until normal completion/retirement or explicit recovery.
|
||||
|
||||
Keep local mode intact. Require the same externally supplied encryption key on
|
||||
replacement; a recovery read must not generate a new key. Broad external-mode
|
||||
startup validation belongs to the parent persistence workstream.
|
||||
|
||||
Extend existing retention reference checks for any new envelope references.
|
||||
Retain active owners, unresolved handoffs/finalization and selected continuation
|
||||
sources. Cleanup must not race a new continuation claim or allow a late writer to
|
||||
resurrect deleted authority. Preserve bounded startup cleanup and zero idle SQL.
|
||||
|
||||
Expose an additive, versioned **native controller durability** diagnostic through
|
||||
the existing internal upgrade report, with explicit unknown/ineligible reasons.
|
||||
It must distinguish active adoption from retained-session continuation, and it
|
||||
must never claim whole-instance volume removal readiness. Keep the existing
|
||||
`ready` meaning compatible; the broader deploy coordinator composes this evidence
|
||||
with workspace and other persistence checks. Derive eligibility from committed
|
||||
records and supported readers, not a persisted boolean that can become stale.
|
||||
Report active adoption, selected retained continuation, outstanding persistence or
|
||||
handoff obligations and unsupported evidence separately. Include observation time,
|
||||
bounded reason counts and coverage/truncation. Use bounded batched metadata/header
|
||||
reads and pagination, and report incomplete inventory as unknown coverage. Add no
|
||||
public endpoint. Establish the first envelope-capable release as the rollback
|
||||
floor; an older local-only reader is not a valid empty-disk rollback target.
|
||||
|
||||
## PostgreSQL load contract
|
||||
|
||||
- Text streaming adds zero recovery SQL, transactions or checkpoint rows per chunk.
|
||||
- Ordinary usage remains coalesced into control checkpoints. Preserve changed-part
|
||||
PRP writes and exact removals; do not add a journal, transcript or archive copy.
|
||||
- Snapshot metadata rides the existing snapshot transaction, including unchanged
|
||||
metadata; no parallel writer, per-agent connection or new pool.
|
||||
- Idle servers add no recurring SQL/network work once obligations settle.
|
||||
- Scoped candidate selection fetches only the selected payload. Diagnostics use
|
||||
bounded headers/metadata, never an automatic scan of historical payloads.
|
||||
|
||||
Keep the existing dedicated checkpoint pool initially with its two-connection
|
||||
default, lazy creation, idle release and shutdown fencing. Two is not an
|
||||
architectural invariant. The invariant is bounded checkpoint traffic that meets
|
||||
recovery deadlines without starving requests or renewals. Include both servers'
|
||||
pools during overlap. Measure queue wait separately from SQL and lock wait. If
|
||||
capacity fails acceptance, compare bounded sizes under the same workload and
|
||||
choose the smallest passing size in a separately reviewable tuning change. Do not
|
||||
increase the cap to conceal slow queries or lock contention.
|
||||
|
||||
Run paired baseline/candidate workloads with identical fixtures and counts and
|
||||
all database clients included. Record statement/transaction classes, changed rows,
|
||||
persisted bytes, WAL, CPU, memory, connections, request latency, checkpoint queue/
|
||||
lock/execution latency, renewal latency/deadlines, throughput, retries and failures.
|
||||
Use the existing 8/512-chunk test plus a stream beyond the retained-event window;
|
||||
require exactly zero extra recovery SQL during text-only intervals. Repeat the
|
||||
documented 300-controller/10,000-historical-task steady and aligned-burst workload,
|
||||
alternating order and including actual session snapshot persistence.
|
||||
|
||||
Gates: unchanged lifecycle sequence has no additional normal checkpoint
|
||||
transactions; no new idle work, missed lease deadlines, lost owners, dropped
|
||||
operations or premature ACKs; existing absolute latency/throughput gates pass.
|
||||
Investigate any repeatable increase above 10% in steady CPU per completed operation,
|
||||
WAL per completed operation or request p99 before handoff; report absolute values
|
||||
too. Measure replacement bursts separately within the 420-second grace. Never
|
||||
relax persistence-before-effect guarantees to meet a performance target.
|
||||
|
||||
## Implementation slices
|
||||
|
||||
| Slice | Owned surfaces and useful result | Dependency / review evidence |
|
||||
| --- | --- | --- |
|
||||
| 1. Complete native recovery state | Native checkpoint adapter, executor persistence/lookup, session-resume owner and focused tests. Normal active, warm and per-turn paths reconstruct with no local marker or supervisor file. | Reuse landed checkpoint work; first reproduce the exact missing path. Test current local mode and incompatible-session barriers. Add only demonstrated missing envelope fields. |
|
||||
| 2. Replacement ownership and compatibility | Native lifecycle, shared recovery scheduling, shutdown wiring, retention and internal upgrade diagnostics. Competing servers hand off safely, and historical sessions have an honest eligibility path. | Slice 1; real Postgres ownership races, natural expiry, stale callbacks, Stop, migration/read compatibility and rollback tests. Keep heartbeat wiring small. |
|
||||
| 3. Application and hosted qualification | Existing restart supervisor/Product E2E suites, continuity docs and acceptance evidence. Two replacements preserve execution and later continuation. | Slices 1–2; workspace durability dependency for full-home cases. Real immutable A/B releases and compatible rollback; no synthetic claims of Daytona qualification. |
|
||||
|
||||
Tests and documentation travel with the behavior they verify. These are review
|
||||
boundaries, not a fixed PR count. Combine tightly coupled implementation where
|
||||
splitting would create an unusable intermediate state. Rebase onto the current
|
||||
default branch and reconcile the active upgrades chat before writing source.
|
||||
|
||||
## Acceptance matrix
|
||||
|
||||
| Scenario | Required result and independently inspectable evidence |
|
||||
| --- | --- |
|
||||
| Fresh active remote run; graceful replacement | Same run, Runner process/start identity, provider session and active turn; one provider start, one final result |
|
||||
| SIGKILL and absent controller storage | Normal startup claims after natural expiry; no manually edited lease, injected recovery marker or manual restore call |
|
||||
| Output-only gap and pending tool/decision | Authenticated cursor recovers; pending input settles once; independent business receipt/effect count confirms no duplicate mutation |
|
||||
| Completion while disconnected or during shutdown | Accepted result, accounting and finalization survive; incomplete persistence resumes without another provider turn |
|
||||
| Completed warm and per-turn sessions | B can discover the correct source from DB; a later C continues it after B's controller storage is removed; no new provider session solely because a marker was missing |
|
||||
| Warm config changes | Verified close/suspend and permitted saved-history resume; never incorrectly reuse changed credentials, tool contract or workspace |
|
||||
| Old and new server overlap | B respects A's live lease; only one successor wins; delayed A cannot checkpoint, ACK, dispatch or finalize after losing ownership |
|
||||
| Stop, revoked access, pause and cross-company attempts | Saved intent and current authorization prevail; no recovery bypass or delayed mutation |
|
||||
| Corrupt/missing checkpoint, key or unsupported version | Precise ineligibility/hold; no silent fallback, replay, key generation or destruction of the last recoverable copy |
|
||||
| Existing records and rollback | Valid local state can be imported under authority; unsupported state stays explicit; compatible prior release reads newly written records |
|
||||
| Cleanup and idle | Selected checkpoints survive cleanup races; settled/idle states issue no new recurring SQL or network work; next legitimate producer reactivates recovery |
|
||||
| Full application with empty homes | A → B → C uses the normal API/admission path and independently verifies the saved result; blocked until required workspace/backup dependencies are remotely durable |
|
||||
|
||||
## Feedback path and verification
|
||||
|
||||
First experiment: extend the existing real-Postgres remote continuation fixture
|
||||
with a completed warm turn, remove both PRP marker and supervisor directories,
|
||||
then submit the next turn through normal admission. Keep the remote Runner alive.
|
||||
Assert the prior run/session chosen, process identity, one new turn and a useful
|
||||
result. Also run the per-turn control. Do not prepopulate the desired recovery
|
||||
state. This distinguishes a lookup/envelope gap from missing workspace material.
|
||||
|
||||
Use the existing local Linux/container harness and scripted provider initially;
|
||||
it avoids paid model calls while preserving database locks and process lifetime.
|
||||
Measure setup separately from edit → run → inspected evidence. Start with a short
|
||||
outage; run the real 300-second outage after the narrow behavior passes. Record
|
||||
elapsed time and the exact revision in this plan when implementation begins.
|
||||
|
||||
Then exercise separate compatible application revisions with distinct native roots
|
||||
and shared Postgres, through normal startup/admission. Remove access to A's native
|
||||
controller storage. Identify surviving non-native dependencies explicitly: this
|
||||
is not evidence of whole-webserver filesystem loss. Test both SIGTERM and SIGKILL, B starting
|
||||
while A still owns the run, decisions/Stop during handoff, and a second B → C
|
||||
replacement. Retain an independent effects ledger and process identity evidence.
|
||||
If this reaches a workspace/backup dependency, keep that failure as explicit
|
||||
parent-workstream evidence; do not hide it with a copied controller directory.
|
||||
|
||||
Qualify one disposable Daytona environment using the existing Product E2E path,
|
||||
including completion and follow-up, plus a compatible rollback. Record server
|
||||
revisions, Runner artifacts, run/session/sandbox identities, lease timestamps,
|
||||
receipt counts, recovery latency and cleanup. Reuse an existing healthy campaign
|
||||
only after checking its owner and exact revision. No fleet rollout is part of
|
||||
this checkpoint feature.
|
||||
|
||||
Required handoff checks for implementation: focused affected suites, then
|
||||
`pnpm -r typecheck`, `pnpm test:run`, and `pnpm build`. If schema changes prove
|
||||
necessary, use `pnpm db:generate` and the migration/snapshot checks. Update
|
||||
`doc/execution-semantics.md`, `doc/testing-native-remote-continuity.md`, relevant
|
||||
developer/database documentation and affected capability maps. Run
|
||||
`node scripts/check-maps.mjs` after map edits. PRs use the complete repo template.
|
||||
Report skipped or unavailable checks without promoting older evidence to a pass.
|
||||
|
||||
## Parent-workstream dependencies and non-goals
|
||||
|
||||
Native checkpoint success alone does not permit removing the webserver volume.
|
||||
Workspace sync still resolves local descriptors/manifests in
|
||||
[`native-workspace-sync.ts`](../../server/src/services/native-runtime/native-workspace-sync.ts).
|
||||
That owner must provide durable logical recovery references and saved generations;
|
||||
this plan consumes that contract rather than moving workspace archives into PRP.
|
||||
Provider backup blobs and prepared resources needed after a process has suspended
|
||||
must likewise resolve without the old server disk.
|
||||
|
||||
Agent files/instructions, logs and accounting journals, imports, mutable packages,
|
||||
global background-work claims, browser live-event delivery, HTTP readiness/drain,
|
||||
and Cloud/Railway volume migration retain their own slices in the broader plan.
|
||||
Do not turn native checkpoints into a second transcript: the existing sparse
|
||||
checkpoint excludes display-only output, and this work does not strengthen the
|
||||
run log's durability guarantee. Destroyed sandbox recovery, legacy local stdio
|
||||
continuity and permanent horizontal scaling are outside this release.
|
||||
|
||||
Railway still prevents overlapping deployments on a service with an attached
|
||||
volume. Its separate overlap/drain controls become useful after all required
|
||||
volume dependencies and concurrent-server behavior are qualified. Sources checked
|
||||
2026-10-30: [volume caveats](https://docs.railway.com/volumes/reference#caveats),
|
||||
[deployment teardown](https://docs.railway.com/deployments/deployment-teardown).
|
||||
|
||||
## Current state and next action
|
||||
|
||||
The observations below retain earlier qualification attempts and their limits.
|
||||
The PR's Verification section is the current acceptance record. The final native
|
||||
runtime candidate is `eaef0d56552c087aaa4fa68a8655282ffbd12667`; documentation-only
|
||||
updates do not change those runtime bytes. Its CI and fresh 5/5 review passed.
|
||||
The corrected source-process test has completed active adoption and a fresh
|
||||
retained follow-up. Distinct compatible rollback, installed-application and live
|
||||
Daytona gates remain under verification as of 2026-10-30 20:28 UTC. Preserve failed
|
||||
attempts when adding later passes. The rollback floor must include the idle-goal
|
||||
reader fix; an envelope-only reader is insufficient.
|
||||
|
||||
Implementation is underway on `codex/native-controller-disk-independent-recovery`.
|
||||
The related Cloud policy work merged as paperclip-cloud PR #1048; it retains
|
||||
compatibility checks and drain fallback and does not implement this component.
|
||||
Verified locally: metadata/source-selection/lock-wait checks (21 tests), readiness
|
||||
and recovery scheduling (55 tests), durable discovery/import/inventory (7 tests),
|
||||
and the complete native executor suite. The combined executor/selection/retention
|
||||
run passed 1,063 tests and exposed two overbroad retention guards; those guards
|
||||
were corrected and all 75 retention cases then passed. The snapshot fence test
|
||||
also checks stale result acceptance and lease expiry during a run-row lock wait.
|
||||
|
||||
Both warm and per-turn fixtures passed three separate controller processes,
|
||||
deleting both native controller directories before B and again before C. They
|
||||
check provider history, Runner identity, invocation counts, and the existing
|
||||
suspension policy. Per-turn uses the production admission selector and snapshot
|
||||
rebinder. These processes use the same candidate source, not distinct release
|
||||
artifacts. Workspace and remote sandbox storage remain available. This is not
|
||||
full-webserver filesystem-loss evidence.
|
||||
|
||||
Both direct and prepared-task transport cases passed the 300-second outage with
|
||||
natural lease expiry. The 8/512/8,192-chunk SQL test passed: each sample used seven
|
||||
control transactions and twelve changed checkpoint records, with zero recovery
|
||||
SQL/transactions/rows during the text interval and zero idle SQL. The long sample
|
||||
included normal lease renewals. Its cluster-wide WAL values include simultaneous
|
||||
test migrations and are diagnostic only, not a load acceptance measurement.
|
||||
|
||||
Workspace typecheck, build, and map structural checks passed on the earlier
|
||||
candidate; the later check dispositions are recorded below. The paired
|
||||
300-controller/10,000-task workload now includes actual snapshot persistence.
|
||||
Its initial host pilot missed throughput gates; the subsequent isolated Linux
|
||||
measurements are recorded below. Database cost acceptance, distinct-release application startup,
|
||||
compatible rollback, and native Daytona qualification remain open. The published Daytona qualification image was independently resolved from successful
|
||||
workflow 40709784515 to `sha256:b38ed051d00fed409092d354eea9ae1b01fb232bcb7d5a187ed763382dda2d0b`
|
||||
(source `8fe5d5defcdfb55a1b9049a801cfa5498629cebc`). The exact Linux Runner was
|
||||
extracted from that image. The local attempt failed during the environment probe before agent creation. A read-only lookup independently returned `DaytonaAuthenticationError: Invalid credentials`. The private database and cleanup obligation remain preserved. The protected single-cell workflow [41093316372](https://github.com/paperclipai/paperclip/actions/runs/41093316372), targeting `512cc6d6dc`, was cancelled during image/artifact builds before provider allocation so acceptance can use the rebased candidate. Its protected secret is separate from the invalid local fixture.
|
||||
No production deployment or complete zero-downtime acceptance is claimed.
|
||||
|
||||
Failures retained in the implementation record: sandbox listener denial and host
|
||||
embedded-Postgres startup limits; a corrected missing `phase` in a new fixture;
|
||||
a corrected unexecuted clock query in result persistence; Linux fixture builds
|
||||
first failed on shared build storage and then missing protocol source files,
|
||||
resolved by a container-local build with the complete inputs. The 8,192-chunk
|
||||
case exposed a test-provider cap of 4,096 (expanded in the test binary), then
|
||||
exceeded the old short-stream wait limit. Its longer measurement now uses the
|
||||
normal 60-second lease and renewal rather than an artificially long owner lease.
|
||||
The long-stream renewal fixture initially omitted the controller PID/start identity;
|
||||
renewal correctly rejected it. Complete fixture identity fixed the test. Per-turn
|
||||
characterization then exposed the missing remote suspended-lifecycle projection;
|
||||
that is now verified against the bound surviving sandbox. The fixture also now
|
||||
uses the existing admission snapshot rebinder, which its warm-only predecessor
|
||||
omitted. The first benchmark attempts lacked ES-module setup and the observer's
|
||||
pg_stat_statements extension; the isolated Linux pilot needed its native Sharp
|
||||
dependency. No database-load acceptance claim has been made from failed attempts.
|
||||
There is no Paperclip issue bound to this local chat; this versioned source plan
|
||||
is the handoff, rather than an upload to an unrelated company or issue.
|
||||
|
||||
The remaining native qualification and compatibility cells require reproducible
|
||||
measurements and a reviewed PR. Full-home application
|
||||
acceptance remains dependent on workspace/backup durability; no controller-only
|
||||
test can waive that gate.
|
||||
|
||||
Qualification update: the legacy and changed-process-configuration warm cases
|
||||
exposed a marker left by suspension before authority rotation. Restoring the
|
||||
committed PRP checkpoint before constructing the replacement backend fixes both;
|
||||
both real-process cases pass. The initial rerun used the wrong fixture executable
|
||||
path and failed before provider startup; its log is retained. All 59 control-plane
|
||||
port tests pass after narrowing the snapshot read to the binding and profile
|
||||
columns it actually needs. The initial paired 300-controller ABBA comparison
|
||||
completed every offered operation with no ownership losses, but candidate latency
|
||||
and CPU increased above the investigation threshold. It is not performance
|
||||
acceptance. A new paired run will measure the narrower snapshot projection.
|
||||
|
||||
|
||||
Qualification update at `512cc6d6dc`: the snapshot port now combines coordinator
|
||||
and run locks into one statement, with a database-clock check after both waits.
|
||||
All 59 real-Postgres port tests pass. Snapshot and PRP transaction counts remain
|
||||
unchanged; the first completed 300-controller baseline/candidate pair each issued
|
||||
74,790 total statements and 7,800 control transactions across all pools. The
|
||||
candidate adds the metadata bytes but no SQL per streamed chunk.
|
||||
|
||||
The dedicated Daytona suite now fixes the active outage at 300 seconds and adds
|
||||
normal UI follow-up after a second replacement following completion. Its oracle
|
||||
requires a new successful native run, retained native/provider session identity,
|
||||
a completed tool receipt and correct saved-project output. Focused E2E support
|
||||
checks passed (56 tests, one optional external-database case skipped), plus all
|
||||
137 Node support tests and E2E typecheck. The explicit external-database fixture
|
||||
was separately exercised against owned Postgres earlier. The broad E2E unit
|
||||
attempt lacked local listener permissions and was stopped; that failure remains
|
||||
in `/tmp/native-controller-e2e-unit.log`, and broad qualification is still open.
|
||||
|
||||
Database load remains **unqualified**. The narrowed-read and combined-query ABBA
|
||||
runs became heavily contended as other local checks ran, including our E2E support
|
||||
suite during the final pair. Both baseline and candidate lost absolute gates;
|
||||
dropped operations invalidate relative comparisons. The combined-query first
|
||||
pair completed all work, but its candidate p99 was 303 ms versus 131 ms and PG CPU
|
||||
35.69 s versus 25.89 s. Unchanged business queries also slowed. These observations
|
||||
require a quiet-host paired rerun; they are neither a performance pass nor proof
|
||||
that the implementation caused all observed regression. The connection limit
|
||||
remains unchanged. Summarized counters are retained in the adjacent
|
||||
`2026-10-30-native-controller-load-evidence.json`; raw measurements remain under
|
||||
`/tmp/native-controller-load/`. The reproducible script records failed absolute
|
||||
and relative gates and marks unequal-operation comparisons ineligible.
|
||||
|
||||
Remaining critical path: finish the protected Daytona attempt and cleanup;
|
||||
complete full repository checks on the final rebased revision; finish the quiet
|
||||
steady/burst/replacement load gates and explicit lock-wait latency measurement;
|
||||
qualify distinct installed A/B/A artifacts; then resolve review on the draft
|
||||
PR. Current server startup smoke and three-controller component tests are useful
|
||||
layer-specific evidence, not substitutes for those remaining gates.
|
||||
|
||||
|
||||
The full local `pnpm test:run` attempt was stopped after roughly 45 minutes while
|
||||
still progressing through the general chat suite. Its output is retained at
|
||||
`/tmp/native-controller-full-tests.log`; it is incomplete, not a pass. Broad
|
||||
checks will run on hosted Linux for the rebased candidate. The broad E2E support
|
||||
rerun with two workers was also stopped before rebasing; focused support checks
|
||||
remain the completed evidence. A separate `server-config.test.ts` diagnostic
|
||||
confirmed the original restricted run's `listen EPERM` cause.
|
||||
|
||||
|
||||
### Rebased review and qualification follow-up
|
||||
|
||||
Draft implementation: https://github.com/paperclipai/paperclip/pull/18343.
|
||||
At rebased head `3aa90fbf0b5553ffa66a9cdc0fb153f348b56d75`, all 66 focused
|
||||
persistence/discovery cases and full `pnpm -r typecheck` passed. The PR's general
|
||||
server shard found 190 executor unit failures where synthetic IDs and a partial
|
||||
Db double reached the new real discovery boundary. That fixture boundary is
|
||||
being corrected; no production validation is relaxed. Greptile scored 3/5 and
|
||||
identified missing prepared-task lifecycle reconstruction and retry after a
|
||||
failed lifecycle read. Corrections and regression cases are under verification.
|
||||
A separately identified retained-owner admission wait now uses the existing
|
||||
same-run retry record/deadline, with unstarted-run and current-authority fences.
|
||||
|
||||
Protected Daytona workflow `41094539946` failed before any test cell ran: the
|
||||
image publisher exhausted disk (`No space left on device`, exit 102), including
|
||||
its cleanup/log writes. It is not live recovery evidence, and repeating the
|
||||
same build without a capacity fix is not justified. The default-branch workflow
|
||||
has no per-run disk-capacity or upgrade-manifest override. CI infrastructure owns
|
||||
that dependency. The invalid local credential attempt remains unresolved evidence.
|
||||
|
||||
Local review-check attempts also encountered exhausted macOS Postgres semaphores;
|
||||
only these owned test processes were stopped and moved to dedicated Docker
|
||||
Postgres fixtures. One additional direct-read retry scenario timed out during
|
||||
its source-controller fixture startup before a Runner appeared. Fixture import
|
||||
readiness is now measured separately from the unchanged 60-second admission
|
||||
limit. These failed/incomplete attempts remain distinct from subsequent passes.
|
||||
|
||||
Review-fix verification: all 1,066 cases across native executor, durable
|
||||
discovery, startup recovery and heartbeat runtime execution passed against owned
|
||||
Docker Postgres. After the final database-clock and operator-Stop fences, all
|
||||
nine durable discovery/import/wait tests passed and server typecheck passed.
|
||||
The direct per-turn failed-read regression passed through normal admission with
|
||||
three real controller processes: the failed read leaves the reconstructed PRP
|
||||
projection, the next admission reconstructs the missing lifecycle receipt, and
|
||||
the next replacement still continues the same provider history. Prepared-task
|
||||
resumed and failed-read/retry cases also pass for Codex and Claude fixtures.
|
||||
These are scripted-provider component results with remote workspace storage
|
||||
available, not live Daytona or distinct-release acceptance.
|
||||
|
||||
|
||||
### Bounded qualification campaign on October 30
|
||||
|
||||
The follow-up campaign began at 18:33 UTC with an approximately one-hour budget.
|
||||
Three GPT-6.1 Sol workers own live Daytona, database load, and distinct-reader
|
||||
replacement/rollback; the coordinator owns integration, artifact builds and review.
|
||||
The exact native implementation under test is `a0ef3ce7493c8bb76c4766ff007e8a390118d145`.
|
||||
Qualification commit `da8c7f6e1837ddd896dfe83a7524812fee813487` adds measurement and
|
||||
installed-journey admission only; its native runtime, Runner, shared and database
|
||||
production files are byte-identical to that implementation.
|
||||
|
||||
The independent Linux database campaign uses an eight-vCPU, 32 GiB disposable
|
||||
host, with PostgreSQL and its Node workload each limited to two CPUs and 1536 MiB.
|
||||
It preserves the existing 10/2/2 request/renewal/checkpoint pool caps per logical
|
||||
server. Steady and aligned-burst ABBA comparisons passed the existing absolute,
|
||||
correctness and 10% relative regression gates with 300 controllers and 10,000
|
||||
historical tasks. Neither changes snapshot/PRP transaction counts. The separate
|
||||
claim-capacity probe waits for natural lease expiry, invokes the production
|
||||
claim path for 300 synthetic executions, restores real encrypted checkpoints
|
||||
under empty controller directories, and measures current-owner checkpoint and
|
||||
renewal work. Synthetic provider references do not establish sandbox existence
|
||||
or authenticated provider reattachment.
|
||||
|
||||
The closed, sanitized measurements and limits are recorded in
|
||||
`2026-10-30-native-controller-qualification-evidence.json`. The earlier failed,
|
||||
contended-host measurements remain in `2026-10-30-native-controller-load-evidence.json`.
|
||||
Lock measurements use calibrated PostgreSQL wait-start timestamps and sampled
|
||||
duration bounds. Observer traffic contributes to the measured database cost;
|
||||
short unsampled waits and full HTTP latency remain outside that measurement.
|
||||
|
||||
Live qualification uses protected campaign
|
||||
[41108162900](https://github.com/paperclipai/paperclip/actions/runs/41108162900).
|
||||
Its initial catalog timeout occurred before provider allocation after Azure
|
||||
reported a dependency-cache egress limit. A scoped exact-entry cache invalidation
|
||||
and targeted retry forced a successful cold install. Only this campaign's
|
||||
ephemeral image publisher received a larger, delete-on-termination disk; image
|
||||
identity, source admission and provider authorization gates remain in force.
|
||||
|
||||
Provider-free package builds pin first reader `59ff89a18fe1312b5f0bb908439bbdb1237b0252`
|
||||
and candidate `a0ef3ce7493c8bb76c4766ff007e8a390118d145` in runs
|
||||
[41108342354](https://github.com/paperclipai/paperclip/actions/runs/41108342354) and
|
||||
[41108352138](https://github.com/paperclipai/paperclip/actions/runs/41108352138).
|
||||
The source-process crash probe independently observed B waiting for A's live
|
||||
lease and claiming the same execution after natural expiry, with no lease edits.
|
||||
Its initial all-turns/single-provider-process assertion also crossed completion
|
||||
contract revision 1 to revision 2. The existing Runner attachment policy can
|
||||
retire a provider at a changed semantic contract while preserving its conversation;
|
||||
phase-specific observations distinguish that boundary from same-contract rollback.
|
||||
Earlier strict failures remain retained until that distinction is demonstrated.
|
||||
|
||||
The final live, reader, installed-artifact, cleanup and CI outcomes are maintained
|
||||
in [PR #18343](https://github.com/paperclipai/paperclip/pull/18343), with their exact
|
||||
revisions and evidence limits. Campaign registration, successful artifact builds
|
||||
and this plan entry do not themselves certify that a live journey passed.
|
||||
@@ -0,0 +1,170 @@
|
||||
--- scripts/benchmark-native-controller-recovery.ts
|
||||
+++ scripts/benchmark-native-controller-recovery.ts
|
||||
@@ -14 +14 @@
|
||||
-const repoRoot = fileURLToPath(new URL("../", import.meta.url));
|
||||
+const repoRoot = path.resolve(required("BENCH_CANDIDATE_ROOT"));
|
||||
@@ -241,0 +242,112 @@
|
||||
+async function claimCapacityPhase(db: any, u: string, items: any[], monitor: any) {
|
||||
+ const { claimNativeRestartRecoveries } = await load(repoRoot, "server/src/services/native-runtime/native-restart-recovery.ts");
|
||||
+ const { loadNativeRunnerCheckpoint } = await load(repoRoot, "server/src/services/native-runtime/native-runner-checkpoint.ts");
|
||||
+ const replacement = createDb(u, { maxConnections: 10, applicationName: "benchmark-claim-replacement" });
|
||||
+ const successor = { bootId: randomUUID(), pid: process.pid, processStartedAt: new Date() };
|
||||
+ const result: any = { schema: "paperclip.native-controller-claim-capacity/v1", ...provenance,
|
||||
+ scope: "Production DB claim function against synthetic prepared-workspace references and real persisted snapshots/PRP records. Natural lease expiry; two server database clients. No provider attachment or external sandbox existence claim.",
|
||||
+ errors: [], claims: [], reads: [], samples: [], sqlByClient: {}, renewedOwnerLosses: 0, restoredCheckpoints: 0,
|
||||
+ retainedOldControllerLeaseMs: 60000, remoteReconnectGraceMs: 420000, checkpointPoolPerServer: 2 };
|
||||
+ const clients = [
|
||||
+ ["old_request", db.$client], ["old_liveness", livenessDb(db).$client], ["old_checkpoint", checkpointDb(db).$client],
|
||||
+ ["new_request", replacement.$client], ["new_liveness", livenessDb(replacement).$client], ["new_checkpoint", checkpointDb(replacement).$client],
|
||||
+ ];
|
||||
+ for (const [name, client] of clients) { result.sqlByClient[name] = 0; client.options.debug = () => { result.sqlByClient[name]++; }; }
|
||||
+ // Data is persisted by the ordinary snapshot port during the preconditioning phase.
|
||||
+ const validity = await db.$client.unsafe("select count(*)::int as valid from heartbeat_runs r join native_run_finalizations f on f.run_id=r.id where r.status='running' and r.runner_profile_json->'nativeWorkspaceSync'->>'state'='prepared' and r.runner_profile_json->'sessionCheckpoint'->'identity'->>'runId'=r.id::text and exists(select 1 from native_runner_checkpoints c where c.run_id=r.id)");
|
||||
+ assert.equal(validity[0].valid, 300, "All claim inputs need real snapshots/checkpoints and prepared references");
|
||||
+ const before = pgResources(), started = performance.now(), requests = new Set<Promise<any>>(), renewals: any[] = [];
|
||||
+ const observed = observeLockWaits(monitor);
|
||||
+ let tick = 0, offered = 0, completed = 0;
|
||||
+ const traffic = setInterval(() => {
|
||||
+ for (let j = 0; j < 20; j++) {
|
||||
+ const index = tick++, item = items[index % items.length], client = index % 2 ? replacement : db;
|
||||
+ const at = performance.now(); offered++;
|
||||
+ const task = loadAgentRunAuthentication(client, item).then(() => { completed++; result.reads.push(performance.now() - at); })
|
||||
+ .catch((e: any) => result.errors.push({ source: "request", message: e.message })).finally(() => requests.delete(task));
|
||||
+ requests.add(task);
|
||||
+ }
|
||||
+ }, 100); // 200 database authentication requests/second, split evenly between servers.
|
||||
+ let sampling: Promise<any> | null = null;
|
||||
+ const sample = setInterval(() => {
|
||||
+ if (sampling) return;
|
||||
+ sampling = monitor`select application_name,count(*)::int as connections,count(*) filter(where state='active')::int as active from pg_stat_activity where datname=current_database() and application_name!='benchmark-observer' group by application_name`
|
||||
+ .then((rows: any) => result.samples.push({ elapsedMs: performance.now() - started, rows, postgresMemoryBytes: pgResources().memoryBytes }))
|
||||
+ .catch((e: any) => result.errors.push({ source: "observer", message: e.message })).finally(() => { sampling = null; });
|
||||
+ }, 1000);
|
||||
+ try {
|
||||
+ const [lease] = await db.$client.unsafe("select greatest(0,extract(epoch from(max(lease_expires_at)-clock_timestamp()))*1000)::float8 as remaining from native_run_finalizations where lease_owner is not null");
|
||||
+ result.naturalExpiryWaitMs = Number(lease.remaining);
|
||||
+ await sleep(result.naturalExpiryWaitMs + 100);
|
||||
+ const [expired] = await db.$client.unsafe("select count(*)::int as expired from native_run_finalizations where lease_owner is not null and lease_expires_at<=clock_timestamp()");
|
||||
+ assert.equal(expired.expired, 300, "Never rewrite or shorten a controller lease for the probe");
|
||||
+ const claimStarted = performance.now();
|
||||
+ const dispositions = await claimNativeRestartRecoveries({ db: replacement, controller: successor, restartKind: "hard",
|
||||
+ runIds: items.map(item => item.runId), limit: 300,
|
||||
+ onClaim: (claim: any, scope: any) => {
|
||||
+ result.claims.push({ runId: claim.runId, kind: claim.kind, elapsedMs: performance.now() - claimStarted, generation: claim.controllerGeneration });
|
||||
+ renewals.push(startNativeSessionExecutionLeaseRenewal({ db: replacement,
|
||||
+ companyId: scope.companyId, issueId: scope.issueId, runId: claim.runId, leaseOwner: claim.leaseOwner,
|
||||
+ attempt: claim.providerAttempt, controller: successor, remote: true,
|
||||
+ onLost: async () => { result.renewedOwnerLosses++; } }));
|
||||
+ },
|
||||
+ });
|
||||
+ result.claimDrainMs = performance.now() - claimStarted;
|
||||
+ assert.equal(dispositions.length, 300);
|
||||
+ assert(dispositions.every((claim: any) => claim.kind === "reattach_remote_runner"), "Every DB claim must independently validate its evidence");
|
||||
+ // Restore persisted records into empty replacement directories, then persist a real new control and snapshot.
|
||||
+ const work = [...dispositions];
|
||||
+ await Promise.all(Array.from({ length: 4 }, async () => {
|
||||
+ while (work.length) {
|
||||
+ const claim: any = work.shift(), item = items.find(i => i.runId === claim.runId);
|
||||
+ const checkpoint = await loadNativeRunnerCheckpoint(replacement, { companyId: item.companyId, issueId: item.issueId, runId: item.runId });
|
||||
+ assert(checkpoint);
|
||||
+ const directory = root + "/replacement-states/" + item.runId;
|
||||
+ checkpoint.restoreSessionDirectory(directory);
|
||||
+ const header = JSON.parse(checkpoint.records.get("header")!);
|
||||
+ const core = new CandidateCore({ stateDirectory: directory, identity: header.state.identity,
|
||||
+ expectedRunnerVersion: checkpoint.runnerVersion, expectedRunnerDigest: checkpoint.runnerDigest });
|
||||
+ const binding = { companyId: item.companyId, issueId: item.issueId, runId: item.runId,
|
||||
+ authorityRunId: item.runId, bootId: successor.bootId, generation: claim.controllerGeneration, leaseOwner: claim.leaseOwner };
|
||||
+ core.setCheckpointPersistence(CandidateWriter(replacement, binding));
|
||||
+ core.queueCommand("semantic_tool.result", { callId: "replacement-capacity", operationId: "fixture_read", result: { body: "x".repeat(4096) }, isError: false }, "replacement-control");
|
||||
+ await core.flushCheckpoint();
|
||||
+ const identity = item.snapshot.identity;
|
||||
+ const port = new CandidatePort(replacement, { ...identity, sourceInstanceId: checkpoint.recoveryEvidence.runnerInstanceId,
|
||||
+ controlPlaneSourceInstanceId: "replacement-controller", completionContractId: item.completionContractId, completionContractSha256: item.completionContractSha256 }, { nativeRecovery: {
|
||||
+ compatibility: { configDigest: "sha256:" + "a".repeat(64), processConfigDigest: null }, owner: binding } });
|
||||
+ await port.openRun({ identity, backendKind: "codex", sourceInstanceId: checkpoint.recoveryEvidence.runnerInstanceId });
|
||||
+ await port.checkpointSession(item.snapshot);
|
||||
+ result.restoredCheckpoints++;
|
||||
+ await core.stop();
|
||||
+ }
|
||||
+ }));
|
||||
+ result.claimAndRestoreDrainMs = performance.now() - claimStarted;
|
||||
+ // Observe one natural 20-second renewal from the last replacement claim.
|
||||
+ await sleep(Math.max(0, 21000 - (performance.now() - claimStarted - result.claimDrainMs)));
|
||||
+ const [saved] = await replacement.$client.unsafe("select count(*)::int as owners,min(controller_generation)::int as min_generation,max(controller_generation)::int as max_generation,min(extract(epoch from(lease_expires_at-clock_timestamp()))*1000)::float8 as minimum_remaining_ms from native_run_finalizations where controller_boot_id=$1 and lease_owner is not null", [successor.bootId]);
|
||||
+ result.postRenewalOwners = saved;
|
||||
+ assert.equal(saved.owners, 300); assert.equal(saved.min_generation, 2); assert.equal(saved.max_generation, 2);
|
||||
+ assert(saved.minimum_remaining_ms > 40000, "Replacement leases should have naturally renewed from the database clock");
|
||||
+ } catch (e: any) { result.errors.push({ source: "claim-or-restore", message: e.message, stack: e.stack }); }
|
||||
+ finally {
|
||||
+ clearInterval(traffic); clearInterval(sample);
|
||||
+ await Promise.allSettled([...requests]); await sampling;
|
||||
+ await Promise.allSettled(renewals.map(r => r.stop()));
|
||||
+ result.lockWait = await observed.stop();
|
||||
+ }
|
||||
+ result.elapsedMs = performance.now() - started;
|
||||
+ result.requestsOffered = offered; result.requestsCompleted = completed; result.requestLatency = summary(result.reads); delete result.reads;
|
||||
+ result.postgresCpuUsec = pgResources().cpuUsec - before.cpuUsec;
|
||||
+ const totalSql = () => Object.values(result.sqlByClient).reduce((a: number, b: any) => a + Number(b), 0);
|
||||
+ const idleStart = totalSql(); await sleep(1000); result.settledIdleSql = totalSql() - idleStart;
|
||||
+ result.passed = result.errors.length === 0 && result.lockWait.errors.length === 0 && result.claims.length === 300
|
||||
+ && result.restoredCheckpoints === 300 && completed === offered && result.renewedOwnerLosses === 0
|
||||
+ && result.claimAndRestoreDrainMs < 420000 && result.requestLatency.p99 < 250 && result.settledIdleSql === 0;
|
||||
+ await writeFile(root + "/claim-capacity.json", JSON.stringify(result, null, 2) + "\n");
|
||||
+ console.log(JSON.stringify({ phase: "claim-capacity-complete", passed: result.passed, claims: result.claims.length,
|
||||
+ claimDrainMs: result.claimDrainMs, claimAndRestoreDrainMs: result.claimAndRestoreDrainMs,
|
||||
+ requests: result.requestLatency, errors: result.errors.slice(0, 3) }));
|
||||
+ if (!result.passed) process.exitCode = 1;
|
||||
+}
|
||||
+
|
||||
@@ -429,0 +542 @@
|
||||
+ runnerProfileJson: { nativeWorkspaceSync: { schema: "paperclip.native-workspace-sync/v3", state: "prepared", descriptorSha256: "a".repeat(64), baselineSha256: "b".repeat(64), finalHostSha256: null, workspaceId: randomUUID(), leaseId: randomUUID(), providerLeaseId: randomUUID(), remoteCwd: "/workspace", resourceDisposition: "destroy" } },
|
||||
@@ -544,0 +658,2 @@
|
||||
+ completionContractId: contractId,
|
||||
+ completionContractSha256: "fixture",
|
||||
@@ -905,0 +1021 @@
|
||||
+ await claimCapacityPhase(db, u, items, monitor);
|
||||
@@ -917,6 +1033 @@
|
||||
- const profile = [
|
||||
- ["baseline", 300],
|
||||
- ["candidate", 300],
|
||||
- ["candidate", 300],
|
||||
- ["baseline", 300],
|
||||
- ] as const;
|
||||
+ const profile = [["candidate", 300]] as const;
|
||||
@@ -953,35 +1064 @@
|
||||
- const perOperation = (r: any, key: string) =>
|
||||
- r[key] / (r.normalCompleted + r.normalWritesCompleted + r.controls + r.admissionsCompleted);
|
||||
- const average = (values: number[]) => values.reduce((a, b) => a + b, 0) / values.length;
|
||||
- const comparable =
|
||||
- result.every((r) => r.correctnessPassed) &&
|
||||
- result.every(
|
||||
- (r) =>
|
||||
- r.controls === result[0].controls &&
|
||||
- r.normalCompleted === result[0].normalCompleted &&
|
||||
- r.normalWritesCompleted === result[0].normalWritesCompleted,
|
||||
- );
|
||||
- const relative = {};
|
||||
- for (const key of ["postgresCpuUsec", "walBytes", "requestP99"]) {
|
||||
- const value = (r: any) => (key === "requestP99" ? r.normalReadSummary.p99 : perOperation(r, key));
|
||||
- const baseline = average(result.filter((r) => r.variant === "baseline").map(value));
|
||||
- const candidate = average(result.filter((r) => r.variant === "candidate").map(value));
|
||||
- relative[key] = {
|
||||
- baseline,
|
||||
- candidate,
|
||||
- increasePercent: comparable ? 100 * (candidate / baseline - 1) : null,
|
||||
- passed: comparable ? candidate <= baseline * 1.1 : null,
|
||||
- };
|
||||
- }
|
||||
- Object.assign(report, {
|
||||
- relative,
|
||||
- comparable,
|
||||
- passed: result.every((r) => r.passed) && Object.values(relative).every((r: any) => r.passed),
|
||||
- measurementLimits: [
|
||||
- "Database request latency is not HTTP latency",
|
||||
- "Pool queue includes client encoding; BEGIN response includes network time",
|
||||
- "pg_stat_statements execution includes lock waits; lock episodes report sampled duration bounds, with short waits potentially missed",
|
||||
- "Peak memory is sampled once a second",
|
||||
- "Use an otherwise idle cluster and host; WAL and cgroup CPU are cluster-wide",
|
||||
- ],
|
||||
- });
|
||||
+ Object.assign(report, { passed: result.every(r => r.passed), scope: "Preconditioning only for the separate claim-capacity.json report" });
|
||||
@@ -0,0 +1,390 @@
|
||||
{
|
||||
"schema": "paperclip.native-controller-load-summary/v1",
|
||||
"baselineRevision": "59e191c368",
|
||||
"candidateRevision": "4f710fd7e7",
|
||||
"qualification": "failed; host contention and unequal completed work invalidate the ABBA comparison",
|
||||
"sourceReport": "/tmp/native-controller-load/combined-steady/report.json",
|
||||
"fixture": {
|
||||
"mode": "steady",
|
||||
"normalReadsPerSecond": 500,
|
||||
"normalWritesPerSecond": 100,
|
||||
"databaseAdmissionEverySeconds": 5,
|
||||
"checkpointControlEverySeconds": 5,
|
||||
"controlPayloadBytes": 4096,
|
||||
"phaseSeconds": 65,
|
||||
"snapshotTransactionsPerControl": 1,
|
||||
"baselineCheckpointPool": 2,
|
||||
"baselineRenewalIntervalSeconds": 20,
|
||||
"candidateRenewalIntervalSeconds": 20,
|
||||
"historicalIssues": 10000
|
||||
},
|
||||
"limits": [
|
||||
"Database request latency is not HTTP latency",
|
||||
"Pool queue includes client encoding; BEGIN response includes network time",
|
||||
"pg_stat_statements execution includes lock waits; samples report concurrent lock waiters, not a lock latency distribution",
|
||||
"Peak memory is sampled once a second",
|
||||
"Use an otherwise idle cluster and host; WAL and cgroup CPU are cluster-wide"
|
||||
],
|
||||
"results": [
|
||||
{
|
||||
"label": "01_baseline_300",
|
||||
"controls": 3900,
|
||||
"normalOffered": 32500,
|
||||
"normalCompleted": 32500,
|
||||
"normalWritesCompleted": 6500,
|
||||
"dropped": 0,
|
||||
"renewalLosses": 0,
|
||||
"checkpointTransactions": 3900,
|
||||
"snapshotTransactions": 3900,
|
||||
"sqlCount": 54687,
|
||||
"livenessSqlCount": 4502,
|
||||
"checkpointSqlCount": 15601,
|
||||
"changedRows": 7800,
|
||||
"changedBytes": 17371500,
|
||||
"walBytes": 63819704,
|
||||
"postgresCpuUsec": 25887716,
|
||||
"postgresPeakSampledMemoryBytes": 697917440,
|
||||
"normalReadSummary": {
|
||||
"n": 32500,
|
||||
"p50": 53.82264099999884,
|
||||
"p95": 83.97040299999935,
|
||||
"p99": 131.4543599999888,
|
||||
"max": 249.75419500000135
|
||||
},
|
||||
"checkpointLatencySummary": {
|
||||
"n": 3900,
|
||||
"p50": 3.5057919999962905,
|
||||
"p95": 14.36212499999965,
|
||||
"p99": 79.32949999999255,
|
||||
"max": 220.70087599998806
|
||||
},
|
||||
"poolQueueByPool": {
|
||||
"request": {
|
||||
"n": 3913,
|
||||
"p50": 0.04241599999659229,
|
||||
"p95": 4.89183399999456,
|
||||
"p99": 25.90625,
|
||||
"max": 210.7756250000093
|
||||
},
|
||||
"liveness": {
|
||||
"n": 900,
|
||||
"p50": 401.07175099999586,
|
||||
"p95": 1987.666584000006,
|
||||
"p99": 2269.4225429999933,
|
||||
"max": 2296.889626000004
|
||||
},
|
||||
"checkpoint": {
|
||||
"n": 3900,
|
||||
"p50": 0.05258299999695737,
|
||||
"p95": 0.14449999999487773,
|
||||
"p99": 41.99183400000038,
|
||||
"max": 188.36633400000574
|
||||
}
|
||||
},
|
||||
"transactionLatencyByPool": {
|
||||
"request": {
|
||||
"n": 3913,
|
||||
"p50": 3.6501250000001164,
|
||||
"p95": 14.756584000002476,
|
||||
"p99": 62.41062499998952,
|
||||
"max": 240.88450000000012
|
||||
},
|
||||
"liveness": {
|
||||
"n": 900,
|
||||
"p50": 406.2205419999955,
|
||||
"p95": 2044.2198760000028,
|
||||
"p99": 2277.324250999998,
|
||||
"max": 2308.3716669999994
|
||||
},
|
||||
"checkpoint": {
|
||||
"n": 3900,
|
||||
"p50": 3.297791999997571,
|
||||
"p95": 14.160208000001148,
|
||||
"p99": 79.12295900000026,
|
||||
"max": 220.50066700000025
|
||||
}
|
||||
},
|
||||
"drainMs": 26.716459000002942,
|
||||
"absoluteGates": {
|
||||
"requestP99": true,
|
||||
"drain": true,
|
||||
"throughput": true,
|
||||
"postgresCpu": true,
|
||||
"postgresMemory": true
|
||||
},
|
||||
"passed": true,
|
||||
"startedAt": "2026-10-30T16:40:56.503Z",
|
||||
"endedAt": "2026-10-30T16:42:01.534Z"
|
||||
},
|
||||
{
|
||||
"label": "02_candidate_300",
|
||||
"controls": 3900,
|
||||
"normalOffered": 32500,
|
||||
"normalCompleted": 32500,
|
||||
"normalWritesCompleted": 6500,
|
||||
"dropped": 0,
|
||||
"renewalLosses": 0,
|
||||
"checkpointTransactions": 3900,
|
||||
"snapshotTransactions": 3900,
|
||||
"sqlCount": 54687,
|
||||
"livenessSqlCount": 4502,
|
||||
"checkpointSqlCount": 15601,
|
||||
"changedRows": 7800,
|
||||
"changedBytes": 17371500,
|
||||
"walBytes": 66746424,
|
||||
"postgresCpuUsec": 35685605,
|
||||
"postgresPeakSampledMemoryBytes": 666578944,
|
||||
"normalReadSummary": {
|
||||
"n": 32500,
|
||||
"p50": 68.02364699999453,
|
||||
"p95": 127.37661100001424,
|
||||
"p99": 303.326441000012,
|
||||
"max": 474.2829830000119
|
||||
},
|
||||
"checkpointLatencySummary": {
|
||||
"n": 3900,
|
||||
"p50": 5.670542000007117,
|
||||
"p95": 241.97500100001344,
|
||||
"p99": 479.3440830000036,
|
||||
"max": 553.2187080000003
|
||||
},
|
||||
"poolQueueByPool": {
|
||||
"request": {
|
||||
"n": 3913,
|
||||
"p50": 0.05162500002188608,
|
||||
"p95": 37.2935839999991,
|
||||
"p99": 136.02920900000026,
|
||||
"max": 420.89487499999814
|
||||
},
|
||||
"liveness": {
|
||||
"n": 900,
|
||||
"p50": 536.7470420000027,
|
||||
"p95": 3620.454543,
|
||||
"p99": 4146.325211000018,
|
||||
"max": 4238.34758500001
|
||||
},
|
||||
"checkpoint": {
|
||||
"n": 3900,
|
||||
"p50": 0.057792000006884336,
|
||||
"p95": 209.87512499999139,
|
||||
"p99": 443.83324999999604,
|
||||
"max": 528.2805419999931
|
||||
}
|
||||
},
|
||||
"transactionLatencyByPool": {
|
||||
"request": {
|
||||
"n": 3913,
|
||||
"p50": 6.640833000012208,
|
||||
"p95": 66.06445900001563,
|
||||
"p99": 184.52537499999744,
|
||||
"max": 449.3999579999945
|
||||
},
|
||||
"liveness": {
|
||||
"n": 900,
|
||||
"p50": 545.1022919999959,
|
||||
"p95": 3665.246376999974,
|
||||
"p99": 4172.158336000022,
|
||||
"max": 4248.078752000001
|
||||
},
|
||||
"checkpoint": {
|
||||
"n": 3900,
|
||||
"p50": 5.484875000009197,
|
||||
"p95": 241.83212600002298,
|
||||
"p99": 478.8787080000038,
|
||||
"max": 553.1449999999895
|
||||
}
|
||||
},
|
||||
"drainMs": 76.8760000000184,
|
||||
"absoluteGates": {
|
||||
"requestP99": false,
|
||||
"drain": true,
|
||||
"throughput": true,
|
||||
"postgresCpu": true,
|
||||
"postgresMemory": true
|
||||
},
|
||||
"passed": false,
|
||||
"startedAt": "2026-10-30T16:42:16.915Z",
|
||||
"endedAt": "2026-10-30T16:43:21.995Z"
|
||||
},
|
||||
{
|
||||
"label": "03_candidate_300",
|
||||
"controls": 3887,
|
||||
"normalOffered": 32500,
|
||||
"normalCompleted": 32379,
|
||||
"normalWritesCompleted": 6470,
|
||||
"dropped": 164,
|
||||
"renewalLosses": 0,
|
||||
"checkpointTransactions": 3887,
|
||||
"snapshotTransactions": 3887,
|
||||
"sqlCount": 54484,
|
||||
"livenessSqlCount": 4502,
|
||||
"checkpointSqlCount": 15549,
|
||||
"changedRows": 7774,
|
||||
"changedBytes": 17313559,
|
||||
"walBytes": 66470080,
|
||||
"postgresCpuUsec": 53715752,
|
||||
"postgresPeakSampledMemoryBytes": 670437376,
|
||||
"normalReadSummary": {
|
||||
"n": 32379,
|
||||
"p50": 87.29533699998865,
|
||||
"p95": 407.7868620000081,
|
||||
"p99": 715.8863589999964,
|
||||
"max": 1413.3376139999891
|
||||
},
|
||||
"checkpointLatencySummary": {
|
||||
"n": 3887,
|
||||
"p50": 35.722666999994544,
|
||||
"p95": 2254.7568759999995,
|
||||
"p99": 4195.890377000003,
|
||||
"max": 4784.590544000006
|
||||
},
|
||||
"poolQueueByPool": {
|
||||
"request": {
|
||||
"n": 3900,
|
||||
"p50": 0.12583400000585243,
|
||||
"p95": 218.51912499999162,
|
||||
"p99": 498.72595799999544,
|
||||
"max": 1368.8183760000102
|
||||
},
|
||||
"liveness": {
|
||||
"n": 900,
|
||||
"p50": 2733.2611259999976,
|
||||
"p95": 7990.602337999997,
|
||||
"p99": 8707.207421,
|
||||
"max": 8915.786796
|
||||
},
|
||||
"checkpoint": {
|
||||
"n": 3887,
|
||||
"p50": 3.155832999997074,
|
||||
"p95": 2207.667333999998,
|
||||
"p99": 4154.626294000016,
|
||||
"max": 4726.676418999996
|
||||
}
|
||||
},
|
||||
"transactionLatencyByPool": {
|
||||
"request": {
|
||||
"n": 3900,
|
||||
"p50": 27.709541999996873,
|
||||
"p95": 276.61000000001513,
|
||||
"p99": 625.9165419999918,
|
||||
"max": 1627.7409179999959
|
||||
},
|
||||
"liveness": {
|
||||
"n": 900,
|
||||
"p50": 2782.7440009999846,
|
||||
"p95": 8019.347628999996,
|
||||
"p99": 8734.34838000001,
|
||||
"max": 8964.291171000019
|
||||
},
|
||||
"checkpoint": {
|
||||
"n": 3887,
|
||||
"p50": 35.49954200000502,
|
||||
"p95": 2254.554459999985,
|
||||
"p99": 4195.5352520000015,
|
||||
"max": 4784.371294000011
|
||||
}
|
||||
},
|
||||
"drainMs": 4928.940960000007,
|
||||
"absoluteGates": {
|
||||
"requestP99": false,
|
||||
"drain": false,
|
||||
"throughput": true,
|
||||
"postgresCpu": true,
|
||||
"postgresMemory": true
|
||||
},
|
||||
"passed": false,
|
||||
"startedAt": "2026-10-30T16:43:41.479Z",
|
||||
"endedAt": "2026-10-30T16:44:51.414Z"
|
||||
},
|
||||
{
|
||||
"label": "04_baseline_300",
|
||||
"controls": 2390,
|
||||
"normalOffered": 32500,
|
||||
"normalCompleted": 10713,
|
||||
"normalWritesCompleted": 1503,
|
||||
"dropped": 28305,
|
||||
"renewalLosses": 0,
|
||||
"checkpointTransactions": 2390,
|
||||
"snapshotTransactions": 2390,
|
||||
"sqlCount": 21797,
|
||||
"livenessSqlCount": 4502,
|
||||
"checkpointSqlCount": 9561,
|
||||
"changedRows": 4780,
|
||||
"changedBytes": 10643952,
|
||||
"walBytes": 31871472,
|
||||
"postgresCpuUsec": 55047197,
|
||||
"postgresPeakSampledMemoryBytes": 643657728,
|
||||
"normalReadSummary": {
|
||||
"n": 10713,
|
||||
"p50": 291.001007999992,
|
||||
"p95": 841.957837999973,
|
||||
"p99": 2601.1680469999555,
|
||||
"max": 3106.3534709999803
|
||||
},
|
||||
"checkpointLatencySummary": {
|
||||
"n": 2390,
|
||||
"p50": 6933.93029400002,
|
||||
"p95": 16553.939049999986,
|
||||
"p99": 17176.082591000013,
|
||||
"max": 17434.499467000016
|
||||
},
|
||||
"poolQueueByPool": {
|
||||
"request": {
|
||||
"n": 2392,
|
||||
"p50": 0.10450000001583248,
|
||||
"p95": 520.4013340000529,
|
||||
"p99": 798.97454200004,
|
||||
"max": 2614.81958499999
|
||||
},
|
||||
"liveness": {
|
||||
"n": 900,
|
||||
"p50": 5497.6995860000025,
|
||||
"p95": 16836.80413299997,
|
||||
"p99": 17748.473174999992,
|
||||
"max": 18895.275049999997
|
||||
},
|
||||
"checkpoint": {
|
||||
"n": 2390,
|
||||
"p50": 6882.121711000043,
|
||||
"p95": 16492.48646600003,
|
||||
"p99": 17097.382091999985,
|
||||
"max": 17352.715134
|
||||
}
|
||||
},
|
||||
"transactionLatencyByPool": {
|
||||
"request": {
|
||||
"n": 2392,
|
||||
"p50": 55.64137500000652,
|
||||
"p95": 638.8517920000013,
|
||||
"p99": 2574.2484169999952,
|
||||
"max": 2879.7436260000104
|
||||
},
|
||||
"liveness": {
|
||||
"n": 900,
|
||||
"p50": 5623.322503000032,
|
||||
"p95": 16872.938590999984,
|
||||
"p99": 18552.84471799998,
|
||||
"max": 18921.25067499996
|
||||
},
|
||||
"checkpoint": {
|
||||
"n": 2390,
|
||||
"p50": 6933.636502000038,
|
||||
"p95": 16553.763133,
|
||||
"p99": 17175.909424000012,
|
||||
"max": 17433.771509000042
|
||||
}
|
||||
},
|
||||
"drainMs": 12508.142965000006,
|
||||
"absoluteGates": {
|
||||
"requestP99": false,
|
||||
"drain": false,
|
||||
"throughput": false,
|
||||
"postgresCpu": true,
|
||||
"postgresMemory": true
|
||||
},
|
||||
"passed": false,
|
||||
"startedAt": "2026-10-30T16:45:39.710Z",
|
||||
"endedAt": "2026-10-30T16:46:57.219Z"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,76 @@
|
||||
--- scripts/benchmark-native-controller-recovery.ts
|
||||
+++ scripts/benchmark-native-controller-recovery.ts
|
||||
@@ -0,0 +1 @@
|
||||
+// Temporary qualification probe derived from frozen benchmark SHA256 abaa29d50eb95013836b54b8dacc94c72e080edf24744cc5d291cc5eb1ff2818.
|
||||
@@ -14 +15 @@
|
||||
-const repoRoot = fileURLToPath(new URL("../", import.meta.url));
|
||||
+const repoRoot = path.resolve(required("BENCH_CANDIDATE_ROOT"));
|
||||
@@ -187 +188 @@
|
||||
- const byPool = Object.fromEntries(["benchmark-application", "paperclip-controller-lease", "paperclip-runner-checkpoint"].map(pool => {
|
||||
+ const byPool = Object.fromEntries(["benchmark-application", "benchmark-replacement-application", "paperclip-controller-lease", "paperclip-runner-checkpoint"].map(pool => {
|
||||
@@ -250,0 +252 @@
|
||||
+ const replacementDb = createDb(u, { maxConnections: 10, applicationName: "benchmark-replacement-application" });
|
||||
@@ -256,0 +259 @@
|
||||
+ const replacementOwner = { ...owner, bootId: randomUUID() };
|
||||
@@ -299,0 +303,3 @@
|
||||
+ ["replacement_request", replacementDb.$client],
|
||||
+ ["replacement_liveness", livenessDb(replacementDb).$client],
|
||||
+ ["replacement_checkpoint", checkpointDb(replacementDb).$client],
|
||||
@@ -369,0 +376 @@
|
||||
+ replacementDb.$client.options.debug = db.$client.options.debug;
|
||||
@@ -386,0 +394,2 @@
|
||||
+ livenessDb(replacementDb).$client.options.debug = livenessDb(db).$client.options.debug;
|
||||
+ checkpointDb(replacementDb).$client.options.debug = checkpointDb(db).$client.options.debug;
|
||||
@@ -389,0 +399,2 @@
|
||||
+ const controllerDb = i % 2 ? replacementDb : db;
|
||||
+ const controllerOwner = i % 2 ? replacementOwner : owner;
|
||||
@@ -443,3 +454,3 @@
|
||||
- controllerBootId: owner.bootId,
|
||||
- controllerPid: owner.pid,
|
||||
- controllerProcessStartedAt: owner.processStartedAt,
|
||||
+ controllerBootId: controllerOwner.bootId,
|
||||
+ controllerPid: controllerOwner.pid,
|
||||
+ controllerProcessStartedAt: controllerOwner.processStartedAt,
|
||||
@@ -467 +478 @@
|
||||
- const writer = (variant === "candidate" ? CandidateWriter : BaselineWriter)(db, {
|
||||
+ const writer = (variant === "candidate" ? CandidateWriter : BaselineWriter)(controllerDb, {
|
||||
@@ -472 +483 @@
|
||||
- bootId: owner.bootId,
|
||||
+ bootId: controllerOwner.bootId,
|
||||
@@ -511 +522 @@
|
||||
- db,
|
||||
+ controllerDb,
|
||||
@@ -531 +542 @@
|
||||
- bootId: owner.bootId,
|
||||
+ bootId: controllerOwner.bootId,
|
||||
@@ -544,0 +556,2 @@
|
||||
+ db: controllerDb,
|
||||
+ owner: controllerOwner,
|
||||
@@ -592 +605 @@
|
||||
- for (let i = 0; i < 100; i++) await loadAgentRunAuthentication(db, items[i % n]);
|
||||
+ for (let i = 0; i < 100; i++) await loadAgentRunAuthentication(items[i % n].db, items[i % n]);
|
||||
@@ -639 +652 @@
|
||||
- const input = { db, ...item, attempt: 1, controller: owner };
|
||||
+ const input = { ...item, attempt: 1, controller: item.owner };
|
||||
@@ -664 +677 @@
|
||||
- const rows = await loadAgentRunAuthentication(db, item);
|
||||
+ const rows = await loadAgentRunAuthentication(item.db, item);
|
||||
@@ -700 +713 @@
|
||||
- await db.update(issues).set({ updatedAt: new Date() }).where(eq(issues.id, item.issueId));
|
||||
+ await item.db.update(issues).set({ updatedAt: new Date() }).where(eq(issues.id, item.issueId));
|
||||
@@ -713 +726 @@
|
||||
- await db.transaction(async (tx: any) => {
|
||||
+ await (metrics.admissionsOffered % 2 ? replacementDb : db).transaction(async (tx: any) => {
|
||||
@@ -917,6 +930 @@
|
||||
- const profile = [
|
||||
- ["baseline", 300],
|
||||
- ["candidate", 300],
|
||||
- ["candidate", 300],
|
||||
- ["baseline", 300],
|
||||
- ] as const;
|
||||
+ const profile = [["baseline", 300], ["candidate", 300], ["candidate", 300], ["baseline", 300]] as const;
|
||||
@@ -929 +937 @@
|
||||
- "Paired baseline/candidate, including real PaperclipControlPlanePort snapshot persistence on every control checkpoint. Database/control-plane component stress; real production checkpoint writer and natural remote lease timers. Same synthetic DB request load. Not full HTTP, harness or hosted capacity qualification.",
|
||||
+ "Two logical servers in one client process, 150 controllers each, separate request/liveness/checkpoint pools at unchanged caps; same aggregate aligned-burst workload. Matched baseline/candidate ABBA pool-overlap component qualification; no takeover latency or provider reattachment claim. Includes real PaperclipControlPlanePort snapshot persistence on every control checkpoint. Database/control-plane component stress; real production checkpoint writer and natural remote lease timers. Same synthetic DB request load. Not full HTTP, harness or hosted capacity qualification.",
|
||||
@@ -930,0 +939 @@
|
||||
+ logicalServers: 2,
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,70 @@
|
||||
{
|
||||
"schema": "paperclip.native-controller.release-qualification-receipt/v1",
|
||||
"finishedAt": "2026-10-30T19:33:20.587023+00:00",
|
||||
"sourceIntegrity": {
|
||||
"A": {
|
||||
"revision": "59ff89a18fe1312b5f0bb908439bbdb1237b0252",
|
||||
"nativeRuntimeSourceBytesUnchanged": true,
|
||||
"mismatches": []
|
||||
},
|
||||
"B": {
|
||||
"revision": "98f05fcd5b452704131a35ff09486f5c04c6959a",
|
||||
"nativeRuntimeSourceBytesUnchanged": true,
|
||||
"mismatches": []
|
||||
}
|
||||
},
|
||||
"sourceAndArtifact": {
|
||||
"A": "59ff89a18fe1312b5f0bb908439bbdb1237b0252",
|
||||
"B": "98f05fcd5b452704131a35ff09486f5c04c6959a",
|
||||
"admittedRunnerSha256": "b1f04caa1ce04e720dec7eb62ede170eb3621948c0849b8f305ea5c5116bced8",
|
||||
"runnerArtifact": {
|
||||
"role": "admitted Runner artifact A, unchanged through every controller",
|
||||
"sourceRevision": "59ff89a18fe1312b5f0bb908439bbdb1237b0252",
|
||||
"binary": "/private/tmp/native-controller-qualification-20261030-1833/installed-A/consumer/node_modules/@paperclipai/server/dist/vendor/paperclip-runner/bin/linux-arm64/paperclip-runnerd",
|
||||
"sha256": "b1f04caa1ce04e720dec7eb62ede170eb3621948c0849b8f305ea5c5116bced8",
|
||||
"bytes": 35345144,
|
||||
"installationReceipt": "/private/tmp/native-controller-qualification-20261030-1833/installed-A/receipt.json"
|
||||
}
|
||||
},
|
||||
"passed": [
|
||||
"Distinct source-controller active SIGKILL followed by natural 60-second lease takeover; exact same run/provider attempt/Runner retained",
|
||||
"Current real-PostgreSQL stream oracle at 8,512,8192 chunks: zero incremental checkpoint SQL/transactions/changed rows; idle SQL zero; positive control passed",
|
||||
"B1 recovered run completed and committed its result with ready native snapshot and valid v1 recovery envelope"
|
||||
],
|
||||
"failed": [
|
||||
"Fresh B2 retained continuation rejected after B1 completed because durable control journal has a null-goal snapshot after run.terminal",
|
||||
"Final 98f05fc live probe still rejects actual paperclip.prp.event.v2 because proposed guard accepts only v1"
|
||||
],
|
||||
"notQualified": [
|
||||
"Complete A/B/A native installed-application startup/rollback gate",
|
||||
"Uninterrupted warm provider process through changed completion-contract boundary",
|
||||
"First compatible published empty-disk rollback reader"
|
||||
],
|
||||
"limits": [
|
||||
"Source controller processes and scripted Codex-compatible provider in Docker; no live-provider or live-Daytona claim",
|
||||
"Qualified immutable A release Runner used; source controller native modules hash-verified, not full installed server execution",
|
||||
"Only native controller projections deleted; non-native configuration, encryption key, workspace/sandbox dependencies preserved",
|
||||
"Local stream WAL/elapsed diagnostics excluded from quantitative performance due concurrent isolated fixture"
|
||||
],
|
||||
"evidence": {
|
||||
"finalProbe": "/tmp/native-controller-release-qualification-qualified-A-20261030/evidence.json",
|
||||
"finalProbeLog": "/tmp/native-controller-release-qualification-qualified-A-20261030/run-final-fix.log",
|
||||
"controlLog": "/tmp/native-controller-release-qualification-qualified-A-20261030/controller.log",
|
||||
"streamOracle": "/tmp/native-controller-release-qualification-20261030/stream-cost-current.json",
|
||||
"streamLog": "/tmp/native-controller-release-qualification-20261030/stream-cost-current.log",
|
||||
"earlierThreeTurnBusinessContinuation": "/tmp/native-controller-release-qualification-20261030/evidence-run-06.json"
|
||||
},
|
||||
"finalProbeFailure": {
|
||||
"message": "Error: runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate\n at deny (/private/tmp/native-controller-release-qualification-qualified-A-20261030/B-98f05f/server/src/services/native-runtime/remote-warm-session-recovery.ts:56:37)\n at inspectRemoteWarmSessionRecovery (/private/tmp/native-controller-release-qualification-qualified-A-20261030/B-98f05f/server/src/services/native-runtime/remote-warm-session-recovery.ts:85:67)\n at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n at async executePaperclipNativeSessionWithinScope (/private/tmp/native-controller-release-qualification-qualified-A-20261030/B-98f05f/server/src/services/native-runtime/native-session-executor.ts:8237:28)\n at async executePaperclipNativeSession (/private/tmp/native-controller-release-qualification-qualified-A-20261030/B-98f05f/server/src/services/native-runtime/native-session-executor.ts:8057:12)\n at async process.<anonymous> (/private/tmp/native-controller-release-qualification-qualified-A-20261030/controller-worker.mjs:36:16)",
|
||||
"cause": null
|
||||
},
|
||||
"candidateCorrection": {
|
||||
"change": "Accept supported PRP event v1 and v2 idle goal suffixes; reject unknown versions",
|
||||
"fullSourceReplay": "not rerun after the final v2 correction within the one-hour qualification window",
|
||||
"focusedTests": {
|
||||
"result": "pass",
|
||||
"cases": 14,
|
||||
"test": "server/src/services/native-runtime/remote-warm-session-recovery.test.ts"
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -109,13 +109,15 @@ not on the ordinary write pool. The helper ends that physical connection in
|
||||
actual refresh settlement. Summary writeback runs watched-query reads on its
|
||||
existing transaction, so a one-connection main pool completes the full flow.
|
||||
|
||||
Migration0419 adds nullable internal `next_eval_reason` to the existing card row.
|
||||
Migration0420 adds nullable internal `next_eval_reason` to the existing card row.
|
||||
Membership deadlines may be reconstructed at startup after a committed source
|
||||
hint was lost. Policy eligibility and the atomically recorded five-minute retry
|
||||
lease retain their saved deadline. Explicit refresh deadlines remain due work.
|
||||
The HTTP card response removes this internal provenance; the public schema stays
|
||||
unchanged.
|
||||
|
||||
Migration0419 retains the published agent-comment retry uniqueness and conversation read-state backfill. The status-card column follows it in migration0420; the combined schema snapshot is generated from both owners.
|
||||
|
||||
For legacy rows, a pending change hash preserves policy eligibility. An untagged
|
||||
active deadline farther than five minutes away cannot be an outstanding
|
||||
five-minute claim, so startup rechecks it. An ambiguous untagged deadline within
|
||||
@@ -157,7 +159,7 @@ eviction, skipped/repeated hours, half-hour DST, date-line changes, rollback,
|
||||
strict boundaries, and actual twenty-card source delivery followed by quiet.
|
||||
|
||||
|
||||
Refresh decisions use the card's PostgreSQL row revision as a private compare-and-swap fence. If a card changes during watched-source reads, the owner rereads it and reconsiders the current policy, for at most three attempts. A replacement future interval deadline remains unchanged; a replacement manual policy does not run automatically. The same fence protects no-change, paused-policy and generation claims. Existing issue idempotency, generation ownership and losing-claim cancellation still apply. The row revision is not exposed through card APIs or shared scheduling contracts.
|
||||
Refresh decisions use the card's PostgreSQL row revision as a private compare-and-swap fence. If a card changes during watched-source reads, the owner rereads it and reconsiders the current policy, for at most three attempts. A replacement future interval deadline remains unchanged; a replacement manual policy does not run automatically. The same fence protects no-change, paused-policy and generation claims. Refresh task idempotency includes that captured revision: concurrent requests for the same revision still share one task, while a later revision gets an independent task even when its watched fingerprint is unchanged. A superseded refresh can therefore cancel its losing task without cancelling a newer revision's not-yet-claimed task. Compile task identity is unchanged. The row revision is not exposed through card APIs or shared scheduling contracts.
|
||||
|
||||
Company source hints do not preempt a future deadline owned by a failed evaluation (`retry`). Membership and policy deadlines remain eligible for source-driven rechecks, and actual due retries still run. An explicit policy edit can replace the retry through the existing card mutation path. Real PostgreSQL regressions hold the actual search while committing an ordinary policy edit, then verify the saved interval still fires. Persistent search failures stay quiet through issue hints and restart, retry at their saved deadline, and recover through an ordinary policy edit.
|
||||
|
||||
|
||||
@@ -7,7 +7,7 @@ and backup validation against a deterministic fake Codex process in Linux. It
|
||||
checks the same runner process identity and one provider process across three warm
|
||||
turns, provider history reuse, and verified suspension at idle close. Separate
|
||||
cases kill the actual controller process and recover the next turn. They reject
|
||||
a still-live controller and a replacement sandbox before reconnecting. Legacy
|
||||
a live execution lease and a replacement sandbox before reconnecting. Legacy
|
||||
checkpoints and changed process configuration require verified suspension and
|
||||
replacement, while preserving the conversation and starting each turn once.
|
||||
Recovery can restart a provider to refresh its tool contract; process reuse is
|
||||
@@ -24,6 +24,39 @@ subscription. The controller must register the connection before subscribing so
|
||||
completed message processing can drain and the suspended session can retire.
|
||||
It does not qualify a live model or a sandbox provider API.
|
||||
|
||||
The `diskless` and `diskless-per-turn` cases delete both native controller
|
||||
directories after A completes, continue in a separate B process, delete those
|
||||
directories after B completes, and continue in C. They use the production source
|
||||
selector and snapshot rebinder for per-turn admission, followed by the real
|
||||
native executor. They do not inject PRP files or edit leases to produce recovery.
|
||||
Warm mode verifies compatible process reuse; per-turn mode verifies the existing
|
||||
suspension and history-resume policy. These component tests preserve workspace
|
||||
and remote sandbox storage, and use one source revision in three processes.
|
||||
They do not qualify distinct application releases or complete webserver disk loss.
|
||||
|
||||
`native-controller-recovery.integration.test.ts` separately verifies encrypted
|
||||
database discovery, one-source import, idempotency, live-owner waiting, corrupt
|
||||
bindings and paginated inventory. `native-session-resume-projection.test.ts`
|
||||
retains the 32-candidate and newer-progress barrier checks. Snapshot transaction
|
||||
counts and stale-generation rejection are exercised by the control-plane port
|
||||
tests; incomplete-publication retention is exercised by the expiration tests.
|
||||
|
||||
The cold Codex transport regressions additionally retain a real provider event
|
||||
after completion, hold its persistence callback and require the original source
|
||||
tuple and Runner outbox to survive until acceptance. A separate case deletes the
|
||||
controller cache after its staged target is durably recorded and resumes through
|
||||
the same source before target attachment. Unknown semantic tails must stay held
|
||||
without target tool dispatch or a fabricated response. The staged restart helper
|
||||
and classifier tests cover target discovery before its first session snapshot,
|
||||
exact encrypted source/header/artifact bindings and the existing compatibility
|
||||
rebinder. Source accounting tests use real PostgreSQL to distinguish immutable
|
||||
receipt replay, equal finalized usage and late usage requiring reconciliation.
|
||||
Handoff crash tests must restore the exact queued command or authenticated
|
||||
receipt, reuse the verified physical Runner and preserve endpoint binding. Test
|
||||
both direct callbacks and the same guest listener when that ingress is enabled;
|
||||
changing an endpoint or losing physical proof must hold recovery.
|
||||
These component cases do not replace the normal three-controller Daytona gate.
|
||||
|
||||
Per-turn plugin continuation uses a stopped checkpoint on a replacement lease.
|
||||
The executor tests cover Claude and Codex, including reopening blocked work.
|
||||
The lease-authority integration test uses PostgreSQL to check that checkpoint
|
||||
@@ -151,14 +184,21 @@ local file becomes a marker that requires restoration from the external
|
||||
checkpoint before restart. Ordinary control changes do not rewrite a second
|
||||
local journal. The sandbox runner keeps its own journal.
|
||||
|
||||
Checkpoint transactions share a separate pool capped at two connections per
|
||||
Checkpoint transactions share a separate pool capped at three connections per
|
||||
server database client. They queue without occupying the application pool or
|
||||
the existing lease-renewal pool. This can delay a control acknowledgement during
|
||||
a burst; it does not release that acknowledgement before its commit. The cost is
|
||||
up to two additional active database connections per instance, not per agent.
|
||||
up to three additional active database connections per instance, not per agent.
|
||||
There is no new SQL per text chunk or while idle, and the number of control
|
||||
transactions is unchanged. SQL cost collection must include the checkpoint pool.
|
||||
|
||||
The cap follows a bounded 2 → 3 → 4 → 2 diagnostic: both cap-2 trials failed
|
||||
the existing burst-drain gate; 3 was the smallest tested passing cap. Retain
|
||||
those failures and the diagnostic separately. The subsequent native feature-cost
|
||||
comparison uses the common current DB factory and equal cap-3 policy for both
|
||||
variants, so its baseline is a controlled baseline rather than the unmodified
|
||||
historical cap-2 configuration. All existing benchmark gates remain unchanged.
|
||||
|
||||
A sparse checkpoint is marked `checkpointEventPolicy: "control-v1"`. Its peer must
|
||||
advertise `recoveryCursorVersion: 1`. During reconnection the controller echoes the
|
||||
runner's durable `resume.ackedSourceSeq` as `recoveryAckedSourceSeq` in the signed
|
||||
@@ -270,8 +310,10 @@ adapters, database loss and destroyed sandboxes are outside this guarantee.
|
||||
|
||||
The empty-controller-cache test uses a real Rust runner and scripted provider
|
||||
in separate directories, removes the controller directory, then restores solely
|
||||
from Postgres. It checks unchanged runner/provider process identities and one
|
||||
provider turn start. Build the host runner and fake provider first, then run:
|
||||
from Postgres. It waits for natural lease expiry and checks unchanged
|
||||
runner/provider process identities and one provider turn start. It explicitly
|
||||
restores the transport checkpoint, so it does not prove application startup
|
||||
discovery by itself. Build the host runner and fake provider first, then run:
|
||||
|
||||
```sh
|
||||
PAPERCLIP_TEST_CONTROLLER_OUTAGE_MS=300000 pnpm exec vitest run \
|
||||
@@ -284,6 +326,85 @@ This is protocol and persistence evidence. The separate
|
||||
qualifies the actual Daytona/provider path, including completion and workspace
|
||||
finalization. Neither test substitutes for the other's failure boundary.
|
||||
|
||||
The explicit `daytona-restart-continuity.runner-codex.daytona.recover-controller`
|
||||
cell fixes the outage at 300 seconds and adds a second replacement after
|
||||
completion. Its new turn executes the saved project through normal admission.
|
||||
The oracle compares hashed native/provider session identities and requires a new
|
||||
completed tool receipt and correct answer. An installed A/B/A artifact manifest
|
||||
uses the last replacement for this retained-session continuation/rollback check.
|
||||
The general everyday suite retains its 125-second default. These intervals are
|
||||
fault tests, not permitted HTTP deployment downtime. Both fixtures retain the
|
||||
workspace and remote sandbox; they do not prove whole-webserver disk loss.
|
||||
|
||||
The dedicated suite's version 6 uses the existing reusable warm environment,
|
||||
with a 15-minute provider idle window and unchanged 60/300/420-second ownership,
|
||||
outage and reconnect limits. Its prior per-turn/no-reuse measurement failed
|
||||
retained native/provider identity and remains failed. The test retires the whole
|
||||
`runtime/paperclip-runner` directory, including provider failover-backup bytes.
|
||||
Per-turn application qualification after sandbox disposal and this disk loss
|
||||
therefore awaits the provider-backup owner's durable storage contract; logical
|
||||
native checkpoints do not replace provider history or weaken the identity oracle.
|
||||
|
||||
The rollback reader must support both the recovery envelope and the settled PRP
|
||||
v2 idle goal suffix. Record its actual source commit and installed tree hash.
|
||||
An older reader rejecting that journal is an unsupported rollback, not a pass.
|
||||
A backported compatibility build is a development fixture until it is released;
|
||||
do not describe it as a previously published version. Distinct source-controller
|
||||
tests, compiled native-module tests, installed CLI startup and the full Product
|
||||
E2E journey provide different evidence and must retain those scope labels.
|
||||
|
||||
## Database cost qualification
|
||||
|
||||
`scripts/benchmark-native-controller-recovery.ts` compares a pinned source
|
||||
baseline with the candidate's real PRP writer, control-plane snapshot port and
|
||||
lease renewal. It uses identical fixtures with 300 controllers and 10,000 prior
|
||||
tasks, in baseline/candidate/candidate/baseline order. `steady` distributes control
|
||||
checkpoints; `burst` aligns them every five seconds. `pilot` checks instrumentation
|
||||
with ten controllers. No provider is called.
|
||||
|
||||
Run on an otherwise idle disposable Linux Postgres cluster with
|
||||
`pg_stat_statements` preloaded and its extension available. Give Postgres and the
|
||||
Node client two CPUs and 1536 MiB each. Set these explicit inputs:
|
||||
|
||||
```sh
|
||||
BENCH_DISPOSABLE_POSTGRES=1 \
|
||||
BENCH_DATABASE_URL=postgres://postgres:fixture-password@127.0.0.1:5432/postgres \
|
||||
BENCH_BASELINE_ROOT=/absolute/path/to/pinned-baseline \
|
||||
BENCH_REPORT_ROOT=/absolute/path/to/new-report-directory \
|
||||
BENCH_POSTGRES_CGROUP=/absolute/path/to/postgres-cgroup \
|
||||
node --import ./cli/node_modules/tsx/dist/loader.mjs \
|
||||
scripts/benchmark-native-controller-recovery.ts steady
|
||||
```
|
||||
|
||||
The baseline needs the same installed dependency versions and built workspace
|
||||
packages. It supplies the baseline writer, snapshot port and PRP controller;
|
||||
the request fixtures, renewal path and pool policy are common controls. When
|
||||
running an exported tree without Git metadata, supply its exact
|
||||
`BENCH_BASELINE_REVISION`, `BENCH_CANDIDATE_REVISION` and
|
||||
`BENCH_CANDIDATE_DIFF`. Record the export source and verify its bytes separately.
|
||||
Use a fresh report directory for every attempt. Only new random databases are
|
||||
created; successful trial databases are dropped, and the owned template remains
|
||||
for inspection until the disposable cluster is removed.
|
||||
|
||||
The report counts SQL across all three pools, actual snapshot/PRP transactions,
|
||||
changed records/bytes, WAL, database CPU, sampled memory, pool admission,
|
||||
transaction latency and database request latency. Pool admission ends at
|
||||
connection assignment/BEGIN encoding; BEGIN response time is separate.
|
||||
Postgres execution time includes lock waits. A 25 ms observer uses PostgreSQL's
|
||||
`pg_locks.waitstart` to report duration bounds for observed lock episodes by pool.
|
||||
The pilot calibrates it against an intentional 250 ms row-lock wait. Short waits
|
||||
can be missed, and delayed samples widen the estimated upper bound. The nominal
|
||||
40 observer queries per second contribute to measured database CPU and statement
|
||||
statistics. Reports include actual sample counts, sampling gaps, CPU pressure,
|
||||
quota throttling and the measurement harness hash. They do not provide exact
|
||||
lock-exclusive SQL execution time or HTTP latency; this is component load
|
||||
qualification.
|
||||
|
||||
Correctness and absolute gates must pass for acceptance.
|
||||
Dropped operations invalidate paired comparisons. The report retains failures
|
||||
and refuses an overall pass if p99, drain, throughput, memory, CPU or the 10%
|
||||
relative regression gate fails. Do not change pool limits to conceal a failure.
|
||||
|
||||
## Required native usage acceptance
|
||||
|
||||
Built-in native remote Runner execution accepts structured accounting before
|
||||
|
||||
Reference in new issue
Block a user