mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-21 00:05:49 +02:00
Record upload failure and warm-owner correction evidence
This commit is contained in:
Dotta committed
1 parent
2a5eca4220
commit
32b092f5c4
5 files changed
+206
-6
No files matched your search
@@ -2,21 +2,21 @@
|
||||
|
||||
**Experimental and unmerged. Local journeys and partial staging qualification passed on the sources below; final-source staging acceptance remains incomplete.** A successful build, probe, or earlier-source journey does not establish the remaining product conditions.
|
||||
|
||||
## Current source boundary (2026-10-11 07:54 UTC)
|
||||
## Current source boundary (2026-10-11 08:18 UTC)
|
||||
|
||||
| Surface | Source and evidence |
|
||||
| --- | --- |
|
||||
| Combined implementation | `ce639505e10211495b4a4ad10c50604f9352cba5`, including bounded turn-start diagnostics. Exact-head review scored 5/5 with no unresolved findings. Initial CI had one browser-fixture deadlock; its single failed-job rerun passed. At 07:52:19 UTC, all 53 checks passed. |
|
||||
| Core branch | `86c1aa2b0c53b1b41888cea6b8e04e42e28dd05c`. Canonical Boat migration 0333 follows unchanged upstream 0332. Turn-start diagnostics passed 146 TypeScript, 13 Rust and two forwarding cases, affected TypeScript checks and independent reviews. Initial CI had one 15-second document-restore test timeout; the unchanged focused suite passed six cases, and one failed-job rerun passed. At 07:52:19 UTC, 54 checks passed and four were skipped. |
|
||||
| Combined implementation | `2a5eca4220bb9050a024d833480287b8296ce2b3`, including shared Boat upload scheduling and the warm-owner orphan-sweep correction. At 08:22:16.280 UTC, all 53 checks passed; exact-head review scored 5/5 with no unresolved findings. Earlier `ce639505` completed 53 successful checks after one documented failed-job rerun. |
|
||||
| Core branch | `086a9dbbd870b631f8b5d70c04acceededb943e3`. Upload scheduling passed 150 boundary and 28 focused cases; the warm-owner sweep correction passed 33 cases and server TypeScript. The final public-module import correction passed the boundary gate and those same checks. At 08:22:16.280 UTC, 54 checks passed and four were skipped; exact-head review scored 5/5 with no unresolved findings. Earlier `86c1aa2b` completed 54 successful checks and four skips after one documented failed-job rerun. |
|
||||
| Verified staging source | Deployment23 verified `a5ed24f7eeb50abd35375c6b1864bb424fe699b3` at 07:22:18.985 UTC through the official workflow. Saved native Codex continuation, state-preserving warm HMR, both legacy jobs, native Claude cold execution, direct browser desktop mouse control and the later native Codex window-input scenario passed. Claude's ordinary follow-up failed before provider work with cleanup quarantine; advertised desktop-target computer actions were rejected before the successful window-input retry. |
|
||||
| Staging candidate | `86994966da` additionally includes turn-start diagnostics. It is pushed but has not been deployed. Applied staging migration history remains unchanged. |
|
||||
| Local retry | Local46 uses compatible controller `ad4b3d18983f71b1fadc3fb8f861d780b0daddb8` and qualified immutable pack `sha256:4ef25756f766016da224f37f68c138c0c4a9ef9d128e6c2fc415f5c57ce38573`. Fresh BOA22 failed during pack upload at 07:43:55.690, before any runner connection or provider event. Local45's cold/warm passes and failed post-idle turn startup remain preserved below. |
|
||||
| Staging candidate | Deployment25 is building `ca07557247533531b85045a1dc20c231dc9cd1de`, including both behavioral corrections and the public-module import fix. Deployment24 was interrupted before rollout after CI found the import violation. Applied staging migration history remains unchanged. |
|
||||
| Local retry | Local47 uses compatible controller `ecc9b2fa9fe28d214d5c6b9db82b46cf19ba22cc` and the unchanged qualified immutable diagnostics pack `sha256:4ef25756f766016da224f37f68c138c0c4a9ef9d128e6c2fc415f5c57ce38573`. BOA22 failed before provider work at 08:11:43.478 while waiting for upload capacity. The scheduling correction alone has not qualified live. Local45/46 outcomes remain preserved below. |
|
||||
|
||||
Final-source live acceptance remains incomplete. Open conditions include Claude warm and post-idle continuation, reliable first-use artifact transfer, embedded human input, and live sibling cancellation isolation. The full local suite remains **34,914 passed, 24 failed, 338 skipped**; no full-suite green result is claimed. The CI reruns retain both original failures and do not change these live results.
|
||||
|
||||
The [consolidated retry receipt](assets/2026-10-10-boat/evidence/retry-qualification-summary.json) preserves earlier sources, timestamps, failures, and receipt hashes. A deployment proves serving source, not product acceptance. No provider stop was forced or cancelled, no quarantine was cleared, and no database history was reset or rewritten. Dated observations below retain their original cutoffs.
|
||||
|
||||
Latest automated checks: [exact-head rerun result](assets/2026-10-10-boat/evidence/ci-86c1-ce639-rerun-passed.json), [original failures](assets/2026-10-10-boat/evidence/ci-86c1-ce639-original-failures.json), and [single-rerun provenance](assets/2026-10-10-boat/evidence/ci-86c1-ce639-rerun-provenance.json). These apply to the recorded code heads, not later edits.
|
||||
Latest automated checks: [current code-head result](assets/2026-10-10-boat/evidence/ci-086a9-2a5eca-passed.json). Earlier [exact-head rerun result](assets/2026-10-10-boat/evidence/ci-86c1-ce639-rerun-passed.json), [original failures](assets/2026-10-10-boat/evidence/ci-86c1-ce639-original-failures.json), and [single-rerun provenance](assets/2026-10-10-boat/evidence/ci-86c1-ce639-rerun-provenance.json). These apply to the recorded code heads, not later edits.
|
||||
|
||||
## Product outcomes
|
||||
|
||||
@@ -323,3 +323,15 @@ The expanded staging controller read at approximately 07:52 UTC found the cold C
|
||||
Normal same-task continuation `c829fe63` resumed saved Codex identities after Boat's completed archive. It obtained fresh Chrome window geometry and used sequential foreground window-targeted `click`, `hotkey` (Ctrl+A), `type_text` (`https://example.org/`) and `press_key` (Enter), with a fresh window-state screenshot after each operation. No operation returned a captured error. After completion, normal Computer Connect independently displayed the page at **example.org**, visibly distinct from the earlier human example.com navigation. No human desktop input intervened: [live Computer panel proof](assets/2026-10-10-boat/staging23-native-computer-example-org.png). The verification viewer was then closed normally.
|
||||
|
||||
This passes the installed agent computer tools' window-input path on staging source `a5ed24f7ee`. It does not erase the earlier advertised desktop-target rejection or establish current embedded human keyboard input. Saved normalized session `06496696`, logical runner `0005a613`, provider thread `01a12881` and proof hash were retained on a new boot/owner with runnerPID30560/start35548 and CodexPID30762/start35742.
|
||||
|
||||
## Upload scheduling and warm-owner cleanup corrections (2026-10-11 08:18 UTC)
|
||||
|
||||
[Correction and test receipt](assets/2026-10-10-boat/evidence/upload-warm-owner-corrections.json).
|
||||
|
||||
Core `8198c9a571` adds transport-owned, per-machine upload slots matching Boat's four bulk SSH slots. Two concurrent 24MiB transfers failed the deterministic regression before the fix and completed byte-identically afterward, preserving control-command capacity. Existing command deadlines, owner checks, cancellation and content validation remain unchanged. This addresses self-queueing under eight global artifact workers; it does not prove that slow active transfers complete reliably.
|
||||
|
||||
Core `d14578ad13` corrects a separate orphan-sweep race: native completion can retain a valid warm computer owner before the heartbeat becomes terminal, while the environment lease remains active during instruction cleanup. The generic sweep could then claim the old lease and retire the retained owner. The correction locks and validates the exact durable binding and retains the existing warm deadline plus existing shutdown grace. Expired or mismatched authority still cleans up. Two regressions failed before the change; all 33 focused cases pass afterward. Independent review also caught that process launch generation and owner generation are independent counters; a launch2/owner1 case now passes. The staging23 controller audit proves the sweep participated in the historical failure, but does not establish its initiating physical retirement time.
|
||||
|
||||
Local47's ordinary same-task retry `6bc40f44` began 08:04:17.255 and failed 08:11:43.478 with “Computer upload timed out waiting for capacity.” Runner artifact preparation lasted 343,560ms; no provider-pack progress, runner connection or provider event was observed. Active transfers retained the available upload slots, but existing logs cannot distinguish SSH transfer, remote owner setup, or process-close cleanup. A 157-second SSH process observation does not itself prove a deadline overrun. Narrow per-chunk phase diagnostics are being added before another attempt. No further deadline or admission-policy change is justified by this result. [Failure receipt](assets/2026-10-10-boat/evidence/local47-upload-failure.json) · [Visible failed task](assets/2026-10-10-boat/local47-upload-capacity-failure.png).
|
||||
|
||||
Deployment24 was stopped before rollout when CI detected an internal-module import violation. The two-line public export/import correction `086a9dbbd8` passed the module gate, 33 affected tests and server TypeScript. Deployment25 targets the corrected staging source; rollout and live lifecycle acceptance are still pending. Previous quarantines, saved sessions, immutable artifacts and migration histories remain intact.
|
||||
@@ -0,0 +1,100 @@
|
||||
{
|
||||
"checkedAt": "2026-10-11T08:22:16.280034+00:00",
|
||||
"prs": [
|
||||
{
|
||||
"pr": 15813,
|
||||
"head": "086a9dbbd870b631f8b5d70c04acceededb943e3",
|
||||
"reviewHead": "086a9dbbd870b631f8b5d70c04acceededb943e3",
|
||||
"mergeable": "MERGEABLE",
|
||||
"counts": {
|
||||
"SKIPPED": 4,
|
||||
"SUCCESS": 54
|
||||
},
|
||||
"failures": [],
|
||||
"focusChecks": [
|
||||
{
|
||||
"__typename": "CheckRun",
|
||||
"completedAt": "2026-10-11T08:15:50Z",
|
||||
"conclusion": "SUCCESS",
|
||||
"detailsUrl": "https://github.com/paperclipai/paperclip/actions/runs/38123798644/job/114424228447",
|
||||
"name": "ci / e2e shard (8/8)",
|
||||
"startedAt": "2026-10-11T08:12:04Z",
|
||||
"status": "COMPLETED",
|
||||
"workflowName": "PR"
|
||||
},
|
||||
{
|
||||
"__typename": "CheckRun",
|
||||
"completedAt": "2026-10-11T08:18:16Z",
|
||||
"conclusion": "SUCCESS",
|
||||
"detailsUrl": "https://app.greptile.com/paperclip-org-3/github/paperclipai/paperclip/-/pull/15813",
|
||||
"name": "Greptile Review",
|
||||
"startedAt": "2026-10-11T08:09:14Z",
|
||||
"status": "COMPLETED",
|
||||
"workflowName": ""
|
||||
}
|
||||
],
|
||||
"pending": [],
|
||||
"unresolved": [],
|
||||
"moreThreads": false,
|
||||
"greptileSummary": [
|
||||
{
|
||||
"url": "https://github.com/paperclipai/paperclip/pull/15813#issuecomment-6099291617",
|
||||
"updatedAt": "2026-10-11T08:18:22Z",
|
||||
"reviewedCommits": [
|
||||
"086a9dbbd870b631f8b5d70c04acceededb943e3"
|
||||
],
|
||||
"score": [
|
||||
"5"
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"pr": 15804,
|
||||
"head": "2a5eca4220bb9050a024d833480287b8296ce2b3",
|
||||
"reviewHead": "2a5eca4220bb9050a024d833480287b8296ce2b3",
|
||||
"mergeable": "MERGEABLE",
|
||||
"counts": {
|
||||
"SUCCESS": 53
|
||||
},
|
||||
"failures": [],
|
||||
"focusChecks": [
|
||||
{
|
||||
"__typename": "CheckRun",
|
||||
"completedAt": "2026-10-11T08:16:03Z",
|
||||
"conclusion": "SUCCESS",
|
||||
"detailsUrl": "https://github.com/paperclipai/paperclip/actions/runs/38123799771/job/114424220747",
|
||||
"name": "ci / e2e shard (8/8)",
|
||||
"startedAt": "2026-10-11T08:11:55Z",
|
||||
"status": "COMPLETED",
|
||||
"workflowName": "PR"
|
||||
},
|
||||
{
|
||||
"__typename": "CheckRun",
|
||||
"completedAt": "2026-10-11T08:19:04Z",
|
||||
"conclusion": "SUCCESS",
|
||||
"detailsUrl": "https://app.greptile.com/paperclip-org-3/github/paperclipai/paperclip/-/pull/15804",
|
||||
"name": "Greptile Review",
|
||||
"startedAt": "2026-10-11T08:09:13Z",
|
||||
"status": "COMPLETED",
|
||||
"workflowName": ""
|
||||
}
|
||||
],
|
||||
"pending": [],
|
||||
"unresolved": [],
|
||||
"moreThreads": false,
|
||||
"greptileSummary": [
|
||||
{
|
||||
"url": "https://github.com/paperclipai/paperclip/pull/15804#issuecomment-6098591173",
|
||||
"updatedAt": "2026-10-11T08:19:10Z",
|
||||
"reviewedCommits": [
|
||||
"2a5eca4220bb9050a024d833480287b8296ce2b3"
|
||||
],
|
||||
"score": [
|
||||
"5"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,39 @@
|
||||
{
|
||||
"checkedAt": "2026-10-11T08:14:57.809071+00:00",
|
||||
"issueId": "0ce8247c-b59d-48c3-aa8c-d5a6ae23e96b",
|
||||
"identifier": "BOA-22",
|
||||
"controllerSource": "ecc9b2fa9fe28d214d5c6b9db82b46cf19ba22cc",
|
||||
"coreCorrections": [
|
||||
"8198c9a571ea24aedc32040d0ebae90535694421",
|
||||
"d14578ad13e681cd5dde0025061c8b1493b0aed1"
|
||||
],
|
||||
"artifactSource": "86c1aa2b0c53b1b41888cea6b8e04e42e28dd05c",
|
||||
"providerPackDigest": "sha256:4ef25756f766016da224f37f68c138c0c4a9ef9d128e6c2fc415f5c57ce38573",
|
||||
"configuredRunnerSha256": "b47ed0f4f29e4c945d12fa4f351c01ee5f03685974bf03591689969bb3b85b73",
|
||||
"runId": "6bc40f44-946c-4ed5-ba4c-b58dea5fe1c1",
|
||||
"nativeSessionId": "007cb3e0-e42b-445d-bcaf-84c9a66943ef",
|
||||
"runnerInstanceId": "ad29bcb9-506b-49eb-bf8c-60646c0788cb",
|
||||
"startedAt": "2026-10-11T08:04:17.255Z",
|
||||
"finishedAt": "2026-10-11T08:11:43.478Z",
|
||||
"status": "failed_before_provider",
|
||||
"errorCode": "native_session_interrupted",
|
||||
"error": "Computer upload timed out waiting for capacity",
|
||||
"providerConnections": 0,
|
||||
"committedProviderEvents": 0,
|
||||
"runnerArtifactPreparationMs": 343560,
|
||||
"providerPackUploadProgressObserved": false,
|
||||
"operatorAction": "Ordinary task comment; no reset or manual session change. Product chose a new session after the preceding pre-provider failure.",
|
||||
"observerPolicy": "Read-only controller/DB/event metadata and task-owned host process names/elapsed times. No guest reads or prewarming.",
|
||||
"diagnosis": {
|
||||
"proven": "An artifact chunk expired waiting for the shared upload slot while other chunks retained slots; runner artifact preparation failed before provider-pack preparation or provider launch.",
|
||||
"hostSnapshot": {
|
||||
"checkedAt": "2026-10-11T08:11:00.088Z",
|
||||
"sshChildCount": 1,
|
||||
"sshElapsedSeconds": 157
|
||||
},
|
||||
"unknown": "No retained per-chunk phase/result log identifies whether active work waited in SSH transfer, remote owner-lock setup, unit launch, foreground completion, or close/cleanup. The first queue error masks later active chunk errors.",
|
||||
"deadlineContract": "Raw SSH timer starts before stdin submission; SIGTERM at deadline, SIGKILL after5s, settlement awaits close and auth cleanup. Host snapshot does not prove a timeout overrun.",
|
||||
"nextEvidence": "Closed per-chunk phase and outcome timings, without payload/argv/path output. No further timeout/backpressure change justified from this result."
|
||||
},
|
||||
"acceptance": "not_passed"
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
{
|
||||
"recordedAt": "2026-10-11T08:21:25.647229+00:00",
|
||||
"source": {
|
||||
"upload": "8198c9a571ea24aedc32040d0ebae90535694421",
|
||||
"warmOwner": "d14578ad13e681cd5dde0025061c8b1493b0aed1",
|
||||
"moduleBoundary": "086a9dbbd870b631f8b5d70c04acceededb943e3",
|
||||
"combined": "2a5eca4220bb9050a024d833480287b8296ce2b3",
|
||||
"stagingCandidate": "ca07557247533531b85045a1dc20c231dc9cd1de",
|
||||
"localServing": "ecc9b2fa9fe28d214d5c6b9db82b46cf19ba22cc"
|
||||
},
|
||||
"upload": {
|
||||
"regression": "two simultaneous transfers rejected under old queue-inclusive command deadlines; both complete byte-identically with shared per-machine slots",
|
||||
"boundaryTestsPassed": 150,
|
||||
"focusedTestsPassed": 28,
|
||||
"adapterAndServerTypecheck": "passed",
|
||||
"independentReview": "no actionable findings",
|
||||
"limits": "Unrelated bulk contention or stalled active SSH can still hit unchanged active-command deadlines; live retry not qualified."
|
||||
},
|
||||
"warmOwner": {
|
||||
"redCases": 2,
|
||||
"finalTestsPassed": 33,
|
||||
"skipped": 0,
|
||||
"serverTypecheck": "passed",
|
||||
"independentReview": "approved after correcting independent launch/owner generation assumption",
|
||||
"protected": "exact company/environment/agent/run/owner/generation binding with valid process metadata and bounded warm deadline/grace",
|
||||
"cleanupPreserved": "expired or mismatched authority; concurrent retention CAS",
|
||||
"causalBoundary": "Staging audit proves orphan sweep participated; deterministic test reproduces premature reclamation. Historical physical retirement time is unavailable."
|
||||
},
|
||||
"moduleBoundary": {
|
||||
"oldHeadPolicyChecks": "failed on direct domain import; preserved",
|
||||
"correction": "use existing public computers module entry point",
|
||||
"moduleGate": "passed",
|
||||
"reaperTestsPassed": 33,
|
||||
"serverTypecheck": "passed"
|
||||
},
|
||||
"live": {
|
||||
"local47": "Failed before provider at08:11:43.478: upload capacity timeout while active chunks retained slots. No provider-pack progress, runner connections or provider events. Scheduling fix alone not qualified.",
|
||||
"deployment24": "local official-helper process interrupted before rollout; preview build may finish; staging unchanged",
|
||||
"deployment25": "Official immutable image build in progress; not verified serving",
|
||||
"acceptance": "incomplete"
|
||||
},
|
||||
"constraints": [
|
||||
"No master merge",
|
||||
"No forced provider stop",
|
||||
"No quarantine/checkpoint/cache reset",
|
||||
"No migration history rewrite",
|
||||
"Warm timeout and existing shutdown grace unchanged"
|
||||
]
|
||||
}
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 94 KiB |
Reference in new issue
Block a user