mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-11 05:11:49 +01:00
test: correct OpenCode and process settlement verification
This commit is contained in:
Dotta committed
1 parent
4db2ec3159
commit
f0a3c395c2
15 files changed
+232
-47
No files matched your search
@@ -18,8 +18,10 @@ All retained raw files and decompressed trace entries passed an exact credential
|
||||
|
||||
The original accepted `visibleDelta` source52 happened during `workspace_finalizing`; it proves pre-terminal visibility only. A read-only review of the immutable full trace found earlier progressive content which the old selector excluded. The two unmodified screencast JPEGs now included show a short partial phrase at00:11:58.573Z and partially completed numbered sentences at00:11:59.142Z. DOM after@call@120 at00:11:58.580Z contains the partial interstitial ending in `1`, matching ordinary source29/30. Later ordinary source31..52 were emitted/committed at00:11:59.037Z through00:12:00.230Z; accepted result was committed at00:12:02.610Z. The recorded UI live-runs read at00:11:58.443Z reported native/running, but omitted nativePhase. This supports visible progressive content before later deltas/result, without inventing a literal provider-active phase predicate. `retained-progressive-streaming-proof.json` supplies original trace hash, resource names, frame IDs, source excerpts, and explicit limits. The nearest first JPEG precedes that DOM snapshot by about7ms and therefore shows only the shorter source29 prefix; it is not claimed to contain the entire slightly later DOM text.
|
||||
|
||||
The old selector returned an empty array atcall120 because the live interstitial had no settled-agent-bubble ancestor. The future helper now scopes to the owned issue/thread, includes normal phase-interstitial/live-text branches, and requires an open started ordinary item plus a fresh running full GET, rejecting known finalization, completed items, accepted results and terminal state. Five offline real-browser DOM/source tests and the existing nine controlled source seams passed. These future helper changes were not executed in the recorded live journey. Original executed e6e/d719 helper copies and the original22-file bundle remain unchanged; the future source and regression receipt are labelled separately in this augmented bundle. No SUT, model, prompt, provider, app or cloud resource was changed or restarted.
|
||||
The old selector returned an empty array atcall120 because the live interstitial had no settled-agent-bubble ancestor. The future helper now scopes to the owned issue/thread, includes normal phase-interstitial/live-text branches, and requires an ordinary item opened by a delta plus a fresh running full GET, rejecting known finalization, completed items, accepted results and terminal state. Six offline real-browser DOM/source tests and the existing nine controlled source seams passed. These future helper changes were not executed in the recorded live journey. Original executed e6e/d719 helper copies and the original22-file bundle remain unchanged; the future source and regression receipt are labelled separately in this augmented bundle. No SUT, model, prompt, provider, app or cloud resource was changed or restarted.
|
||||
|
||||
## Public packaging
|
||||
|
||||
Resource-specific absolute checkout/evidence/temporary-home paths were replaced with descriptive references. `public-projections.json` records original and public hashes for each projection. Revisions, event identities, times, counts, hashes, cleanup outcomes and limits are unchanged. The launcher retains its already published generic macOS temporary-directory prefix, not a resource-specific private path. The two argument-adapted monitor/checker copies were syntax checked only; original executed hashes remain distinct. Run the future offline verification explicitly with `node --test scripts/native-ingestion/live-browser-streaming-check.mjs`; it is an opt-in Node/Playwright check outside the current default Vitest projects. The original22-file proof bundle remains unchanged.
|
||||
|
||||
The reviewed follow-up also accepts OpenCode ordinary deltas without an artificial `item.started` prerequisite while retaining explicit completion/finalization/terminal rejection. The birth-settlement reproduction helper now requires one successful complete process-list snapshot and propagates command failures. `verification-follow-up.json` records the offline regressions; `independent-successful-process-snapshot.json` separately selects the original successful monitor output for all four captured process identities. These corrected helpers were not executed in a new live journey.
|
||||
+1
-1
@@ -10,7 +10,7 @@
|
||||
},
|
||||
"browser": {
|
||||
"beforeSha256": "cf34b0f7bb8a4f84786a38f1c90b750e7a336b126e6e2bc25647095c1eac9fab",
|
||||
"afterSha256": "5ff4cd9b187829e706ff770b96a110f654b087c92a5a97c8f4ce0a1319dcf2be"
|
||||
"afterSha256": "d5aac1e02c23fd294a4bcb11c0441f92f2c2c7309d9f4f4b4c3e2030de9456a2"
|
||||
}
|
||||
},
|
||||
"cases": [
|
||||
|
||||
+7
-6
@@ -3,16 +3,17 @@
|
||||
"scope": "Offline verification-only, network aborted, no app/provider/database/resources",
|
||||
"command": "node --test scripts/native-ingestion/live-browser-streaming-check.mjs",
|
||||
"exit": 0,
|
||||
"testsPassed": 5,
|
||||
"checkedAt": "2026-11-04T00:47:02.143566+00:00",
|
||||
"helperSha256": "5ff4cd9b187829e706ff770b96a110f654b087c92a5a97c8f4ce0a1319dcf2be",
|
||||
"testSha256": "3f26b2df27f9f40e5f941706a38623cf391fe92bb4091f903745a17d963e315c",
|
||||
"testsPassed": 6,
|
||||
"checkedAt": "2026-11-04T01:03:21.963140+00:00",
|
||||
"helperSha256": "d5aac1e02c23fd294a4bcb11c0441f92f2c2c7309d9f4f4b4c3e2030de9456a2",
|
||||
"testSha256": "75965e259fe067ebec45f3f6de87458f568baa1820cf4e8ecd77f8dd1ff74e49",
|
||||
"cases": [
|
||||
"capturedpartialDOMoldselectormisses/newobserves",
|
||||
"live-tail/settled compatibility",
|
||||
"hidden/human/otherthread rejection",
|
||||
"openitem/freshrunning positive/no closedphaseallowlist",
|
||||
"finalizing/accepted/terminal/closeditem rejection"
|
||||
"ordinarydelta/freshrunning positive/no closedphaseallowlist",
|
||||
"finalizing/accepted/terminal/closeditem rejection",
|
||||
"actualretainedOpenCodedeltaswithoutsyntheticitemstart"
|
||||
],
|
||||
"renamedOptInCommandVerified": true
|
||||
}
|
||||
+14
-5
@@ -54,7 +54,7 @@ test('hidden output, human prompts and a different thread cannot prove owned liv
|
||||
});
|
||||
|
||||
async function observation({ phase = 'observed', status = 'running', finishedAt = null,
|
||||
accepted = false, completedItem = false, startedItem = true, terminal = false, kind = 'agentMessage' } = {}) {
|
||||
accepted = false, completedItem = false, startedItem = false, terminal = false, kind = 'agentMessage', recordedOpenCode = false } = {}) {
|
||||
const start = source.indexOf(' const streaming = await poll(');
|
||||
const end = source.indexOf(' await page.screenshot(', start);
|
||||
const normalizer = source.split('\n').find(line => line.startsWith('const normalizeVisible = '));
|
||||
@@ -62,10 +62,11 @@ async function observation({ phase = 'observed', status = 'running', finishedAt
|
||||
assert.ok(start >= 0 && end > start && normalizer && summaryStart >= 0 && summaryEnd > summaryStart);
|
||||
const execute = new AsyncFunction('poll', 'taskRuns', 'visibleAgentText', 'eventsFor', 'api', 'created', 'journey', 'expect',
|
||||
normalizer + '\n' + source.slice(summaryStart, summaryEnd) + '\n' + source.slice(start, end) + '\nreturn journey;');
|
||||
const active = { id: 'owned-run', status: 'running', runtimeMode: 'native' };
|
||||
const recorded = recordedOpenCode ? JSON.parse(await readFile(new URL('../../doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/retained-progressive-streaming-proof.json', import.meta.url), 'utf8')) : null;
|
||||
const active = { id: recorded?.runId ?? 'owned-run', status: 'running', runtimeMode: 'native' };
|
||||
const identity = { runId: active.id, sourceInstanceId: 'source-1', turnId: 'turn-1', itemId: 'item-1' };
|
||||
const row = event => ({ eventType: event.eventType, payload: { prpEvent: { ...identity, ...event } } });
|
||||
const events = [
|
||||
const events = recorded ? recorded.matchedAndLaterOrdinaryEvents.slice(0, 2) : [
|
||||
...(startedItem ? [row({ eventType: 'item.started', sourceEventId: 'started' })] : []),
|
||||
row({ eventType: 'item.delta', sourceEventId: 'delta-29', sourceSeq: 29, payload: { kind, text: 'The progress message was sent and' } }),
|
||||
...(completedItem ? [row({ eventType: 'item.completed' })] : []),
|
||||
@@ -81,7 +82,7 @@ async function observation({ phase = 'observed', status = 'running', finishedAt
|
||||
value => ({ toBe(expected) { assert.equal(value, expected); } }));
|
||||
}
|
||||
|
||||
test('fresh running observation with a still-open ordinary item proves streaming', async () => {
|
||||
test('fresh running observation with an ordinary delta and no close proves streaming', async () => {
|
||||
const value = await observation();
|
||||
assert.equal(value.visibleBeforeTerminal, true);
|
||||
assert.equal(value.visibleDelta.sourceEventId, 'delta-29');
|
||||
@@ -92,7 +93,15 @@ test('fresh running observation with a still-open ordinary item proves streaming
|
||||
|
||||
test('known finalization, accepted results, terminal state and closed items are rejected', async () => {
|
||||
for (const options of [{ phase: 'workspace_finalizing' }, { accepted: true }, { terminal: true },
|
||||
{ completedItem: true }, { startedItem: false }, { status: 'succeeded' }, { finishedAt: 'fixture-time' }, { kind: 'reasoning' }]) {
|
||||
{ completedItem: true }, { status: 'succeeded' }, { finishedAt: 'fixture-time' }, { kind: 'reasoning' }]) {
|
||||
await assert.rejects(observation(options), /streaming_observation_not_proven/);
|
||||
}
|
||||
});
|
||||
|
||||
// OpenCode's real driver emits ordinary deltas without an item.started event.
|
||||
// The retained normalized rows carry actual run/source/item identities.
|
||||
test('recorded OpenCode ordinary deltas qualify without a synthetic item start', async () => {
|
||||
const value = await observation({ recordedOpenCode: true });
|
||||
assert.equal(value.visibleDelta.sourceSeq, 29);
|
||||
assert.equal(value.visibleBeforeTerminal, true);
|
||||
});
|
||||
-1
@@ -89,7 +89,6 @@ try {
|
||||
const delta = events.find(event => event?.eventType === "item.delta" && event.runId === active.id
|
||||
&& event.payload.kind === "agentMessage" && normalizeVisible(event.payload.text).length >= 12
|
||||
&& text.includes(normalizeVisible(event.payload.text)) && typeof event.itemId === "string"
|
||||
&& events.some(item => item.eventType === "item.started" && sameItem(item, event))
|
||||
&& !events.some(item => ["item.completed", "item.failed", "item.cancelled"].includes(item.eventType) && sameItem(item, event)));
|
||||
if (!delta) return null;
|
||||
// Re-read status after observing the matching UI text. A completed run
|
||||
|
||||
+14
-3
@@ -12,10 +12,21 @@ assert.equal(gate.runId,stop.activeRunId);assert.equal(gate.command,'sleep 60');
|
||||
assert.ok(gate.observedProcessAgeMs+gate.verifiedBeforeClickAtMs-gate.observedAtMs<20000);
|
||||
assert.equal(gate.chain[0].pid,gate.sleepPid);assert.equal(gate.chain.at(-1).pid,gate.runnerPid);
|
||||
assert.ok(gate.chain.some(proc=>/verified-executable|opencode/.test(proc.executable)));
|
||||
// A successful full process-list read is required. Command failures never
|
||||
// become absence evidence, including status1, signals and spawn errors.
|
||||
const processList = execFileSync('ps', ['-axo', 'pid=,ppid=,stat=,lstart=,comm='],
|
||||
{encoding:'utf8',stdio:['ignore','pipe','pipe']});
|
||||
const lines = processList.split('\n').filter(line => line.trim());
|
||||
assert.ok(lines.length > 0, 'Process-list snapshot is empty');
|
||||
const present = new Set(lines.map(line => {
|
||||
const match = line.match(/^\s*(\d+)\s+/);
|
||||
assert.ok(match, 'Process-list snapshot contains an invalid row');
|
||||
return Number(match[1]);
|
||||
}));
|
||||
assert.ok(present.has(process.pid), 'Process-list snapshot does not include the verifier');
|
||||
const checks=gate.chain.map(proc=>{
|
||||
let text='';try{text=execFileSync('ps',['-p',String(proc.pid),'-o','pid=,ppid=,stat=,lstart=,comm='],{encoding:'utf8',stdio:['ignore','pipe','ignore']}).trim();}catch{}
|
||||
assert.equal(text,'','Exact owned birth PID remains present after settlement');
|
||||
assert.equal(present.has(proc.pid),false,'Exact owned birth PID remains present after settlement');
|
||||
return {pid:proc.pid,observedBirthTime:proc.birthTime,observedExecutable:proc.executable,absent:true};
|
||||
});
|
||||
const output={verified:true,checkedAt:new Date().toISOString(),runId:gate.runId,causalOrder:'Owned live sleep60 birth identity gate completed before ordinary UI Stop click',source:'Independent postsettlement ps by exact preStop birth-identity chain',checks};
|
||||
const output={verified:true,checkedAt:new Date().toISOString(),runId:gate.runId,causalOrder:'Owned live sleep60 birth identity gate completed before ordinary UI Stop click',source:'Successful complete ps snapshot checked against exact preStop birth-identity chain',checks};
|
||||
await writeFile(path.join(directory,'independent-birth-settlement-proof.json'),JSON.stringify(output,null,2));console.log(JSON.stringify(output,null,2));
|
||||
+64
@@ -0,0 +1,64 @@
|
||||
{
|
||||
"schema": "paperclip.retained-successful-process-snapshot.v1",
|
||||
"scope": "Read-only selection from original executed successful full process-list monitor; no new live journey",
|
||||
"sourceRevision": "914129afb141d81c0d6d8c5f0ea11c20bd69a3da",
|
||||
"runId": "51ad4429-6d34-4df9-bdb9-a54b91842d5a",
|
||||
"originalMonitorOutputSha256": "a221f6dc6dc80b91f4660808587d296719b07d08a7fc2e516cd35718fe93ba40",
|
||||
"exactExecutedMonitorSha256": "4f003d4db2ce70006028265735ecb10151710fccb8b6c701837ba1a21b150ab0",
|
||||
"complete": true,
|
||||
"startedAtMs": 1793751091096,
|
||||
"finishedAtMs": 1793751143470,
|
||||
"checks": [
|
||||
{
|
||||
"pid": 90624,
|
||||
"parentPid": 58087,
|
||||
"processGroupId": 90624,
|
||||
"state": "Ss",
|
||||
"birthTime": "Tue Nov 3 18:12:05 2026",
|
||||
"executable": "paperclip-runnerd",
|
||||
"firstSeen": 1793751126043,
|
||||
"lastSeen": 1793751139287,
|
||||
"aliveAtFinalSnapshot": false,
|
||||
"matchesCapturedPreStopBirthTime": true
|
||||
},
|
||||
{
|
||||
"pid": 90674,
|
||||
"parentPid": 90624,
|
||||
"processGroupId": 90674,
|
||||
"state": "S",
|
||||
"birthTime": "Tue Nov 3 18:12:06 2026",
|
||||
"executable": "(.paperclip-verif)",
|
||||
"firstSeen": 1793751126605,
|
||||
"lastSeen": 1793751139287,
|
||||
"aliveAtFinalSnapshot": false,
|
||||
"matchesCapturedPreStopBirthTime": true
|
||||
},
|
||||
{
|
||||
"pid": 90708,
|
||||
"parentPid": 90674,
|
||||
"processGroupId": 90674,
|
||||
"state": "S",
|
||||
"birthTime": "Tue Nov 3 18:12:06 2026",
|
||||
"executable": "(launch)",
|
||||
"firstSeen": 1793751126970,
|
||||
"lastSeen": 1793751139287,
|
||||
"aliveAtFinalSnapshot": false,
|
||||
"matchesCapturedPreStopBirthTime": true
|
||||
},
|
||||
{
|
||||
"pid": 92068,
|
||||
"parentPid": 90708,
|
||||
"processGroupId": 92068,
|
||||
"state": "Ss",
|
||||
"birthTime": "Tue Nov 3 18:12:17 2026",
|
||||
"executable": "sleep",
|
||||
"firstSeen": 1793751137985,
|
||||
"lastSeen": 1793751138192,
|
||||
"aliveAtFinalSnapshot": false,
|
||||
"matchesCapturedPreStopBirthTime": true
|
||||
}
|
||||
],
|
||||
"successBoundary": "Executed monitor assigns finalOutput=execFileSync(ps,[-axo,pid=,comm=]) without a catch before computing aliveAtFinalSnapshot and saving this complete result",
|
||||
"limit": "Original full ps stdout was not retained separately; the successful monitor output, exact executed source hash and exit receipt establish the recorded PID membership. No later same-number PID probe is substituted.",
|
||||
"executableLabelLimit": "macOS comm labels can be truncated or parenthesized; captured PID, birth time and parent chain supply identity, not literal equality of comm and full executable labels."
|
||||
}
|
||||
@@ -7,8 +7,8 @@
|
||||
"files": [
|
||||
{
|
||||
"path": "README.md",
|
||||
"bytes": 6764,
|
||||
"sha256": "e8c050cb081f4aabef32a3a832bcd90921811b4cc6adc54eb4d00f9a75a58921"
|
||||
"bytes": 7357,
|
||||
"sha256": "349c8a64444f41f06942959950afd0f480c724ee539e8a8b76523be5a3481050"
|
||||
},
|
||||
{
|
||||
"path": "after-live-stop.png",
|
||||
@@ -48,12 +48,12 @@
|
||||
{
|
||||
"path": "future-helper-review-regressions.json",
|
||||
"bytes": 2064,
|
||||
"sha256": "234994fab5fa62ee849bc76c6d4f12de3558bae299c4d91c701d4889bd504781"
|
||||
"sha256": "1a13f8703bcace769a087499714952be7caaadbd24f7af326147d7ee065e397f"
|
||||
},
|
||||
{
|
||||
"path": "future-helper-streaming-regressions.json",
|
||||
"bytes": 765,
|
||||
"sha256": "201949be1dc6992410373f84c0feb387d28662062c96a6f01b54eea63f689022"
|
||||
"bytes": 831,
|
||||
"sha256": "26d5b8af0a13f79b20b8ce672f83abb2e73a9033887b38bedbd0108cdf04df8b"
|
||||
},
|
||||
{
|
||||
"path": "helper-source-hashes.json",
|
||||
@@ -62,13 +62,13 @@
|
||||
},
|
||||
{
|
||||
"path": "helpers/live-browser-streaming-check.mjs",
|
||||
"bytes": 7179,
|
||||
"sha256": "3f26b2df27f9f40e5f941706a38623cf391fe92bb4091f903745a17d963e315c"
|
||||
"bytes": 7906,
|
||||
"sha256": "75965e259fe067ebec45f3f6de87458f568baa1820cf4e8ecd77f8dd1ff74e49"
|
||||
},
|
||||
{
|
||||
"path": "helpers/live-browser.future.mjs",
|
||||
"bytes": 13230,
|
||||
"sha256": "5ff4cd9b187829e706ff770b96a110f654b087c92a5a97c8f4ce0a1319dcf2be"
|
||||
"bytes": 13141,
|
||||
"sha256": "d5aac1e02c23fd294a4bcb11c0441f92f2c2c7309d9f4f4b4c3e2030de9456a2"
|
||||
},
|
||||
{
|
||||
"path": "helpers/live-browser.guarded.mjs",
|
||||
@@ -92,8 +92,8 @@
|
||||
},
|
||||
{
|
||||
"path": "helpers/verify-birth-settlement.mjs",
|
||||
"bytes": 1752,
|
||||
"sha256": "3eca4c0bd960e87225fee9c74a6f9ef0eb785b3d3766bc8435900aaa266d4d46"
|
||||
"bytes": 2308,
|
||||
"sha256": "6072f66a8fedb085f6a321a11c90fbc98b4ee95dfc9b4c3c42010feaf0ae342e"
|
||||
},
|
||||
{
|
||||
"path": "independent-birth-settlement-proof.json",
|
||||
@@ -115,6 +115,11 @@
|
||||
"bytes": 6764,
|
||||
"sha256": "8e68827a9b29107202f5529f0f7ef3d59751336a0922fa8bc5efccefdf0d2ee6"
|
||||
},
|
||||
{
|
||||
"path": "independent-successful-process-snapshot.json",
|
||||
"bytes": 2584,
|
||||
"sha256": "ee4a8658657b965e666d71d3bd214a15d896911d6a84dcfb9a181763ebdc429a"
|
||||
},
|
||||
{
|
||||
"path": "live-browser-stop-gate.patch",
|
||||
"bytes": 1542,
|
||||
@@ -142,13 +147,13 @@
|
||||
},
|
||||
{
|
||||
"path": "public-projections.json",
|
||||
"bytes": 1955,
|
||||
"sha256": "ead153cab4eb082664e48b4f92cd9ee844c67342823c9bd7a016c8682f7efea6"
|
||||
"bytes": 2533,
|
||||
"sha256": "766392319226de5a2338fa9c7127f24305036580b77ce00e15a40cb394e21dc4"
|
||||
},
|
||||
{
|
||||
"path": "qualification.json",
|
||||
"bytes": 11671,
|
||||
"sha256": "48ed5f00aa5ee8beceae638217c7df5e8db14841ffb241ba80110292e0555dad"
|
||||
"bytes": 11814,
|
||||
"sha256": "210a6c1e2ea617cf07ba3269490bb4c356988e75b5527989bc5641d8159539c8"
|
||||
},
|
||||
{
|
||||
"path": "retained-progressive-streaming-proof.json",
|
||||
@@ -159,6 +164,11 @@
|
||||
"path": "runtime-provenance.json",
|
||||
"bytes": 1226,
|
||||
"sha256": "de542183cddfa7e64fc9eb47064ec91ac2e98a1780a07fe62a8ab7bc31495a01"
|
||||
},
|
||||
{
|
||||
"path": "verification-follow-up.json",
|
||||
"bytes": 1676,
|
||||
"sha256": "1c87e25e957f032ac3462441a8b20a6ca07985bf5e0e396fed8e49c1596f36ff"
|
||||
}
|
||||
]
|
||||
}
|
||||
+11
-5
@@ -1,5 +1,5 @@
|
||||
{
|
||||
"scope": "Public metadata/path-only helper adaptations; not a new live acceptance run",
|
||||
"scope": "Declared public metadata/path adaptations and offline verification follow-ups; not a new live acceptance run",
|
||||
"projections": [
|
||||
{
|
||||
"path": "cleanup.json",
|
||||
@@ -15,8 +15,8 @@
|
||||
},
|
||||
{
|
||||
"path": "future-helper-review-regressions.json",
|
||||
"originalSha256": "81a64f34ab1e635309d2f71f4bb55072e5f7b860e349a44efecc8e10ef5b79c9",
|
||||
"publicSha256": "234994fab5fa62ee849bc76c6d4f12de3558bae299c4d91c701d4889bd504781",
|
||||
"originalSha256": "4c4752d1211431a1e3ea94887a8ff2e2cfee440f3af80d6b93f89a5419b9b49e",
|
||||
"publicSha256": "1a13f8703bcace769a087499714952be7caaadbd24f7af326147d7ee065e397f",
|
||||
"reason": "Remove a local evidence-directory path; controlled regression results and source hashes unchanged"
|
||||
},
|
||||
{
|
||||
@@ -28,8 +28,14 @@
|
||||
{
|
||||
"path": "helpers/verify-birth-settlement.mjs",
|
||||
"originalSha256": "ce6a5aca9a0fdeef21c0e5b785d2d3ac643da62c992465f8b9b4e4c41b947cef",
|
||||
"publicSha256": "3eca4c0bd960e87225fee9c74a6f9ef0eb785b3d3766bc8435900aaa266d4d46",
|
||||
"reason": "Public reproduction path adaptation only; executed private source SHA remains in execution receipt; adaptation was not used for another live claim"
|
||||
"publicSha256": "6072f66a8fedb085f6a321a11c90fbc98b4ee95dfc9b4c3c42010feaf0ae342e",
|
||||
"reason": "Public evidence-directory argument adaptation plus reviewed process-list failure handling correction; original executed private source SHA stays in its receipt; corrected copy has five offline source regressions and was not used for a new live claim"
|
||||
},
|
||||
{
|
||||
"path": "qualification.json",
|
||||
"originalSha256": "48ed5f00aa5ee8beceae638217c7df5e8db14841ffb241ba80110292e0555dad",
|
||||
"publicSha256": "210a6c1e2ea617cf07ba3269490bb4c356988e75b5527989bc5641d8159539c8",
|
||||
"reason": "Only explicitly unexecuted future-helper metadata updated for the reviewed delta-without-start correction and six offline cases; original live observations and executed source hashes unchanged"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -283,11 +283,13 @@
|
||||
"futureVerificationHelper": {
|
||||
"executedInThisLiveJourney": false,
|
||||
"scope": "Verification-only selector/open-item/finalizing guards; offline regression, no new app/provider run",
|
||||
"helperSha256": "5ff4cd9b187829e706ff770b96a110f654b087c92a5a97c8f4ce0a1319dcf2be",
|
||||
"helperSha256": "d5aac1e02c23fd294a4bcb11c0441f92f2c2c7309d9f4f4b4c3e2030de9456a2",
|
||||
"regressionCases": {
|
||||
"focusedDomAndSource": 5,
|
||||
"focusedDomAndSource": 6,
|
||||
"existingControlledSeams": 9
|
||||
}
|
||||
},
|
||||
"previousHelperSha256": "5ff4cd9b187829e706ff770b96a110f654b087c92a5a97c8f4ce0a1319dcf2be",
|
||||
"followUp": "verification-follow-up.json"
|
||||
},
|
||||
"streamingEvidenceScope": "Original selected source52 was preterminal at workspace_finalizing. Earlier retained JPEG/DOM/source excerpts establish progressive visible content before subsequent ordinary deltas and result acceptance; no literal provider-active nativePhase is asserted."
|
||||
}
|
||||
+25
@@ -0,0 +1,25 @@
|
||||
{
|
||||
"schema": "paperclip.live-verification-review-follow-up.v1",
|
||||
"checkedAt": "2026-11-04T01:03:21.963140+00:00",
|
||||
"scope": "Offline source/DOM verification only; no app, provider, database or cloud resource restarted",
|
||||
"issues": [
|
||||
{
|
||||
"reviewComment": 4462307523,
|
||||
"correction": "Treat first ordinary delta as opening an item; retain matching close, run-finalization and accepted/terminal checks. OpenCode does not require item.started for text.",
|
||||
"regression": "Six offline real-browser checks pass, including actual retained OpenCode source29/30 without invented item.started.",
|
||||
"helperSha256": "d5aac1e02c23fd294a4bcb11c0441f92f2c2c7309d9f4f4b4c3e2030de9456a2"
|
||||
},
|
||||
{
|
||||
"reviewComment": 4462307531,
|
||||
"correction": "Require a successful complete process-list read with verifier PID; all command failures propagate.",
|
||||
"regression": "Five actual-source checks pass: original false-positive reproduced; successful absent snapshot; command failures status1/status2/ENOENT/EACCES/SIGTERM rejected; malformed/empty/incomplete snapshot rejected; live captured PID rejected.",
|
||||
"helperSha256": "6072f66a8fedb085f6a321a11c90fbc98b4ee95dfc9b4c3c42010feaf0ae342e",
|
||||
"testSha256": "3783686a4e3334a8ada07ec157b201c28d07412c7746b73d26a807ba230f9aeb",
|
||||
"independentOriginalEvidence": "independent-successful-process-snapshot.json"
|
||||
}
|
||||
],
|
||||
"limits": [
|
||||
"These helper corrections do not rewrite the exact original live journey source or create a second live execution claim.",
|
||||
"The independent successful full process snapshot is retained from the original journey, not produced by the corrected checker."
|
||||
]
|
||||
}
|
||||
@@ -173,4 +173,6 @@ database invocation must retain its own revision/environment/cleanup receipt.
|
||||
|
||||
The current [914 functional packet](../../doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/README.md) records the exact executed helper hashes, real model/tool/result/reload, and independent live-process Stop settlement. Its initially selected source52 observation was preterminal during finalization. Original retained JPEGs/DOM/source timestamps independently prove earlier progressive content before later deltas/result; the retained UI response omitted native phase. The original executed helpers and historical proof manifests remain unchanged.
|
||||
|
||||
The current browser helper includes the owned thread's normal live interstitial and live-text branches and requires an open started ordinary item plus a fresh running full GET, rejecting known finalization, accepted results, terminal state and closed items. These follow-up checks were verified offline and were not retroactively executed in the recorded journey. Run the five DOM/source checks explicitly with `node --test scripts/native-ingestion/live-browser-streaming-check.mjs`; this starts an offline browser with network aborted and belongs outside default Vitest. The current nine-case source-seam checker is `live-helper-contract-check.mjs`; the archived historical checker and receipt retain their original bytes.
|
||||
The current browser helper includes the owned thread's normal live interstitial and live-text branches and requires an ordinary item opened by a delta plus a fresh running full GET, rejecting known finalization, accepted results, terminal state and closed items. These follow-up checks were verified offline and were not retroactively executed in the recorded journey. Run the six DOM/source checks explicitly with `node --test scripts/native-ingestion/live-browser-streaming-check.mjs`; this starts an offline browser with network aborted and belongs outside default Vitest. The current nine-case source-seam checker is `live-helper-contract-check.mjs`; the archived historical checker and receipt retain their original bytes.
|
||||
|
||||
The current process-settlement reproduction check rejects failed, empty or incomplete process-list reads. Run its five source regressions explicitly with `node --test scripts/native-ingestion/live-process-check.mjs`. The current proof packet separately retains the original successful full-process monitor result; the helper fix makes no new live execution claim.
|
||||
@@ -54,7 +54,7 @@ test('hidden output, human prompts and a different thread cannot prove owned liv
|
||||
});
|
||||
|
||||
async function observation({ phase = 'observed', status = 'running', finishedAt = null,
|
||||
accepted = false, completedItem = false, startedItem = true, terminal = false, kind = 'agentMessage' } = {}) {
|
||||
accepted = false, completedItem = false, startedItem = false, terminal = false, kind = 'agentMessage', recordedOpenCode = false } = {}) {
|
||||
const start = source.indexOf(' const streaming = await poll(');
|
||||
const end = source.indexOf(' await page.screenshot(', start);
|
||||
const normalizer = source.split('\n').find(line => line.startsWith('const normalizeVisible = '));
|
||||
@@ -62,10 +62,11 @@ async function observation({ phase = 'observed', status = 'running', finishedAt
|
||||
assert.ok(start >= 0 && end > start && normalizer && summaryStart >= 0 && summaryEnd > summaryStart);
|
||||
const execute = new AsyncFunction('poll', 'taskRuns', 'visibleAgentText', 'eventsFor', 'api', 'created', 'journey', 'expect',
|
||||
normalizer + '\n' + source.slice(summaryStart, summaryEnd) + '\n' + source.slice(start, end) + '\nreturn journey;');
|
||||
const active = { id: 'owned-run', status: 'running', runtimeMode: 'native' };
|
||||
const recorded = recordedOpenCode ? JSON.parse(await readFile(new URL('../../doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/retained-progressive-streaming-proof.json', import.meta.url), 'utf8')) : null;
|
||||
const active = { id: recorded?.runId ?? 'owned-run', status: 'running', runtimeMode: 'native' };
|
||||
const identity = { runId: active.id, sourceInstanceId: 'source-1', turnId: 'turn-1', itemId: 'item-1' };
|
||||
const row = event => ({ eventType: event.eventType, payload: { prpEvent: { ...identity, ...event } } });
|
||||
const events = [
|
||||
const events = recorded ? recorded.matchedAndLaterOrdinaryEvents.slice(0, 2) : [
|
||||
...(startedItem ? [row({ eventType: 'item.started', sourceEventId: 'started' })] : []),
|
||||
row({ eventType: 'item.delta', sourceEventId: 'delta-29', sourceSeq: 29, payload: { kind, text: 'The progress message was sent and' } }),
|
||||
...(completedItem ? [row({ eventType: 'item.completed' })] : []),
|
||||
@@ -81,7 +82,7 @@ async function observation({ phase = 'observed', status = 'running', finishedAt
|
||||
value => ({ toBe(expected) { assert.equal(value, expected); } }));
|
||||
}
|
||||
|
||||
test('fresh running observation with a still-open ordinary item proves streaming', async () => {
|
||||
test('fresh running observation with an ordinary delta and no close proves streaming', async () => {
|
||||
const value = await observation();
|
||||
assert.equal(value.visibleBeforeTerminal, true);
|
||||
assert.equal(value.visibleDelta.sourceEventId, 'delta-29');
|
||||
@@ -92,7 +93,15 @@ test('fresh running observation with a still-open ordinary item proves streaming
|
||||
|
||||
test('known finalization, accepted results, terminal state and closed items are rejected', async () => {
|
||||
for (const options of [{ phase: 'workspace_finalizing' }, { accepted: true }, { terminal: true },
|
||||
{ completedItem: true }, { startedItem: false }, { status: 'succeeded' }, { finishedAt: 'fixture-time' }, { kind: 'reasoning' }]) {
|
||||
{ completedItem: true }, { status: 'succeeded' }, { finishedAt: 'fixture-time' }, { kind: 'reasoning' }]) {
|
||||
await assert.rejects(observation(options), /streaming_observation_not_proven/);
|
||||
}
|
||||
});
|
||||
|
||||
// OpenCode's real driver emits ordinary deltas without an item.started event.
|
||||
// The retained normalized rows carry actual run/source/item identities.
|
||||
test('recorded OpenCode ordinary deltas qualify without a synthetic item start', async () => {
|
||||
const value = await observation({ recordedOpenCode: true });
|
||||
assert.equal(value.visibleDelta.sourceSeq, 29);
|
||||
assert.equal(value.visibleBeforeTerminal, true);
|
||||
});
|
||||
@@ -89,7 +89,6 @@ try {
|
||||
const delta = events.find(event => event?.eventType === "item.delta" && event.runId === active.id
|
||||
&& event.payload.kind === "agentMessage" && normalizeVisible(event.payload.text).length >= 12
|
||||
&& text.includes(normalizeVisible(event.payload.text)) && typeof event.itemId === "string"
|
||||
&& events.some(item => item.eventType === "item.started" && sameItem(item, event))
|
||||
&& !events.some(item => ["item.completed", "item.failed", "item.cancelled"].includes(item.eventType) && sameItem(item, event)));
|
||||
if (!delta) return null;
|
||||
// Re-read status after observing the matching UI text. A completed run
|
||||
|
||||
@@ -0,0 +1,46 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import { test } from 'node:test';
|
||||
import { readFile } from 'node:fs/promises';
|
||||
import { execFileSync } from 'node:child_process';
|
||||
|
||||
const file = 'doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/helpers/verify-birth-settlement.mjs';
|
||||
const source = await readFile(new URL('../../' + file, import.meta.url), 'utf8');
|
||||
const before = execFileSync('git', ['show', 'd05f9b0db71eaee823b72136ae54611b12a72f35:' + file], { encoding: 'utf8' });
|
||||
const gate = { chain: [{ pid: 101, birthTime: 'fixture-birth', executable: 'fixture-provider' }] };
|
||||
function check(text, ps) {
|
||||
const start = text.includes('const processList = ') ? text.indexOf('const processList = ') : text.indexOf('const checks=gate.chain.map');
|
||||
const end = text.indexOf('const output=', start);
|
||||
assert.ok(start >= 0 && end > start);
|
||||
return new Function('execFileSync', 'assert', 'process', 'gate', text.slice(start, end) + '\nreturn checks;')(
|
||||
ps, assert, { pid: 777 }, gate,
|
||||
);
|
||||
}
|
||||
const failure = details => Object.assign(new Error('injected process-list failure'), details);
|
||||
|
||||
test('original checker falsely accepted a failed ps call', () => {
|
||||
const result = check(before, () => { throw failure({ status: 1 }); });
|
||||
assert.equal(result[0].absent, true);
|
||||
});
|
||||
test('successful complete snapshot proves absent captured PIDs', () => {
|
||||
let calls = 0;
|
||||
const result = check(source, (command, args) => {
|
||||
calls++;
|
||||
assert.equal(command, 'ps');
|
||||
assert.deepEqual(args, ['-axo', 'pid=,ppid=,stat=,lstart=,comm=']);
|
||||
return ' 777 1 S Tue Nov 3 18:00:00 2026 node\n 888 1 S Tue Nov 3 18:00:00 2026 ps\n';
|
||||
});
|
||||
assert.equal(calls, 1);
|
||||
assert.equal(result[0].absent, true);
|
||||
});
|
||||
test('process-list command failures never prove absence', () => {
|
||||
for (const details of [{ status: 1 }, { status: 2 }, { code: 'ENOENT' }, { code: 'EACCES' }, { signal: 'SIGTERM', status: null }]) {
|
||||
const error = failure(details);
|
||||
assert.throws(() => check(source, () => { throw error; }), value => value === error);
|
||||
}
|
||||
});
|
||||
test('empty, malformed and incomplete snapshots fail closed', () => {
|
||||
for (const text of ['', 'invalid row\n', ' 888 1 S ps\n']) assert.throws(() => check(source, () => text));
|
||||
});
|
||||
test('a captured process still present rejects settlement', () => {
|
||||
assert.throws(() => check(source, () => ' 777 1 S node\n 101 1 S fixture-provider\n'), /remains present/);
|
||||
});
|
||||
Reference in new issue
Block a user