mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-08 01:57:31 +01:00
Fix Pi failure notices and internal file completion
Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
0d65753fe1
commit
f62b8510a8
9 files changed
+220
-60
No files matched your search
@@ -13,6 +13,13 @@ remote workspace. Supply a workspace-relative `contentRef`, basename `filename`,
|
||||
The tool verifies the file, stores an attachment and artifact work product, and
|
||||
binds it to the response. Generic API tools and a legacy API key are unnecessary.
|
||||
|
||||
Managed personal memory and a file explicitly requested only for internal
|
||||
verification do not require a downloadable attachment. The completion gate uses
|
||||
the current task objective to distinguish those from requested file outputs;
|
||||
an internal file cannot excuse a separate download request. Expected denied
|
||||
native writes in a negative test also do not represent a requested deliverable.
|
||||
Requested downloads still require verified accessible publication evidence.
|
||||
|
||||
Wait for the receipt. It includes `attachmentId`, `contentPath`, and
|
||||
`downloadPath`, along with the existing command, revision, entity references,
|
||||
and disposition. Reuse the original key after an ambiguous result. A receipt
|
||||
|
||||
@@ -21,13 +21,13 @@ qualification. The existing draft stack must be reviewed in dependency order.
|
||||
|
||||
| Gate | Acceptance evidence | Current result |
|
||||
| --- | --- | --- |
|
||||
| Runtime identity and admission | Exact runtime/profile/model; verified effective thinking; startup below the 60-second admission limit; unsupported configuration fails before a prompt | Normal public installation and credential-free profile-14 admission pass at frozen runtime `b148b73ea` on ARM Mac (8.043 s), Intel Mac (37.681 s), and Linux (5.698 s). No prompt or binary override is used. |
|
||||
| Runtime identity and admission | Exact runtime/profile/model; verified effective thinking; startup below the 60-second admission limit; unsupported configuration fails before a prompt | Normal public installation and credential-free profile-14 admission pass at shipping source `0d65753fe` on ARM Mac (7.788 s), Intel Mac (31.185 s), and Linux (8.071 s). No prompt or binary override is used. Subsequent notice/completion corrections require fresh qualification. |
|
||||
| Live lifecycle | Pending native question survives controller restart in the original process; one creation and resolution; stale answer after provider death rejected; Stop retires owned processes; three warm turns keep process identity | Installed profile-14 same-turn steering passes at `dc2b053f3`. Restart and three-turn warm continuity pass at historical `f5f57e380`; pending-permission Stop and stale response rejection pass at `03dd6ef93`. Controller restart passes at frozen runtime `b148b73ea`. Warm continuity fails on turn 2 with `run.attach` timeout; the campaign stops. Remaining final-candidate lifecycle and provider-death evidence remain required. |
|
||||
| Product workflows | All 26 explicit Pi Product E2E cells (13 local, 13 Daytona); screenshots, public state, independent artifacts, terminal and cleanup evidence | Frozen `b148b73ea` campaign has one local pass and one retained local failure. Eleven local cells are unexecuted. All 13 Daytona cells remain held for the immutable image. The full 26-cell gate is incomplete. |
|
||||
| Product workflows | All 26 explicit Pi Product E2E cells (13 local, 13 Daytona); screenshots, public state, independent artifacts, terminal and cleanup evidence | At `0d65753fe`, five local cells pass: warm continuity, controller restart, pending-permission Stop, steering, and native questions. Agent-files times out after completion is rejected. The campaign closes; seven local cells and all 13 Daytona cells remain unexecuted. A subsequent shipping correction needs fresh qualification. |
|
||||
| Runner protocol | Pi roster passes through the native packaged runner and authenticated mock control plane; lifecycle/denial/control cases remain distinct from Product tests | All seven cases pass through installed `b148b73ea` with pinned definitions `a9e0e7e0`, profile 14 and native-confirmed low thinking. The governed-wait retry follows the tested idle-close correction. Each case has one attempt and zero automatic retries. Its original failure remains unchanged. |
|
||||
| Installed distribution | Public CLI/server tars on ARM Mac, Intel Mac and Linux; normal Pi setup; exact Linux companion and immutable Daytona image imported without binary override | ARM, Intel and Linux public installation pass at `b148b73ea`. Exact-source Linux companion and immutable Daytona proof remain required. Image job 37099122706 is queued on EC2; retain that handle. Its older superseded image job is terminal/cancelled. |
|
||||
| Installed distribution | Public CLI/server tars on ARM Mac, Intel Mac and Linux; normal Pi setup; exact Linux companion and immutable Daytona image imported without binary override | ARM, Intel and Linux public installation pass at `0d65753fe`. Exact-source Linux companion and immutable Daytona proof remain required. Image job 37102904889 remains queued on EC2. Its predecessor 37099122706 is terminal/cancelled. Any shipping correction requires source-bound replacement after this handle is terminal. |
|
||||
| Governance and spend | Company isolation, human-only permission, duplicate/stale answers, Stop and budget hard stop; pricing estimates never become claimed bills | 58 focused governance/cost/session invariants pass at `f5f57e380`. Live pending-permission Stop passes at `03dd6ef93`. The qualification key cap is not proof of Paperclip budget enforcement. |
|
||||
| Integration and rollout | Review each prerequisite; final-head typecheck, tests, build and CI pass; no unresolved review; exact artifacts; rollback recorded | All 55 CI checks and Greptile 5/5 pass at `3d75626bf`, which differs from frozen shipping source only in the Linux fence test. Two #14924 UI findings have tested downstream corrections. Prerequisite PRs #14921–#14924 remain open; qualification and review dispositions remain required. Later edits require fresh-head CI/review. No merge or release. |
|
||||
| Integration and rollout | Review each prerequisite; final-head typecheck, tests, build and CI pass; no unresolved review; exact artifacts; rollback recorded | All 55 CI checks and Greptile 5/5 pass at shipping source `0d65753fe`. Two #14924 UI findings have tested downstream corrections. Prerequisite PRs #14921–#14924 remain open; qualification and review dispositions remain required. Later edits require fresh-head CI/review. No merge or release. |
|
||||
|
||||
## Bounded execution
|
||||
|
||||
@@ -336,9 +336,13 @@ prices are estimates. Unpriced usage must stay unpriced in the ledger and UI.
|
||||
533 passed files and 163 skipped files. Private Postgres library symlinks are
|
||||
missing, and the restricted test PATH omits macOS lsof. Both setup causes are
|
||||
corrected only in the task-owned dependency environment. The focused DB and
|
||||
process-owner checks pass 95 tests with three existing skips. The full suite
|
||||
then reruns with the corrected environment. The failed log remains intact; it is not
|
||||
a passing full local test or complete integration claim.
|
||||
process-owner checks pass 95 tests with three existing skips. The corrected
|
||||
broad run completes with 738 passed files, four skipped files, 14,984 passed
|
||||
tests and 84 skips. One native integration suite cannot start because Cargo
|
||||
is absent from the restricted PATH. With the pinned Rust toolchain added,
|
||||
that native Codex result/resume integration test passes. Both failed broad
|
||||
logs remain intact; neither is a passing full-command claim. The final-source
|
||||
run must include the pinned Rust toolchain from the start.
|
||||
- The local Product controller-restart cell passes with canonical evidence.
|
||||
The next cell, warm-three-turn, completes turn 1 but fails before turn 2
|
||||
provider work: native `run.attach` reaches its 30-second command timeout.
|
||||
@@ -357,21 +361,78 @@ prices are estimates. Unpriced usage must stay unpriced in the ledger and UI.
|
||||
controller uses Pi's existing absolute 60 s cold-process admission budget for
|
||||
warm attachment and aborts admission on close. The same fixture then completes
|
||||
turn 2, retains one Runner, starts exactly two sidecars without a retry, and
|
||||
passes strict cleanup. TypeScript passes; the full transport/eval-session
|
||||
suite is running. The native 60 s bound and profile-14 bytes are unchanged.
|
||||
passes strict cleanup. TypeScript, all 201 transport tests, and 37 eval-session
|
||||
contract/entrypoint tests pass. The native 60 s bound and profile-14 bytes are unchanged.
|
||||
The paid three-turn failure remains failed. Fresh exact-source packaging and
|
||||
one explicit paid retry are still required; no live resolution is claimed.
|
||||
- All 55 CI checks and Greptile 5/5 are terminal at `96a0e329b`, with zero
|
||||
unresolved draft-PR threads. The immutable image job 37099122706 remains
|
||||
queued. Track it without a duplicate dispatch. Qualification, prerequisites
|
||||
and image/companion gates still hold production.
|
||||
- The corrected shipping candidate at `0d65753fe` passes full build and typecheck,
|
||||
normal public CLI/server packaging and fresh Runner pack/vendor equivalence.
|
||||
ARM, Intel and Linux public installation pass in 7.788, 31.185 and 8.071 s.
|
||||
The prior candidate manifests and Runner tar are preserved.
|
||||
- All 55 CI checks and Greptile 5/5 are terminal at `96a0e329b`. Greptile is
|
||||
also 5/5 at `0d65753fe`; all 55 current-head CI checks are terminal and green. The superseded
|
||||
immutable image job 37099122706 is canceled because shipping inputs changed.
|
||||
Its one replacement, 37102904889, is authorized and queued for the corrected
|
||||
source. Track this exact handle without another dispatch. Qualification,
|
||||
prerequisites and image/companion gates still hold production.
|
||||
|
||||
## Qualification follow-up — 2026-10-03 02:15 CDT
|
||||
|
||||
- The explicitly authorized corrected-source warm retry at `0d65753fe` passes
|
||||
all nine matchers with three succeeded turns, one Runner process and session,
|
||||
exact independent file bytes, and strict cleanup. The original `b148b73ea`
|
||||
warm timeout stays failed. No automatic paid retry occurs.
|
||||
- The final-source local matrix then passes controller restart,
|
||||
pending-permission Stop, same-turn steering, and native questions. Its next
|
||||
agent-files attempt fails with `native_session_interrupted` after the unchanged
|
||||
120-second bound; cleanup passes. The canonical grade remains
|
||||
`transient_infrastructure`. Retained tool outcomes prove the managed-memory
|
||||
write/read and expected cross-root denial succeeded. Completion is rejected
|
||||
because the server incorrectly recognizes these internal/negative-test
|
||||
instructions as a requested downloadable output. The provider continues after
|
||||
that rejection until the timeout. Treat the completion false positive as the
|
||||
observed product cause; preserve the machine classifier separately.
|
||||
- The campaign closes after that failure. Its held coordinator is retired only
|
||||
after the provider attempt is terminal and cleanup has passed. The key keeps
|
||||
its $5 lifetime cap, zero BYOK usage and about $4.87 remaining. Its API deltas
|
||||
remain provisional. Five local passes are retained, without regrading the
|
||||
agent-files failure. No paid retry is authorized by a presentation-only change.
|
||||
- The prerequisite #14921 exit/catch notice finding is reproduced through the
|
||||
actual extension-turn binding: two identical failure inputs produce two
|
||||
canonical notices. The correction coalesces only consecutive identical
|
||||
display notices within one turn. Different reasons/severity, intervening retry
|
||||
activity and later turns remain observable. The patched wrapper, profile 14,
|
||||
native deadlines, terminal events and approval authority remain unchanged.
|
||||
All 115 extension/driver regressions pass.
|
||||
- The completion regression reproduces four false positives without model calls:
|
||||
file-tool names, managed personal memory, an explicitly internal assertion
|
||||
file, and an expected denied native-write attempt. The correction preserves
|
||||
actual downloadable output requirements, mixed requests, and publication
|
||||
evidence enforcement. All 59 unit and database integration tests pass. A free replay of the exact
|
||||
retained server-bound objective changes from a false download requirement to
|
||||
the intended internal outcome, with no grader or deadline changes.
|
||||
- The `0d65753fe` broad local suite is terminal with 736 passed files, four
|
||||
skipped files, 14,982 passed tests and three failures: a Git scan load
|
||||
single-flight count (497 versus 498) and two HTTP socket resets. One associated
|
||||
unhandled socket rejection is retained. This is not a passing full-command
|
||||
claim. The two socket-reset suites pass in isolation. The Git load failure repeats
|
||||
(496 joins versus 498). Its fixture creates 500 ephemeral listeners; the
|
||||
correction sends all 500 concurrent requests through one listening server and
|
||||
handles every rejection during teardown. The unchanged assertions then prove
|
||||
500 HTTP 200 responses, two scans, 498 joins and 2.4 ms health p99. All 138
|
||||
load/redaction/recovery tests pass. The full failed command remains retained.
|
||||
- Image run 37102904889 remains queued. It is not an image receipt or Daytona
|
||||
qualification. Finish the current source corrections before superseding its
|
||||
source-bound build. No workflow edit, lockfile commit, new model, fallback key,
|
||||
merge or release occurs.
|
||||
|
||||
## Remaining work in order
|
||||
|
||||
1. Retain the governed-wait failure and its corrected-source pass. Diagnose
|
||||
the new completed-turn warm-attachment failure without paid retries. Make
|
||||
and prove a concrete correction, then rebuild exact-source public packages
|
||||
before one explicit warm-continuity retry. Recheck final-source lifecycle and
|
||||
1. Retain the governed-wait failure and its corrected-source pass. The free
|
||||
completed-turn warm-attachment correction is proved. The explicit corrected-source warm retry passes. Complete the tested
|
||||
duplicate-notice and internal-file completion corrections, then rebuild and
|
||||
verify exact-source public packages before one explicit agent-files retry.
|
||||
Recheck final-source lifecycle and
|
||||
classify the observed Mac ownership contention. Produce the normal Linux
|
||||
companion receipt,
|
||||
then import an immutable exact-source Daytona image without an override.
|
||||
|
||||
@@ -66,6 +66,11 @@ while retaining the existing Runner authority. Live adoption and ordinary
|
||||
commands retain their 30-second bounds. Closing the transport cancels admission;
|
||||
an acknowledgement received after the admission deadline cannot revive it.
|
||||
|
||||
Pi's display projection coalesces consecutive identical runtime-failure notices
|
||||
within one turn, including the exit handler and its late prompt rejection.
|
||||
Distinct failure details, intervening retry activity and later turns remain
|
||||
visible. This does not change terminal settlement or the pinned wrapper bytes.
|
||||
|
||||
Cursor candidate configuration accepts `acpxSessionMode: "agent" | "plan" | "ask"`
|
||||
(default `agent`). This selects the native Cursor mode independently of
|
||||
`acpxPermissionMode` and Paperclip task planning or company approvals. The mode
|
||||
|
||||
@@ -1,7 +1,40 @@
|
||||
import { expect, it } from "vitest";
|
||||
import { createPiProfileExtensionAdapter, PI_NOTICE_METHOD } from "./pi-extension-adapter.js";
|
||||
import { validateAcpxRichEvent } from "./profile-extensions.js";
|
||||
import { bindAcpxExtensionTurn, validateAcpxRichEvent } from "./profile-extensions.js";
|
||||
const context = { workspacePath: "/fixture", sessionId: "session", turnId: "turn" };
|
||||
it("projects the exit handler and late prompt rejection as one failure in the same turn", async () => {
|
||||
const events: unknown[] = [];
|
||||
const binding = bindAcpxExtensionTurn({
|
||||
adapter: createPiProfileExtensionAdapter(context),
|
||||
active: () => true,
|
||||
sessionId: context.sessionId,
|
||||
waitForInput: async () => { throw new Error("failure notices cannot request input"); },
|
||||
emit: event => events.push(event),
|
||||
});
|
||||
const notice = { sessionId: "session", category: "runtime_failure", severity: "error",
|
||||
summary: "Pi process exited with code 4", details: { reason: "native_process_exited" } };
|
||||
binding.onExtensionNotification(PI_NOTICE_METHOD, notice);
|
||||
binding.onExtensionNotification(PI_NOTICE_METHOD, structuredClone(notice));
|
||||
await binding.drain();
|
||||
expect(events).toHaveLength(1);
|
||||
expect(events[0]).toMatchObject({ eventType: "provider.notice.recorded",
|
||||
payload: { category: "pi.runtime_failure", severity: "error", summary: notice.summary } });
|
||||
});
|
||||
|
||||
it("keeps distinct failure details, intervening activity, and later turns observable", async () => {
|
||||
const adapter = createPiProfileExtensionAdapter(context);
|
||||
const notice = { sessionId: "session", category: "runtime_failure", severity: "error",
|
||||
summary: "Provider request failed", details: { reason: "provider_http_401" } };
|
||||
expect(await adapter.notification(PI_NOTICE_METHOD, notice)).toHaveLength(1);
|
||||
expect(await adapter.notification(PI_NOTICE_METHOD, { ...notice, details: { reason: "provider_http_429" } })).toHaveLength(1);
|
||||
expect(await adapter.notification(PI_NOTICE_METHOD, { ...notice, severity: "warning" })).toHaveLength(1);
|
||||
for (let i = 0; i < 2; i++) {
|
||||
expect(await adapter.notification(PI_NOTICE_METHOD, { ...notice, category: "auto_retry_start" })).toHaveLength(1);
|
||||
}
|
||||
expect(await adapter.notification(PI_NOTICE_METHOD, notice)).toHaveLength(1);
|
||||
expect(await createPiProfileExtensionAdapter({ ...context, turnId: "next-turn" })
|
||||
.notification(PI_NOTICE_METHOD, notice)).toHaveLength(1);
|
||||
});
|
||||
it("preserves bounded native notice severity, category and meaningful metadata without assistant output", async () => {
|
||||
const adapter = createPiProfileExtensionAdapter(context);
|
||||
for (const severity of ["info", "warning", "error"]) {
|
||||
|
||||
@@ -10,6 +10,7 @@ const safeText = (value: string, limit = 4000) => String(redactPaperclipSemantic
|
||||
* content nor authority to complete a turn, execute tools, or grant approval. */
|
||||
export function createPiProfileExtensionAdapter(context: AcpxProfileExtensionContext): AcpxProfileExtensionAdapter {
|
||||
let sequence = 0;
|
||||
let previousFailure: string | null = null;
|
||||
return {
|
||||
async request() { throw new Error("Pi has no qualified inbound extension request"); },
|
||||
async notification(method, params) {
|
||||
@@ -27,6 +28,14 @@ export function createPiProfileExtensionAdapter(context: AcpxProfileExtensionCon
|
||||
for (const key of ["success", "aborted", "willRetry", "enabled"]) if (typeof native[key] === "boolean") details.push({ name: key, value: String(native[key]) });
|
||||
for (const key of ["reason", "errorMessage"]) if (typeof native[key] === "string") details.push({ name: key, value: safeText(native[key] as string) });
|
||||
}
|
||||
// The pinned wrapper reports process exit, then the rejected prompt can
|
||||
// report that same failure again. Coalesce consecutive identical display
|
||||
// notices within this turn; distinct failures and retry activity remain.
|
||||
const failure = params.category === "runtime_failure"
|
||||
? JSON.stringify([params.summary, params.severity, details])
|
||||
: null;
|
||||
if (failure !== null && failure === previousFailure) return [];
|
||||
previousFailure = failure;
|
||||
const noticeId = `pi-notice-${createHash("sha256").update(JSON.stringify([context.sessionId, context.turnId, ++sequence])).digest("hex")}`;
|
||||
return [{ eventType: "provider.notice.recorded", itemId: noticeId, payload: {
|
||||
schema: "paperclip.provider.notice.v1", noticeId, severity: params.severity,
|
||||
|
||||
@@ -134,50 +134,65 @@ describe("workspace Git scan route load regression", () => {
|
||||
...unavailableMethods(),
|
||||
};
|
||||
const app = createLoadApp(db, companyId, service);
|
||||
const pendingResponses = Array.from({ length: 500 }, (_, index) => request(app)
|
||||
// Model one busy API server, not 500 independent ephemeral listeners whose
|
||||
// accept/close races can reset clients before they reach the scheduler.
|
||||
const server = app.listen(0, "127.0.0.1");
|
||||
await new Promise<void>(resolve => server.once("listening", resolve));
|
||||
const pendingResponses = Array.from({ length: 500 }, (_, index) => request(server)
|
||||
.get(`/api/issues/issue-${index}/file-resources/list`)
|
||||
.set("x-test-actor", `actor-${index % 73}`)
|
||||
.query({ mode: "changed" })
|
||||
.then((response) => response));
|
||||
// Observe every client rejection immediately, including on assertion failure.
|
||||
const settledResponses = Promise.allSettled(pendingResponses);
|
||||
try {
|
||||
await vi.waitFor(
|
||||
() => expect(scheduler.snapshot().totals.singleFlightJoins).toBe(498),
|
||||
{ timeout: 15_000, interval: 20 },
|
||||
);
|
||||
const loadedSnapshot = scheduler.snapshot();
|
||||
expect(loadedSnapshot).toMatchObject({ activeCount: 2, queuedCount: 0, inFlightCount: 2 });
|
||||
|
||||
await vi.waitFor(
|
||||
() => expect(scheduler.snapshot().totals.singleFlightJoins).toBe(498),
|
||||
{ timeout: 15_000, interval: 20 },
|
||||
);
|
||||
const loadedSnapshot = scheduler.snapshot();
|
||||
expect(loadedSnapshot).toMatchObject({ activeCount: 2, queuedCount: 0, inFlightCount: 2 });
|
||||
const healthLatencies: number[] = [];
|
||||
for (let index = 0; index < 25; index += 1) {
|
||||
const startedAt = performance.now();
|
||||
const response = await request(server).get("/api/health");
|
||||
healthLatencies.push(performance.now() - startedAt);
|
||||
expect(response.status).toBe(200);
|
||||
}
|
||||
healthLatencies.sort((left, right) => left - right);
|
||||
const healthP99Ms = healthLatencies[Math.ceil(healthLatencies.length * 0.99) - 1]!;
|
||||
expect(healthP99Ms).toBeLessThan(250);
|
||||
|
||||
const healthLatencies: number[] = [];
|
||||
for (let index = 0; index < 25; index += 1) {
|
||||
const startedAt = performance.now();
|
||||
const response = await request(app).get("/api/health");
|
||||
healthLatencies.push(performance.now() - startedAt);
|
||||
expect(response.status).toBe(200);
|
||||
releaseScans();
|
||||
const responses = (await settledResponses).map(result => {
|
||||
if (result.status === "rejected") throw result.reason;
|
||||
return result.value;
|
||||
});
|
||||
const outcomeCounts = responses.reduce<Record<number, number>>((counts, response) => {
|
||||
counts[response.status] = (counts[response.status] ?? 0) + 1;
|
||||
return counts;
|
||||
}, {});
|
||||
expect(outcomeCounts).toEqual({ 200: 500 });
|
||||
expect({ runnerCalls, peakActive }).toEqual({ runnerCalls: 2, peakActive: 2 });
|
||||
expect(scheduler.snapshot()).toMatchObject({ activeCount: 0, queuedCount: 0, inFlightCount: 0 });
|
||||
|
||||
console.info("workspace Git scan regression metrics", {
|
||||
requests: responses.length,
|
||||
repositories: roots.length,
|
||||
underlyingScans: runnerCalls,
|
||||
peakActive,
|
||||
peakQueued: loadedSnapshot.queuedCount,
|
||||
outcomes: outcomeCounts,
|
||||
healthP99Ms: Math.round(healthP99Ms * 100) / 100,
|
||||
unreapedChildCount: 0,
|
||||
});
|
||||
} finally {
|
||||
releaseScans();
|
||||
server.closeAllConnections();
|
||||
await settledResponses;
|
||||
await new Promise<void>((resolve, reject) => server.close(error => error ? reject(error) : resolve()));
|
||||
}
|
||||
healthLatencies.sort((left, right) => left - right);
|
||||
const healthP99Ms = healthLatencies[Math.ceil(healthLatencies.length * 0.99) - 1]!;
|
||||
expect(healthP99Ms).toBeLessThan(250);
|
||||
|
||||
releaseScans();
|
||||
const responses = await Promise.all(pendingResponses);
|
||||
const outcomeCounts = responses.reduce<Record<number, number>>((counts, response) => {
|
||||
counts[response.status] = (counts[response.status] ?? 0) + 1;
|
||||
return counts;
|
||||
}, {});
|
||||
expect(outcomeCounts).toEqual({ 200: 500 });
|
||||
expect({ runnerCalls, peakActive }).toEqual({ runnerCalls: 2, peakActive: 2 });
|
||||
expect(scheduler.snapshot()).toMatchObject({ activeCount: 0, queuedCount: 0, inFlightCount: 0 });
|
||||
|
||||
console.info("workspace Git scan regression metrics", {
|
||||
requests: responses.length,
|
||||
repositories: roots.length,
|
||||
underlyingScans: runnerCalls,
|
||||
peakActive,
|
||||
peakQueued: loadedSnapshot.queuedCount,
|
||||
outcomes: outcomeCounts,
|
||||
healthP99Ms: Math.round(healthP99Ms * 100) / 100,
|
||||
unreapedChildCount: 0,
|
||||
});
|
||||
}, 60_000);
|
||||
|
||||
it.each([
|
||||
|
||||
@@ -12,6 +12,10 @@ describe("explicit file output requirements", () => {
|
||||
"Make a file but do not send it to anyone else.",
|
||||
"Export a summary of this PDF as CSV.",
|
||||
"Create no temporary files; export the results as CSV.",
|
||||
"Write report.pdf. This is an internal verification file, not a deliverable. Also export the results as CSV.",
|
||||
"Write a downloadable report.pdf. This is personal memory, not a task deliverable.",
|
||||
"Attempt native write to report.txt with content requested.",
|
||||
"Write report.pdf and checklist.md. This is an internal assertion file, not a deliverable.",
|
||||
])("recognizes an explicit output request: %s", objective => {
|
||||
expect(explicitlyRequestsFileOutput(objective)).toBe(true);
|
||||
});
|
||||
@@ -31,6 +35,10 @@ describe("explicit file output requirements", () => {
|
||||
"Create no files.",
|
||||
"Generate no attachments and answer in chat.",
|
||||
"Write a reply without any files.",
|
||||
"Use native write/read file tools for this task, not bash or the instructions API.",
|
||||
"Write a memory entry to memory/pi-native.txt inside the registered AGENT_HOME. This is personal memory, not a task deliverable.",
|
||||
"Use native write to copy those exact bytes into pi-agent-memory-proof.txt in the task workspace. This is an internal assertion file, not a deliverable.",
|
||||
"Before finishing, attempt native write exactly once to /outside/pi-unassigned.txt with content forbidden. This intentionally unassigned root must be denied.",
|
||||
])("does not require a file for a text or source-review request: %s", objective => {
|
||||
expect(explicitlyRequestsFileOutput(objective)).toBe(false);
|
||||
});
|
||||
|
||||
@@ -63,12 +63,13 @@ async function hasCurrentPublicationReceipt(db: Db, companyId: string, receipts:
|
||||
* an output requirement or erase a user's request for a file.
|
||||
*/
|
||||
export function explicitlyRequestsFileOutput(objective: string): boolean {
|
||||
return objective.split(/(?:[.!?](?:\s|$)|\n|[;,]|\bbut\b)/iu).some(clause => {
|
||||
const sentences = objective.replaceAll("\\_", "_").split(/(?:[.!?](?:\s|$)|\n)/iu);
|
||||
return sentences.some((sentence, index) => sentence.split(/(?:[;,]|\bbut\b)/iu).some(clause => {
|
||||
const file = /\b(?:files?|attachments?|downloads?|pdf|spreadsheets?|workbooks?|slide decks?|powerpoints?|docx|xlsx|csv)\b|\b[^\s/]+\.(?:md|txt|pdf|docx?|xlsx?|csv|pptx?|png|jpe?g|svg|zip)\b/giu;
|
||||
const create = /\b(?:create|make|write|save|export|attach|send|generate|produce|prepare|provide|give|return|build)\b/iu.exec(clause);
|
||||
if (create && /\b(?:do not|don't|never|no need to)\s*$/iu.test(clause.slice(0, create.index))) return false;
|
||||
const output = create ? clause.slice(create.index + create[0].length) : "";
|
||||
const fileObject = [...output.matchAll(file)].some(match => {
|
||||
const objects = [...output.matchAll(file)].filter(match => {
|
||||
const prefix = output.slice(0, match.index);
|
||||
const suffix = output.slice(match.index + match[0].length);
|
||||
// "Create no files" is a prohibition, even though it contains a creation
|
||||
@@ -78,12 +79,18 @@ export function explicitlyRequestsFileOutput(objective: string): boolean {
|
||||
// Explicit export destinations still count after such input references.
|
||||
const destination = /\b(?:as|into|to)\s+(?:(?:a|an|the|new|separate|markdown|word|excel)\s+)*$/iu.test(prefix);
|
||||
if (!destination && /\b(?:of|about|on|from|using|for|with)\b/iu.test(prefix)) return false;
|
||||
if (/^files?$/iu.test(match[0]) && /^\s+(?:permissions?|systems?|formats?|names?|paths?|types?|sizes?|descriptors?)\b/iu.test(suffix)) return false;
|
||||
if (/^files?$/iu.test(match[0]) && /^\s+(?:tools?|permissions?|systems?|formats?|names?|paths?|types?|sizes?|descriptors?)\b/iu.test(suffix)) return false;
|
||||
return true;
|
||||
});
|
||||
return fileObject ||
|
||||
// An immediately following "This ..." can qualify one internal file, not
|
||||
// erase a separate output request. Only the authoritative objective counts.
|
||||
const internal = objects.length === 1 && /^\s*This is (?:an? )?(?:personal memory|internal (?:assertion|verification) file)\b[^.!?]*\bnot a (?:task )?deliverable\b/iu.test(sentences[index + 1] ?? "");
|
||||
const deniedAttempt = /\b(?:attempt|try)\s+(?:the\s+)?native\s+(?:write|edit)\b/iu.test(clause)
|
||||
&& /\b(?:must be denied|denial is (?:the )?expected|(?:this|the) negative test)\b/iu.test(objective);
|
||||
if ((internal || deniedAttempt) && !/\b(?:downloadable|attached|attach|export|send|provide|return|give)\b/iu.test(clause)) return false;
|
||||
return objects.length > 0 ||
|
||||
(!/\b(?:no|without)\s+(?:downloadable|attached)/iu.test(clause) && /\b(?:downloadable|attached)\s+(?:file|report|document|checklist|draft)\b/iu.test(clause));
|
||||
});
|
||||
}));
|
||||
}
|
||||
|
||||
/** Files cited as completed output must be reachable outside the agent workspace. */
|
||||
|
||||
@@ -332,6 +332,21 @@ describe("native runner file handoff", () => {
|
||||
}
|
||||
});
|
||||
|
||||
it.each([
|
||||
"Use native write/read file tools. Write memory/pi-native.txt inside AGENT_HOME. This is personal memory, not a task deliverable.",
|
||||
"Use native write to copy bytes into pi-agent-memory-proof.txt. This is an internal assertion file, not a deliverable.",
|
||||
"Attempt native write once to /outside/pi-unassigned.txt. This intentionally unassigned root must be denied.",
|
||||
])("accepts an internal file outcome without treating it as published output: %s", async objective => {
|
||||
await db.update(heartbeatRuns).set({ contextSnapshot: { issueId, executionContinuation: { objective } } })
|
||||
.where(eq(heartbeatRuns.id, runId));
|
||||
try {
|
||||
await expect(nativeCompletionFeedback(db, runId, doneReport([])))
|
||||
.resolves.toContain("Completion report accepted");
|
||||
} finally {
|
||||
await db.update(heartbeatRuns).set({ contextSnapshot: { issueId } }).where(eq(heartbeatRuns.id, runId));
|
||||
}
|
||||
});
|
||||
|
||||
it("prepares one verified same-run attachment and replays without duplicates", async () => {
|
||||
const body = Buffer.from("native runner file handoff\n", "utf8");
|
||||
await mkdir(path.join(workspaceRoot, "out"), { recursive: true });
|
||||
|
||||
Reference in new issue
Block a user