mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
fix(evals): admit the in-flight chat completion follow-up on a settled source (#21313)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat catalog evals check that a chat request reaches the correct saved outcome > - The `source-settled-with-active-work` run policy checks that the CEO chat is settled while delegated work can continue > - After a delegated task finishes, the product starts its normal `chat_task_completed` follow-up on the source chat > - The policy required every source run to be `succeeded` and the source to be `waiting`, so a re-grade during that follow-up failed > - This pull request admits that one product follow-up while it is in flight, and keeps every other source run strict > - The benefit is that the eval grades the saved private-delegation outcome, not the timing of the reload ## Linked Issues or Issue Description **What happened?** The OpenCode cell of `chat-features.runner-opencode.local.tasks.private-task` failed with "Independent oracle survives browser reload: expected true, received false". The saved state was correct. The CEO answered, Morgan saved the private draft in the private project, and the unrelated member could not read it. The grade passed for its full 5-second stability window. Morgan's run then finished, and 73 ms later the product started the CEO `chat_task_completed` follow-up on the source chat. The harness reloaded the page and re-graded while that follow-up was running. The only failed check was `catalog-settled-source-separate-work`, because it required all source runs to be `succeeded`. **Expected behavior** The case already counts this follow-up in its run bound (`expectedRunCount: 3`, `minimumRunCount: 2`). A queued or running product follow-up on an answered source must not fail the case. **Steps to reproduce** Run the private-task case. Let the worker finish within about 5 seconds of the first passing grade. The post-reload re-grade sees the follow-up as `running` and fails. ## What Changed - Add `chatCatalogSourceCompletionFollowUp` to `tests/runner-e2e/chat-catalog-run-policy.ts`. It identifies the lead's `chat_task_completed` run on the source whose every completion update names separate work in the run ledger. - In the `source-settled-with-active-work` check, admit at most one such follow-up while it is `queued`, `running` or `succeeded`. Accept a source `conversationState` of `active` only while that follow-up is in flight. - Keep the other rules strict. The original answer must exist and must have succeeded. Any other active, failed or extra source run still fails. The declared run-count bound still applies. - Also admit the follow-up before its turn is prepared. The server claims the run first and saves the completion updates after that, so a queued run, or a running run in preparation, has only its delivery IDs. It qualifies only while queued or running, when automation woke it, and after a separate worker run in the ledger succeeded. - Add a red/green test that reproduces the observed race and covers the negative cases. ## Verification - `npx vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog-run-policy.test.ts`: the new test fails before the change (1 failed, 13 passed) and passes after it (14 passed). The queued-snapshot cases were added after review. - The related runner-e2e tests (`chat-catalog-run-policy`, `chat-catalog-readiness`, `chat-catalog-failure-retirement`, `runtime-dependency`, `internal`) pass: 200 tests. - `pnpm test:e2e:runner:typecheck` passes. - Paid run [43374419951](https://github.com/paperclipai/paperclip/actions/runs/43374419951) on the first commit of this branch: `chat-features.runner-opencode.local.tasks.private-task` passed. ## Risks - Low risk. The change is in eval grading only. It does not change product code. - The check accepts a follow-up that is still running at grade time. A later failure of that follow-up is not observed by this case. Completion delivery is covered by its own completion cases. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
23c2343015
commit
74096c19f4
2 files changed
+64
-2
No files matched your search
@@ -29,6 +29,38 @@ it("admits a lead run outside the source only as a declared case handback", () =
|
||||
expect(gradeChatCatalogRuns({ ...scope, implementation: declared, runs: [succeeded, done, { ...handback, contextSnapshot: { issueId: "task", wakeReason: "issue_commented" } }] }).passed).toBe(false);
|
||||
expect(gradeChatCatalogRuns({ ...scope, implementation: { ...declared, expectedRunCount: 4 }, runs: [succeeded, done, handback, { ...handback, id: "again" }] }).passed).toBe(false);
|
||||
});
|
||||
it("admits at most one in-flight product completion follow-up on a settled source, and nothing else active there", () => {
|
||||
const implementation = { expectedRunCount: 3, minimumRunCount: 2, runPolicy: { kind: "source-settled-with-active-work" as const, maximumActiveWorkerRuns: 1 } };
|
||||
const done = { ...worker, status: "succeeded" };
|
||||
const followUp: ChatRun = { id: "follow-up", companyId: "company", agentId: "ceo", status: "running", runtimeMode: "native",
|
||||
contextSnapshot: { issueId: "source", wakeReason: "chat_task_completed", chatCompletionUpdates: [{ id: "task", status: "done" }] } };
|
||||
const active = { ...scope, source: { ...scope.source, conversationState: "active" as const } };
|
||||
// The private-task race: Morgan finished, and the product's follow-up started on the answered source.
|
||||
expect(gradeChatCatalogRuns({ ...active, implementation, runs: [succeeded, done, followUp] }).passed).toBe(true);
|
||||
expect(gradeChatCatalogRuns({ ...active, implementation, runs: [succeeded, done, { ...followUp, status: "queued" }] }).passed).toBe(true);
|
||||
expect(gradeChatCatalogRuns({ ...scope, implementation, runs: [succeeded, done, { ...followUp, status: "succeeded" }] }).passed).toBe(true);
|
||||
for (const bad of [
|
||||
{ ...followUp, status: "failed" },
|
||||
{ ...followUp, contextSnapshot: { ...followUp.contextSnapshot, chatCompletionUpdates: [{ id: "unrelated" }] } },
|
||||
{ ...followUp, contextSnapshot: { ...followUp.contextSnapshot, chatCompletionUpdates: [] } },
|
||||
{ ...followUp, contextSnapshot: { ...followUp.contextSnapshot, wakeReason: "issue_commented" } },
|
||||
{ ...followUp, agentId: "worker" },
|
||||
]) expect(gradeChatCatalogRuns({ ...active, implementation, runs: [succeeded, done, bad] }).passed).toBe(false);
|
||||
// Before the turn is prepared, the queued run carries only its saved delivery IDs (actual server snapshot shape).
|
||||
const pending: ChatRun = { ...followUp, status: "queued",
|
||||
contextSnapshot: { issueId: "source", wakeReason: "chat_task_completed", wakeSource: "automation", chatCompletionDeliveryIds: ["delivery"] } };
|
||||
// The server claims the run (running) before it saves the updates, so a running run in preparation also qualifies.
|
||||
for (const status of ["queued", "running"]) expect(gradeChatCatalogRuns({ ...active, implementation, runs: [succeeded, done, { ...pending, status }] }).passed).toBe(true);
|
||||
for (const bad of [{ ...pending, status: "succeeded" }, { ...pending, status: "failed" }, { ...pending, contextSnapshot: { ...pending.contextSnapshot, chatCompletionDeliveryIds: [] } },
|
||||
{ ...pending, contextSnapshot: { ...pending.contextSnapshot, wakeSource: "manual" } }])
|
||||
expect(gradeChatCatalogRuns({ ...active, implementation, runs: [succeeded, done, bad] }).passed).toBe(false);
|
||||
expect(gradeChatCatalogRuns({ ...active, implementation, runs: [succeeded, worker, pending] }).passed).toBe(false);
|
||||
// The original answer must exist and have succeeded; a second follow-up and an unexplained active source stay rejected.
|
||||
expect(gradeChatCatalogRuns({ ...active, implementation, runs: [done, followUp] }).passed).toBe(false);
|
||||
expect(gradeChatCatalogRuns({ ...active, implementation, runs: [{ ...succeeded, status: "running" }, done, followUp] }).passed).toBe(false);
|
||||
expect(gradeChatCatalogRuns({ ...active, implementation: { ...implementation, expectedRunCount: 4 }, runs: [succeeded, done, followUp, { ...followUp, id: "again" }] }).passed).toBe(false);
|
||||
expect(gradeChatCatalogRuns({ ...active, implementation, runs: [succeeded, done, { ...followUp, status: "succeeded" }] }).passed).toBe(false);
|
||||
});
|
||||
|
||||
function reassignment() {
|
||||
const receipt = { runId: "worker-run", taskId: "task", workerId: "worker", sourceRunId: "final-source" };
|
||||
|
||||
@@ -74,6 +74,31 @@ export function chatCatalogProviderRuns(runs: ChatRun[], source?: ChatIssue): Ch
|
||||
const superseded = new Set(chatCatalogSupersededQueuedRuns(runs, source).map(run => run.id));
|
||||
return runs.filter(run => !superseded.has(run.id));
|
||||
}
|
||||
/**
|
||||
* The product's automatic CEO follow-up on the source chat after a delegated
|
||||
* task it created completed (`chat_task_completed`). The person's request was
|
||||
* already answered; this later turn may start at any moment after the worker
|
||||
* finishes, so a settled source can briefly show it queued or running. Only a
|
||||
* lead run on the source whose every completion update names separate work in
|
||||
* this ledger qualifies; any other source run must still have succeeded. The
|
||||
* server records the updates only while it prepares the already-claimed turn,
|
||||
* so a queued run, or a running run that is still being prepared, carries only
|
||||
* its saved delivery IDs. It qualifies only while queued or running, woken by
|
||||
* automation, after a separate worker run in this ledger succeeded.
|
||||
*/
|
||||
export function chatCatalogSourceCompletionFollowUp(run: ChatRun, input: { leadId: string; source?: ChatIssue; runs: ChatRun[] }): boolean {
|
||||
const snapshot = record(run.contextSnapshot);
|
||||
const updates = Array.isArray(snapshot.chatCompletionUpdates) ? snapshot.chatCompletionUpdates : record(snapshot.paperclipWake).chatCompletionUpdates;
|
||||
const work = input.runs.filter(other => other.agentId !== input.leadId && typeof taskId(other) === "string" && taskId(other) !== input.source?.id);
|
||||
const workTasks = new Set(work.map(taskId));
|
||||
if (!input.source || run.agentId !== input.leadId || taskId(run) !== input.source.id || snapshot.wakeReason !== "chat_task_completed") return false;
|
||||
if (updates === undefined || updates === null) {
|
||||
const deliveries = snapshot.chatCompletionDeliveryIds;
|
||||
return ["queued", "running"].includes(run.status) && snapshot.wakeSource === "automation" && Array.isArray(deliveries) && deliveries.length > 0
|
||||
&& deliveries.every(id => typeof id === "string" && id.length > 0) && work.some(other => other.status === "succeeded");
|
||||
}
|
||||
return Array.isArray(updates) && updates.length > 0 && updates.every(update => workTasks.has(record(update).id as string));
|
||||
}
|
||||
export function gradeChatCatalogRuns(input: {
|
||||
implementation: Pick<ChatCatalogCase, "expectedRunCount" | "minimumRunCount" | "runPolicy"> & CancellationCase;
|
||||
companyId: string; leadId: string; source?: ChatIssue; runs: ChatRun[];
|
||||
@@ -103,6 +128,9 @@ export function gradeChatCatalogRuns(input: {
|
||||
detail: "The independently observed ledger matches every declared successful/cancelled/failed terminal outcome count (or its declared role-bound range)." });
|
||||
} else if (c.runPolicy?.kind === "source-settled-with-active-work") {
|
||||
const sourceRuns = runs.filter(run => taskId(run) === source?.id);
|
||||
const followUps = sourceRuns.filter(run => chatCatalogSourceCompletionFollowUp(run, { leadId: input.leadId, source, runs }));
|
||||
const answers = sourceRuns.filter(run => !followUps.includes(run));
|
||||
const activeFollowUp = followUps.some(run => ["queued", "running"].includes(run.status));
|
||||
const handbacks = runs.filter(run => taskId(run) !== source?.id && run.agentId === input.leadId && run.contextSnapshot?.wakeReason === "case_handoff");
|
||||
const workers = runs.filter(run => taskId(run) !== source?.id && !handbacks.includes(run));
|
||||
const cancelled = runs.filter(run => run.status === "cancelled");
|
||||
@@ -125,11 +153,13 @@ export function gradeChatCatalogRuns(input: {
|
||||
checks.push({ id: "catalog-exact-owned-worker-cancellation", passed: cancelled.length === 0 ? expected === undefined : declared && exact,
|
||||
detail: "Only the exact independently proved task-reassignment cancellation is accepted; final actor, chronology and saved-state proof remains mandatory in the case callback." });
|
||||
checks.push({ id: "catalog-settled-source-separate-work", passed: !!source && source.companyId === input.companyId
|
||||
&& source.conversationState === "waiting" && sourceRuns.length > 0 && sourceRuns.every(run => run.agentId === input.leadId && run.status === "succeeded")
|
||||
&& (source.conversationState === "waiting" || source.conversationState === "active" && activeFollowUp)
|
||||
&& answers.length > 0 && answers.every(run => run.agentId === input.leadId && run.status === "succeeded")
|
||||
&& followUps.length <= 1 && followUps.every(run => ["queued", "running", "succeeded"].includes(run.status))
|
||||
&& workers.every(run => typeof taskId(run) === "string" && run.agentId !== input.leadId)
|
||||
&& workers.filter(run => ["queued", "running"].includes(run.status)).length <= c.runPolicy.maximumActiveWorkerRuns
|
||||
&& handbacks.length <= (c.runPolicy.maximumLeadCaseHandbackRuns ?? 0) && handbacks.every(run => typeof taskId(run) === "string"),
|
||||
detail: "The CEO source is settled; unfinished worker runs are separately task-bound and bounded, and any CEO run outside the source is a declared case handback. The case oracle must additionally bind their exact task and assignee." });
|
||||
detail: "The CEO source is settled; unfinished worker runs are separately task-bound and bounded, and any CEO run outside the source is a declared case handback. At most one product completion follow-up for that separate work may still be in flight on the source. The case oracle must additionally bind their exact task and assignee." });
|
||||
}
|
||||
return { passed: checks.every(check => check.passed), checks };
|
||||
}
|
||||
|
||||
Reference in new issue
Block a user