mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-13 23:39:21 +02:00
## Thinking Path > - Paperclip lets people manage AI agents and their work. > - Agent Chat needs reliable native execution before native runners become the onboarding default. > - Existing stories covered idle reassignment and controller restart, but not an executing worker handoff or worker process loss. > - Status answer tests also need to reject stale claims and invented facts. > - This pull request adds six opt-in full-stack cells with independent state assertions and retained evidence. > - The probes exposed a misleading Retry across server projection and recovery-banner paths; the fix reports the blocked recovery honestly. > - The tests preserve failures without changing recovery policy, production prompts, or onboarding defaults. ## Linked Issues or Issue Description Refs: #13762. Related: #13765 (Retry targets the latest failed attempt), #13753 (task context ownership), #13746 (native recovery work). ## What Changed - Add active reassignment with saved draft and plan preservation, old-worker cancellation, and successor completion checks. - Preserve recovery-needed projection when native cleanup fails before its coordinator exists, refuse a generic retry that would immediately fail again, and replace the recovery banner's misleading Retry with Inspect run. - Add verified local worker process loss with a required successful continuation; retain a failing qualification result when recovery is unavailable, while independently verifying the UI/API refuse doomed retries. - Add two-turn factual answer checks for current blockers, stale claims, inactive backlog work, and unknown facts. Retain prose for separate semantic review. - Add positive and negative oracle calibration and document fault isolation, cleanup, billing, and qualification limits. ## Verification - Eval TypeScript check passes. - All 442 eval support tests pass locally. The 89 focused server tests and server typecheck pass. Six recovery-banner UI tests and token gates pass. - Initial new-cell campaign: https://github.com/paperclipai/paperclip/actions/runs/35657128077. All six results are retained; four failed on fixture-contract issues and two exposed real worker cleanup quarantine. - All 26 existing native onboarding cells: https://github.com/paperclipai/paperclip/actions/runs/35656761484 (26/26 passed on master846336e5a, all cleanup passed). - Intermediate handoff/fault campaign: https://github.com/paperclipai/paperclip/actions/runs/35657945095 (four retained failures: two overly strict draft oracles, two real crash quarantines). - Final active handoff: https://github.com/paperclipai/paperclip/actions/runs/35659014397 (2/2 passed on cf6d4ae3a; both cleanup passed). - Clarified answer-quality fixtures: https://github.com/paperclipai/paperclip/actions/runs/35658262695 (2/2 passed on 4a26f10be; both cleanup passed; all four answers semantically reviewed). - Quarantine guard regression campaign: https://github.com/paperclipai/paperclip/actions/runs/35659580100 (both API requests correctly refused with 409/no second run, but exposed a separate misleading Retry in the recovery banner and a fixture wait on a non-admitted run). - Final quarantine guard verification: https://github.com/paperclipai/paperclip/actions/runs/35661067305 (147e42f7e: both providers verify Inspect run/no Retry, HTTP 409, one retained run, unchanged saved plan, and successful disposable cleanup. Both evals intentionally remain red with `worker_crash_recovery_unqualified`; no successful continuation exists). The preceding campaign 35658772755 never ran provider cases because GitHub artifact finalization returned HTTP 403. - Full repository CI passes on147e42f7e: typecheck, tests, build, and browser gates. One unchanged local-service-supervisor readiness test failed initially; its six-test file passed in isolation and the failed shard passed on its single rerun. Latest-head rollup: 54 successful, 2 intentionally skipped, no failed or pending checks. Greptile is 5/5 with zero unresolved findings. - See tests/runner-e2e/QUALIFICATION-2026-09-21.md for retained attempts and semantic review. Published reports: [onboarding](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-35656761484-1/), [handoff](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-35659014397-1/), [grounded answers](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-35658262695-1/), [crash guards and unqualified recovery](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-35661067305-1/). ## Risks - Paid cells are explicit-only and local-only. The fault fixture signals only the exact native run PID after checking its identity. - Live worker-loss probes currently fail on cleanup quarantine for both providers. The eval must remain red until there is a usable recovery, even when preservation and refusal checks pass. Verified cleanup with a fresh attempt versus exact-session resume remains a product decision. - Structured facts alone do not qualify prose quality; semantic review remains separate. - Onboarding uses the existing runtime switch after the real wizard and before provider execution. Native UI selection and public defaults remain unchanged. ## Model Used OpenAI GPT-6 through Codex, with reasoning, tool use, and code execution. The exact served snapshot and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
241 lines
9.5 KiB
TypeScript
241 lines
9.5 KiB
TypeScript
import { describe, expect, it } from "vitest";
|
|
import { projectExecution } from "./execution-projection.js";
|
|
|
|
type Run = Parameters<typeof projectExecution>[0];
|
|
type Coordinator = NonNullable<Parameters<typeof projectExecution>[1]>;
|
|
const now = new Date("2026-09-08T12:00:00Z");
|
|
const run = (values: Partial<Run> = {}) =>
|
|
({
|
|
status: "running",
|
|
runtimeMode: "native",
|
|
errorCode: null,
|
|
nextAction: null,
|
|
lastUsefulActionAt: now,
|
|
lastOutputAt: null,
|
|
startedAt: now,
|
|
scheduledRetryAt: null,
|
|
scheduledRetryAttempt: 1,
|
|
retryOfRunId: null,
|
|
executionControlDeadlineAt: null,
|
|
...values,
|
|
}) as Run;
|
|
const coordinator = (values: Partial<Coordinator> = {}) =>
|
|
({
|
|
phase: "observed",
|
|
attempt: 1,
|
|
failureCode: null,
|
|
failureDetail: {},
|
|
nextAttemptAt: null,
|
|
leaseExpiresAt: new Date(now.getTime() + 30_000),
|
|
controlDeadlineAt: null,
|
|
...values,
|
|
}) as Coordinator;
|
|
const project = (
|
|
r = run(),
|
|
c: Coordinator | undefined = coordinator(),
|
|
pending: { kind: string }[] = [],
|
|
) => projectExecution(r, c, pending, undefined, now);
|
|
|
|
describe("execution truth projection", () => {
|
|
it("keeps cleanup quarantine visible without a coordinator or recovery row", () => {
|
|
const stopped = run({ id: "stopped", status: "failed", errorCode: "native_session_cleanup_quarantined" });
|
|
expect(projectExecution(stopped, undefined, [], undefined, now)).toMatchObject({
|
|
phase: "recovery_needed", cause: "native_session_cleanup_quarantined", recoveryOwner: "board",
|
|
permittedActions: ["inspect_run", "inspect_recovery"],
|
|
});
|
|
expect(projectExecution(stopped, coordinator({ phase: "retryable_failure" }), [], undefined, now).phase).toBe("recovery_needed");
|
|
expect(projectExecution({ ...stopped, finishedAt: now }, coordinator({
|
|
phase: "terminal_failure", failureCode: "native_provider_terminal_failed",
|
|
}), [], undefined, now)).toMatchObject({ phase: "recovery_needed", recoveryOwner: "board" });
|
|
expect(projectExecution({ ...stopped, errorCode: "provider_frame_too_large" }, undefined, [], undefined, now).phase).toBe("failed");
|
|
expect(projectExecution(stopped, undefined, [], { status: "resolved", cause: "native_session_cleanup_quarantined", nextAction: "Verify prior execution",
|
|
evidence: { executionReconciliation: { providerStopped: true }, continuationRunId: "successor" } }, now))
|
|
.toMatchObject({ phase: "completed", successorRunId: "successor", cause: null });
|
|
});
|
|
it("shows subscription contention as a resource wait without failed provider attempts", () => {
|
|
expect(projectExecution(run({ runtimeMode: "legacy", status: "scheduled_retry", scheduledRetryReason: "ai_connection_busy",
|
|
scheduledRetryAttempt: 12, contextSnapshot: { failureRetriesBeforeAiConnectionWait: 0 } }), undefined, [], undefined, now))
|
|
.toMatchObject({ label: "Waiting for AI subscription", phase: "retry_scheduled", attempt: 1, recoveryOwner: null });
|
|
});
|
|
it("shows a workspace wait without presenting its deferral count as failed attempts", () => {
|
|
expect(projectExecution(run({ runtimeMode: "legacy", status: "scheduled_retry", scheduledRetryReason: "workspace_busy",
|
|
scheduledRetryAttempt: 12, contextSnapshot: { failureRetriesBeforeWorkspaceWait: 1 } }), undefined, [], undefined, now))
|
|
.toMatchObject({ label: "Waiting for workspace", phase: "retry_scheduled", attempt: 2, recoveryOwner: null });
|
|
});
|
|
it.each([
|
|
{ status: "resolved", previousRunId: "old", nextRunId: "next", continued: true },
|
|
{ status: "active", previousRunId: "old", nextRunId: "next", continued: false },
|
|
{ status: "resolved", previousRunId: "other", nextRunId: "next", continued: false },
|
|
{ status: "resolved", previousRunId: "old", nextRunId: "old", continued: false },
|
|
{ status: "resolved", previousRunId: "old", nextRunId: null, continued: false },
|
|
])("projects the recorded explicit successor without hiding unresolved recovery: $status/$previousRunId/$nextRunId", ({ status, previousRunId, nextRunId, continued }) => {
|
|
const source = run({ id: "old", status: "failed", errorCode: "adapter_failed" });
|
|
expect(projectExecution(source, coordinator({ phase: "terminal_failure" }), [], {
|
|
status,
|
|
cause: "native_continuation_requires_reconciliation",
|
|
nextAction: "Inspect the stopped execution.",
|
|
evidence: {
|
|
automaticRecovery: { policy: "preserve_without_replay_v1" },
|
|
explicitUserContinuation: { previousRunId, runId: nextRunId },
|
|
},
|
|
}, now)).toMatchObject(continued ? {
|
|
phase: "completed", label: "Continued in another run", successorRunId: "next",
|
|
cause: "native_continuation_requires_reconciliation", nextAction: null,
|
|
} : { phase: "recovery_needed", successorRunId: null });
|
|
expect(source.status).toBe("failed");
|
|
});
|
|
|
|
it("shows a reconciled continuation as queued until its durable delivery is recorded", () => {
|
|
const action = {
|
|
cause: "native_session_retry_exhausted",
|
|
nextAction: "Old recovery action",
|
|
status: "resolved" as const,
|
|
evidence: {
|
|
executionReconciliation: { runId: "old" },
|
|
continuationDelivery: "pending",
|
|
},
|
|
};
|
|
expect(
|
|
projectExecution(
|
|
run({ status: "failed" }),
|
|
coordinator({ phase: "terminal_failure" }),
|
|
[],
|
|
action,
|
|
now,
|
|
),
|
|
).toMatchObject({
|
|
phase: "queued",
|
|
label: "Continuation queued",
|
|
cause: null,
|
|
});
|
|
expect(
|
|
projectExecution(
|
|
run({ status: "failed" }),
|
|
coordinator({ phase: "terminal_failure" }),
|
|
[],
|
|
{
|
|
...action,
|
|
evidence: { ...action.evidence, continuationRunId: "next" },
|
|
},
|
|
now,
|
|
),
|
|
).toMatchObject({ phase: "completed", successorRunId: "next" });
|
|
expect(
|
|
projectExecution(
|
|
run({ status: "failed" }),
|
|
coordinator({ phase: "terminal_failure" }),
|
|
[],
|
|
{
|
|
...action,
|
|
evidence: { ...action.evidence, continuationDelivery: "invalidated" },
|
|
},
|
|
now,
|
|
),
|
|
).toMatchObject({ phase: "failed", label: "Continuation cancelled" });
|
|
});
|
|
it("requires current execution evidence, without treating output silence as failure", () => {
|
|
expect(project(run({ lastOutputAt: new Date(0) })).phase).toBe("working");
|
|
expect(
|
|
project(run(), coordinator({ leaseExpiresAt: new Date(0) })),
|
|
).toMatchObject({ phase: "reconnecting", label: "Confirming execution" });
|
|
expect(
|
|
project(
|
|
run({ runtimeMode: "legacy", processPid: process.pid }),
|
|
undefined,
|
|
).phase,
|
|
).toBe("working");
|
|
});
|
|
it("distinguishes provider work, finalization and timed retry", () => {
|
|
expect(
|
|
project(
|
|
run({ executionControlDeadlineAt: new Date(now.getTime() + 60_000) }),
|
|
).phase,
|
|
).toBe("finishing");
|
|
expect(
|
|
project(
|
|
run({ status: "failed" }),
|
|
coordinator({
|
|
phase: "retryable_failure",
|
|
nextAttemptAt: new Date(now.getTime() + 30_000),
|
|
attempt: 2,
|
|
}),
|
|
),
|
|
).toMatchObject({ phase: "retry_scheduled", attempt: 2, maxAttempts: 3 });
|
|
expect(
|
|
project(
|
|
run({ status: "failed" }),
|
|
coordinator({ phase: "retryable_failure", nextAttemptAt: new Date(0) }),
|
|
).phase,
|
|
).toBe("reconnecting");
|
|
});
|
|
it("includes the original legacy attempt in the displayed incident budget", () => {
|
|
expect(projectExecution(run({ runtimeMode: "legacy", status: "scheduled_retry", scheduledRetryAttempt: 1 }), undefined, [], undefined, now)).toMatchObject({ attempt: 2, maxAttempts: 3 });
|
|
});
|
|
it("shows pending interaction only after productive work has stopped", () => {
|
|
expect(
|
|
project(run(), coordinator(), [{ kind: "connection_intent" }]).phase,
|
|
).toBe("working");
|
|
expect(
|
|
project(run({ status: "succeeded" }), undefined, [
|
|
{ kind: "connection_intent" },
|
|
]).phase,
|
|
).toBe("waiting_for_access");
|
|
expect(
|
|
project(run({ status: "succeeded" }), undefined, [
|
|
{ kind: "ask_user_questions" },
|
|
]).phase,
|
|
).toBe("waiting_for_answer");
|
|
});
|
|
it("retains replacement lineage and exposes operator-owned failures for legacy runs", () => {
|
|
expect(
|
|
project(
|
|
run({ status: "failed" }),
|
|
coordinator({
|
|
phase: "terminal_failure",
|
|
failureDetail: { successorRunId: "replacement" },
|
|
}),
|
|
),
|
|
).toMatchObject({ phase: "completed", successorRunId: "replacement" });
|
|
expect(
|
|
projectExecution(
|
|
run({ runtimeMode: "legacy", status: "failed" }),
|
|
undefined,
|
|
[],
|
|
{
|
|
cause: "uncertain_external_action",
|
|
nextAction: "Reconcile the email delivery.",
|
|
},
|
|
now,
|
|
),
|
|
).toMatchObject({
|
|
phase: "recovery_needed",
|
|
recoveryOwner: "board",
|
|
nextAction: "Reconcile the email delivery.",
|
|
});
|
|
});
|
|
it("bounds recovery checking and surfaces its next action if verification never finishes", () => {
|
|
const c = coordinator({
|
|
phase: "terminal_failure",
|
|
failureCode: "native_provider_terminal_failed",
|
|
});
|
|
expect(project(run({ status: "failed", finishedAt: now }), c).label).toBe(
|
|
"Checking recovery",
|
|
);
|
|
expect(
|
|
project(
|
|
run({ status: "failed", finishedAt: new Date(now.getTime() - 60_000) }),
|
|
c,
|
|
).phase,
|
|
).toBe("recovery_needed");
|
|
expect(
|
|
project(
|
|
run({ status: "failed", finishedAt: now }),
|
|
coordinator({
|
|
...c,
|
|
failureDetail: { replacementDenied: "uncertain_provider_action" },
|
|
}),
|
|
).phase,
|
|
).toBe("recovery_needed");
|
|
});
|
|
});
|