mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-11 13:24:55 +01:00
test(e2e): cover chat reload and fresh sessions (#18784)
## Thinking Path > - Paperclip keeps agent work attached to saved company records. > - Agent Chat keeps one conversation for each person and agent. > - Reload must preserve context without starting another turn. > - The ordinary `/new` command must create fresh provider context and preserve history. > - Existing local chat journeys cover these behaviors, but the deployment-targeted daily catalog does not. > - This PR adds one reusable journey with independent saved-state and rendered-reply checks. ## Linked Issues or Issue Description Refs: #2242, #18655. **What existing behavior does this improve?** Product E2E coverage for Agent Chat persistence and session reset on an attached deployment. **Current behavior** The deployment-targeted catalog has no bounded check for recall after reopening and a fresh session in the same saved conversation. **Proposed behavior** Remember a random phrase, reopen and reload the chat, then ask for the phrase without supplying it. Send `/new` through the ordinary composer. Require a new native session and a fresh-context answer while the original chat and history remain saved. ## What Changed - Add the `chat-continuity` flow and independent evidence grader. - Reuse the existing chat sender and run-bound native reply reader. - Require exact board-user message attribution, three successful native provider turns and one synthetic reset row. Count all four persisted run IDs. A reset with provider input or a native session cannot pass as synthetic work. - Use the original bound issue-run summary to prove reset completion when the detail result is null or compact. Preserve the raw detail and require its reset context, company, agent, issue and wake comment to agree. Native provider turns require committed native state and independent output evidence; an optional duplicate result field is not required. - Require an empty agent-filtered run history before the first send. Reused agents are unsupported for this case. Validate every agent-scoped activity row and reject mixed or truncated responses; do not filter global history into a pass. - Require the fresh native detail to contain an explicit null `sessionIdBefore`. Missing or empty admission evidence cannot prove a fresh provider session. Keep optional result blobs optional. - Check provider, model, environment, native session, finalization and reset generation. Reject stale replies, duplicate sends, extra runs, missing history and unexpected work. - Permit authenticated chat screenshots through the existing same-origin and mounted-Layout guard. Authentication and unrelated settings pages remain excluded. - Register the case, exact checks, source hash inputs, discovery and documentation. Feature-map files are unchanged. ## Verification - Normal merge commit `ddc37bb05b87cbc1fa449e4b2fc5a4677f0fe3b6` has exactly two real parents: the staging-qualified chat source `4c6cc9d67d2ae577e3a8ae7856bd6ae22a8489c1` and lifecycle squash `8771606fd87f69ed4057779e3dce3a9bca8088e6` from #18782. Thirteen dedicated case, support and reused runtime/oracle files are byte-for-byte preserved. The eight shared registry placements match the finite integration forecast. The nine required checks and four-run limit are unchanged. - The integrated source passes 99 focused tests across three files and 925 related Product E2E/catalog/native-reply tests across 42 files. Credential-free discovery lists 22 unique journeys, including chat and lifecycle exactly once. The PR diff against the actual lifecycle squash passes its whitespace check and remains the original 17 chat paths. - The current canonical `node scripts/typecheck-test-projects.mjs --low-memory tests/runner-e2e` passes, including Runner TypeScript and server declaration prerequisites. The DB migration checks and build pass against the actual squash's execution-admission schema and migration 0428. No Rust build was run locally. Canonical generated outputs are bound to current tracked inputs and were rechecked after tests. - The first direct typecheck found a stale generated Runner export, `taskDocumentKeySchema`; the current canonical prerequisites resolved it. A frozen offline install lacked a cached locked tarball, and an initial child command reached the global pnpm version switch and failed signature retrieval. Fetching only locked packages and pinning child pnpm to 9.15.4 corrected the setup. The original diagnostics remain retained; no tracked source was changed to bypass them. - The unchanged chat implementation passed a genuine staging campaign, workflow 41725181387, at source `4c6cc9d67d2ae577e3a8ae7856bd6ae22a8489c1`: all nine checks, all five phases, three provider turns plus one synthetic reset, all four usage IDs, and closed cleanup. This evidence remains bound to that original source. The new integration head requires its own current gates and coordinator qualification decision; this PR does not enroll anything in the daily plan. - The earlier staging workflow 41701828968 remains failed. Its nine behavior checks passed, but private collection rejected the valid reset's capture of the old session and cleanup failed. The private adapter fix and evidence-first recovery were handled separately. The original failed grade and archive are preserved. Reset prior-session capture is distinct from the third native turn's explicit-null fresh admission. - Three missing/undefined/empty fresh-admission controls reproduced false passes on `247827a066af01fa583ff030df79b61cb2c87730`, then passed after the strict null fix. Four earlier controls reproduced compact-detail, mixed-activity, missing fresh-agent receipt and unscoped-activity problems. Their original failures remain retained. Positive and wrong outcomes still enforce bound reset summaries, committed native state, configured identity and independent rendered output. Board provisioning uses the acting-agent filter; it does not grandfather earlier work by the test agent. - The first broad support attempt passed 861 tests and failed one report-generator check. A separate reproduction showed `listen EPERM` for the tsx local IPC socket before report output; the unchanged support suite passed with normal local IPC permission. An early flow fixture also returned an object where the events API returns an array; the real event reader rejected it, and only that fixture was corrected. Both original failures remain recorded. - Original commit `d6226b30acaf5a2dad28030f5b295ac1dd119565` failed full CI [41653171240](https://github.com/paperclipai/paperclip/actions/runs/41653171240): a CLI child-server runtime-file wait and two 15-second attention-retention timeouts, followed by database setup/cleanup errors. Their exact causes remain unresolved; the failed jobs and logs were retained and not retried. A later pass does not explain those timeouts. - Strict-admission source `055a7fa99e7cb8aad12b85fccf23880f0dc98a3f` failed full CI [41669527762](https://github.com/paperclipai/paperclip/actions/runs/41669527762): private-invitation and pending-shell access fixtures. The retained assertions match the account-identity fixture repair in #18799, now included through master, together with #18798 and #18650. No old-head job was retried. Subsequent source `4c6cc9d67d2ae577e3a8ae7856bd6ae22a8489c1` passed full CI 41680411215 and current-source review at 5/5. - Current full CI [41743580793, attempt 2](https://github.com/paperclipai/paperclip/actions/runs/41743580793/attempts/2) completed successfully on unchanged `ddc37bb05b87cbc1fa449e4b2fc5a4677f0fe3b6` at 2026-11-04 00:34:52 UTC: 48 successful jobs and one intentional selector skip. Attempt 1 remains failed: three run-bound runners were independently confirmed to have lost Spot capacity, and the two aggregate failures reflected those interrupted lanes. The build and boot health smoke had passed before interruption. Exactly one failed-job recheck was used; no source change or further retry was made. Fresh exact-head Greptile review is 5/5 with no new actionable issues and both review threads resolved. The separate Ponytail assessment says Ship with no must-fix or should-fix item; its optional direct board-row grader control remains documented. Whole-workspace recursive typecheck, full tests and full build were covered by hosted verification and were not repeated locally; the bounded local checks above passed. These gates do not claim that a later master merge tree has been live-qualified. ## Risks The attached target must supply a fresh disposable native agent and durable pre-send ownership for its chat. Its private adapter must collect all four run IDs, distinguish the synthetic reset from the three provider turns, and stop and clean up owned resources after evidence is saved. This case verifies browser reload, reopen and explicit session reset. It does not verify controller restart. Unknown setup or provider cost remains unknown. No shared QA source agent, model setting, production behavior or budget policy is changed by this public PR. ## Model Used OpenAI GPT-6 through Codex with reasoning, code execution and repository tools. The runtime does not expose a more specific model identifier or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
fbb7055b90
commit
d7283e8757
17 files changed
+785
-5
No files matched your search
+4
-2
@@ -117,8 +117,10 @@ successor. Registration and calibration are separate from a live staging pass.
|
||||
deployment targets and starting-state recipes. The shared task-properties
|
||||
journey and task-document create/edit/reopen journey use no model; Cloud
|
||||
onboarding supplies reusable verified state. The document journey checks exact
|
||||
saved content and both revision records through independent API reads. Cloud
|
||||
fleet operations and trusted CI remain private. Local eval gateway qualification
|
||||
saved content and both revision records through independent API reads. The explicit `chat-continuity` selection verifies browser reload/reopen, prompt-hidden recall,
|
||||
ordinary `/new` reset, fresh native context and retained history against an attached target.
|
||||
It requires a fresh owned agent with no prior work runs and uses three provider turns plus one synthetic reset row; it does not restart a shared stack.
|
||||
Cloud fleet operations and trusted CI remain private. Local eval gateway qualification
|
||||
does not qualify Cloud routes, and these selections stay outside `--all`.
|
||||
|
||||
The explicit-only [case handoff workflow](../tests/runner-e2e/README.md#persistent-case-handoffs-explicit-only)
|
||||
|
||||
@@ -31,6 +31,7 @@ never start a server or depend on a tenant filesystem.
|
||||
| `task-create` | Company and signed-in board user | Create a task through the shared production composer, assign it to yourself, reopen and reload it, verify the saved prompt and no agent work |
|
||||
| `project-task-context` | Company, signed-in board user, task-write capability; no model runs | Create a project and a self-assigned task with the project selected through the production dialogs. Reopen the project, open its task, reload both, and verify exact saved company, project, prompt, assignment, standard work mode and zero runs |
|
||||
| `ask-question` | Company, configured agent capability, positive model allowance and exactly one permitted work run | Reuse the existing cross-model Ask question prompt; require one successful run with matching company/task/agent/model/environment, one saved exact answer, and visible readback after reload |
|
||||
| `chat-continuity` | Fresh owned native agent with no prior work runs; Agent Chat enabled; fresh owned user chat; three provider turns and one synthetic reset | Recall prompt-hidden context after reopening, then use `/new` to advance the same saved conversation to a fresh native session, preserve history and verify the final reply after reload. See [CHAT-CONTINUITY.md](product-journeys/CHAT-CONTINUITY.md) |
|
||||
| `question-answer-resume` | Native agent; up to three work runs | Reuse the classic two-question story: answer a choice and a free-text question through the browser, verify each native continuation, then read the exact saved answers in an agent-authored task document after reload |
|
||||
| `task-properties` | Company, signed-in board user, task-write capability, prepared run-owned backlog task | Open its properties, assign it to the signed-in user, reload, check both UI and saved assignment; verify no agent run |
|
||||
| `task-edit-lifecycle` | Company, signed-in board user, task-write capability; target support for run-owned labels and exact predeclared `taskEdit.revisedTitle` cleanup; zero model runs | Create a self-assigned task, edit its inline header title and label, reload, complete and reopen it; independently verify exact saved state, unchanged context and zero runs. Open a closed Properties panel through its shipped toggle |
|
||||
|
||||
@@ -136,6 +136,7 @@ describe("runner E2E catalog", () => {
|
||||
"task-create",
|
||||
"project-task-context",
|
||||
"ask-question",
|
||||
"chat-continuity",
|
||||
"question-answer-resume",
|
||||
"task-properties",
|
||||
"task-document-persistence",
|
||||
|
||||
@@ -26,6 +26,7 @@ associate(["product-journeys"], {
|
||||
"inbox-triage": ["inbox-search/views", "inbox-search/triage"], "task-create": ["tasks/create"], "ask-question": ["questions-and-approvals/questions"],
|
||||
"question-answer-resume": ["questions-and-approvals/questions", "questions-and-approvals/continuation"],
|
||||
"followup-while-running": ["steering/queue"],
|
||||
"chat-continuity": ["agent-chat/history", "runs-adapters/session"],
|
||||
"hire-delegate-reuse": ["agents/hire", "documents-artifacts/documents"],
|
||||
"agent-settings-persistence": ["agents/instructions", "tasks/completion"],
|
||||
"project-task-context": ["projects/lifecycle", "projects/task-context", "tasks/create"],
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
# Chat reload and session continuity
|
||||
|
||||
`chat-continuity` is a deployment-targeted Product E2E journey. It uses the
|
||||
ordinary Agent Chat composer on a fresh, campaign-owned native agent. It does
|
||||
not restart a persistent Cloud instance or alter a shared source agent.
|
||||
The agent-filtered heartbeat history must be empty before the first send.
|
||||
An agent with earlier completed, failed, queued or active work is unsupported;
|
||||
the fixture does not subtract or hide those runs. A target that reuses one
|
||||
agent across several journeys must supply a separate fresh agent for this case.
|
||||
|
||||
The first exchange contains a random phrase. After leaving the chat, reopening
|
||||
it and reloading, the next message asks for that phrase without supplying it.
|
||||
The independently attributed reply must contain the phrase, and both native
|
||||
turns must retain the same session identity.
|
||||
|
||||
The user then sends the ordinary `/new` command. Paperclip preserves the same
|
||||
backing chat and saved history while advancing `conversationSessionGeneration`
|
||||
from zero to one. The final request does not reveal the remembered phrase. Its
|
||||
reply must use the fresh-context marker, with a different native session, no
|
||||
prior session on admission, and no old phrase in the new native task context.
|
||||
The independent run detail must include its `sessionIdBefore` column explicitly
|
||||
set to `null`; a missing or empty field is not session-boundary evidence.
|
||||
Saved history is not removed or rewritten by the fixture.
|
||||
|
||||
Every reply is read from the actual rendered chat after reload using the
|
||||
existing run-bound native reply oracle. Board-user messages must each persist
|
||||
exactly once and bind to exactly one observed run. The original comments remain
|
||||
unchanged through the final reload. The owned agent must create no extra task,
|
||||
project or teammate.
|
||||
|
||||
The closed accounting contract is **three successful native provider turns
|
||||
plus one synthetic reset row**, four persisted run IDs in total. The reset must
|
||||
have its normal reset context and a matching original `/issues/:id/runs`
|
||||
summary receipt and must not publish an assistant response. That receipt binds
|
||||
the request's company and issue to the same successful summary run, agent,
|
||||
context issue and wake comment as the independently read detail. The raw run
|
||||
detail is not rewritten. Its optional `resultJson` can be absent or compact;
|
||||
an explicit contradictory reset flag is rejected. Real provider turns require
|
||||
the authoritative committed `nativePhase`, not an optional duplicate result
|
||||
field. Their output still requires a bound native-final event or the selected
|
||||
attributed completion comment and an actual reloaded rendered reply.
|
||||
The reset is not counted as an additional provider turn. Source model, provider and
|
||||
managed environment must match all three native turns. The target provider owns
|
||||
authentication, the disposable agent, pre-send chat ownership, complete run and
|
||||
usage collection, stopping and evidence-first cleanup. Unknown setup or model
|
||||
cost remains unknown; a run limit is not proof of an absolute spend limit.
|
||||
|
||||
The company activity request includes the exact owned agent filter before the
|
||||
server's 500-row limit. Every returned row must match that company and agent,
|
||||
have its canonical activity identity, and leave the response below that limit.
|
||||
Mixed or truncated results fail; the client never filters a global response
|
||||
into an apparent pass. Unrelated company history therefore does not limit this
|
||||
fresh agent's proof, while any owned task, project or teammate creation fails.
|
||||
Activity identifies the acting agent, not the created entity. Board provisioning
|
||||
therefore has no agent actor even when its `entityId` is this new agent. Creation
|
||||
attributed to the owned agent is rejected, including activity before the first
|
||||
chat message; the fresh-agent contract does not grandfather that work.
|
||||
|
||||
This case covers browser reload/reopen and explicit session reset. Controller
|
||||
restart remains covered by the separate existing Agent Chat journeys. Local
|
||||
oracle and catalog checks do not establish a staging pass; daily admission
|
||||
requires the actual original staging attempt and complete cleanup evidence.
|
||||
@@ -24,6 +24,7 @@ type AgentJourney = Exclude<keyof typeof agentJourneyRuns,
|
||||
"ask-question"
|
||||
| "question-answer-resume"
|
||||
| "followup-while-running"
|
||||
| "chat-continuity"
|
||||
| "file-input-answer"
|
||||
| "routine-scheduled-run"
|
||||
| "agent-settings-persistence"
|
||||
|
||||
@@ -11,6 +11,7 @@ export const productJourneys = [
|
||||
{ id: "task-create", label: "Create a task for yourself and reopen it", requires: ["company", "board-user", "task-write"], capabilities: ["tasks/create"], paid: false },
|
||||
{ id: "project-task-context", label: "Create a project and reopen its self-assigned task", requires: ["company", "board-user", "task-write"], capabilities: ["projects/lifecycle", "projects/task-context", "tasks/create"], paid: false },
|
||||
{ id: "ask-question", label: "Ask an agent a question and read its answer", requires: ["company", "board-user", "task-write", "agent"], capabilities: ["questions-and-approvals/questions"], paid: true },
|
||||
{ id: "chat-continuity", label: "Reopen a chat, recall its context, then start a fresh session", requires: ["company", "board-user", "task-write", "agent"], capabilities: ["agent-chat/history", "runs-adapters/session"], paid: true },
|
||||
{ id: "question-answer-resume", label: "Answer agent questions and verify the saved result", requires: ["company", "board-user", "task-write", "agent"], capabilities: ["questions-and-approvals/questions"], paid: true },
|
||||
{ id: "task-properties", label: "Assign a task to the signed-in user", requires: ["company", "board-user", "task-write", "backlog-task"], capabilities: ["tasks/properties"], paid: false },
|
||||
{ id: "task-document-persistence", label: "Create, edit, and reopen a task document", requires: ["company", "board-user", "task-write", "backlog-task"], capabilities: ["documents-artifacts/documents"], paid: false },
|
||||
|
||||
@@ -0,0 +1,135 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { CHAT_CONTINUITY_CHECKS, chatContinuityIssueId, chatContinuityScenario, gradeChatContinuity,
|
||||
} from "./chat-continuity-evidence.js";
|
||||
|
||||
import { chatContinuityFixture, compactChatContinuityFixture } from "./chat-continuity.test-fixture.js";
|
||||
import { parseTargetConfig } from "./catalog.js";
|
||||
import { journeyRequiredChecks } from "./success-contract.js";
|
||||
|
||||
type Row = Record<string, any>;
|
||||
const allPass = (f: ReturnType<typeof chatContinuityFixture>) => gradeChatContinuity(f.scope, f.points).every(check => check.passed);
|
||||
|
||||
describe("chat reload and new-session evidence", () => {
|
||||
it("requires the actual three-turn outcome, one reset and run-bound reloaded replies", () => {
|
||||
const f = chatContinuityFixture();
|
||||
expect(gradeChatContinuity(f.scope, f.points).map(row => row.id)).toEqual(CHAT_CONTINUITY_CHECKS);
|
||||
expect(allPass(f)).toBe(true);
|
||||
});
|
||||
it("accepts normal compact run details only with bound native-final events and the original reset summary", () => {
|
||||
expect(allPass(compactChatContinuityFixture())).toBe(true);
|
||||
});
|
||||
it("rejects an unscoped activity response even below the truncation limit", () => {
|
||||
const f = chatContinuityFixture();
|
||||
f.points[5]!.activity.push({ id: "foreign-event", companyId: "foreign", agentId: "other-agent", action: "project.created",
|
||||
entityType: "project", entityId: "foreign-project", runId: "other-run" });
|
||||
expect(allPass(f)).toBe(false);
|
||||
});
|
||||
it("requires the fresh agent's pre-send empty-history receipt", () => {
|
||||
const f = chatContinuityFixture(); delete f.scope.agentHistory;
|
||||
expect(allPass(f)).toBe(false);
|
||||
});
|
||||
it("never supplies the old phrase in the recall or fresh-context request", () => {
|
||||
const s = chatContinuityScenario("random-123");
|
||||
expect(s.remember).toContain(s.remembered);
|
||||
expect(s.recall).not.toContain(s.remembered);
|
||||
expect(s.fresh).not.toContain(s.remembered);
|
||||
expect(s.reset).toBe("/new");
|
||||
});
|
||||
it("registers the exact measured checks and counts all four persisted runs", () => {
|
||||
expect(journeyRequiredChecks["chat-continuity"]).toEqual(CHAT_CONTINUITY_CHECKS);
|
||||
const config = { version: 1, id: "qa-chat", mode: "existing-cloud", adapterModule: "/private/provider.mjs",
|
||||
executionEnvironment: "daytona", journeys: ["chat-continuity"], baseline: { features: {}, connections: [] },
|
||||
limits: { durationMs: 900_000, maxRuns: 4, modelBudgetCents: 500 } };
|
||||
expect(parseTargetConfig(config).limits.maxRuns).toBe(4);
|
||||
for (const maxRuns of [0, 3, 5]) expect(() => parseTargetConfig({ ...config, limits: { ...config.limits, maxRuns } })).toThrow("agent_journeys_require_bounded_runs");
|
||||
});
|
||||
const mutants: ReadonlyArray<readonly [string, (f: ReturnType<typeof chatContinuityFixture>) => void]> = [
|
||||
["missing point", (f: ReturnType<typeof chatContinuityFixture>) => { f.points.pop(); }],
|
||||
["duplicate phase", (f) => { f.points[5]!.phase = "fresh"; }],
|
||||
["foreign chat", (f) => { f.points[1]!.chat.companyId = "foreign"; }],
|
||||
["different user", (f) => { f.points[3]!.chat.conversationUserId = "foreign"; }],
|
||||
["replaced backing issue", (f) => { f.points[5]!.chat.id = "replacement"; }],
|
||||
["active conversation", (f) => { f.points[5]!.chat.conversationState = "running"; }],
|
||||
["scheduled retry", (f) => { f.points[5]!.chat.scheduledRetry = { state: "pending" }; }],
|
||||
["duplicate user send", (f) => { f.points[5]!.comments.push({ ...f.points[5]!.comments[0], id: "duplicate" }); }],
|
||||
["foreign author", (f) => { f.points[5]!.comments[0]!.authorUserId = "foreign"; }],
|
||||
["wrong remembered phrase", (f) => { f.points[2]!.reply!.body = "I do not remember"; }],
|
||||
["unbound first reply", (f) => { f.points[0]!.reply!.anchor = "unbound"; }],
|
||||
["first reply missing after reopen", (f) => { delete f.points[1]!.reply; }],
|
||||
["stale recall response", (f) => { f.points[2]!.reply!.runId = "run-0"; }],
|
||||
["foreign recall run", (f) => { f.points[2]!.runs[1]!.agentId = "foreign"; }],
|
||||
["different provider", (f) => { f.points[2]!.runs[1]!.runnerProfileJson.nativeExecutionInput.provider.kind = "other"; }],
|
||||
["different model", (f) => { f.points[5]!.runs[3]!.runnerProfileJson.nativeExecutionInput.provider.model = "other"; }],
|
||||
["different environment", (f) => { f.points[5]!.runs[3]!.contextSnapshot.paperclipEnvironment.id = "other"; }],
|
||||
["uncommitted native phase", (f) => { f.points[5]!.runs[3]!.nativePhase = "finalizing"; }],
|
||||
["missing session proof", (f) => { delete f.points[0]!.runs[0]!.nativeSessionId; }],
|
||||
["unintended session replacement on reload", (f) => { f.points[2]!.runs[1]!.nativeSessionId = "replacement"; }],
|
||||
["reset generation absent", (f) => { f.points[3]!.chat.conversationSessionGeneration = 0; }],
|
||||
["fresh run generation stale", (f) => { f.points[4]!.runs[3]!.contextSnapshot.conversationSessionGeneration = 0; }],
|
||||
["fresh turn resumes old session", (f) => { f.points[4]!.runs[3]!.nativeSessionId = "original-session"; }],
|
||||
["previous session supplied on fresh admission", (f) => { f.points[4]!.runs[3]!.sessionIdBefore = "original-session"; }],
|
||||
["missing fresh-admission session evidence", (f) => { delete f.points[4]!.runs[3]!.sessionIdBefore; }],
|
||||
["undefined fresh-admission session evidence", (f) => { f.points[4]!.runs[3]!.sessionIdBefore = undefined; }],
|
||||
["empty fresh-admission session evidence", (f) => { f.points[4]!.runs[3]!.sessionIdBefore = ""; }],
|
||||
["old phrase leaked in native context", (f) => { f.points[4]!.runs[3]!.contextSnapshot.paperclipTaskMarkdown += f.scenario.remembered; }],
|
||||
["old phrase returned after reset", (f) => { f.points[4]!.reply!.body = f.scenario.remembered; }],
|
||||
["fresh marker absent", (f) => { f.points[4]!.reply!.body = "I have no phrase"; }],
|
||||
["extra run", (f) => { f.points[5]!.runs.push({ ...f.points[5]!.runs[3], id: "extra" }); }],
|
||||
["duplicate run ID", (f) => { f.points[5]!.runs[3]!.id = "run-0"; }],
|
||||
["missing reset", (f) => { f.points[5]!.runs.splice(2, 1); }],
|
||||
["missing synthetic reset receipt", (f) => { f.points[5]!.resetReceipts.splice(0); }],
|
||||
["foreign reset summary agent", (f) => { f.points[5]!.resetReceipts[0]!.agentId = "foreign"; }],
|
||||
["foreign reset summary company", (f) => { f.points[5]!.resetReceipts[0]!.companyId = "foreign"; }],
|
||||
["foreign reset summary issue", (f) => { f.points[5]!.resetReceipts[0]!.contextIssueId = "foreign"; }],
|
||||
["wrong reset summary run", (f) => { f.points[5]!.resetReceipts[0]!.runId = "run-0"; }],
|
||||
["wrong reset summary wake", (f) => { f.points[5]!.resetReceipts[0]!.wakeCommentId = "user-0"; }],
|
||||
["reset summary not successful", (f) => { f.points[5]!.resetReceipts[0]!.status = "running"; }],
|
||||
["contradictory reset detail", (f) => { f.points[5]!.runs[2]!.resultJson.conversationReset = false; }],
|
||||
["provider work disguised as reset", (f) => { f.points[5]!.runs[2]!.runnerProfileJson = { nativeExecutionInput: { provider: { kind: f.scope.provider, model: f.scope.model } } }; }],
|
||||
["native session attached to synthetic reset", (f) => { f.points[5]!.runs[2]!.nativeSessionId = "unexpected-provider-session"; }],
|
||||
["reset publishes an answer", (f) => { f.points[5]!.comments.push({ id: "reset-answer", authorAgentId: f.scope.agentId, createdByRunId: "run-2", body: "Hi" }); }],
|
||||
["unrequested completion callback", (f) => { f.points[5]!.runs[3]!.contextSnapshot.source = "automation"; }],
|
||||
["wrong wake comment", (f) => { f.points[5]!.runs[3]!.contextSnapshot.wakeCommentId = "user-0"; }],
|
||||
["run has conflicting issue IDs", (f) => { f.points[5]!.runs[3]!.issueId = "foreign"; }],
|
||||
["changed original history", (f) => { f.points[5]!.comments[0]!.body = "changed"; }],
|
||||
["missing original history", (f) => { f.points[5]!.comments.splice(1, 1); }],
|
||||
["reopen repeats the send", (f) => { f.points[5]!.runs.push({ ...f.points[5]!.runs[3], id: "repeated" }); }],
|
||||
["reply not observed after reload", (f) => { f.points[5]!.reply!.observedAfterReload = false; }],
|
||||
["hidden final reply", (f) => { f.points[5]!.reply!.visible = false; }],
|
||||
["created work", (f) => { f.points[5]!.tasks.push({ id: "unexpected" }); }],
|
||||
["created project", (f) => { f.points[5]!.activity.push({ id: "created-project", companyId: f.scope.companyId, agentId: f.scope.agentId,
|
||||
runId: "run-1", action: "project.created", entityType: "project", entityId: "unexpected" }); }],
|
||||
["truncated creation history", (f) => { f.points[5]!.activity = Array.from({ length: 500 }, () => ({})); }],
|
||||
["missing run collection", (f) => { f.points[5]!.runs = undefined as unknown as Row[]; }],
|
||||
["malformed history row", (f) => { f.points[5]!.comments.push(null as unknown as Row); }],
|
||||
];
|
||||
it.each(mutants)("rejects %s", (_name, mutate) => {
|
||||
const f = chatContinuityFixture();
|
||||
mutate(f);
|
||||
expect(allPass(f)).toBe(false);
|
||||
});
|
||||
it("rejects another agent's activity rather than filtering an unscoped response", () => {
|
||||
const f = chatContinuityFixture();
|
||||
f.points[5]!.activity.push({ companyId: f.scope.companyId, agentId: "other-agent", runId: "other-run", action: "project.created" });
|
||||
expect(allPass(f)).toBe(false);
|
||||
});
|
||||
it("accepts complete owned activity and rejects unaccounted creation by that same agent", () => {
|
||||
const f = chatContinuityFixture();
|
||||
const row = { id: "owned-event", companyId: f.scope.companyId, agentId: f.scope.agentId,
|
||||
runId: "run-0", action: "heartbeat.run_completed", entityType: "heartbeat_run", entityId: "run-0" };
|
||||
f.points[5]!.activity.push(row);
|
||||
expect(allPass(f)).toBe(true);
|
||||
row.action = "project.created"; row.runId = "unaccounted-owned-run";
|
||||
expect(allPass(f)).toBe(false);
|
||||
});
|
||||
it("does not treat a compact run as sufficient without actual native output proof", () => {
|
||||
const f = compactChatContinuityFixture();
|
||||
f.points[5]!.events = [];
|
||||
expect(allPass(f)).toBe(false);
|
||||
});
|
||||
it("retains backward-compatible issue aliases only when they agree", () => {
|
||||
expect(chatContinuityIssueId({ nativeIssueId: "chat", contextSnapshot: { issueId: "chat" } })).toBe("chat");
|
||||
expect(chatContinuityIssueId({ issueId: "foreign", nativeIssueId: "chat" })).toBeUndefined();
|
||||
expect(chatContinuityIssueId({})).toBeUndefined();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,157 @@
|
||||
import { chatMarker } from "../chat-cases.js";
|
||||
import { taskReplyBelongsToRun, type TaskReplyObservation } from "../native-task-reply.js";
|
||||
|
||||
type Row = Record<string, any>;
|
||||
export const CHAT_CONTINUITY_CHECKS = [
|
||||
"chat-fresh-owned-conversation", "chat-user-messages-stored-once", "chat-reload-preserved-context",
|
||||
"chat-new-session-boundary", "chat-fresh-context-answer", "chat-history-preserved-after-reload",
|
||||
"chat-no-unrequested-work", "chat-exact-turn-accounting", "configured-model-and-environment",
|
||||
] as const;
|
||||
|
||||
export function chatContinuityScenario(campaignId: string) {
|
||||
const remembered = chatMarker("OLDCONTEXT", campaignId), marker = chatMarker("CHAT", campaignId);
|
||||
return {
|
||||
remembered, marker,
|
||||
remember: `For this conversation only, remember the phrase ${remembered}. Just acknowledge briefly; no project or task is needed.`,
|
||||
recall: "What phrase did I just ask you to remember? Reply with the phrase only. No project or task is needed.",
|
||||
reset: "/new",
|
||||
fresh: `Without reading older history or files, if you have a remembered phrase in your current context return it; otherwise reply exactly ${marker}. Do not look it up. No project or task is needed.`,
|
||||
};
|
||||
}
|
||||
|
||||
export interface ChatContinuityScope {
|
||||
companyId: string; userId: string; agentId: string; agentName: string;
|
||||
environmentId: string; provider: string; model: string; campaignId: string;
|
||||
agentHistory?: { source: "company-agent-runs"; companyId: string; agentId: string; limit: 1; runIds: string[]; observedBeforeSend: true };
|
||||
}
|
||||
export interface ChatContinuityResetReceipt {
|
||||
source: "issue-runs"; companyId: string; issueId: string;
|
||||
runId: string; agentId: string; status: string; contextIssueId: string; wakeCommentId: string;
|
||||
resultJson: { conversationReset: true };
|
||||
}
|
||||
export interface ChatContinuityPoint {
|
||||
phase: "initial" | "returned" | "recalled" | "reset" | "fresh" | "reopened";
|
||||
chat: Row; runs: Row[]; comments: Row[]; tasks: Row[]; activity: Row[];
|
||||
reply?: TaskReplyObservation & { runId: string; agentId: string; observedAfterReload: boolean };
|
||||
events?: Row[];
|
||||
resetReceipts: ChatContinuityResetReceipt[];
|
||||
}
|
||||
const nonempty = (value: unknown): value is string => typeof value === "string" && value.length > 0;
|
||||
export const chatContinuityResetRun = (run: Row) => run.contextSnapshot?.conversationReset === true || run.resultJson?.conversationReset === true;
|
||||
export const chatContinuityIssueId = (run: Row): string | undefined => {
|
||||
const ids = [run.issueId, run.nativeIssueId, run.contextSnapshot?.issueId, run.contextSnapshot?.taskId].filter(nonempty);
|
||||
return ids.length && new Set(ids).size === 1 ? ids[0] : undefined;
|
||||
};
|
||||
const userBody = (row: Row) => typeof row.body === "string" ? row.body.replaceAll("\\_", "_").trim() : "";
|
||||
const sameIds = (left: Row[], right: Row[]) => left.length === right.length && left.every(row => right.some(other => row.id === other.id));
|
||||
const commentContent = (row: Row) => [row.id, row.companyId, row.issueId, row.body, row.authorAgentId,
|
||||
row.authorUserId, row.createdByRunId, row.conversationSessionGeneration];
|
||||
const sameRows = (left: Row[], right: Row[]) => left.every(row => right.some(other => other.id === row.id
|
||||
&& JSON.stringify(commentContent(other)) === JSON.stringify(commentContent(row))));
|
||||
|
||||
export function chatContinuityActivityComplete(scope: Pick<ChatContinuityScope, "companyId" | "agentId">, rows: Row[]): boolean {
|
||||
return Array.isArray(rows) && rows.length < 500 && new Set(rows.map(row => row?.id)).size === rows.length
|
||||
&& rows.every(row => row && nonempty(row.id) && row.companyId === scope.companyId && row.agentId === scope.agentId
|
||||
&& nonempty(row.action) && nonempty(row.entityType) && nonempty(row.entityId));
|
||||
}
|
||||
|
||||
export function chatContinuityResetReceiptMatches(scope: Pick<ChatContinuityScope, "companyId" | "agentId">,
|
||||
chatId: string, run: Row, receipt: ChatContinuityResetReceipt): boolean {
|
||||
const fullResult = run.resultJson;
|
||||
const optionalFlagValid = fullResult == null || typeof fullResult === "object" && !Array.isArray(fullResult)
|
||||
&& (!Object.hasOwn(fullResult, "conversationReset") || fullResult.conversationReset === true);
|
||||
return receipt?.source === "issue-runs" && receipt.companyId === scope.companyId && receipt.issueId === chatId
|
||||
&& receipt.runId === run.id && receipt.agentId === scope.agentId && receipt.status === "succeeded"
|
||||
&& receipt.contextIssueId === chatId && nonempty(receipt.wakeCommentId)
|
||||
&& receipt.wakeCommentId === run.contextSnapshot?.wakeCommentId && receipt.resultJson?.conversationReset === true
|
||||
&& run.companyId === scope.companyId && run.agentId === scope.agentId && chatContinuityIssueId(run) === chatId
|
||||
&& run.status === "succeeded" && run.contextSnapshot?.conversationReset === true && optionalFlagValid
|
||||
&& !run.runnerProfileJson?.nativeExecutionInput && !run.nativeSessionId;
|
||||
}
|
||||
|
||||
export function gradeChatContinuity(scope: ChatContinuityScope, points: ChatContinuityPoint[]) {
|
||||
const scenario = chatContinuityScenario(scope.campaignId);
|
||||
const phases = ["initial", "returned", "recalled", "reset", "fresh", "reopened"] as const;
|
||||
const complete = points.length === phases.length && points.every((point, index) => point?.phase === phases[index]
|
||||
&& point.chat && [point.runs, point.comments, point.tasks, point.activity, point.resetReceipts].every(rows => Array.isArray(rows)
|
||||
&& rows.every(row => row && typeof row === "object" && !Array.isArray(row))));
|
||||
if (!complete) return CHAT_CONTINUITY_CHECKS.map(id => ({ id, passed: false }));
|
||||
const [initial, returned, recalled, reset, fresh, reopened] = complete ? points : [];
|
||||
const chatId = initial?.chat?.id;
|
||||
const scoped = complete && [scope.companyId, scope.userId, scope.agentId, scope.agentName, scope.environmentId,
|
||||
scope.provider, scope.model, scope.campaignId].every(nonempty) && nonempty(chatId)
|
||||
&& scope.agentHistory?.source === "company-agent-runs" && scope.agentHistory.companyId === scope.companyId
|
||||
&& scope.agentHistory.agentId === scope.agentId && scope.agentHistory.limit === 1 && scope.agentHistory.observedBeforeSend === true
|
||||
&& Array.isArray(scope.agentHistory.runIds) && scope.agentHistory.runIds.length === 0
|
||||
&& points.every(point => point.chat?.id === chatId && point.chat.companyId === scope.companyId
|
||||
&& point.chat.assigneeAgentId === scope.agentId && point.chat.conversationAgentId === scope.agentId
|
||||
&& point.chat.conversationUserId === scope.userId && point.chat.createdByUserId === scope.userId
|
||||
&& point.chat.title === `Chat with ${scope.agentName}` && point.chat.status === "in_review"
|
||||
&& point.chat.conversationState === "waiting" && !point.chat.executionRunId && !point.chat.scheduledRetry && !point.chat.activeRecoveryAction);
|
||||
const requested = [scenario.remember, scenario.recall, scenario.reset, scenario.fresh];
|
||||
const finalComments = reopened?.comments ?? [];
|
||||
const board = finalComments.filter(row => row.authorUserId === scope.userId && !row.authorAgentId);
|
||||
const userMessages = scoped && board.length === requested.length && new Set(board.map(row => row.id)).size === requested.length
|
||||
&& requested.every(body => board.filter(row => userBody(row) === body && row.companyId === scope.companyId && row.issueId === chatId).length === 1)
|
||||
&& finalComments.every(row => !row.authorUserId || row.authorUserId === scope.userId);
|
||||
const runs = reopened?.runs ?? [], providerRuns = runs.filter(run => !chatContinuityResetRun(run)), resetRuns = runs.filter(chatContinuityResetRun);
|
||||
const runBinding = scoped && points.every(point => point.runs.every(run => nonempty(run.id) && run.companyId === scope.companyId
|
||||
&& run.agentId === scope.agentId && chatContinuityIssueId(run) === chatId && run.status === "succeeded"));
|
||||
const providerBinding = runBinding && providerRuns.length === 3 && points.flatMap(point => point.runs)
|
||||
.filter(run => !chatContinuityResetRun(run)).every(run => {
|
||||
const provider = run.runnerProfileJson?.nativeExecutionInput?.provider;
|
||||
return run.runtimeMode === "native" && provider?.kind === scope.provider && provider?.model === scope.model
|
||||
&& run.contextSnapshot?.paperclipEnvironment?.id === scope.environmentId
|
||||
&& run.nativePhase === "committed"
|
||||
&& nonempty(run.nativeSessionId);
|
||||
});
|
||||
const expectedCounts = [1, 1, 2, 3, 4, 4];
|
||||
const exactTurns = runBinding && providerBinding && resetRuns.length === 1 && runs.length === 4 && new Set(runs.map(run => run.id)).size === 4
|
||||
&& points.every(point => point.resetReceipts.length === point.runs.filter(chatContinuityResetRun).length
|
||||
&& point.runs.filter(chatContinuityResetRun).every(run => point.resetReceipts
|
||||
.filter(receipt => chatContinuityResetReceiptMatches(scope, chatId, run, receipt)).length === 1))
|
||||
&& points.every((point, index) => point.runs.length === expectedCounts[index]
|
||||
&& point.runs.every(run => runs.some(final => final.id === run.id && final.status === run.status)))
|
||||
&& runs.every(run => {
|
||||
const request = board.find(comment => comment.id === run.contextSnapshot?.wakeCommentId);
|
||||
return request && run.contextSnapshot?.source === "issue.comment"
|
||||
&& (chatContinuityResetRun(run) ? userBody(request) === scenario.reset : userBody(request) !== scenario.reset);
|
||||
}) && new Set(runs.map(run => run.contextSnapshot?.wakeCommentId)).size === 4;
|
||||
const pointReply = (point: ChatContinuityPoint | undefined) => {
|
||||
const receipt = point?.reply, run = point?.runs.find(run => run.id === receipt?.runId);
|
||||
return receipt && run && receipt.agentId === scope.agentId && receipt.observedAfterReload === true
|
||||
&& taskReplyBelongsToRun(receipt, run, scope.agentId, point!.comments, point!.events ?? []) ? receipt : undefined;
|
||||
};
|
||||
const originalReply = pointReply(initial), returnedReply = pointReply(returned), recalledReply = pointReply(recalled), freshReply = pointReply(fresh), reopenedReply = pointReply(reopened);
|
||||
const firstRuns = initial?.runs.filter(run => !chatContinuityResetRun(run)) ?? [];
|
||||
const recallRuns = recalled?.runs.filter(run => !chatContinuityResetRun(run) && run.id !== firstRuns[0]?.id) ?? [];
|
||||
const freshRuns = fresh?.runs.filter(run => !chatContinuityResetRun(run) && !recalled?.runs.some(old => old.id === run.id)) ?? [];
|
||||
const first = firstRuns[0], second = recallRuns[0], last = freshRuns[0], resetRun = resetRuns[0];
|
||||
const recall = exactTurns && Boolean(originalReply && returnedReply && recalledReply && originalReply.runId === first?.id
|
||||
&& returnedReply.runId === originalReply.runId && returnedReply.body === originalReply.body && recalledReply.runId === second?.id
|
||||
&& recalledReply.body.includes(scenario.remembered) && first?.nativeSessionId === second?.nativeSessionId
|
||||
&& initial!.chat.conversationSessionGeneration === 0 && returned!.chat.conversationSessionGeneration === 0
|
||||
&& recalled!.chat.conversationSessionGeneration === 0 && first?.contextSnapshot?.conversationSessionGeneration === 0
|
||||
&& second?.contextSnapshot?.conversationSessionGeneration === 0 && sameIds(initial!.runs, returned!.runs)
|
||||
&& sameRows(initial!.comments, returned!.comments));
|
||||
const boundary = exactTurns && Boolean(first && second && last && resetRun && freshRuns.length === 1
|
||||
&& reset!.chat.conversationSessionGeneration === 1 && fresh!.chat.conversationSessionGeneration === 1
|
||||
&& reopened!.chat.conversationSessionGeneration === 1 && last.contextSnapshot?.conversationSessionGeneration === 1
|
||||
&& resetRun.contextSnapshot?.conversationSessionGeneration === 1
|
||||
&& Object.hasOwn(last, "sessionIdBefore") && last.sessionIdBefore === null
|
||||
&& last.nativeSessionId !== first.nativeSessionId && last.nativeSessionId !== second.nativeSessionId
|
||||
&& typeof last.contextSnapshot?.paperclipTaskMarkdown === "string"
|
||||
&& !last.contextSnapshot.paperclipTaskMarkdown.includes(scenario.remembered)
|
||||
&& finalComments.every(row => row.createdByRunId !== resetRun.id));
|
||||
const freshAnswer = boundary && Boolean(freshReply && freshReply.runId === last?.id
|
||||
&& freshReply.body.trim() === scenario.marker && !freshReply.body.includes(scenario.remembered));
|
||||
const history = scoped && Boolean(reopenedReply && freshReply && reopenedReply.runId === freshReply.runId
|
||||
&& reopenedReply.body === freshReply.body && sameIds(fresh!.runs, reopened!.runs)
|
||||
&& sameRows(initial!.comments, reopened!.comments) && sameRows(recalled!.comments, reopened!.comments)
|
||||
&& sameRows(fresh!.comments, reopened!.comments));
|
||||
const noWork = scoped && points.every(point => Array.isArray(point.tasks) && point.tasks.length === 0
|
||||
&& chatContinuityActivityComplete(scope, point.activity)
|
||||
&& !point.activity.some(row => ["issue.created", "issue.child_created", "project.created", "agent.created", "agent.hire_created"].includes(row.action)));
|
||||
const values = [scoped, userMessages, recall, boundary, freshAnswer, history, noWork, exactTurns, providerBinding];
|
||||
return CHAT_CONTINUITY_CHECKS.map((id, index) => ({ id, passed: Boolean(values[index]) }));
|
||||
}
|
||||
@@ -0,0 +1,180 @@
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
import { mkdtemp, readFile, rm } from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import type { JourneyContext, JourneyEvidence, TargetSession } from "./types.js";
|
||||
import { chatContinuityFixture, compactChatContinuityFixture } from "./chat-continuity.test-fixture.js";
|
||||
|
||||
const mocks = vi.hoisted(() => ({ send: vi.fn(), reply: vi.fn(), capture: vi.fn() }));
|
||||
vi.mock("../chat-flow.js", () => ({ sendChatMessage: mocks.send }));
|
||||
vi.mock("../native-task-reply.js", async importOriginal => ({
|
||||
...await importOriginal<typeof import("../native-task-reply.js")>(), readBoundTaskFinalReply: mocks.reply,
|
||||
}));
|
||||
vi.mock("./product-screenshot.js", () => ({ captureProductScreenshot: mocks.capture }));
|
||||
vi.mock("@playwright/test", () => ({ expect: (locator: { isVisible(): Promise<boolean> }) => ({
|
||||
async toBeVisible() { expect(await locator.isVisible()).toBe(true); },
|
||||
}) }));
|
||||
import { runChatContinuity } from "./chat-continuity-flow.js";
|
||||
|
||||
const directories: string[] = [];
|
||||
afterEach(async () => {
|
||||
vi.restoreAllMocks();
|
||||
vi.resetAllMocks();
|
||||
for (const directory of directories.splice(0)) await rm(directory, { recursive: true, force: true });
|
||||
});
|
||||
async function fixture(options: { preexisting?: boolean; intentFails?: boolean; sendFails?: boolean; foreignRun?: boolean;
|
||||
compactRuns?: boolean; unrelatedActivity?: number; ignoreActivityScope?: boolean; priorAgentStatus?: string; mixedAgentHistory?: boolean;
|
||||
provisioningActor?: "board" | "owned-agent" } = {}) {
|
||||
const f = options.compactRuns ? compactChatContinuityFixture() : chatContinuityFixture();
|
||||
const directory = await mkdtemp(path.join(os.tmpdir(), "chat-continuity-flow-"));
|
||||
directories.push(directory);
|
||||
let stage = options.preexisting ? 0 : -1;
|
||||
const messages: string[] = [], actions: string[] = [];
|
||||
const realTimeout = globalThis.setTimeout;
|
||||
vi.spyOn(globalThis, "setTimeout").mockImplementation(((callback: (...args: unknown[]) => void, delay?: number, ...args: unknown[]) =>
|
||||
realTimeout(callback, delay === 1000 ? 0 : delay, ...args)) as typeof setTimeout);
|
||||
const current = () => f.points[stage]!;
|
||||
const companyActivity = [
|
||||
...Array.from({ length: options.unrelatedActivity ?? 0 }, (_, index) => ({ id: `unrelated-${index}`,
|
||||
companyId: f.scope.companyId, agentId: "other-agent", runId: "other-run", action: "project.created", entityType: "project", entityId: `other-project-${index}` })),
|
||||
...(options.provisioningActor ? [{ id: "provisioned-agent", companyId: f.scope.companyId,
|
||||
agentId: options.provisioningActor === "board" ? null : f.scope.agentId, runId: null,
|
||||
action: "agent.created", entityType: "agent", entityId: f.scope.agentId }] : []),
|
||||
];
|
||||
const companyRuns = Array.from({ length: options.unrelatedActivity ?? 0 }, (_, index) => ({ id: `unrelated-run-${index}`,
|
||||
companyId: f.scope.companyId, agentId: "other-agent", status: "succeeded" }));
|
||||
if (options.priorAgentStatus) companyRuns.push({ id: "prior-owned-run", companyId: f.scope.companyId, agentId: f.scope.agentId, status: options.priorAgentStatus });
|
||||
const get = vi.fn(async (route: string) => {
|
||||
if (route === "/api/instance/settings/experimental") return { enableAgentChat: true };
|
||||
if (route === "/api/companies/company/heartbeat-runs?agentId=agent&limit=1&summary=true") {
|
||||
actions.push("history");
|
||||
if (options.mixedAgentHistory) return [{ id: "foreign-run", companyId: "foreign-company", agentId: "other-agent", status: "running" }];
|
||||
return companyRuns.filter(run => run.agentId === f.scope.agentId).slice(0, 1);
|
||||
}
|
||||
if (route === "/api/companies/company/chats/agent") return stage < 0 ? null : structuredClone(current().chat);
|
||||
if (route === "/api/issues/chat/runs") return current().runs.map(run => ({ runId: run.id, agentId: run.agentId, status: run.status,
|
||||
contextIssueId: run.contextSnapshot.issueId, wakeCommentId: run.contextSnapshot.wakeCommentId,
|
||||
resultJson: run.contextSnapshot.conversationReset ? { conversationReset: true } : {} }));
|
||||
if (route === "/api/issues/chat/comments?order=asc") return structuredClone(current().comments);
|
||||
if (route === "/api/companies/company/issues?view=index&assigneeAgentId=agent&includeAgentCreatedTasks=true&limit=2") return [];
|
||||
if (route === "/api/companies/company/activity?agentId=agent&limit=500") {
|
||||
return (options.ignoreActivityScope ? companyActivity : companyActivity.filter(row => row.agentId === f.scope.agentId)).slice(0, 500);
|
||||
}
|
||||
if (/\/api\/heartbeat-runs\/run-\d\/events\?/.test(route)) return structuredClone(current().events ?? []);
|
||||
const id = route.match(/^\/api\/heartbeat-runs\/(run-\d)$/)?.[1];
|
||||
if (id) {
|
||||
const run = structuredClone(current().runs.find(row => row.id === id)!);
|
||||
if (options.foreignRun) run.nativeIssueId = "foreign";
|
||||
return run;
|
||||
}
|
||||
throw new Error(`unexpected GET ${route}`);
|
||||
});
|
||||
const page = { goto: vi.fn(async (_url: string) => { actions.push("navigate"); }), reload: vi.fn(async () => { actions.push("reload"); }),
|
||||
getByTestId: vi.fn(() => ({ isVisible: async () => true })) };
|
||||
const own = vi.fn(async (resource: { id?: string }) => {
|
||||
actions.push(resource.id ? "receipt" : "intent");
|
||||
if (options.intentFails && !resource.id) throw new Error("ownership intent failed");
|
||||
});
|
||||
const context = { page, api: { get, origin: "https://qa.example.test" },
|
||||
company: { id: f.scope.companyId, issuePrefix: "QA" }, user: { id: f.scope.userId, name: "QA" },
|
||||
environmentId: f.scope.environmentId, agent: { id: f.scope.agentId, name: f.scope.agentName,
|
||||
provider: f.scope.provider, model: f.scope.model, runtimeMode: "native" } } as unknown as JourneyContext;
|
||||
const evidence: JourneyEvidence = { outcome: "passed", startingState: "recipe", phases: [], checks: [], resourceDisposition: "released", privacy: "private" };
|
||||
mocks.send.mockImplementation(async (_page, message: string) => {
|
||||
actions.push("send"); messages.push(message);
|
||||
if (options.sendFails) throw new Error("uncertain send response");
|
||||
const index = [f.scenario.remember, f.scenario.recall, f.scenario.reset, f.scenario.fresh].indexOf(message);
|
||||
expect(index).toBe(messages.length - 1);
|
||||
stage = [0, 2, 3, 4][index]!;
|
||||
});
|
||||
mocks.reply.mockImplementation(async input => {
|
||||
await input.page.reload();
|
||||
const data = await input.load();
|
||||
expect(data.run.id).toBe(input.runId);
|
||||
const point = f.points.find(point => point.reply?.runId === input.runId)!;
|
||||
return structuredClone(point.reply!);
|
||||
});
|
||||
mocks.capture.mockResolvedValue(undefined);
|
||||
return { ...f, context, evidence, own, directory, messages, actions, page, get, companyActivity,
|
||||
run: () => runChatContinuity(context, { own } as unknown as TargetSession, f.scope.campaignId, evidence,
|
||||
{ directory, deadlineAt: Date.now() + 10_000 }) };
|
||||
}
|
||||
describe("chat continuity journey controls", () => {
|
||||
it("uses ordinary user messages once, owns the fresh chat before dispatch, and retains the independently graded result", async () => {
|
||||
const f = await fixture();
|
||||
await f.run();
|
||||
expect(f.messages).toEqual([f.scenario.remember, f.scenario.recall, "/new", f.scenario.fresh]);
|
||||
expect(f.actions.indexOf("intent")).toBeLessThan(f.actions.indexOf("send"));
|
||||
expect(f.actions.indexOf("history")).toBeLessThan(f.actions.indexOf("intent"));
|
||||
expect(f.own).toHaveBeenCalledTimes(2);
|
||||
expect(f.own.mock.calls[0]![0]).toMatchObject({ journeyId: "chat-continuity", freshChat: { absent: true, userId: f.scope.userId } });
|
||||
expect(f.own.mock.calls[1]![0]).toMatchObject({ id: "chat", journeyId: "chat-continuity" });
|
||||
expect(f.evidence.checks).toHaveLength(9);
|
||||
expect(f.evidence.checks.every(check => check.passed)).toBe(true);
|
||||
expect(mocks.reply).toHaveBeenCalledTimes(5);
|
||||
expect(mocks.capture).toHaveBeenCalledTimes(6);
|
||||
expect(f.page.goto.mock.calls.flat().every(url => typeof url !== "string" || url.startsWith("https://qa.example.test/"))).toBe(true);
|
||||
const saved = JSON.parse(await readFile(path.join(f.directory, "chat-continuity.json"), "utf8"));
|
||||
expect(saved.points.map((point: { phase: string }) => point.phase)).toEqual(["initial", "returned", "recalled", "reset", "fresh", "reopened"]);
|
||||
expect(saved.checks.every((check: { passed: boolean }) => check.passed)).toBe(true);
|
||||
});
|
||||
it("accepts compact details with original reset summaries despite more than 500 unrelated company events", async () => {
|
||||
const f = await fixture({ compactRuns: true, unrelatedActivity: 750 });
|
||||
await f.run();
|
||||
expect(f.evidence.checks.every(check => check.passed)).toBe(true);
|
||||
const saved = JSON.parse(await readFile(path.join(f.directory, "chat-continuity.json"), "utf8"));
|
||||
expect(saved.points.every((point: { activity: unknown[] }) => point.activity.length === 0)).toBe(true);
|
||||
expect(saved.points[5].resetReceipts[0]).toMatchObject({ source: "issue-runs", runId: "run-2", agentId: f.scope.agentId,
|
||||
contextIssueId: "chat", wakeCommentId: "user-2", resultJson: { conversationReset: true } });
|
||||
expect(saved.points[5].runs[2].resultJson).toBeNull();
|
||||
});
|
||||
it.each([1, 750])("rejects an ignored agent filter instead of filtering %s unrelated global activity rows", async unrelatedActivity => {
|
||||
const f = await fixture({ unrelatedActivity, ignoreActivityScope: true });
|
||||
await expect(f.run()).rejects.toThrow("chat_continuity_observation_incomplete");
|
||||
expect(f.messages).toEqual([f.scenario.remember]);
|
||||
});
|
||||
it("accepts board provisioning whose created entity is the owned agent, rather than treating its entity id as the actor", async () => {
|
||||
const f = await fixture({ provisioningActor: "board", unrelatedActivity: 750 });
|
||||
expect(f.companyActivity.at(-1)).toMatchObject({ action: "agent.created", agentId: null, entityId: f.scope.agentId });
|
||||
await f.run();
|
||||
expect(f.evidence.checks.every(check => check.passed)).toBe(true);
|
||||
const saved = JSON.parse(await readFile(path.join(f.directory, "chat-continuity.json"), "utf8"));
|
||||
expect(saved.points.every((point: { activity: unknown[] }) => point.activity.length === 0)).toBe(true);
|
||||
});
|
||||
it("rejects creation attributed to the fresh owned agent even if that activity predates the first send", async () => {
|
||||
const f = await fixture({ provisioningActor: "owned-agent" });
|
||||
await expect(f.run()).rejects.toThrow("chat_continuity_evidence_failed");
|
||||
expect(f.evidence.checks.find(check => check.id === "chat-no-unrequested-work")?.passed).toBe(false);
|
||||
});
|
||||
it.each(["succeeded", "running", "queued", "failed"])("rejects prior %s work on a reused agent before any send", async priorAgentStatus => {
|
||||
const f = await fixture({ priorAgentStatus, unrelatedActivity: 750 });
|
||||
await expect(f.run()).rejects.toThrow("fresh_chat_agent_required");
|
||||
expect(f.messages).toEqual([]);
|
||||
expect(f.own).not.toHaveBeenCalled();
|
||||
});
|
||||
it("rejects mixed or foreign agent history before any send", async () => {
|
||||
const f = await fixture({ mixedAgentHistory: true });
|
||||
await expect(f.run()).rejects.toThrow("chat_continuity_agent_history_scope_mismatch");
|
||||
expect(f.messages).toEqual([]);
|
||||
expect(f.own).not.toHaveBeenCalled();
|
||||
});
|
||||
it.each(["preexisting", "intentFails"] as const)("does not dispatch model work when %s rejects ownership", async option => {
|
||||
const f = await fixture({ [option]: true });
|
||||
await expect(f.run()).rejects.toThrow();
|
||||
expect(f.messages).toEqual([]);
|
||||
expect(f.evidence.checks.every(check => !check.passed)).toBe(true);
|
||||
});
|
||||
it("never replays an uncertain send", async () => {
|
||||
const f = await fixture({ sendFails: true });
|
||||
await expect(f.run()).rejects.toThrow("uncertain send response");
|
||||
expect(f.messages).toEqual([f.scenario.remember]);
|
||||
expect(mocks.send).toHaveBeenCalledTimes(1);
|
||||
expect(f.evidence.checks.some(check => !check.passed)).toBe(true);
|
||||
});
|
||||
it("rejects a foreign native run before sending another message", async () => {
|
||||
const f = await fixture({ foreignRun: true });
|
||||
await expect(f.run()).rejects.toThrow("chat_continuity_run_mismatch");
|
||||
expect(f.messages).toEqual([f.scenario.remember]);
|
||||
expect(f.evidence.checks.some(check => !check.passed)).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,164 @@
|
||||
import { mkdir, writeFile } from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { expect } from "@playwright/test";
|
||||
import { sendChatMessage } from "../chat-flow.js";
|
||||
import { readBoundTaskFinalReply } from "../native-task-reply.js";
|
||||
import { collectRunEvents } from "../run-observations.js";
|
||||
import { chatContinuityActivityComplete, chatContinuityIssueId, chatContinuityResetReceiptMatches,
|
||||
chatContinuityResetRun, chatContinuityScenario, gradeChatContinuity,
|
||||
type ChatContinuityPoint, type ChatContinuityScope, type ChatContinuityResetReceipt } from "./chat-continuity-evidence.js";
|
||||
import { createObservedReader } from "./observation-read.js";
|
||||
import { captureProductScreenshot } from "./product-screenshot.js";
|
||||
import { JourneyError, type JourneyContext, type JourneyEvidence, type TargetSession } from "./types.js";
|
||||
|
||||
type Row = Record<string, any>;
|
||||
interface Options { deadlineAt: number; directory: string }
|
||||
|
||||
export async function runChatContinuity(context: JourneyContext, session: TargetSession,
|
||||
campaignId: string, evidence: JourneyEvidence, options: Options) {
|
||||
const agent = context.agent;
|
||||
if (!agent || agent.runtimeMode !== "native" || !agent.model || !agent.provider || !context.environmentId) {
|
||||
throw new JourneyError("unsupported", "native_agent_and_environment_required");
|
||||
}
|
||||
const scope: ChatContinuityScope = { companyId: context.company.id, userId: context.user.id,
|
||||
agentId: agent.id, agentName: agent.name, environmentId: context.environmentId,
|
||||
provider: agent.provider, model: agent.model, campaignId };
|
||||
const scenario = chatContinuityScenario(campaignId), reader = createObservedReader(context.api, options.deadlineAt);
|
||||
const chatPath = `/api/companies/${scope.companyId}/chats/${agent.id}`;
|
||||
const chatUrl = `${context.api.origin}/${encodeURIComponent(context.company.issuePrefix)}/chats/${encodeURIComponent(agent.id)}`;
|
||||
const points: ChatContinuityPoint[] = [];
|
||||
let chatId: string | undefined, owned = false;
|
||||
const remaining = () => {
|
||||
const value = options.deadlineAt - Date.now();
|
||||
if (value <= 0) throw new JourneyError("failed", "chat_continuity_deadline");
|
||||
return value;
|
||||
};
|
||||
await mkdir(options.directory, { recursive: true, mode: 0o700 });
|
||||
const retain = async () => writeFile(path.join(options.directory, "chat-continuity.json"),
|
||||
JSON.stringify({ scope, points, checks: gradeChatContinuity(scope, points) }, null, 2), { mode: 0o600 });
|
||||
const freshChat = { companyId: scope.companyId, userId: scope.userId, leadAgentId: agent.id, absent: true as const };
|
||||
const read = async (): Promise<Omit<ChatContinuityPoint, "phase"> | undefined> => {
|
||||
const chat = await reader.get<Row | null>(chatPath);
|
||||
if (!chat) return;
|
||||
if (!chat.id || chat.companyId !== scope.companyId || chat.assigneeAgentId !== agent.id
|
||||
|| chat.conversationAgentId !== agent.id || chat.conversationUserId !== scope.userId
|
||||
|| chat.createdByUserId !== scope.userId || chat.title !== `Chat with ${agent.name}` || chatId && chat.id !== chatId) {
|
||||
throw new JourneyError("failed", "chat_continuity_scope_mismatch");
|
||||
}
|
||||
chatId ??= chat.id;
|
||||
if (!owned) {
|
||||
await session.own({ kind: "issue", name: chat.title, id: chat.id, journeyId: "chat-continuity", freshChat });
|
||||
owned = true;
|
||||
}
|
||||
const [summaries, comments, tasks, activity] = await Promise.all([
|
||||
reader.get<Row[]>(`/api/issues/${chat.id}/runs`), reader.get<Row[]>(`/api/issues/${chat.id}/comments?order=asc`),
|
||||
reader.get<Row[]>(`/api/companies/${scope.companyId}/issues?view=index&assigneeAgentId=${agent.id}&includeAgentCreatedTasks=true&limit=2`),
|
||||
reader.get<Row[]>(`/api/companies/${scope.companyId}/activity?agentId=${agent.id}&limit=500`),
|
||||
]);
|
||||
if (![summaries, comments, tasks, activity].every(Array.isArray) || summaries.length > 4 || !chatContinuityActivityComplete(scope, activity)
|
||||
|| new Set(summaries.map(row => row.runId)).size !== summaries.length) {
|
||||
throw new JourneyError("failed", "chat_continuity_observation_incomplete");
|
||||
}
|
||||
const runs = await Promise.all(summaries.map(async row => {
|
||||
if (typeof row.runId !== "string" || !row.runId) throw new JourneyError("failed", "chat_continuity_run_mismatch");
|
||||
if (row.agentId !== agent.id || row.contextIssueId !== chat.id) throw new JourneyError("failed", "chat_continuity_summary_scope_mismatch");
|
||||
const run = await reader.get<Row>(`/api/heartbeat-runs/${row.runId}`);
|
||||
if (run.id !== row.runId || run.companyId !== scope.companyId || run.agentId !== agent.id
|
||||
|| chatContinuityIssueId(run) !== chat.id || !["queued", "running", "succeeded"].includes(run.status)) {
|
||||
throw new JourneyError("failed", "chat_continuity_run_mismatch");
|
||||
}
|
||||
return run;
|
||||
}));
|
||||
const resetReceipts: ChatContinuityResetReceipt[] = summaries.flatMap(summary => summary.resultJson?.conversationReset === true ? [{
|
||||
source: "issue-runs" as const, companyId: scope.companyId, issueId: chat.id,
|
||||
runId: summary.runId, agentId: summary.agentId, status: summary.status,
|
||||
contextIssueId: summary.contextIssueId, wakeCommentId: summary.wakeCommentId, resultJson: { conversationReset: true as const },
|
||||
}] : []);
|
||||
return { chat, runs, comments, tasks, activity, resetReceipts };
|
||||
};
|
||||
const wait = async (providerCount: number, resetCount: number) => {
|
||||
let stable = "";
|
||||
while (remaining() > 0) {
|
||||
const state = await read();
|
||||
if (state) {
|
||||
const providerRuns = state.runs.filter(run => !chatContinuityResetRun(run));
|
||||
const resetRuns = state.runs.filter(chatContinuityResetRun);
|
||||
if (providerRuns.length > providerCount || resetRuns.length > resetCount) throw new JourneyError("failed", "chat_continuity_extra_run");
|
||||
const ready = providerRuns.length === providerCount && resetRuns.length === resetCount
|
||||
&& state.runs.every(run => run.status === "succeeded")
|
||||
&& providerRuns.every(run => run.nativePhase === "committed")
|
||||
&& resetRuns.every(run => state.resetReceipts.filter(receipt => chatContinuityResetReceiptMatches(scope, state.chat.id, run, receipt)).length === 1)
|
||||
&& state.chat.status === "in_review" && state.chat.conversationState === "waiting"
|
||||
&& !state.chat.executionRunId && !state.chat.scheduledRetry && !state.chat.activeRecoveryAction;
|
||||
const key = ready ? JSON.stringify([state.chat.id, state.chat.conversationSessionGeneration,
|
||||
state.runs.map(run => [run.id, run.status]), state.comments.map(row => row.id)]) : "";
|
||||
if (key && stable === key) return state;
|
||||
stable = key;
|
||||
}
|
||||
await new Promise(resolve => setTimeout(resolve, Math.min(1000, remaining())));
|
||||
}
|
||||
throw new JourneyError("failed", "chat_continuity_not_settled");
|
||||
};
|
||||
const snapshot = async (phase: ChatContinuityPoint["phase"], state: Awaited<ReturnType<typeof wait>>, runId?: string) => {
|
||||
let point: ChatContinuityPoint = { phase, ...state };
|
||||
if (runId) {
|
||||
const load = async () => {
|
||||
const latest = await read();
|
||||
const run = latest?.runs.find(run => run.id === runId);
|
||||
if (!latest || !run) throw new JourneyError("failed", "chat_continuity_reply_run_missing");
|
||||
const events = await collectRunEvents<Row>((afterSeq, limit) =>
|
||||
reader.get(`/api/heartbeat-runs/${runId}/events?afterSeq=${afterSeq}&limit=${limit}`));
|
||||
return { run, comments: latest.comments, events };
|
||||
};
|
||||
const reply = await readBoundTaskFinalReply({ page: context.page, runId, agentId: agent.id,
|
||||
deadlineAt: options.deadlineAt, knownCompanyPrefixes: [context.company.issuePrefix], load });
|
||||
const latest = await read(), runEvidence = await load();
|
||||
if (!latest) throw new JourneyError("failed", "chat_continuity_reply_chat_missing");
|
||||
point = { phase, ...latest, reply, events: runEvidence.events };
|
||||
}
|
||||
points.push(point);
|
||||
await retain();
|
||||
await captureProductScreenshot(context.page, context.api.origin, path.join(options.directory, `chat-${phase}.png`), options.deadlineAt);
|
||||
return point;
|
||||
};
|
||||
const newProviderRun = (state: Awaited<ReturnType<typeof wait>>, old: Row[] = []) => {
|
||||
const added = state.runs.filter(run => !chatContinuityResetRun(run) && !old.some(before => before.id === run.id));
|
||||
if (added.length !== 1) throw new JourneyError("failed", "chat_continuity_turn_ambiguous");
|
||||
return added[0]!;
|
||||
};
|
||||
try {
|
||||
const settings = await reader.get<Row>("/api/instance/settings/experimental");
|
||||
if (settings.enableAgentChat !== true) throw new JourneyError("unsupported", "agent_chat_required");
|
||||
const history = await reader.get<Row[]>(`/api/companies/${scope.companyId}/heartbeat-runs?agentId=${agent.id}&limit=1&summary=true`);
|
||||
if (!Array.isArray(history) || history.some(run => !run || run.companyId !== scope.companyId || run.agentId !== agent.id)) {
|
||||
throw new JourneyError("failed", "chat_continuity_agent_history_scope_mismatch");
|
||||
}
|
||||
if (history.length) throw new JourneyError("unsupported", "fresh_chat_agent_required");
|
||||
scope.agentHistory = { source: "company-agent-runs", companyId: scope.companyId, agentId: agent.id, limit: 1, runIds: [], observedBeforeSend: true };
|
||||
if (await reader.get(chatPath) !== null) throw new JourneyError("unsupported", "fresh_agent_chat_required");
|
||||
await session.own({ kind: "issue", name: `Chat with ${agent.name}`, journeyId: "chat-continuity", freshChat });
|
||||
await context.page.goto(chatUrl, { waitUntil: "domcontentloaded", timeout: remaining() });
|
||||
await expect(context.page.getByTestId("task-chat-composer-input")).toBeVisible({ timeout: remaining() });
|
||||
await sendChatMessage(context.page, scenario.remember);
|
||||
const first = await wait(1, 0), initial = await snapshot("initial", first, newProviderRun(first).id);
|
||||
await context.page.goto(`${context.api.origin}/${encodeURIComponent(context.company.issuePrefix)}/issues`, { waitUntil: "domcontentloaded", timeout: remaining() });
|
||||
await context.page.goto(chatUrl, { waitUntil: "domcontentloaded", timeout: remaining() });
|
||||
await context.page.reload({ waitUntil: "domcontentloaded", timeout: remaining() });
|
||||
await expect(context.page.getByTestId("task-chat-composer-input")).toBeVisible({ timeout: remaining() });
|
||||
await snapshot("returned", await wait(1, 0), initial.reply!.runId);
|
||||
await sendChatMessage(context.page, scenario.recall);
|
||||
const second = await wait(2, 0), recalled = await snapshot("recalled", second, newProviderRun(second, initial.runs).id);
|
||||
await sendChatMessage(context.page, scenario.reset);
|
||||
await snapshot("reset", await wait(2, 1));
|
||||
await sendChatMessage(context.page, scenario.fresh);
|
||||
const third = await wait(3, 1), freshRun = newProviderRun(third, recalled.runs);
|
||||
await snapshot("fresh", third, freshRun.id);
|
||||
await context.page.goto(chatUrl, { waitUntil: "domcontentloaded", timeout: remaining() });
|
||||
await snapshot("reopened", await wait(3, 1), freshRun.id);
|
||||
} finally {
|
||||
evidence.checks.push(...gradeChatContinuity(scope, points));
|
||||
await retain();
|
||||
await reader.preserve(options.directory);
|
||||
}
|
||||
if (evidence.checks.some(check => !check.passed)) throw new JourneyError("failed", "chat_continuity_evidence_failed");
|
||||
}
|
||||
@@ -0,0 +1,66 @@
|
||||
import { chatContinuityScenario, type ChatContinuityPoint, type ChatContinuityScope } from "./chat-continuity-evidence.js";
|
||||
|
||||
type Row = Record<string, any>;
|
||||
export function chatContinuityFixture() {
|
||||
const scope: ChatContinuityScope = { companyId: "company", userId: "board", agentId: "agent", agentName: "Owned agent",
|
||||
environmentId: "environment", provider: "opencode", model: "configured-model", campaignId: "synthetic-continuity-123",
|
||||
agentHistory: { source: "company-agent-runs", companyId: "company", agentId: "agent", limit: 1, runIds: [], observedBeforeSend: true } };
|
||||
const scenario = chatContinuityScenario(scope.campaignId);
|
||||
const user = (id: string, body: string, generation: number) => ({ id, body, authorUserId: scope.userId,
|
||||
authorAgentId: null, companyId: scope.companyId, issueId: "chat", conversationSessionGeneration: generation });
|
||||
const answer = (id: string, runId: string, body: string, generation: number) => ({ id, body,
|
||||
authorAgentId: scope.agentId, authorUserId: null, createdByRunId: runId,
|
||||
companyId: scope.companyId, issueId: "chat", conversationSessionGeneration: generation });
|
||||
const prompts = [scenario.remember, scenario.recall, scenario.reset, scenario.fresh];
|
||||
const board = prompts.map((body, index) => user(`user-${index}`, body, index >= 2 ? 1 : 0));
|
||||
const replies = [answer("reply-0", "run-0", "Remembered.", 0), answer("reply-1", "run-1", scenario.remembered, 0),
|
||||
answer("reply-3", "run-3", scenario.marker, 1)];
|
||||
const run = (index: number): Row => ({ id: `run-${index}`, companyId: scope.companyId, agentId: scope.agentId,
|
||||
nativeIssueId: "chat", status: "succeeded", runtimeMode: "native", nativePhase: "committed",
|
||||
nativeSessionId: index === 3 ? "new-session" : "original-session", sessionIdBefore: null,
|
||||
contextSnapshot: { issueId: "chat", source: "issue.comment", wakeCommentId: board[index]!.id,
|
||||
conversationSessionGeneration: index >= 2 ? 1 : 0, paperclipTaskMarkdown: index === 3 ? scenario.fresh : scenario.remember,
|
||||
paperclipEnvironment: { id: scope.environmentId } },
|
||||
runnerProfileJson: { nativeExecutionInput: { provider: { kind: scope.provider, model: scope.model } } },
|
||||
resultJson: { finalizationPhase: "committed", presentationDecision: { commentId: `reply-${index}` } } });
|
||||
const runs = [run(0), run(1), run(2), run(3)];
|
||||
runs[2]!.contextSnapshot.conversationReset = true;
|
||||
runs[2]!.resultJson = { conversationReset: true };
|
||||
delete runs[2]!.runnerProfileJson;
|
||||
delete runs[2]!.nativeSessionId;
|
||||
delete runs[2]!.nativePhase;
|
||||
delete runs[2]!.runtimeMode;
|
||||
const chat = { id: "chat", companyId: scope.companyId, assigneeAgentId: scope.agentId, conversationAgentId: scope.agentId,
|
||||
conversationUserId: scope.userId, createdByUserId: scope.userId, title: `Chat with ${scope.agentName}`,
|
||||
status: "in_review", conversationState: "waiting", conversationSessionGeneration: 0 };
|
||||
const comments = [board[0]!, replies[0]!, board[1]!, replies[1]!, board[2]!, board[3]!, replies[2]!];
|
||||
const makePoint = (phase: ChatContinuityPoint["phase"], count: number, commentCount: number, generation: number, replyIndex?: number): ChatContinuityPoint => {
|
||||
const reply = replyIndex === undefined ? undefined : replies[replyIndex]!;
|
||||
return structuredClone({ phase, chat: { ...chat, conversationSessionGeneration: generation }, runs: runs.slice(0, count),
|
||||
comments: comments.slice(0, commentCount), tasks: [], activity: [], events: [], resetReceipts: count < 3 ? [] : [{
|
||||
source: "issue-runs", companyId: scope.companyId, issueId: "chat", runId: "run-2", agentId: scope.agentId,
|
||||
status: "succeeded", contextIssueId: "chat", wakeCommentId: "user-2", resultJson: { conversationReset: true },
|
||||
}],
|
||||
...(reply ? { reply: { runId: reply.createdByRunId, agentId: scope.agentId, observedAfterReload: true,
|
||||
body: reply.body, anchor: reply.id, visible: true, settled: true, nativeFinal: false } } : {}) });
|
||||
};
|
||||
const points = [makePoint("initial", 1, 2, 0, 0), makePoint("returned", 1, 2, 0, 0), makePoint("recalled", 2, 4, 0, 1),
|
||||
makePoint("reset", 3, 5, 1), makePoint("fresh", 4, 7, 1, 2), makePoint("reopened", 4, 7, 1, 2)];
|
||||
return { scope, scenario, points };
|
||||
}
|
||||
|
||||
export function compactChatContinuityFixture() {
|
||||
const fixture = chatContinuityFixture();
|
||||
for (const point of fixture.points) {
|
||||
for (const run of point.runs) run.resultJson = null;
|
||||
if (!point.reply) continue;
|
||||
point.reply.anchor = `${point.reply.runId}:turn`;
|
||||
point.reply.nativeFinal = true;
|
||||
point.events = [{ seq: 1, runId: point.reply.runId, companyId: fixture.scope.companyId, agentId: fixture.scope.agentId,
|
||||
eventType: "item.completed", protocolSchemaVersion: 1, sourceEventId: "event", sourceInstanceId: "instance", sourceSeq: 1,
|
||||
payload: { prpEvent: { schema: "paperclip.prp.event.v1", schemaVersion: 1, runId: point.reply.runId, sourceKind: "runner",
|
||||
eventType: "item.completed", sourceEventId: "event", sourceInstanceId: "instance", sourceSeq: 1,
|
||||
payload: { kind: "agentMessage", channel: "final", text: point.reply.body } } } }];
|
||||
}
|
||||
return fixture;
|
||||
}
|
||||
@@ -18,6 +18,7 @@ import { projectTaskContext } from "./project-context-flow.js";
|
||||
import { runAgentJourney } from "./agent-flows.js";
|
||||
import { runActiveFollowup } from "./active-followup-flow.js";
|
||||
import { runClarificationJourney } from "./clarification-flow.js";
|
||||
import { runChatContinuity } from "./chat-continuity-flow.js";
|
||||
import { runRepositoryContinuation } from "./repository-continuation-flow.js";
|
||||
import { runScheduledRoutine } from "./routine-schedule-flow.js";
|
||||
import { runInputFile } from "./input-file-flow.js";
|
||||
@@ -164,6 +165,9 @@ export async function launchProductJourneys(args: string[], repositoryRoot: stri
|
||||
"../question-resume-scoring.ts",
|
||||
"../runtime-question-readiness.ts",
|
||||
"../chat-flow.ts",
|
||||
"chat-continuity-flow.ts",
|
||||
"chat-continuity-evidence.ts",
|
||||
"../chat-cases.ts",
|
||||
"../continuation-readiness.ts",
|
||||
"../continuation-cases.ts",
|
||||
"success-contract.ts",
|
||||
@@ -297,6 +301,8 @@ export async function launchProductJourneys(args: string[], repositoryRoot: stri
|
||||
await mkdir(directory, { recursive: true, mode: 0o700 });
|
||||
await captureProductScreenshot(ready!.page, ready!.api.origin, path.join(directory, file), deadlineAt);
|
||||
} });
|
||||
else if (id === "chat-continuity") await runChatContinuity(ready, session!, campaignId, evidence,
|
||||
{ deadlineAt, directory: path.join(outputDir, "private-stories", id) });
|
||||
else await runAgentJourney(id, ready, session!, campaignId, evidence, { deadlineAt,
|
||||
directory: path.join(outputDir, "private-stories", id),
|
||||
capture: async (_id, _label, file) => {
|
||||
|
||||
@@ -13,12 +13,12 @@ function fixture(options: { mounted?: boolean; redirect?: string; initialPath?:
|
||||
return { page, locator, waitFor, screenshot };
|
||||
}
|
||||
describe("product screenshot boundary", () => {
|
||||
it.each(["/QA/inbox/unread", "/QA/routines"])("retains an authenticated journey failure on %s", async initialPath => {
|
||||
it.each(["/QA/inbox/unread", "/QA/routines", "/QA/chats/owned-agent"])("retains an authenticated journey failure on %s", async initialPath => {
|
||||
const f = fixture({ initialPath });
|
||||
await captureProductScreenshot(f.page, origin, "failed.png");
|
||||
expect(f.screenshot).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
it.each(["/login", "/QA/settings/keys", "/QA/inboxing", "/QA/routines-export"])("excludes unrelated or authentication paths: %s", async initialPath => {
|
||||
it.each(["/login", "/QA/settings/keys", "/QA/inboxing", "/QA/routines-export", "/QA/chats-export"])("excludes unrelated or authentication paths: %s", async initialPath => {
|
||||
const f = fixture({ initialPath });
|
||||
await expect(captureProductScreenshot(f.page, origin, "failed.png")).rejects.toThrow("product_screen_unavailable");
|
||||
expect(f.waitFor).not.toHaveBeenCalled();
|
||||
|
||||
@@ -3,7 +3,7 @@ import { JourneyError } from "./types.js";
|
||||
|
||||
export function isProductScreenshotUrl(value: string, origin: string): boolean {
|
||||
const url = new URL(value);
|
||||
return url.origin === origin && /^\/[^/]+\/(issues|search|inbox|routines)(\/|$)/.test(url.pathname);
|
||||
return url.origin === origin && /^\/[^/]+\/(issues|search|inbox|routines|chats)(\/|$)/.test(url.pathname);
|
||||
}
|
||||
|
||||
export async function captureProductScreenshot(page: Page, origin: string, file: string, deadlineAt = Date.now() + 10_000) {
|
||||
|
||||
@@ -4,6 +4,7 @@ export const agentJourneyRuns = {
|
||||
"routine-scheduled-run": 1,
|
||||
"file-input-answer": 1,
|
||||
"ask-question": 1,
|
||||
"chat-continuity": 4,
|
||||
"plan-revise-accept": 3,
|
||||
"build-revise": 2,
|
||||
"followup-while-running": 2,
|
||||
@@ -34,6 +35,7 @@ export const journeyRequiredChecks: Record<string, readonly string[]> = {
|
||||
"project-task-context": ["browser-created-project-persisted", "browser-created-project-task-persisted", "project-task-association-reloaded", "project-task-context-visible", "no-agent-work"],
|
||||
"task-search-filter": ["task-search-results-exact", "task-status-filter-excludes-controls", "task-search-filter-reloaded", "task-search-result-reopened", "no-agent-work"],
|
||||
"ask-question": ["question-answer-persisted", "one-bound-successful-run", "configured-model-and-environment"],
|
||||
"chat-continuity": ["chat-fresh-owned-conversation", "chat-user-messages-stored-once", "chat-reload-preserved-context", "chat-new-session-boundary", "chat-fresh-context-answer", "chat-history-preserved-after-reload", "chat-no-unrequested-work", "chat-exact-turn-accounting", "configured-model-and-environment"],
|
||||
"question-answer-resume": ["question-resume.forms", "question-resume.choice.answer", "question-resume.choice.path",
|
||||
"question-resume.text.answer", "question-resume.text.path", "question-resume.complete-evidence", "question-no-premature-output",
|
||||
"question-answers-owned-by-board", "question-document-attributed", "question-document-reloaded", "configured-model-and-environment"],
|
||||
|
||||
@@ -9,6 +9,7 @@ export type JourneyId =
|
||||
| "project-task-context"
|
||||
| "task-document-persistence"
|
||||
| "ask-question"
|
||||
| "chat-continuity"
|
||||
| "cloud-onboarding"
|
||||
| "question-answer-resume"
|
||||
| "task-search-filter"
|
||||
|
||||
Reference in new issue
Block a user