mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-04 11:01:16 +01:00
test(e2e): verify saved agent instructions on the current QA harness (#18891)
## Thinking Path > - Paperclip lets users configure agents and verify their work. > - Daily QA must exercise saved instructions on the QA staging instance. > - A fresh owned agent can test an instruction change without changing a shared agent. > - The test must prove that saved instructions caused the next run's output. > - It must also prove that the original instructions were restored. > - This pull request brings the qualified settings journey onto the current shared registry layout. > - The benefit is a reusable daily QA candidate with independent evidence and fresh current-source checks. ## Linked Issues or Issue Description Refs #18788. This is the replacement review PR for the same settings journey after integration with the merged lifecycle registry layout in #18782. **What happened?** The settings journey passed its original staging qualification at source898ae8ab4e49d2408688088ce82e3316b91cc09c. The earlier PR's review workspace remains retained for file preservation. That workspace and its recovery evidence remain intact. **Expected behavior** Review the integrated settings source through this PR's own normal CI and fresh reviews. The journey saves instructions through ordinary UI controls, verifies their effect on one current native run, and restores the original bytes. **Steps to reproduce** Use the documented dedicated agent-settings-persistence campaign. The provider must supply a fresh owned agent and the agent-settings-write capability. Save the instruction edit, reload it, complete the ordinary assigned task, verify its independently saved result and current-run attribution, then restore and verify the original instruction entry. ## What Changed - Add the settings browser journey, its independent evidence grader, fixtures, retained evidence contract and focused controls from #18788. - Require one fresh dedicated agent. Select its exact configured instruction entry through the file tree. Confirm both board saves through persisted source and audit evidence. - Require one committed native run on the configured provider, model and environment. Keep the synthetic instruction reference out of the task request. - Restore the original instruction bytes through the same ordinary UI and verify the saved result after reload. - Normally merge actual lifecycle squash8771606fd87f69ed4057779e3dce3a9bca8088e6 once. Place only settings registrations at the documented distinct anchors. Preserve all eight dedicated source/oracle/fixture/docs blobs from qualified898 exactly. ## Verification - Current integrated head: 330 focused checks pass across eight settings, catalog, contract, observation, result/report and operations test files. - Current E2E types pass: `node scripts/typecheck-test-projects.mjs --low-memory --reuse-declarations tests/runner-e2e`. - Declaration reuse was bound to all11091 matching package/script/server/UI/root inputs, installed lock and TypeScript, plus all source and target generated-output hashes. No source or module-resolution links were copied. All copied output hashes remain unchanged after local checks. - Maintained product-journey discovery passes and includes both lifecycle and settings. The integrated catalog has22 entries, including the separate fresh-cloud onboarding case. - Static preservation checks pass for seven shared registry ASTs after only declared order changes. The settings dispatch body, required seven checks, one-run bound and all dedicated implementation/oracle blobs remain intact. - Historical staging evidence: [QA workflow41689186453](https://github.com/paperclipai/paperclip-cloud/actions/runs/41689186453) passed all seven settings checks and all five phases at source898ae8ab4e49d2408688088ce82e3316b91cc09c. This is the original qualification receipt. It is not a new run of this integrated head. - This PR must earn its own current-head full CI, fresh5/5 review, resolved threads and terminal passing normal check contexts. Those checks are pending. No prior review or check status was copied. - Full workspace checks and a new live QA/model run were not run locally. The coordinator owns current-source acceptance and daily-plan admission. ## Risks - The journey has one paid provider run and needs a fresh dedicated agent. It rejects a campaign that shares this agent with another paid journey. - Browser saves and restoration must persist before navigation. Wrong, missing, foreign or incomplete evidence fails the independent grader. Failed or active work is preserved for normal owned cleanup. - Registration and hash-input order changed during integration, so the definition/source identity changes. The historical QA receipt remains bound to its original source. - The retained earlier review workspace is still held for file preservation. This PR does not recover, reset, replace or destroy it, and submits no turn to it. - No product runtime, permission, model, shared agent, configuration, feature map, QA pin or TestOps-plan changes are included. ## Model Used - OpenAI GPT-6 (Codex), with reasoning, tool use and code execution. The host does not expose the exact deployment identifier or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub issue/PR references; historical QA workflow is linked as qualification evidence) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed
1 parent
7b1c53f8f1
commit
985b175e83
17 files changed
+705
-1
No files matched your search
@@ -1,5 +1,7 @@
|
||||
# Paperclip evaluation guide
|
||||
|
||||
The explicit deployment-targeted [saved agent instructions journey](../tests/runner-e2e/product-journeys/AGENT-SETTINGS.md) checks an owned agent’s ordinary UI save, persisted instruction bytes, one current-run output, and restoration. The task request omits its instruction-only reference. It reuses the existing Product E2E target and report path; local controls do not establish a staging pass.
|
||||
|
||||
The explicit-only [verified API artifact transfer](../tests/runner-e2e/README.md#verified-api-artifact-transfer)
|
||||
cell checks a native Codex agent on Daytona installing an authorized saved API
|
||||
response and delivering unchanged bytes. The oracle binds the source capture,
|
||||
|
||||
@@ -43,6 +43,7 @@ never start a server or depend on a tenant filesystem.
|
||||
| `stop-redirect` | Native agent; three work runs; qualified Docker oracle | Download a verified checkpoint, observe an active native command with current-turn usage, Stop it, verify cancellation and settled accounting, then send one new direction and complete without original-run activity |
|
||||
| `followup-while-running` | Native agent; a maximum of two work runs | Save a follow-up while an independently observed native shell command is still active, then require one attributed final reply with the late requirement after reload; in-turn steering and one normal successor are accepted |
|
||||
| `hire-delegate-reuse` | Native lead with a fresh user chat and scoped hiring permission; Agent Chat enabled; direct hiring policy; seven work runs | Hire one teammate, delegate a saved checklist, reuse the same teammate for a saved review, verify native run and revision authorship, preserve the original, and follow the lead's saved result link after reload |
|
||||
| `agent-settings-persistence` | Fresh dedicated native agent with the campaign ID in its name; `agent-settings-write`; no previous runs/proof block; sole paid journey; one work run | Save and reload an instruction edit through the browser, verify a task uses the instruction-only reference, reopen its run-authored document, then restore the original bytes. See [AGENT-SETTINGS.md](product-journeys/AGENT-SETTINGS.md) |
|
||||
| `cloud-onboarding` | Signed-in admitted Cloud account; fresh workspace; qualified bounded provider route | Create workspace in the portal, connect API key and native Codex agent in the wizard, complete a task, download and independently check its artifact |
|
||||
|
||||
The deployment-targeted Stop journey asks the revision to finish one harmless
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
# Paid runner full-stack E2E
|
||||
|
||||
The deployment-targeted [saved agent instructions journey](product-journeys/AGENT-SETTINGS.md) verifies browser save/reload, one instruction-dependent native task, independently attributed output, and restoration of the owned agent’s original instructions. Discovery and local grader controls remain separate from QA staging qualification.
|
||||
|
||||
The explicit-only `cursor-cloud` suite uses a local controller and hosted Cursor
|
||||
Cloud work. Its four required cells check a downloaded registered artifact,
|
||||
question/answer continuation on the same Cloud agent, UI Stop of an
|
||||
|
||||
@@ -147,6 +147,7 @@ describe("runner E2E catalog", () => {
|
||||
"stop-redirect",
|
||||
"followup-while-running",
|
||||
"hire-delegate-reuse",
|
||||
"agent-settings-persistence",
|
||||
"cloud-onboarding"
|
||||
]);
|
||||
expect(runnerSuites.some(suite => suite.id === "product-journeys")).toBe(false);
|
||||
|
||||
@@ -27,6 +27,7 @@ associate(["product-journeys"], {
|
||||
"question-answer-resume": ["questions-and-approvals/questions", "questions-and-approvals/continuation"],
|
||||
"followup-while-running": ["steering/queue"],
|
||||
"hire-delegate-reuse": ["agents/hire", "documents-artifacts/documents"],
|
||||
"agent-settings-persistence": ["agents/instructions", "tasks/completion"],
|
||||
"project-task-context": ["projects/lifecycle", "projects/task-context", "tasks/create"],
|
||||
"task-edit-lifecycle": ["tasks/properties", "tasks/completion"],
|
||||
"task-properties": ["tasks/properties"], "task-document-persistence": ["documents-artifacts/documents"], "task-search-filter": ["tasks/organize"], "cloud-onboarding": ["onboarding/first-task"] });
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
# Saved agent instructions
|
||||
|
||||
`agent-settings-persistence` verifies an ordinary instruction edit on an owned
|
||||
native agent and the result of its next task. It has one provider work run and
|
||||
requires the `agent-settings-write` target capability. The target provider grants
|
||||
that capability only for a fresh disposable agent dedicated to this case. Its
|
||||
exact name must match the prepared context and contain the current campaign ID.
|
||||
It must have no previous work runs or QA settings proof block. The name is an
|
||||
additional fixture binding; the target's capability grant establishes ownership.
|
||||
Target configuration rejects campaigns that combine this case with another paid
|
||||
journey before acquiring a target. Browser-only journeys may share the campaign.
|
||||
The journey
|
||||
does not edit the shared source agent, provider, model, accounts, or feature flags.
|
||||
|
||||
1. Read the original managed instruction entry, exact bytes, content hash, and
|
||||
content revision token. Require an empty work-run history.
|
||||
Empty or whitespace-only files use the ordinary editor value because the
|
||||
read-only page shows an empty-state message. Preserve those exact bytes too.
|
||||
2. Open Instructions, select **Edit** for the entry, change the **Instruction
|
||||
file editor**, and use **Save changes**. The new procedure contains a fresh
|
||||
synthetic reference that does not occur in the task request.
|
||||
Select the configured entry through the file tree on every open or reload,
|
||||
expanding its parent folders. An existing `AGENTS.md` must not divert the edit
|
||||
from a different configured entry.
|
||||
Confirm persisted bytes and the new save audit before navigating away. The
|
||||
same confirmation applies when restoring the original instructions.
|
||||
3. Reload Instructions. Require the exact saved bytes in both the browser and
|
||||
the normal read API. Verify a new `agent.files_updated` audit event with the
|
||||
signed-in user, exact owned agent, entry path, and content hash.
|
||||
Activity reads use both entity filters before the API limit. Unrelated company
|
||||
history cannot consume that page. A full owned page or foreign entity fails
|
||||
the coverage check rather than being silently filtered.
|
||||
4. Create one ordinary task through the shared composer. The agent must use the
|
||||
saved procedure to produce the `settings-proof` task document. Require the
|
||||
exact reference, one committed native run on the configured provider, model,
|
||||
and environment, and current-run document authorship. Pending questions,
|
||||
checkout, or recovery cannot pass.
|
||||
The dedicated `nativePhase` column must be `committed`. The grader does not
|
||||
require `resultJson.finalizationPhase`, which normal bounded run reads can omit.
|
||||
5. Reload the task and read the saved document. Verify the entry file is still
|
||||
unchanged after the agent's task.
|
||||
6. Restore the original bytes through the same Instructions controls. Verify the
|
||||
restored browser/API content, its original content token, a distinct board
|
||||
save audit, and no second task run. Return to the task result for the ordinary
|
||||
authenticated screenshot.
|
||||
|
||||
Current instruction revisions are content tokens. Identical restored bytes have
|
||||
the original token; the journey does not invent a database parent/child revision
|
||||
chain. The independently retained board audit establishes each actual save.
|
||||
|
||||
The independent grader rejects wrong or missing settings, leaked task inputs,
|
||||
foreign agents/tasks/runs/documents, a changed entry, unknown runtime identity,
|
||||
uncommitted work, extra runs, and an unconfirmed restoration. A failed work run
|
||||
remains failed. The public flow does not edit instructions during failed or active
|
||||
work; the target's normal cleanup stops and terminates that owned agent.
|
||||
|
||||
Private evidence consists of `agent-settings-observations.json` and
|
||||
`observation-read-failures.json` under this case's `private-stories` directory.
|
||||
The first contains closed IDs, hashes, the synthetic reference, checks, and both
|
||||
matched board-save receipts. Each receipt includes its audit ID, actor, agent
|
||||
entity, entry path, content hash, and save source. It
|
||||
does not contain raw instruction files, account bindings, or provider payloads.
|
||||
Discovery and local controls do not qualify a staging run. The daily plan may
|
||||
admit this case only after a genuine staging pass and reviewed source merge.
|
||||
@@ -26,6 +26,7 @@ type AgentJourney = Exclude<keyof typeof agentJourneyRuns,
|
||||
| "followup-while-running"
|
||||
| "file-input-answer"
|
||||
| "routine-scheduled-run"
|
||||
| "agent-settings-persistence"
|
||||
| "repository-continuation">;
|
||||
interface TaskState { currentIssue: Row & { id: string; status: string }; taskRuns: Array<Row & { status: string }>; comments: Row[]; interactions: Array<Row & { id: string; status: string }>; attachments: StoryDelivery[] }
|
||||
interface Options { deadlineAt: number; directory: string; capture(id: string, label: string, file: string): Promise<void> }
|
||||
|
||||
@@ -0,0 +1,102 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { AGENT_SETTINGS_CHECKS, gradeAgentSettings, settingsProofHeading, settingsTaskPrompt, settingsResultMatches } from "./agent-settings-evidence.js";
|
||||
import { settingsEvidenceFixture } from "./agent-settings-test-fixture.js";
|
||||
import { agentJourneyRunCountValid, journeyRequiredChecks } from "./success-contract.js";
|
||||
import { parseTargetConfig } from "./catalog.js";
|
||||
|
||||
describe("saved agent settings independent evidence", () => {
|
||||
it.each([["ask-question", "agent-settings-persistence"], ["agent-settings-persistence", "ask-question"]])(
|
||||
"rejects a shared paid campaign before target acquisition: %j", (...journeys) => {
|
||||
expect(() => parseTargetConfig({ version: 1, id: "settings-target", mode: "managed-self-hosted", journeys,
|
||||
executionEnvironment: "local", baseline: { features: {}, connections: [] },
|
||||
limits: { durationMs: 30000, modelBudgetCents: 500, maxRuns: 2 } })).toThrow("agent_settings_requires_dedicated_campaign");
|
||||
});
|
||||
it("permits browser-only journeys with the dedicated settings campaign", () => {
|
||||
expect(parseTargetConfig({ version: 1, id: "settings-target", mode: "managed-self-hosted",
|
||||
journeys: ["task-create", "agent-settings-persistence"], executionEnvironment: "local",
|
||||
baseline: { features: {}, connections: [] }, limits: { durationMs: 30000, modelBudgetCents: 500, maxRuns: 1 } }).journeys)
|
||||
.toEqual(["task-create", "agent-settings-persistence"]);
|
||||
});
|
||||
it("requires every independent check and exactly one work run in retained results", () => {
|
||||
expect(journeyRequiredChecks["agent-settings-persistence"]).toEqual(AGENT_SETTINGS_CHECKS);
|
||||
expect(agentJourneyRunCountValid("agent-settings-persistence", 1)).toBe(true);
|
||||
for (const count of [undefined, 0, 2, -1, 1.5, "1"]) {
|
||||
expect(agentJourneyRunCountValid("agent-settings-persistence", count)).toBe(false);
|
||||
}
|
||||
});
|
||||
it("accepts the actual content-token contract, board save audits and one bound result", () => {
|
||||
const f = settingsEvidenceFixture();
|
||||
expect(gradeAgentSettings(f.scope, f.evidence)).toEqual(AGENT_SETTINGS_CHECKS.map(id => ({ id, passed: true })));
|
||||
expect(f.evidence.restored!.file.revision.id).toBe(f.evidence.original!.file.revision.id);
|
||||
expect(settingsTaskPrompt).not.toContain(f.scope.reference);
|
||||
});
|
||||
it.each([null, {}, { result: "bounded output" }])("accepts committed current-run evidence when resultJson is %j", resultJson => {
|
||||
const f = settingsEvidenceFixture();
|
||||
Object.assign(f.run, { resultJson });
|
||||
expect(gradeAgentSettings(f.scope, f.evidence).every(row => row.passed)).toBe(true);
|
||||
});
|
||||
it("accepts committed current-run evidence when resultJson is omitted", () => {
|
||||
const f = settingsEvidenceFixture();
|
||||
delete (f.run as Record<string, unknown>).resultJson;
|
||||
expect(gradeAgentSettings(f.scope, f.evidence).every(row => row.passed)).toBe(true);
|
||||
});
|
||||
it("rejects an internally consistent baseline containing a previous QA proof block", () => {
|
||||
const f = settingsEvidenceFixture(undefined, `# Agent\n${settingsProofHeading}\nStale previous reference\n`);
|
||||
expect(gradeAgentSettings(f.scope, f.evidence).every(row => row.passed)).toBe(false);
|
||||
});
|
||||
const changes: Array<[string, (f: ReturnType<typeof settingsEvidenceFixture>) => void]> = [
|
||||
["foreign company", f => { f.evidence.original!.agent.companyId = "foreign"; }],
|
||||
["missing campaign-bearing target name", f => { f.scope.agentName = "Agent without campaign name"; f.evidence.original!.agent.name = f.scope.agentName; }],
|
||||
["past work", f => { f.evidence.original!.runs.push(f.run); }],
|
||||
["wrong instruction entry", f => { f.evidence.saved!.bundle.entryFile = "OTHER.md"; }],
|
||||
["false content hash", f => { f.evidence.saved!.file.contentHash = "0".repeat(64); }],
|
||||
["wrong saved bytes", f => { f.evidence.saved!.file.content += "wrong"; }],
|
||||
["missing save audit", f => { f.evidence.saved!.activity = []; }],
|
||||
["unscoped activity response", f => { f.evidence.original!.activity.push({ ...f.firstSave, entityId: "foreign-agent" }); }],
|
||||
["full owned activity page", f => { f.evidence.original!.activity = Array.from({ length: 200 }, (_, index) => ({ ...f.firstSave, id: `old-${index}` })); }],
|
||||
["foreign board actor", f => { f.firstSave.actorId = "someone-else"; }],
|
||||
["foreign audit entity", f => { f.firstSave.entityId = "other-agent"; }],
|
||||
["duplicate save evidence", f => { f.evidence.saved!.activity.push({ ...f.firstSave, id: "duplicate" }); }],
|
||||
["rendered draft only", f => { f.evidence.reloaded!.file.content = f.original; }],
|
||||
["missing rendered content", f => { delete f.evidence.reloaded!.renderedContent; }],
|
||||
["prompt leaks reference", f => { f.evidence.task!.issue.description += f.scope.reference; }],
|
||||
["foreign task", f => { f.evidence.task!.issue.companyId = "other"; }],
|
||||
["wrong assignee", f => { f.evidence.task!.issue.assigneeAgentId = "other"; }],
|
||||
["incomplete task", f => { f.evidence.task!.issue.status = "in_progress"; }],
|
||||
["active checkout", f => { f.evidence.task!.issue.executionRunId = "active"; }],
|
||||
["recovery pending", f => { f.evidence.task!.issue.activeRecoveryAction = {}; }],
|
||||
["question pending", f => { f.evidence.task!.interactions = [{ status: "pending" }]; }],
|
||||
["extra run", f => { f.evidence.task!.runs.push({ ...f.run, id: "extra" }); }],
|
||||
["failed run", f => { f.run.status = "failed"; }],
|
||||
["uncommitted run", f => { f.run.nativePhase = "applied"; }],
|
||||
["missing committed phase", f => { delete (f.run as Record<string, unknown>).nativePhase; }],
|
||||
["uncommitted run with omitted result", f => { f.run.nativePhase = "applied"; Object.assign(f.run, { resultJson: null }); }],
|
||||
["legacy run", f => { f.run.runtimeMode = "legacy"; }],
|
||||
["conflicting issue bindings", f => { Object.assign(f.run, { issueId: "foreign" }); }],
|
||||
["wrong provider", f => { f.run.runnerProfileJson.nativeExecutionInput.provider.kind = "other"; }],
|
||||
["wrong model", f => { f.run.runnerProfileJson.nativeExecutionInput.provider.model = "other"; }],
|
||||
["wrong environment", f => { f.run.contextSnapshot.paperclipEnvironment.id = "other"; }],
|
||||
["wrong nonce", f => { f.evidence.task!.document.body = JSON.stringify({ reference: "wrong", settingsSaved: true }); }],
|
||||
["human-authored output", f => { f.evidence.task!.document.createdByUserId = "user"; }],
|
||||
["foreign revision", f => { f.evidence.task!.revision.createdByRunId = "other"; }],
|
||||
["task changes instructions", f => { f.evidence.afterWork!.file.content = "changed"; }],
|
||||
["wrong reopened revision", f => { f.evidence.task!.reopenedDocument!.latestRevisionId = "old"; }],
|
||||
["missing rendered result", f => { delete f.evidence.task!.renderedReference; }],
|
||||
["unrestored instructions", f => { f.evidence.restored!.file.content = f.saved; }],
|
||||
["missing restore audit", f => { f.evidence.restored!.activity = [f.firstSave]; }],
|
||||
["extra work during restore", f => { f.evidence.restored!.runs.push({ ...f.run, id: "later" }); }],
|
||||
];
|
||||
it.each(changes)("rejects %s", (_name, change) => {
|
||||
const f = settingsEvidenceFixture(); change(f);
|
||||
expect(gradeAgentSettings(f.scope, f.evidence).every(row => row.passed)).toBe(false);
|
||||
});
|
||||
it.each(["original", "saved", "reloaded", "afterWork", "restored", "task"] as const)("rejects missing %s evidence", key => {
|
||||
const f = settingsEvidenceFixture(); delete f.evidence[key];
|
||||
expect(gradeAgentSettings(f.scope, f.evidence).every(row => row.passed)).toBe(false);
|
||||
});
|
||||
it("rejects plausible malformed result bodies and accepts one optional JSON fence", () => {
|
||||
expect(settingsResultMatches('```json\n{"reference":"r","settingsSaved":true}\n```', "r")).toBe(true);
|
||||
for (const body of [null, "", "not json", "[]", '{"reference":"r"}', '{"reference":"r","settingsSaved":"true"}',
|
||||
'{"reference":"r","settingsSaved":true,"extra":1}']) expect(settingsResultMatches(body, "r")).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,125 @@
|
||||
import { createHash } from "node:crypto";
|
||||
|
||||
type Row = Record<string, any>;
|
||||
export const AGENT_SETTINGS_CHECKS = [
|
||||
"agent-settings-browser-save", "agent-settings-reload-persisted", "agent-settings-instruction-only-input",
|
||||
"agent-settings-result-attributed", "agent-settings-result-reopened", "agent-settings-original-restored",
|
||||
"configured-model-and-environment",
|
||||
] as const;
|
||||
export const settingsDocumentKey = "settings-proof";
|
||||
export const settingsProofHeading = "## QA settings persistence proof";
|
||||
export const settingsTaskPrompt = "Follow the QA settings proof procedure in your saved agent instructions. Save the requested task document, then finish the task. No other tasks or agents are needed. Do not change your instruction files.";
|
||||
export const instructionDigest = (content: string) => createHash("sha256").update(content).digest("hex");
|
||||
export function settingsInstructionContent(original: string, reference: string) {
|
||||
return original + `\n\n${settingsProofHeading}\nWhen a task asks for the QA settings proof, save a task document with key ${settingsDocumentKey} and title Settings proof. Its body must be JSON with exactly two fields: reference equal to ${reference}, and settingsSaved equal to true. Keep these instructions unchanged.\n`;
|
||||
}
|
||||
export function settingsResultMatches(body: unknown, reference: string) {
|
||||
if (typeof body !== "string") return false;
|
||||
const text = body.trim(), json = /^```(?:json)?\s*\n([\s\S]*?)\n```$/.exec(text)?.[1] ?? text;
|
||||
try {
|
||||
const parsed = JSON.parse(json);
|
||||
return parsed && !Array.isArray(parsed) && Object.keys(parsed).sort().join(",") === "reference,settingsSaved"
|
||||
&& parsed.reference === reference && parsed.settingsSaved === true;
|
||||
} catch { return false; }
|
||||
}
|
||||
export interface AgentSettingsScope {
|
||||
campaignId: string; companyId: string; userId: string; agentId: string; agentName: string;
|
||||
provider: string; model: string; environmentId: string; reference: string;
|
||||
}
|
||||
export interface SettingsSnapshot { agent: Row; bundle: Row; file: Row; runs: Row[]; activity: Row[]; renderedContent?: string }
|
||||
export interface SettingsTaskSnapshot {
|
||||
issue: Row; runs: Row[]; interactions: Row[]; document: Row; revision: Row;
|
||||
reopenedDocument?: Row; reopenedRevision?: Row; renderedReference?: string;
|
||||
}
|
||||
export interface AgentSettingsEvidence {
|
||||
original?: SettingsSnapshot; saved?: SettingsSnapshot; reloaded?: SettingsSnapshot;
|
||||
afterWork?: SettingsSnapshot; restored?: SettingsSnapshot; task?: SettingsTaskSnapshot;
|
||||
}
|
||||
const present = (value: unknown): value is string => typeof value === "string" && value.length > 0;
|
||||
const revisionId = (point?: SettingsSnapshot) => point?.file?.revision?.id;
|
||||
function snapshotBound(point: SettingsSnapshot | undefined, scope: AgentSettingsScope, entry?: string) {
|
||||
return Boolean(point && point.agent?.id === scope.agentId && point.agent.companyId === scope.companyId
|
||||
&& point.agent.name === scope.agentName && point.agent.adapterType === "paperclip_runner"
|
||||
&& point.agent.adapterConfig?.provider === scope.provider && point.agent.adapterConfig?.model === scope.model
|
||||
&& point.bundle?.agentId === scope.agentId && point.bundle.companyId === scope.companyId
|
||||
&& point.bundle.mode === "managed" && point.bundle.editable === true && present(point.bundle.entryFile)
|
||||
&& (!entry || point.bundle.entryFile === entry) && point.file?.path === point.bundle.entryFile
|
||||
&& point.file.isEntryFile === true && point.file.editable === true && point.file.binary !== true
|
||||
&& typeof point.file.content === "string" && point.file.contentHash === instructionDigest(point.file.content)
|
||||
&& point.file.size === Buffer.byteLength(point.file.content)
|
||||
&& present(revisionId(point)) && point.file.revision.companyId === scope.companyId
|
||||
&& point.file.revision.agentId === scope.agentId && point.file.revision.entryFile === point.bundle.entryFile
|
||||
&& point.file.revision.contentHash === point.file.contentHash
|
||||
&& point.file.revision.byteLength === point.file.size && Array.isArray(point.runs)
|
||||
&& Array.isArray(point.activity) && point.activity.length < 200
|
||||
&& point.activity.every(row => present(row.id) && row.companyId === scope.companyId
|
||||
&& row.entityType === "agent" && row.entityId === scope.agentId));
|
||||
}
|
||||
export function matchedBoardSaveAudit(point: SettingsSnapshot | undefined, before: SettingsSnapshot | undefined, scope: AgentSettingsScope) {
|
||||
const old = new Set(before?.activity?.map(row => row.id));
|
||||
const added = point?.activity?.filter(row => !old.has(row.id) && row.action === "agent.files_updated"
|
||||
&& row.entityType === "agent" && row.entityId === scope.agentId && row.details?.path === point.bundle.entryFile);
|
||||
if (!(Array.isArray(added) && added.length === 1 && present(added[0]?.id)
|
||||
&& added[0].companyId === scope.companyId && added[0].actorType === "user" && added[0].actorId === scope.userId
|
||||
&& added[0].agentId === null && added[0].runId === null && added[0].details.contentHash === point!.file.contentHash
|
||||
&& added[0].details.source === "board")) return null;
|
||||
const row = added[0];
|
||||
return { id: row.id, companyId: row.companyId, actorType: row.actorType, actorId: row.actorId,
|
||||
agentId: row.agentId, runId: row.runId, action: row.action, entityType: row.entityType, entityId: row.entityId,
|
||||
path: row.details.path, contentHash: row.details.contentHash, source: row.details.source };
|
||||
}
|
||||
function boundIssue(run: Row) {
|
||||
const ids = [run.issueId, run.nativeIssueId, run.contextSnapshot?.issueId, run.contextSnapshot?.taskId].filter(value => value != null);
|
||||
return ids.length && ids.every(present) && new Set(ids).size === 1 ? ids[0] : undefined;
|
||||
}
|
||||
export function gradeAgentSettings(scope: AgentSettingsScope, value: AgentSettingsEvidence) {
|
||||
const { original, saved, reloaded, afterWork, restored, task } = value;
|
||||
const entry = original?.bundle?.entryFile;
|
||||
const scoped = [scope.campaignId, scope.companyId, scope.userId, scope.agentId, scope.agentName,
|
||||
scope.provider, scope.model, scope.environmentId, scope.reference].every(present)
|
||||
&& scope.agentName.includes(scope.campaignId) && snapshotBound(original, scope)
|
||||
&& original!.runs.length === 0 && !original!.file.content.includes(scope.reference)
|
||||
&& !original!.file.content.includes(settingsProofHeading);
|
||||
const expected = scoped ? settingsInstructionContent(original!.file.content, scope.reference) : undefined;
|
||||
const save = scoped && snapshotBound(saved, scope, entry) && saved!.file.content === expected
|
||||
&& revisionId(saved) !== revisionId(original) && matchedBoardSaveAudit(saved, original, scope) && saved!.runs.length === 0;
|
||||
const reload = save && snapshotBound(reloaded, scope, entry) && reloaded!.file.content === expected
|
||||
&& revisionId(reloaded) === revisionId(saved) && reloaded!.renderedContent === expected && reloaded!.runs.length === 0;
|
||||
const issue = task?.issue, runs = task?.runs;
|
||||
const inputOnly = reload && issue && present(issue.id) && issue.companyId === scope.companyId
|
||||
&& issue.assigneeAgentId === scope.agentId && issue.assigneeUserId === null && issue.createdByUserId === scope.userId
|
||||
&& issue.createdByAgentId === null && issue.workMode === "standard" && issue.description === settingsTaskPrompt
|
||||
&& !issue.description.includes(scope.reference);
|
||||
const run = Array.isArray(runs) && runs.length === 1 ? runs[0] : undefined;
|
||||
const provider = run?.runnerProfileJson?.nativeExecutionInput?.provider;
|
||||
const configured = Boolean(inputOnly && run && present(run.id) && run.companyId === scope.companyId
|
||||
&& run.agentId === scope.agentId && boundIssue(run) === issue.id && run.status === "succeeded"
|
||||
&& run.runtimeMode === "native" && run.nativePhase === "committed"
|
||||
&& provider?.kind === scope.provider && provider.model === scope.model
|
||||
&& run.contextSnapshot?.paperclipEnvironment?.id === scope.environmentId);
|
||||
const document = task?.document, revision = task?.revision;
|
||||
const output = configured && issue && issue.status === "done" && !issue.executionRunId && !issue.scheduledRetry && !issue.activeRecoveryAction
|
||||
&& Array.isArray(task!.interactions) && task!.interactions.every(row => row.status !== "pending")
|
||||
&& document && present(document.id) && document.companyId === scope.companyId && document.issueId === issue.id
|
||||
&& document.key === settingsDocumentKey && document.createdByAgentId === scope.agentId && document.createdByUserId === null
|
||||
&& present(document.latestRevisionId) && settingsResultMatches(document.body, scope.reference)
|
||||
&& revision && revision.id === document.latestRevisionId && revision.documentId === document.id
|
||||
&& revision.createdByAgentId === scope.agentId && revision.createdByUserId === null && revision.createdByRunId === run!.id
|
||||
&& settingsResultMatches(revision.body, scope.reference)
|
||||
&& snapshotBound(afterWork, scope, entry) && afterWork!.file.content === expected && revisionId(afterWork) === revisionId(saved)
|
||||
&& afterWork!.runs.length === 1 && afterWork!.runs[0]?.id === run!.id;
|
||||
const reopened = task?.reopenedDocument, reopenedRevision = task?.reopenedRevision;
|
||||
const resultReload = output && issue && reopened && reopened.id === document.id && reopened.companyId === scope.companyId
|
||||
&& reopened.issueId === issue.id && reopened.key === settingsDocumentKey && reopened.latestRevisionId === document.latestRevisionId
|
||||
&& reopened.createdByAgentId === scope.agentId && reopened.createdByUserId === null && settingsResultMatches(reopened.body, scope.reference)
|
||||
&& reopenedRevision && reopenedRevision.id === revision.id && reopenedRevision.documentId === document.id
|
||||
&& reopenedRevision.createdByRunId === run!.id && reopenedRevision.createdByAgentId === scope.agentId
|
||||
&& reopenedRevision.createdByUserId === null && settingsResultMatches(reopenedRevision.body, scope.reference)
|
||||
&& task!.renderedReference === scope.reference;
|
||||
const restore = resultReload && snapshotBound(restored, scope, entry) && restored!.file.content === original!.file.content
|
||||
&& restored!.renderedContent === original!.file.content && restored!.file.contentHash === original!.file.contentHash
|
||||
&& revisionId(restored) === revisionId(original) && revisionId(restored) !== revisionId(saved) && matchedBoardSaveAudit(restored, afterWork, scope)
|
||||
&& restored!.runs.length === 1 && restored!.runs[0]?.id === run!.id;
|
||||
return [save, reload, inputOnly, output, resultReload, restore, configured]
|
||||
.map((passed, index) => ({ id: AGENT_SETTINGS_CHECKS[index]!, passed: Boolean(passed) }));
|
||||
}
|
||||
@@ -0,0 +1,180 @@
|
||||
import { mkdtemp, readFile, rm } from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
import type { JourneyContext, JourneyEvidence, TargetSession } from "./types.js";
|
||||
import { settingsEvidenceFixture } from "./agent-settings-test-fixture.js";
|
||||
import { settingsTaskPrompt } from "./agent-settings-evidence.js";
|
||||
|
||||
const controls = vi.hoisted(() => ({ createTask: vi.fn(), openDocument: vi.fn() }));
|
||||
vi.mock("../user-actions.js", () => ({ createTaskThroughUi: controls.createTask }));
|
||||
vi.mock("./task-document-flow.js", () => ({ openDocumentAuthoring: controls.openDocument }));
|
||||
vi.mock("@playwright/test", () => ({ expect: (locator: any) => ({
|
||||
async toBeVisible() { expect(locator.visible).toBe(true); },
|
||||
async toBeEnabled() { expect(locator.enabled()).toBe(true); },
|
||||
async toBeDisabled() { expect(locator.enabled()).toBe(false); },
|
||||
async toHaveValue(value: string) { expect(locator.value()).toBe(value); },
|
||||
async toContainText(value: string) { expect(await locator.textContent()).toContain(value); },
|
||||
}) }));
|
||||
import { runAgentSettings } from "./agent-settings-flow.js";
|
||||
|
||||
const directories: string[] = [];
|
||||
afterEach(async () => { await Promise.all(directories.splice(0).map(dir => rm(dir, { recursive: true, force: true }))); });
|
||||
|
||||
async function fixture(nativeFailure = false, unscopedActivity = false, delayedRestore = false, original?: string, entryFile = "AGENTS.md") {
|
||||
let f = settingsEvidenceFixture(undefined, original, entryFile), content = f.original, rendered = content, draft = content, dirty = false, dispatched = false;
|
||||
const originalContent = f.original;
|
||||
const availablePaths = new Set(f.evidence.original!.bundle.files.map((file: any) => file.path));
|
||||
const neighbor = "# AGENTS.md sibling must remain unchanged\n";
|
||||
let selectedFile = "AGENTS.md";
|
||||
const expandedDirs = new Set<string>(), selectedPaths: string[] = [], savedPaths: string[] = [];
|
||||
let pendingContent: string | undefined;
|
||||
const saves: string[] = [], owns: any[] = [], apiWrites: string[] = [], activityReads: string[] = [];
|
||||
const unrelatedActivity = Array.from({ length: 600 }, (_, index) => ({ id: `unrelated-${index}`,
|
||||
companyId: f.scope.companyId, entityType: "agent", entityId: "another-agent", details: {} }));
|
||||
const current = () => content === f.original
|
||||
? dispatched ? f.evidence.restored! : f.evidence.original!
|
||||
: dispatched ? f.evidence.afterWork! : f.evidence.saved!;
|
||||
const editor = { visible: true, value: () => draft, async inputValue() { return draft; }, async fill(value: string) { draft = value; dirty = true; } };
|
||||
const raw = { get visible() { return Boolean(rendered.trim()); }, async textContent() { return rendered; } };
|
||||
const edit = { get visible() { return selectedFile === entryFile; }, async click() { draft = content; } };
|
||||
const save = { enabled: () => dirty, async click() {
|
||||
expect(selectedFile).toBe(entryFile); savedPaths.push(selectedFile);
|
||||
saves.push(draft); dirty = false;
|
||||
if (delayedRestore && saves.length === 2) pendingContent = draft;
|
||||
else content = draft;
|
||||
const nonce = /reference equal to ([a-f0-9]+),/.exec(draft)?.[1];
|
||||
if (nonce) f = settingsEvidenceFixture(nonce, originalContent, entryFile);
|
||||
} };
|
||||
const document = { visible: true, async textContent() { return f.evidence.task!.document.body; } };
|
||||
const page = { async goto() {
|
||||
if (pendingContent !== undefined) throw new Error("navigation_before_save_committed");
|
||||
selectedFile = "AGENTS.md"; expandedDirs.clear(); rendered = entryFile === "AGENTS.md" ? content : neighbor;
|
||||
}, async reload() { selectedFile = "AGENTS.md"; expandedDirs.clear(); rendered = entryFile === "AGENTS.md" ? content : neighbor; },
|
||||
getByTestId(id: string) { expect(id).toBe("instructions-raw-source"); return raw; },
|
||||
getByRole(role: string, options: { name: string }) {
|
||||
if (role === "button" && options.name === `Edit ${entryFile}`) return edit;
|
||||
if (role === "button" && options.name === `Stop editing ${entryFile}`) return { async click() {} };
|
||||
if (role === "button" && options.name === "Save changes") return save;
|
||||
if (role === "textbox" && options.name === "Instruction file editor") return editor;
|
||||
throw new Error("unexpected_locator");
|
||||
}, locator(selector: string) {
|
||||
if (selector === "#document-settings-proof") return document;
|
||||
const match = /^\[role="treeitem"\]\[data-file-tree-path=("(?:\\.|[^"\\])*")\]$/.exec(selector);
|
||||
if (!match) throw new Error("unexpected_selector:" + selector);
|
||||
const filePath: string = JSON.parse(match[1]!);
|
||||
const directory = entryFile.startsWith(`${filePath}/`);
|
||||
const parents = filePath.split("/").slice(0, -1);
|
||||
return { get visible() { return (directory || availablePaths.has(filePath))
|
||||
&& parents.every((_, index) => expandedDirs.has(parents.slice(0, index + 1).join("/"))); },
|
||||
async getAttribute(name: string) { expect(name).toBe("aria-expanded"); return directory ? String(expandedDirs.has(filePath)) : null; },
|
||||
async click() {
|
||||
if (directory) expandedDirs.add(filePath);
|
||||
else { selectedFile = filePath; selectedPaths.push(filePath); rendered = filePath === entryFile ? content : neighbor; }
|
||||
} };
|
||||
} };
|
||||
const api = { origin: "https://qa.example.test", async get(route: string) {
|
||||
if (route === "/api/agents/agent") return structuredClone(current().agent);
|
||||
if (route === "/api/agents/agent/instructions-bundle") return structuredClone(current().bundle);
|
||||
if (route === `/api/agents/agent/instructions-bundle/file?path=${encodeURIComponent(entryFile)}`) {
|
||||
if (pendingContent !== undefined) { content = pendingContent; pendingContent = undefined; }
|
||||
return structuredClone(current().file);
|
||||
}
|
||||
if (route.includes("/heartbeat-runs?agentId=agent")) return structuredClone(current().runs);
|
||||
if (route.startsWith("/api/companies/company/activity?")) {
|
||||
activityReads.push(route);
|
||||
const query = new URL(route, api.origin).searchParams;
|
||||
expect(query.get("entityType")).toBe("agent"); expect(query.get("entityId")).toBe("agent");
|
||||
expect(query.get("limit")).toBe("200");
|
||||
if (unscopedActivity) return [unrelatedActivity[0]];
|
||||
return structuredClone([...unrelatedActivity, ...current().activity]
|
||||
.filter(row => row.entityType === query.get("entityType") && row.entityId === query.get("entityId"))
|
||||
.slice(0, Number(query.get("limit"))).map(row => ({ ...row,
|
||||
details: { ...row.details, privateFixturePayload: "must-not-be-retained" } })));
|
||||
}
|
||||
if (route === "/api/issues/task") return structuredClone(f.evidence.task!.issue);
|
||||
if (route === "/api/issues/task/runs") return [{ runId: "run" }];
|
||||
if (route === "/api/issues/task/interactions") return [];
|
||||
if (route === "/api/heartbeat-runs/run") return { ...structuredClone(f.run), ...(nativeFailure ? { status: "failed" } : {}) };
|
||||
if (route === "/api/issues/task/documents/settings-proof") return structuredClone(f.evidence.task!.document);
|
||||
if (route === "/api/issues/task/documents/settings-proof/revisions/document-revision") return structuredClone(f.evidence.task!.revision);
|
||||
throw new Error("unexpected_read:" + route);
|
||||
}, async put() { apiWrites.push("put"); throw new Error("tested save must use browser"); },
|
||||
async patch() { apiWrites.push("patch"); throw new Error("tested save must use browser"); } };
|
||||
const context = { page, api, company: { id: "company", issuePrefix: "QA" }, user: { id: "user", name: "QA" },
|
||||
agent: { id: "agent", name: "Owned agent campaign-1", provider: "opencode", model: "model", runtimeMode: "native" },
|
||||
environmentId: "environment", capabilities: ["company", "board-user", "task-write", "agent", "agent-settings-write"] } as unknown as JourneyContext;
|
||||
controls.createTask.mockReset(); controls.openDocument.mockReset();
|
||||
controls.createTask.mockImplementation(async (input: any) => {
|
||||
expect(saves).toHaveLength(1); expect(input.prompt).toBe(settingsTaskPrompt);
|
||||
expect(input.prompt).not.toContain(f.scope.reference);
|
||||
await input.beforeDispatch(); dispatched = true; return { issueId: "task" };
|
||||
});
|
||||
const session = { async own(resource: any) { owns.push(resource); } } as TargetSession;
|
||||
const evidence: JourneyEvidence = { outcome: "passed", startingState: "recipe", phases: [], checks: [], resourceDisposition: "retained", privacy: "private" };
|
||||
const directory = await mkdtemp(path.join(os.tmpdir(), "qa-settings-flow-")); directories.push(directory);
|
||||
return { context, session, evidence, directory, saves, savedPaths, selectedPaths, availablePaths, owns, apiWrites, activityReads, unrelatedActivity, original: f.original,
|
||||
currentContent: () => content, options: { directory, deadlineAt: Date.now() + 30000 } };
|
||||
}
|
||||
|
||||
describe("agent settings production-surface orchestration", () => {
|
||||
it.each(["WORKFLOW.md", "team guide/qa [settings].md", "team/nested/INSTRUCTIONS.md"])(
|
||||
"selects configured entry %s when AGENTS.md also exists and is initially selected", async entryFile => {
|
||||
const f = await fixture(false, false, false, undefined, entryFile);
|
||||
expect(f.availablePaths).toEqual(new Set(["AGENTS.md", entryFile]));
|
||||
await runAgentSettings(f.context, f.session, "campaign-1", f.evidence, f.options);
|
||||
expect(f.evidence.checks.every(row => row.passed)).toBe(true);
|
||||
expect(f.savedPaths).toEqual([entryFile, entryFile]);
|
||||
expect(f.selectedPaths.length).toBeGreaterThan(2); expect(new Set(f.selectedPaths)).toEqual(new Set([entryFile]));
|
||||
expect(f.currentContent()).toBe(f.original); expect(f.apiWrites).toEqual([]);
|
||||
const receipt = JSON.parse(await readFile(path.join(f.directory, "agent-settings-observations.json"), "utf8"));
|
||||
expect(receipt.saveAudits.saved.path).toBe(entryFile); expect(receipt.saveAudits.restored.path).toBe(entryFile);
|
||||
});
|
||||
it.each(["", " \n\t\n"])("reads and restores the exact empty-file editor value %j", async original => {
|
||||
const f = await fixture(false, false, false, original);
|
||||
await runAgentSettings(f.context, f.session, "campaign-1", f.evidence, f.options);
|
||||
expect(f.saves).toHaveLength(2); expect(f.saves[1]).toBe(original); expect(f.currentContent()).toBe(original);
|
||||
expect(f.evidence.checks.every(row => row.passed)).toBe(true);
|
||||
});
|
||||
it("saves and restores only through browser controls, then leaves the result ready for screenshot", async () => {
|
||||
const f = await fixture();
|
||||
await runAgentSettings(f.context, f.session, "campaign-1", f.evidence, f.options);
|
||||
expect(f.evidence.checks).toHaveLength(7); expect(f.evidence.checks.every(row => row.passed)).toBe(true);
|
||||
expect(f.saves).toHaveLength(2); expect(f.saves[1]).toBe(f.original); expect(f.apiWrites).toEqual([]);
|
||||
expect(f.owns.map(row => row.id ?? "intent")).toEqual(["intent", "task"]);
|
||||
const artifact = await readFile(path.join(f.directory, "agent-settings-observations.json"), "utf8");
|
||||
expect(artifact).not.toContain(f.original); expect(artifact).not.toContain("adapterConfig");
|
||||
expect(artifact).not.toContain("must-not-be-retained");
|
||||
const saved = JSON.parse(artifact); expect(saved.checks.every((row: any) => row.passed)).toBe(true);
|
||||
expect(saved.checkpoints.original.contentHash).toBe(saved.checkpoints.restored.contentHash);
|
||||
expect(saved.saveAudits).toEqual({ saved: { id: "save", companyId: "company", actorType: "user", actorId: "user",
|
||||
agentId: null, runId: null, action: "agent.files_updated", entityType: "agent", entityId: "agent", path: "AGENTS.md",
|
||||
contentHash: saved.checkpoints.saved.contentHash, source: "board" }, restored: { id: "restore", companyId: "company",
|
||||
actorType: "user", actorId: "user", agentId: null, runId: null, action: "agent.files_updated", entityType: "agent",
|
||||
entityId: "agent", path: "AGENTS.md", contentHash: saved.checkpoints.restored.contentHash, source: "board" } });
|
||||
expect(f.unrelatedActivity).toHaveLength(600); expect(f.activityReads.length).toBeGreaterThan(0);
|
||||
});
|
||||
it("rejects a wrongly unscoped activity response before any settings edit", async () => {
|
||||
const f = await fixture(false, true);
|
||||
await expect(runAgentSettings(f.context, f.session, "campaign-1", f.evidence, f.options)).rejects.toThrow("agent_settings_read_coverage_unavailable");
|
||||
expect(f.saves).toEqual([]); expect(f.owns).toEqual([]);
|
||||
});
|
||||
it("confirms a delayed restore audit and bytes before navigating away from the editor", async () => {
|
||||
const f = await fixture(false, false, true);
|
||||
await runAgentSettings(f.context, f.session, "campaign-1", f.evidence, f.options);
|
||||
expect(f.saves).toHaveLength(2); expect(f.currentContent()).toBe(f.original);
|
||||
expect(f.evidence.checks.every(row => row.passed)).toBe(true);
|
||||
});
|
||||
it("requires scoped settings-write capability before any configuration edit", async () => {
|
||||
const f = await fixture(); f.context.capabilities = ["company", "board-user", "task-write", "agent"];
|
||||
await expect(runAgentSettings(f.context, f.session, "campaign-1", f.evidence, f.options)).rejects.toThrow("owned_agent_settings_capability_required");
|
||||
expect(f.saves).toEqual([]); expect(f.owns).toEqual([]);
|
||||
});
|
||||
it("preserves a native failure and does not restore instructions while failed work may require cleanup", async () => {
|
||||
const f = await fixture(true);
|
||||
await expect(runAgentSettings(f.context, f.session, "campaign-1", f.evidence, f.options)).rejects.toThrow("agent_settings_native_run_failed");
|
||||
expect(f.saves).toHaveLength(1); expect(f.currentContent()).not.toBe(f.original);
|
||||
expect(JSON.parse(await readFile(path.join(f.directory, "agent-settings-observations.json"), "utf8")).checks.every((row: any) => row.passed)).toBe(false);
|
||||
expect(f.apiWrites).toEqual([]);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,174 @@
|
||||
import { randomBytes } from "node:crypto";
|
||||
import { mkdir, writeFile } from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { expect } from "@playwright/test";
|
||||
import { createTaskThroughUi } from "../user-actions.js";
|
||||
import { waitFor } from "./agent-flows.js";
|
||||
import { createObservedReader } from "./observation-read.js";
|
||||
import { openDocumentAuthoring } from "./task-document-flow.js";
|
||||
import { AGENT_SETTINGS_CHECKS, gradeAgentSettings, instructionDigest, matchedBoardSaveAudit, settingsInstructionContent,
|
||||
settingsDocumentKey, settingsProofHeading, settingsTaskPrompt, type AgentSettingsEvidence, type AgentSettingsScope,
|
||||
type SettingsSnapshot } from "./agent-settings-evidence.js";
|
||||
import { JourneyError, type JourneyContext, type JourneyEvidence, type TargetSession } from "./types.js";
|
||||
|
||||
type Row = Record<string, any>;
|
||||
const id = "agent-settings-persistence";
|
||||
|
||||
export async function runAgentSettings(context: JourneyContext, session: TargetSession, campaignId: string,
|
||||
evidence: JourneyEvidence, options: { deadlineAt: number; directory: string }) {
|
||||
const agent = context.agent;
|
||||
if (!agent || agent.runtimeMode !== "native" || !context.environmentId || !context.capabilities.includes("agent-settings-write")) {
|
||||
throw new JourneyError("unsupported", "owned_agent_settings_capability_required");
|
||||
}
|
||||
const scope: AgentSettingsScope = { campaignId, companyId: context.company.id, userId: context.user.id,
|
||||
agentId: agent.id, agentName: agent.name, provider: agent.provider, model: agent.model,
|
||||
environmentId: context.environmentId, reference: randomBytes(16).toString("hex") };
|
||||
const observations: AgentSettingsEvidence = {};
|
||||
const read = createObservedReader(context.api, options.deadlineAt), agentRoute = `/api/agents/${agent.id}`;
|
||||
const agentRunsRoute = `/api/companies/${scope.companyId}/heartbeat-runs?agentId=${agent.id}&limit=100`;
|
||||
const activityRoute = `/api/companies/${scope.companyId}/activity?entityType=agent&entityId=${agent.id}&limit=200`;
|
||||
const snapshot = async (): Promise<SettingsSnapshot> => {
|
||||
const [current, bundle, runs, activity] = await Promise.all([
|
||||
read.get<Row>(agentRoute), read.get<Row>(`${agentRoute}/instructions-bundle`),
|
||||
read.get<Row[]>(agentRunsRoute), read.get<Row[]>(activityRoute),
|
||||
]);
|
||||
if (typeof bundle?.entryFile !== "string" || !bundle.entryFile || !Array.isArray(runs) || runs.length >= 100
|
||||
|| !Array.isArray(activity) || activity.length >= 200
|
||||
|| activity.some(row => row.companyId !== scope.companyId || row.entityType !== "agent" || row.entityId !== agent.id)) {
|
||||
throw new JourneyError("failed", "agent_settings_read_coverage_unavailable");
|
||||
}
|
||||
const file = await read.get<Row>(`${agentRoute}/instructions-bundle/file?path=${encodeURIComponent(bundle.entryFile)}`);
|
||||
return { agent: current, bundle, file, runs, activity };
|
||||
};
|
||||
const openInstructions = async (entry: string, expectedContent: string, reload = false) => {
|
||||
await context.page.goto(`${context.api.origin}/${context.company.issuePrefix}/agents/${agent.id}/instructions`, { waitUntil: "domcontentloaded" });
|
||||
if (reload) await context.page.reload();
|
||||
const segments = entry.split("/");
|
||||
const treeRow = (filePath: string) => context.page.locator(`[role="treeitem"][data-file-tree-path=${JSON.stringify(filePath)}]`);
|
||||
for (let depth = 1; depth < segments.length; depth++) {
|
||||
const directory = treeRow(segments.slice(0, depth).join("/"));
|
||||
await expect(directory).toBeVisible();
|
||||
if (await directory.getAttribute("aria-expanded") !== "true") await directory.click();
|
||||
}
|
||||
const entryRow = treeRow(entry);
|
||||
await expect(entryRow).toBeVisible();
|
||||
await entryRow.click();
|
||||
const edit = context.page.getByRole("button", { name: `Edit ${entry}`, exact: true });
|
||||
await expect(edit).toBeVisible();
|
||||
if (!expectedContent.trim()) {
|
||||
await edit.click();
|
||||
const editor = context.page.getByRole("textbox", { name: "Instruction file editor", exact: true });
|
||||
await expect(editor).toHaveValue(expectedContent);
|
||||
const renderedContent = await editor.inputValue();
|
||||
await context.page.getByRole("button", { name: `Stop editing ${entry}`, exact: true }).click();
|
||||
return { edit, renderedContent };
|
||||
}
|
||||
const raw = context.page.getByTestId("instructions-raw-source");
|
||||
await expect(raw).toBeVisible();
|
||||
const renderedContent = await raw.textContent();
|
||||
if (renderedContent !== expectedContent) throw new JourneyError("failed", "agent_settings_rendered_source_mismatch");
|
||||
return { edit, renderedContent };
|
||||
};
|
||||
const saveThroughUi = async (beforePoint: SettingsSnapshot, content: string) => {
|
||||
const entry = beforePoint.bundle.entryFile, before = beforePoint.file.content;
|
||||
const { edit } = await openInstructions(entry, before);
|
||||
await edit.click();
|
||||
const editor = context.page.getByRole("textbox", { name: "Instruction file editor", exact: true });
|
||||
await expect(editor).toHaveValue(before);
|
||||
await editor.fill(content);
|
||||
const save = context.page.getByRole("button", { name: "Save changes", exact: true });
|
||||
await expect(save).toBeEnabled();
|
||||
await save.click();
|
||||
await expect(save).toBeDisabled({ timeout: 30_000 });
|
||||
// Disabled can also mean saving. Confirm persisted bytes and the new audit
|
||||
// before navigating, including during restoration.
|
||||
return waitFor(options.deadlineAt, snapshot, point => point.file.content === content
|
||||
&& Boolean(matchedBoardSaveAudit(point, beforePoint, scope)), "agent_settings_save_not_persisted");
|
||||
};
|
||||
const check = (name: typeof AGENT_SETTINGS_CHECKS[number]) => {
|
||||
const result = gradeAgentSettings(scope, observations).find(row => row.id === name)!;
|
||||
evidence.checks.push(result);
|
||||
if (!result.passed) throw new JourneyError("failed", name.replaceAll("-", "_"));
|
||||
};
|
||||
let failed = false;
|
||||
try {
|
||||
observations.original = await snapshot();
|
||||
const original = observations.original, entry = original.bundle.entryFile;
|
||||
if (original.agent.id !== agent.id || original.agent.companyId !== scope.companyId || original.agent.name !== agent.name
|
||||
|| !agent.name.includes(campaignId) || original.agent.adapterType !== "paperclip_runner"
|
||||
|| original.agent.adapterConfig?.provider !== agent.provider || original.agent.adapterConfig?.model !== agent.model
|
||||
|| original.bundle.mode !== "managed" || original.bundle.editable !== true || original.file.editable !== true
|
||||
|| typeof original.file.content !== "string" || original.file.content.includes("\r")
|
||||
|| original.file.content.includes(settingsProofHeading) || original.runs.length !== 0) {
|
||||
throw new JourneyError("failed", "agent_settings_fixture_not_fresh_owned");
|
||||
}
|
||||
const edited = settingsInstructionContent(original.file.content, scope.reference);
|
||||
observations.saved = await saveThroughUi(original, edited);
|
||||
check("agent-settings-browser-save");
|
||||
const reopenedSource = await openInstructions(entry, edited, true);
|
||||
observations.reloaded = { ...await snapshot(), renderedContent: reopenedSource.renderedContent };
|
||||
check("agent-settings-reload-persisted");
|
||||
|
||||
const title = `Journey QA agent settings ${campaignId}`;
|
||||
const task = await createTaskThroughUi({ page: context.page, origin: context.api.origin, issuePrefix: context.company.issuePrefix,
|
||||
agentName: agent.name, title, prompt: settingsTaskPrompt, workMode: "standard", requireExplicitTitle: true,
|
||||
beforeDispatch: () => session.own({ kind: "issue", name: title, journeyId: id }) });
|
||||
await session.own({ kind: "issue", name: title, id: task.issueId, journeyId: id });
|
||||
const route = `/api/issues/${task.issueId}`, documentRoute = `${route}/documents/${settingsDocumentKey}`;
|
||||
const state = await waitFor(options.deadlineAt, async () => {
|
||||
const [issue, summaries, interactions] = await Promise.all([
|
||||
read.get<Row>(route), read.get<Row[]>(`${route}/runs`), read.get<Row[]>(`${route}/interactions`),
|
||||
]);
|
||||
if (!Array.isArray(summaries) || summaries.length > 1 || !Array.isArray(interactions)) throw new JourneyError("failed", "agent_settings_run_coverage_mismatch");
|
||||
const runs = await Promise.all(summaries.map(row => read.get<Row>(`/api/heartbeat-runs/${row.runId}`)));
|
||||
if (runs.some(run => ["failed", "cancelled", "timed_out"].includes(run.status))) throw new JourneyError("failed", "agent_settings_native_run_failed");
|
||||
return { issue, runs, interactions };
|
||||
}, point => point.issue.status === "done" && point.runs.length === 1 && point.runs[0]!.status === "succeeded"
|
||||
&& point.runs[0]!.nativePhase === "committed" && point.interactions.every(row => row.status !== "pending"), "agent_settings_task_not_completed");
|
||||
const document = await read.get<Row>(documentRoute);
|
||||
const revision = await read.get<Row>(`${documentRoute}/revisions/${document.latestRevisionId}`);
|
||||
observations.task = { ...state, document, revision };
|
||||
observations.afterWork = await snapshot();
|
||||
check("agent-settings-instruction-only-input");
|
||||
check("configured-model-and-environment");
|
||||
check("agent-settings-result-attributed");
|
||||
await context.page.goto(`${context.api.origin}/${context.company.issuePrefix}/issues/${state.issue.identifier ?? task.issueId}`, { waitUntil: "domcontentloaded" });
|
||||
await context.page.reload();
|
||||
await openDocumentAuthoring(context);
|
||||
const displayed = context.page.locator(`#document-${settingsDocumentKey}`);
|
||||
await expect(displayed).toContainText(scope.reference);
|
||||
observations.task.reopenedDocument = await read.get<Row>(documentRoute);
|
||||
observations.task.reopenedRevision = await read.get<Row>(`${documentRoute}/revisions/${document.latestRevisionId}`);
|
||||
observations.task.renderedReference = (await displayed.textContent())?.includes(scope.reference) ? scope.reference : undefined;
|
||||
check("agent-settings-result-reopened");
|
||||
|
||||
await saveThroughUi(observations.afterWork, original.file.content);
|
||||
const restoredSource = await openInstructions(entry, original.file.content);
|
||||
observations.restored = { ...await snapshot(), renderedContent: restoredSource.renderedContent };
|
||||
check("agent-settings-original-restored");
|
||||
// The shared screenshot policy permits only the task result, not settings.
|
||||
await context.page.goto(`${context.api.origin}/${context.company.issuePrefix}/issues/${state.issue.identifier ?? task.issueId}`, { waitUntil: "domcontentloaded" });
|
||||
await openDocumentAuthoring(context);
|
||||
await expect(context.page.locator(`#document-${settingsDocumentKey}`)).toContainText(scope.reference);
|
||||
} catch (error) { failed = true; throw error; }
|
||||
finally {
|
||||
try {
|
||||
await read.preserve(options.directory);
|
||||
await mkdir(options.directory, { recursive: true, mode: 0o700 });
|
||||
await writeFile(path.join(options.directory, "agent-settings-observations.json"), JSON.stringify({
|
||||
companyId: scope.companyId, agentId: scope.agentId, reference: scope.reference,
|
||||
checkpoints: Object.fromEntries(["original", "saved", "reloaded", "afterWork", "restored"].map(key => {
|
||||
const point = observations[key as Exclude<keyof AgentSettingsEvidence, "task">];
|
||||
return [key, point ? { entryFile: point.file.path, contentHash: point.file.contentHash,
|
||||
independentlyComputedHash: typeof point.file.content === "string" ? instructionDigest(point.file.content) : null,
|
||||
revisionId: point.file.revision?.id ?? null, runIds: point.runs.map(row => row.id ?? row.runId) } : null];
|
||||
})),
|
||||
issueId: observations.task?.issue.id ?? null, runIds: observations.task?.runs.map(row => row.id) ?? [],
|
||||
documentId: observations.task?.document.id ?? null, documentRevisionId: observations.task?.revision.id ?? null,
|
||||
saveAudits: { saved: matchedBoardSaveAudit(observations.saved, observations.original, scope),
|
||||
restored: matchedBoardSaveAudit(observations.restored, observations.afterWork, scope) },
|
||||
checks: gradeAgentSettings(scope, observations),
|
||||
}, null, 2), { mode: 0o600 });
|
||||
} catch (error) { if (!failed) throw error; }
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,40 @@
|
||||
import { instructionDigest, settingsInstructionContent, settingsDocumentKey, settingsTaskPrompt, type AgentSettingsEvidence, type AgentSettingsScope } from "./agent-settings-evidence.js";
|
||||
|
||||
export function settingsEvidenceFixture(reference = "1a2b3c4d5e6f", original = "# Ordinary instructions\nFollow the assigned task.\n", entryFile = "AGENTS.md") {
|
||||
const scope: AgentSettingsScope = { campaignId: "campaign-1", companyId: "company", userId: "user", agentId: "agent",
|
||||
agentName: "Owned agent campaign-1", environmentId: "environment", provider: "opencode", model: "model", reference };
|
||||
const saved = settingsInstructionContent(original, scope.reference);
|
||||
const run = { id: "run", companyId: scope.companyId, agentId: scope.agentId, nativeIssueId: "task", status: "succeeded",
|
||||
runtimeMode: "native", nativePhase: "committed", resultJson: { finalizationPhase: "committed" },
|
||||
contextSnapshot: { issueId: "task", paperclipEnvironment: { id: scope.environmentId } },
|
||||
runnerProfileJson: { nativeExecutionInput: { provider: { kind: scope.provider, model: scope.model } } } };
|
||||
const snapshot = (content: string, runs: any[] = [], activity: any[] = []) => {
|
||||
const hash = instructionDigest(content);
|
||||
return { agent: { id: scope.agentId, companyId: scope.companyId, name: scope.agentName, adapterType: "paperclip_runner",
|
||||
adapterConfig: { provider: scope.provider, model: scope.model } },
|
||||
bundle: { agentId: scope.agentId, companyId: scope.companyId, entryFile, mode: "managed", editable: true,
|
||||
files: [{ path: entryFile, isEntryFile: true, editable: true },
|
||||
...(entryFile !== "AGENTS.md" ? [{ path: "AGENTS.md", isEntryFile: false, editable: true }] : [])] },
|
||||
file: { path: entryFile, content, contentHash: hash, size: Buffer.byteLength(content), isEntryFile: true, editable: true,
|
||||
revision: { id: `agent-files:${hash}`, companyId: scope.companyId, agentId: scope.agentId, entryFile,
|
||||
contentHash: hash, byteLength: Buffer.byteLength(content), parentRevisionId: null, baseRevisionId: null, source: "api" } },
|
||||
runs, activity, renderedContent: content };
|
||||
};
|
||||
const audit = (id: string, content: string) => ({ id, companyId: scope.companyId, actorType: "user", actorId: scope.userId,
|
||||
agentId: null, runId: null, action: "agent.files_updated", entityType: "agent", entityId: scope.agentId,
|
||||
details: { path: entryFile, contentHash: instructionDigest(content), source: "board" } });
|
||||
const firstSave = audit("save", saved), restore = audit("restore", original);
|
||||
const body = JSON.stringify({ reference: scope.reference, settingsSaved: true });
|
||||
const document = { id: "document", companyId: scope.companyId, issueId: "task", key: settingsDocumentKey,
|
||||
body, latestRevisionId: "document-revision", createdByAgentId: scope.agentId, createdByUserId: null };
|
||||
const revision = { id: "document-revision", documentId: document.id, body, createdByRunId: run.id,
|
||||
createdByAgentId: scope.agentId, createdByUserId: null };
|
||||
const evidence: AgentSettingsEvidence = {
|
||||
original: snapshot(original), saved: snapshot(saved, [], [firstSave]), reloaded: snapshot(saved, [], [firstSave]),
|
||||
afterWork: snapshot(saved, [run], [firstSave]), restored: snapshot(original, [run], [firstSave, restore]),
|
||||
task: { issue: { id: "task", companyId: scope.companyId, assigneeAgentId: scope.agentId, assigneeUserId: null,
|
||||
createdByUserId: scope.userId, createdByAgentId: null, workMode: "standard", description: settingsTaskPrompt, status: "done" },
|
||||
runs: [run], interactions: [], document, revision, reopenedDocument: { ...document }, reopenedRevision: { ...revision }, renderedReference: scope.reference },
|
||||
};
|
||||
return { scope, evidence, original, saved, run, firstSave, restore };
|
||||
}
|
||||
@@ -22,6 +22,7 @@ export const productJourneys = [
|
||||
{ id: "stop-redirect", label: "Stop active work and send a new direction once", requires: ["company", "board-user", "task-write", "agent"], capabilities: ["steering/stop-and-pause"], paid: true },
|
||||
{ id: "followup-while-running", label: "Send a follow-up during active work and verify its saved result", requires: ["company", "board-user", "task-write", "agent"], capabilities: ["steering/queue"], paid: true },
|
||||
{ id: "hire-delegate-reuse", label: "Hire, delegate and reuse one teammate", requires: ["company", "board-user", "task-write", "agent"], capabilities: ["agents/hire", "documents-artifacts/documents"], paid: true },
|
||||
{ id: "agent-settings-persistence", label: "Save a fresh dedicated agent's instructions and verify its first task", requires: ["company", "board-user", "task-write", "agent", "agent-settings-write"], capabilities: ["agents/instructions", "tasks/completion"], paid: true },
|
||||
{ id: "cloud-onboarding", label: "Create a Cloud workspace and complete a first task", requires: ["cloud-account", "fresh-workspace", "provider-key"], capabilities: ["onboarding/first-task"], paid: true },
|
||||
] as const;
|
||||
|
||||
@@ -57,6 +58,8 @@ export function parseTargetConfig(value: unknown): TargetConfig {
|
||||
if (c.adapterModule !== undefined && (typeof c.adapterModule !== "string" || !c.adapterModule)) throw new Error("invalid_adapter_module");
|
||||
if (c.adapterConfig !== undefined && !object(c.adapterConfig)) throw new Error("invalid_adapter_configuration");
|
||||
if (c.modelProfileId !== undefined && !["runner-codex", "runner-codex-mini"].includes(c.modelProfileId)) throw new Error("unsupported_model_profile");
|
||||
if (c.journeys.includes("agent-settings-persistence") && c.journeys.some(id => id !== "agent-settings-persistence"
|
||||
&& productJourneys.find(journey => journey.id === id)!.paid)) throw new Error("agent_settings_requires_dedicated_campaign");
|
||||
if (c.mode === "new-cloud" && c.journeys[0] !== "cloud-onboarding") throw new Error("new_cloud_requires_onboarding_first");
|
||||
if (c.mode === "new-cloud" && c.baseline.connections.length) throw new Error("onboarding_requires_ui_connection_setup");
|
||||
if (c.mode !== "new-cloud" && c.journeys.includes("cloud-onboarding")) throw new Error("onboarding_requires_fresh_cloud_workspace");
|
||||
|
||||
@@ -11,7 +11,7 @@ export function journeyFixtures(session: TargetSession, campaignId?: string) {
|
||||
id: "ready-context",
|
||||
async setup() { return ready ??= await (preparation ??= session.prepare()); },
|
||||
});
|
||||
for (const id of ["company", "board-user", "task-write", "agent", "skill-write", "review-service", "inbox-author", "company-access"]) registry.register({
|
||||
for (const id of ["company", "board-user", "task-write", "agent", "agent-settings-write", "skill-write", "review-service", "inbox-author", "company-access"]) registry.register({
|
||||
id, dependencies: ["ready-context"],
|
||||
async setup(resolved) {
|
||||
const context = resolved.get("ready-context") as JourneyContext;
|
||||
|
||||
@@ -21,6 +21,7 @@ import { runClarificationJourney } from "./clarification-flow.js";
|
||||
import { runRepositoryContinuation } from "./repository-continuation-flow.js";
|
||||
import { runScheduledRoutine } from "./routine-schedule-flow.js";
|
||||
import { runInputFile } from "./input-file-flow.js";
|
||||
import { runAgentSettings } from "./agent-settings-flow.js";
|
||||
import { companyAccessIsolation } from "./company-access-flow.js";
|
||||
import { inboxTriage } from "./inbox-triage-flow.js";
|
||||
import { JourneyError, type JourneyContext, type JourneyEvidence, type JourneyOutcome, type TargetLifecycleInput, type TargetProvider, type TargetSession } from "./types.js";
|
||||
@@ -150,6 +151,8 @@ export async function launchProductJourneys(args: string[], repositoryRoot: stri
|
||||
"project-context-evidence.ts",
|
||||
"agent-flows.ts",
|
||||
"hire-delegate-flow.ts",
|
||||
"agent-settings-flow.ts",
|
||||
"agent-settings-evidence.ts",
|
||||
"hire-delegate-composer.ts",
|
||||
"hire-delegate-evidence.ts",
|
||||
"active-followup-flow.ts",
|
||||
@@ -278,6 +281,8 @@ export async function launchProductJourneys(args: string[], repositoryRoot: stri
|
||||
result.runtimeMode = ready.agent.runtimeMode;
|
||||
if (id === "repository-continuation") await runRepositoryContinuation(ready, session!, campaignId, evidence,
|
||||
{ deadlineAt, directory: path.join(outputDir, "private-stories", id) });
|
||||
else if (id === "agent-settings-persistence") await runAgentSettings(ready, session!, campaignId, evidence,
|
||||
{ deadlineAt, directory: path.join(outputDir, "private-stories", id) });
|
||||
else if (id === "routine-scheduled-run") await runScheduledRoutine(ready, session!, campaignId, evidence,
|
||||
{ deadlineAt, directory: path.join(outputDir, "private-stories", id) });
|
||||
else if (id === "file-input-answer") await runInputFile(ready, session!, campaignId, evidence,
|
||||
|
||||
@@ -10,6 +10,7 @@ export const agentJourneyRuns = {
|
||||
"protected-actions": 4,
|
||||
"create-skill-studio": 1,
|
||||
"stop-redirect": 3,
|
||||
"agent-settings-persistence": 1,
|
||||
"question-answer-resume": 3,
|
||||
"hire-delegate-reuse": 7
|
||||
} as const;
|
||||
@@ -45,5 +46,6 @@ export const journeyRequiredChecks: Record<string, readonly string[]> = {
|
||||
"stopped-run-no-later-events", "stop-preserved-download", "redirect-consumed-once", "user-reply-stored-once", "configured-model-and-environment"],
|
||||
"followup-while-running": ["followup-native-command-active", "followup-stored-once", "followup-consumed-once", "followup-final-answer-persisted", "followup-one-or-two-native-runs", "configured-model-and-environment"],
|
||||
"hire-delegate-reuse": ["one-attributed-native-hire", "first-delegated-task-complete", "checklist-authored-by-hired-run", "same-teammate-reused", "review-authored-by-same-hired-run", "original-checklist-preserved", "hire-user-messages-stored-once", "delegated-result-visible-after-reload", "configured-model-and-environment"],
|
||||
"agent-settings-persistence": ["agent-settings-browser-save", "agent-settings-reload-persisted", "agent-settings-instruction-only-input", "agent-settings-result-attributed", "agent-settings-result-reopened", "agent-settings-original-restored", "configured-model-and-environment"],
|
||||
"cloud-onboarding": ["fresh-workspace", "ui-connection-and-native-agent", "first-task-complete", "downloadable-artifact-verified", "native-trace-verified"],
|
||||
};
|
||||
@@ -19,6 +19,7 @@ export type JourneyId =
|
||||
| "create-skill-studio"
|
||||
| "stop-redirect"
|
||||
| "hire-delegate-reuse"
|
||||
| "agent-settings-persistence"
|
||||
| "file-input-answer"
|
||||
| "inbox-triage"
|
||||
| "routine-scheduled-run"
|
||||
|
||||
Reference in new issue
Block a user