mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-06 21:05:21 +02:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Users can delegate work through onboarding and Agent Chat. > - A completed task does not prove that its result reached the original conversation. > - Existing tests do not isolate completion after the source chat becomes idle. > - This pull request adds four explicit native-runner probes across Claude and Codex. > - The probes preserve the result and reply so we can separate delivery failures from inaccurate answers. ## Linked Issues or Issue Description Refs #13775. Refs #13813. These evals extend native-runner qualification. They measure completion updates before we choose a product change. ## What Changed - Add the opt-in `completion-updates` suite with two stories for each native provider. - Test completion in the existing onboarding task flow and after an Agent Chat handoff becomes idle. - Gate the chat worker on a brief inside its managed project workspace. Prove the source is idle before releasing the worker. - Check durable task completion, saved output, a subsequent source reply, and rendered access to the result. - Preserve replies, task state, screenshots, run events, and a separate semantic review rubric. - Add grader regression tests and update the documented eval contract. - Preserve the suites added on master and include four completion cases in the 306-cell catalog. Production behavior and prompts are unchanged. ## Verification - Passed all 565 eval support tests across 45 files after merging current master: `node node_modules/vitest/vitest.mjs run --config tests/runner-e2e/vitest.config.ts`. - Passed eval TypeScript: `node node_modules/typescript/bin/tsc -p tests/runner-e2e/tsconfig.json`. - Confirmed four selected cells: `node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/launch.ts --list --suite completion-updates`. - Four-cell behavior campaign on source `ad47cf1da2b1e36f19f4227cfeb53998720b0b5b`: https://github.com/paperclipai/paperclip/actions/runs/36072337485. - A screenshot-only follow-up waits for the restored source reply to render after result-link navigation. Its one-cell Claude onboarding verification passed on final head: https://github.com/paperclipai/paperclip/actions/runs/36075716141. Corrected report: https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-36075716141-1/. The original four-cell onboarding screenshots caught navigation loading; its saved reply evidence remains valid. The follow-up again found stale wording: "That work will run next" was posted 38 seconds after the child was Done. The four-cell campaign keeps its original source and measurements. - Published evidence: https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-36072337485-1/. - Suite definition: `afba4d85d6c53d9f64c08b37a2e9cc20481b78f5bd7e2fa012045e2c69444d9d`, version 6. Models: native `gpt-5.6-sol` and `claude-sonnet-5`, local execution, one attempt per cell. All four cleanup checks passed. Onboarding billing coverage is partial; reported zero cost must not be read as a free run. | Story | Automated delivery/access | Separate semantic review | | --- | --- | --- | | Codex onboarding | Pass | Pass: accurate completion reply with an accessible result | | Claude onboarding | Pass | Fail: reply says it will save the note once the task runs, after the note is already saved and the task is Done | | Codex idle chat handoff | Fail | Worker completed and saved the note; no completion reply during the full observation window | | Claude idle chat handoff | Fail | Worker completed and saved the note; no completion reply during the full observation window | Both chat cases positively recorded the source waiting and the worker at the brief gate before release. Both saved outputs include the brief-only start time. The opt-in campaign is red because it exposes current behavior. It is not a required merge gate. The PR does not fix that product behavior. Semantic review is a recorded human/agent assessment of retained evidence; it is not an automated prose-quality judge. - Second campaign: https://github.com/paperclipai/paperclip/actions/runs/36071065098. Codex chat reached the idle boundary and completed its task, then received no completion reply during the full window. Claude onboarding again returned a stale handoff answer. Claude chat exceeded the prior 110-second handoff setup budget; this revision raises that bounded setup window to 180 seconds. - Retained baseline: https://github.com/paperclipai/paperclip/actions/runs/36069427676. Onboarding passed delivery/access for both providers, but Claude gave a stale handoff answer. Chat cases stopped at fixture problems; they do not establish a completion-delivery failure. This revision fixes the workspace path and competing reference requirements. - On the previous head `4023a2a3c28d45c9eb2c42d452ce99ffba5c7b73`, 54 PR checks passed and two were skipped, including typecheck, tests, and build. Broad checks ran in CI, not locally. That head received Greptile 5/5 with no unresolved findings. The unchanged mobile repository-settings browser test passed on one targeted retry after a detached/disabled Save-button timeout. - Merged current master in `9b4491e1f` and resolved the catalog-count conflict. Eval support tests and eval TypeScript pass locally. All individual CI jobs passed on this merge commit, including build, typecheck, server tests, runner checks, and browser shards. The final aggregate check also passed: 54 checks passed and two were skipped. Greptile reviewed this exact commit at 5/5 with no unresolved findings. ## Risks - These explicit probes can expose current product failures. They do not change the default paid test selection. - Mechanical delivery and result access do not establish answer accuracy. The preserved reply still requires semantic review. - A fixture failure before the idle boundary or worker completion cannot establish a completion-update failure. - The handoff setup window lasts three minutes. The worker brief wait is bounded at four minutes. The observation window lasts two minutes after worker completion. It retains later replies without erasing earlier accessible delivery. ## Model Used OpenAI Codex, GPT-6 (`gpt-6-astra`), with reasoning, repository inspection, code execution, and GitHub tool use. The runtime does not expose the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
93 lines
3.3 KiB
TypeScript
93 lines
3.3 KiB
TypeScript
import { normalizePaperclipOperationalSkillPreference } from "../../packages/adapter-utils/src/server-utils.js";
|
|
import type { RunnerApi } from "./api.js";
|
|
import type { LiveFixtureValues } from "./live-fixtures.js";
|
|
import type { CredentialName, MatrixExecution } from "./types.js";
|
|
|
|
/** Provision only credentials for the UI-created company. The production wizard
|
|
* remains the sole creator/configurer of its first agent and onboarding task. */
|
|
export async function provisionFirstTaskFixtures(input: {
|
|
api: Pick<RunnerApi, "get" | "postSensitive">;
|
|
execution: MatrixExecution;
|
|
nonce: string;
|
|
company: LiveFixtureValues["company"];
|
|
credentials: Partial<Record<CredentialName, string>>;
|
|
}): Promise<LiveFixtureValues> {
|
|
const { api, execution, nonce, company, credentials } = input;
|
|
if (
|
|
!["first-task", "completion-updates"].includes(execution.suite.id) || execution.task.flow !== "first_task" ||
|
|
execution.environment.id !== "local" ||
|
|
![
|
|
"legacy-codex",
|
|
"legacy-claude",
|
|
"runner-codex",
|
|
"runner-acpx-claude",
|
|
].includes(execution.profile.id)
|
|
) {
|
|
throw new Error(
|
|
"First-task fixtures require a supported local onboarding profile",
|
|
);
|
|
}
|
|
const credential = execution.profile.credential;
|
|
const value = credentials[credential];
|
|
if (!value) throw new Error(`Missing credential ${credential}`);
|
|
const environments = await api.get<Array<LiveFixtureValues["environment"]>>(
|
|
`/api/companies/${company.id}/environments?driver=local`,
|
|
);
|
|
const environment = environments.find((e) => e.driver === "local");
|
|
if (!environment)
|
|
throw new Error("Onboarding fixture local environment missing");
|
|
const secret = await api.postSensitive<{ id: string }>(
|
|
`/api/companies/${company.id}/secrets`,
|
|
{
|
|
name: `First task ${credential}`,
|
|
key: credential,
|
|
value,
|
|
},
|
|
);
|
|
return {
|
|
company,
|
|
environment,
|
|
secretRefs: {
|
|
[credential]: {
|
|
type: "secret_ref",
|
|
secretId: secret.id,
|
|
version: "latest",
|
|
},
|
|
},
|
|
agent: { id: "", name: `Garden lead ${nonce}`, companyId: company.id },
|
|
teardown: async () => {}, // Existing launcher removes the complete isolated instance.
|
|
};
|
|
}
|
|
|
|
/** Reuse the qualified runtime configuration only, never the QA persona from
|
|
* buildAgent. Native onboarding is not offered by the production wizard yet. */
|
|
export function firstTaskNativeRuntimePatch(
|
|
execution: MatrixExecution,
|
|
fixtures: LiveFixtureValues,
|
|
agent: Record<string, any>,
|
|
) {
|
|
if (!["runner-codex", "runner-acpx-claude"].includes(execution.profile.id))
|
|
throw new Error("Unsupported native first-task profile");
|
|
const built = execution.profile.buildAgent({
|
|
environmentId: fixtures.environment.id,
|
|
environmentFixtureId: "local",
|
|
workspacePath: agent.adapterConfig?.cwd ?? "",
|
|
secretRefs: fixtures.secretRefs,
|
|
executionId: execution.id,
|
|
});
|
|
const config = {
|
|
...agent.adapterConfig,
|
|
...(built.adapterConfig as Record<string, unknown>),
|
|
};
|
|
// Preserve the wizard's model choice (including its unset provider default).
|
|
if (agent.adapterConfig?.model == null) delete config.model;
|
|
else config.model = agent.adapterConfig.model;
|
|
return {
|
|
adapterType: "paperclip_runner",
|
|
adapterConfig: normalizePaperclipOperationalSkillPreference(
|
|
"paperclip_runner",
|
|
config,
|
|
),
|
|
};
|
|
}
|