From 8424a510181984c753bdf205d8f01056288711dc Mon Sep 17 00:00:00 2001 From: AlexFrontegg <140060214+AlexFrontegg@users.noreply.github.com> Date: Mon, 2 Nov 2026 18:49:08 +0200 Subject: [PATCH 01/35] fix(redaction): scan long letter/digit runs in linear time (#16999) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip removes secrets from run logs, tool results and diagnostics with `redactSensitiveText` before it stores or shows them. > - The JSON field-name patterns in that function start with an unanchored `[A-Za-z0-9_-]*`. Each start position inside a long run of letters and digits scans to the end of the run, so the cost grows with the square of the run length. > - Base64 of text is one long run of letters and digits. A 40 KB base64 blob of source code in a shell command takes about 5 s to redact, and a large Gmail RAW tool result does not finish. > - The redaction runs on the main event loop, so one such chunk blocks every request and every run on the instance. > - This pull request makes these scans start only at the start of a run. The scans become linear, and the redacted output does not change. > - The benefit is that large base64 payloads no longer freeze the server, and every secret that was redacted before is still redacted. ## Linked Issues or Issue Description Fixes #15393. The tool content guard runs `redactSensitiveText` on tool results. A Gmail RAW body is base64url of a MIME message, which is one long letter/digit run. `validateToolContent` on a Gmail-shaped result with a 68 KB `raw` field takes 59 s on master and 19 ms with this change; a 5.4 MB `raw` field takes 0.65 s. Thanks @ts-beat for the clear report in #15393. Refs #12309 (false positives in the same patterns; this pull request does not change which text matches). I searched issues and pull requests for "redaction", "redactSensitiveText", "quadratic", "backtracking", "hangs the server" and "100% CPU". #15393 reports the hang. No open pull request fixes it. **Subsystem affected:** secret redaction (`server/src/redaction.ts`, `packages/adapter-utils/src/command-redaction.ts`), used by run logs, tool content guards and run diagnostics. **Problem:** `compactRunLogChunk` redacts each run log chunk before it truncates the chunk. A shell command such as `printf '%s' '' | base64 -d > file.ts` with base64 of source text makes one long letter/digit run. Base64 of ASCII text contains `+` or `/` only where every third byte is `>`, `?`, `~` or DEL, so random base64 does not show the problem. One `redactSensitiveText` call on such a tool-call line takes (local measurement): | Blob | Before | After | | --- | --- | --- | | 4 KB | 66 ms | 1.1 ms | | 8 KB | 272 ms | 0.5 ms | | 16 KB | 1.1 s | 1.0 ms | | 40 KB | 5.4 s | 2.5 ms | | 1 MB | not measured | 36 ms | The `\b`-anchored command patterns (`KEY=value`, JWT, `Authorization`) and the CLI option pattern have the same quadratic cost on dash-separated runs, for example base64url data. **Proposed solution:** start the unanchored field-name scans only where no name character precedes them (`(?` are still redacted, and `my-token is here` is not changed. - Without the change, the timing tests fail: 9.3 s, 9.8 s, 32 s and 60 s against the 1 s budget. The correctness tests pass with and without the change. - Mutation check: I reverted each of the six pattern changes one at a time. Each revert makes at least one test fail. - Differential fuzzing: 1.1 million random inputs built from secret names, quotes, escaped quotes, dashes, dots, colons and JWT parts gave the same output from the old and new `redactSensitiveText`, `redactCommandText` and `redactDiagnosticText`. - `cd server && npx vitest run `: all 33 server test files that use the redaction helpers, with embedded Postgres, 1730 passed. - `npx vitest run --project @paperclipai/adapter-utils packages/adapter-utils/src/command-redaction.test.ts`: 25 passed. - `npx vitest run --project @paperclipai/adapter-utils`, complete suite: 80 of 82 files passed in the parallel run. The 2 other files (`github-launcher-environment`, `sandbox-managed-runtime`) had 5 s timeouts in git-based tests and passed on rerun. ## Risks - Lookbehind assertions need ES2018. `redactCommandText` also runs in the UI bundle. Node and current browsers support lookbehind, and the server (`server/src/log-redaction.ts`) and the UI (`ui/src/components/task-chat/task-chat-attachments.ts`) already use it. - A change to a redaction pattern could leak a secret. The reasoning above, the unchanged existing tests and the differential fuzzing show the same output as before. - Not changed here, found while checking: `redactDiagnosticText` backtracks exponentially on an escaped JSON secret value made of many escaped backslashes with no closing quote, and `redactSensitiveText` redacts a value only up to its first quote of either kind (`{"apiKey":"abc\"tail"}` keeps `tail"`). Both behave the same before and after this pull request. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) by Anthropic, in Claude Code, with tool use (shell, code search, and test runs) and extended thinking. A human directed the design choices. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --- .../src/command-redaction.test.ts | 39 +++++++++++ .../adapter-utils/src/command-redaction.ts | 11 +-- .../src/__tests__/heartbeat-run-log.test.ts | 18 +++++ server/src/__tests__/redaction.test.ts | 70 +++++++++++++++++++ server/src/redaction.ts | 7 +- 5 files changed, 138 insertions(+), 7 deletions(-) diff --git a/packages/adapter-utils/src/command-redaction.test.ts b/packages/adapter-utils/src/command-redaction.test.ts index 321866bf7c7..2afd383a6d9 100644 --- a/packages/adapter-utils/src/command-redaction.test.ts +++ b/packages/adapter-utils/src/command-redaction.test.ts @@ -145,3 +145,42 @@ second-line\" status=401`; expect(output).toContain(REDACTED_COMMAND_TEXT_VALUE); }); }); + +describe("redactCommandText on long runs", () => { + const jwt = + "eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIxMjM0NTY3ODkwIn0.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c"; + + function redactWithinTimeBudget(input: string) { + const startedAt = performance.now(); + const output = redactCommandText(input); + expect(performance.now() - startedAt).toBeLessThan(1_000); + return output; + } + + it("keeps redacting options, assignments, and tokens inside dash-separated runs", () => { + const cases = [ + ["x--token flag-secret", `x--token ${REDACTED_COMMAND_TEXT_VALUE}`], + ["--x--token flag-secret", `--x--token ${REDACTED_COMMAND_TEXT_VALUE}`], + [ + "a-b--api-key=assigned-secret --password 'quoted-secret'", + `a-b--api-key=${REDACTED_COMMAND_TEXT_VALUE} --password '${REDACTED_COMMAND_TEXT_VALUE}'`, + ], + ["prefix-MY_API_KEY=env-secret", `prefix-MY_API_KEY=${REDACTED_COMMAND_TEXT_VALUE}`], + [`--${jwt}`, `--${REDACTED_COMMAND_TEXT_VALUE}`], + [`session=${jwt}`, `session=${REDACTED_COMMAND_TEXT_VALUE}`], + ["my-token is here", "my-token is here"], + ]; + + for (const [input, expected] of cases) { + expect(redactCommandText(input)).toBe(expected); + } + }); + + it("scans long letter/digit and dash-separated runs in linear time", () => { + const base64Url = Buffer.from(Array.from({ length: 90_000 }, (_, index) => (index * 7919) % 256)).toString("base64url"); + for (const run of ["a".repeat(120_000), "a-".repeat(60_000), "a--".repeat(40_000), base64Url]) { + expect(redactWithinTimeBudget(`${run} --token run-secret`)).toBe(`${run} --token ${REDACTED_COMMAND_TEXT_VALUE}`); + expect(redactWithinTimeBudget(`${run}--token run-secret`)).toBe(`${run}--token ${REDACTED_COMMAND_TEXT_VALUE}`); + } + }); +}); diff --git a/packages/adapter-utils/src/command-redaction.ts b/packages/adapter-utils/src/command-redaction.ts index 46eeb6f5040..451f0ccf58e 100644 --- a/packages/adapter-utils/src/command-redaction.ts +++ b/packages/adapter-utils/src/command-redaction.ts @@ -11,16 +11,19 @@ export function isPublicExecutorToolSelector(value: string): boolean { return PUBLIC_EXECUTOR_TOOL_SELECTORS.has(value); } -const SECRET_NAME_PATTERN = String.raw`[A-Za-z0-9_-]*(?:api[-_]?key|(?:access[-_]?|auth[-_]?)?token|token|authorization(?:[-_]?code)?|bearer|secrets?|passwd|passwords?|credentials?|jwt|private[-_]?key|cookie|connectionstring)(?:[-_]?(?:value|header|prod(?:uction)?|dev(?:elopment)?|test|staging|primary|secondary))*`; +const SECRET_NAME_WORDS = String.raw`(?:api[-_]?key|(?:access[-_]?|auth[-_]?)?token|token|authorization(?:[-_]?code)?|bearer|secrets?|passwd|passwords?|credentials?|jwt|private[-_]?key|cookie|connectionstring)(?:[-_]?(?:value|header|prod(?:uction)?|dev(?:elopment)?|test|staging|primary|secondary))*`; +const SECRET_NAME_PATTERN = String.raw`[A-Za-z0-9_-]*${SECRET_NAME_WORDS}`; +const SECRET_NAME_START = String.raw`(? { expect(compacted).not.toContain("paperclip-flag-secret"); }); }); + +describe("compactRunLogChunk with long base64 commands", () => { + it("redacts a tool call that writes base64 of source text without stalling", () => { + const typescriptSource = "export const limit = 65536;\nexport function keep(value: string) {\n return value.slice(0, limit);\n}\n"; + const blob = Buffer.from(typescriptSource.repeat(1_000)).toString("base64").slice(0, 48 * 1024); + expect(blob).toMatch(/^[A-Za-z0-9]+$/); + const command = `printf '%s' '${blob}' | base64 -d > src/keep.ts && echo '{"GITHUB_TOKEN":"tool-call-secret"}'`; + const chunk = `${JSON.stringify({ type: "assistant", message: { content: [{ type: "tool_use", input: { command } }] } })}\n`; + + const startedAt = performance.now(); + const compacted = compactRunLogChunk(chunk); + + expect(performance.now() - startedAt).toBeLessThan(1_000); + expect(compacted).toContain(blob); + expect(compacted).not.toContain("tool-call-secret"); + expect(compacted).toContain("***REDACTED***"); + }); +}); diff --git a/server/src/__tests__/redaction.test.ts b/server/src/__tests__/redaction.test.ts index 4ab09bbd80e..5d060d67d1a 100644 --- a/server/src/__tests__/redaction.test.ts +++ b/server/src/__tests__/redaction.test.ts @@ -899,3 +899,73 @@ second-line\" status=401`, }); }); }); + +describe("redaction of long letter/digit runs", () => { + const typescriptSource = [ + "export function compactRunLogChunk(chunk: string, maxChars = 65536) {", + " const normalized = redactSensitiveText(chunk);", + " if (normalized.length <= maxChars) return normalized;", + " return normalized.slice(0, maxChars);", + "}", + "", + ].join("\n"); + + function base64OfTypeScriptSource(length: number) { + const repeatedSource = typescriptSource.repeat(Math.ceil(length / typescriptSource.length)); + return Buffer.from(repeatedSource).toString("base64").slice(0, length); + } + + function redactWithinTimeBudget(input: string) { + const startedAt = performance.now(); + const output = redactSensitiveText(input); + expect(performance.now() - startedAt).toBeLessThan(1_000); + return output; + } + + it("redacts secret field names that carry a prefix in plain and escaped JSON", () => { + const cases = [ + [`{"myApiKey": "plain-prefixed-secret"}`, `{"myApiKey": "${REDACTED_EVENT_VALUE}"}`], + [`{"x_auth_token":"snake-prefixed-secret","status":"ok"}`, `{"x_auth_token":"${REDACTED_EVENT_VALUE}","status":"ok"}`], + [`GITHUB_TOKEN: "upper-prefixed-secret"`, `GITHUB_TOKEN: "${REDACTED_EVENT_VALUE}"`], + [`{'db_password': 'single-quoted-secret'}`, `{'db_password': '${REDACTED_EVENT_VALUE}'}`], + [`text"apiKey":"glued-quote-secret"`, `text"apiKey":"${REDACTED_EVENT_VALUE}"`], + [`aGVsbG8apiKey:"glued-run-secret"`, `aGVsbG8apiKey:"${REDACTED_EVENT_VALUE}"`], + [String.raw`{\"myApiKey\":\"escaped-prefixed-secret\"}`, String.raw`{\"myApiKey\":\"${REDACTED_EVENT_VALUE}\"}`], + [ + String.raw`{\"x_auth_token\":\"escaped-snake-secret\",\"status\":\"ok\"}`, + String.raw`{\"x_auth_token\":\"${REDACTED_EVENT_VALUE}\",\"status\":\"ok\"}`, + ], + [String.raw`aGVsbG8\"GITHUB_TOKEN\":\"escaped-glued-secret\"`, String.raw`aGVsbG8\"GITHUB_TOKEN\":\"${REDACTED_EVENT_VALUE}\"`], + ]; + + for (const [input, expected] of cases) { + expect(redactSensitiveText(input)).toBe(expected); + expect(redactSensitiveText(expected)).toBe(expected); + } + }); + + it("keeps base64 of source text intact and fast while redacting a secret glued to its end", () => { + const blob = base64OfTypeScriptSource(48 * 1024); + expect(blob).toMatch(/^[A-Za-z0-9]+$/); + + const shellCommand = `printf '%s' '${blob}' | base64 -d > src/compact.ts`; + expect(redactWithinTimeBudget(shellCommand)).toBe(shellCommand); + + const toolCallLine = JSON.stringify({ type: "tool_use", input: { command: shellCommand } }); + expect(redactWithinTimeBudget(toolCallLine)).toBe(toolCallLine); + const serializedToolCallLine = JSON.stringify(toolCallLine); + expect(redactWithinTimeBudget(serializedToolCallLine)).toBe(serializedToolCallLine); + + const gluedSecret = redactWithinTimeBudget(`${blob}apiKey:"run-glued-secret"`); + expect(gluedSecret).toBe(`${blob}apiKey:"${REDACTED_EVENT_VALUE}"`); + }); + + it("scans long dash-separated runs in linear time", () => { + for (const run of ["a-".repeat(60_000), "a--".repeat(40_000)]) { + expect(redactWithinTimeBudget(`${run} Authorization: Bearer dash-run-secret`)) + .toBe(`${run} Authorization: ${REDACTED_EVENT_VALUE}`); + expect(redactWithinTimeBudget(`${run}Authorization: Bearer glued-run-secret`)) + .toBe(`${run}Authorization: ${REDACTED_EVENT_VALUE}`); + } + }); +}); diff --git a/server/src/redaction.ts b/server/src/redaction.ts index 28693d13fa0..dd3edf95680 100644 --- a/server/src/redaction.ts +++ b/server/src/redaction.ts @@ -2,6 +2,7 @@ import { redactCommandText } from "@paperclipai/adapter-utils"; import { isPublicExecutorToolSelector, looksLikeCredentialJwt } from "@paperclipai/adapter-utils/command-redaction"; const SECRET_FIELD_NAME_PATTERN = String.raw`[A-Za-z0-9_-]*(?:api[-_]?key|access[-_]?token|auth(?:_?token)?|token|authorization(?:[-_]?code)?|bearer|secrets?|passwd|passwords?|credentials?|jwt|private[-_]?key|cookie|connectionstring|browser[-_]?code|login[-_]?url)(?:[-_]?(?:value|header|prod(?:uction)?|dev(?:elopment)?|test|staging|primary|secondary))*`; +const SECRET_FIELD_NAME_START = String.raw`(? Date: Mon, 2 Nov 2026 11:00:42 -0600 Subject: [PATCH 02/35] fix(runner): keep requests responsive during retired backup cleanup (#18619) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Remote native runs keep durable harness backups for continuation and recovery. > - Publishing a new backup retires earlier backup directories in the server process. > - The synchronous permission walk and recursive removal block unrelated requests. > - This pull request makes that cleanup asynchronous and keeps publication waiting for completion. > - The benefit is continued request progress while large retired backups are removed. ## Linked Issues or Issue Description **What happened?** Publishing a backup with many small files blocks the Node event loop during directory permission repair and removal. A local comparison with 20,000 retired files reproduced 1.37–1.95 second event-loop stalls. This is a pause candidate. A short hosted profile has not established it as the dominant source of sustained CPU use. **Expected behavior** Unrelated requests can progress during cleanup. Publication still waits for complete removal, preserves rollback on a failed lease stamp, and leaves live files and symlink targets unchanged. **Steps to reproduce** Run the benchmark in `doc/performance/native-backup-publication.md`. It compiles the actual publication service from a pinned baseline and the working tree, prepares immutable retired backups, and sends foreground HTTP probes from a separate process. **Paperclip version or commit** Measured baseline: `774ddc72713d6aa7cb516691c71f1fc23bcfa5ff`. The measured treatment source is committed in this branch. Current head: `0a69cf2d43c2f9fcf032d96c1dceb56163bf21ba`. The runtime patch is unchanged from the measured source; later commits only strengthen the test and benchmark evidence paths. **Deployment mode** Local compiled service on macOS arm64 with Node v24.22.2. No hosted rollout has occurred. Related work: #18380 changed retention scheduling. #13676 included the existing immutable-backup cleanup behavior. Neither change makes publication cleanup asynchronous. The GitHub duplicate search also found #11929, which concerns database dumps rather than native harness backups. No matching open event-loop cleanup issue was found. ## What Changed - Use asynchronous filesystem operations for retired-backup cleanup. Visit directories serially and skip unnecessary metadata calls for file entries. - Await cleanup before publication proceeds, during rollback, after successful rotation, and in executor finalization. - Exercise immutable current and previous backups in the publication test. Preserve the existing symlink-isolation and rollback tests. Add a real separate-process HTTP probe that must complete while publication is still active. - Add a repeatable service benchmark and document its fixture, metrics, and acceptance limits. ## Verification Draft status: focused correctness tests, the local comparison, and repository typecheck pass. The complete local test gate is incomplete. The build passed. Complete CI, hosted acceptance, and final-head review remain pending. - Final focused Vitest run: 2 files, 4 tests passed in 649 ms with the normal server configuration. The regression warms a separate Node client, waits for its HTTP request during the stamp callback, queues the response for the next event-loop turn, and then allows final retired-tree removal to start. The async treatment serves that response before publication completes. The original synchronous cleanup fails the same final regression in a separate source fixture. That expected control failure is retained. - Syntax and `git diff --check` passed. - `pnpm -r typecheck` passed on head `332c57b88c` in 152.18 seconds, including the native Runner build. Validation uses Node v24.22.2 and the repository-pinned pnpm v9.15.4 with isolated dependencies. - `pnpm test:run` was intentionally stopped after 1,390.24 seconds (exit 143). All six chat shards passed, covering all 1,817 parameterized cases across their exact partitions. The general server phase was advancing but incomplete. The repository timing manifest estimates another roughly two hours for the serial remainder, so validation moved to final-head parallel CI. This is not a full local test pass. The command began on `332c57b88c`; later changes only refine the regression and benchmark, and the final regression has its own focused pass. - Benchmark failure/metadata checks pass in a tiny isolated Git fixture: an injected worker exit 17 is reported in 0.44 seconds without hanging, and the result includes a staged source change when the unstaged diff is empty. - `pnpm build` passed in 41.80 seconds on `9f7e4e669b`. The final commit only releases a pending test probe on failure; runtime and build sources are unchanged. An injected client exit verifies that final test failure path exits promptly in 0.97 seconds instead of hanging. - Three alternating baseline/treatment pairs used fresh fixtures with 100 directories × 100 files in each of two retired backups. Each publication removed 20,000 files. A separate process sent HTTP probes at fixed 5 ms intervals, including overdue arrivals. - Every run verified the replacement manifest, complete cleanup, and unchanged symlink-target contents and mode. All 2,395 HTTP requests completed without errors. The largest measured generator delay was 8.8 ms. | Metric | Baseline median (range), ms | Treatment median (range), ms | | --- | ---: | ---: | | Foreground request p95 | 1,357.7 (1,287.4–2,014.0) | 1.8 (1.7–25.9) | | Maximum event-loop delay | 1,439.7 (1,371.5–1,954.5) | 19.9 (17.4–40.7) | | Publication completion | 1,429.1 (1,360.2–1,952.9) | 1,439.5 (897.9–1,649.6) | | Process CPU during publication | 1,194.1 (1,041.8–1,225.5) | 2,423.1 (1,995.7–2,514.7) | The treatment served 180–329 requests while publication was active. The baseline served zero during that interval. Process CPU includes foreground work that the baseline cannot serve until cleanup completes. The result supports responsiveness, with a CPU tradeoff. It does not establish lower total CPU or a hosted latency gain. Reproduce with Node v24.22.2: ```sh pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-harness-backup-cleanup.test.ts src/services/native-runtime/native-harness-backup-publication.test.ts node server/scripts/benchmark-native-backup-publication.mjs --baseline-ref 774ddc72713d6aa7cb516691c71f1fc23bcfa5ff --output /tmp/native-backup-publication.json ``` The benchmark records the baseline SHA, treatment HEAD and source diff. Raw local evidence remains outside the repository. A first small smoke run hit a sandbox restriction on loopback listeners; the authorized retry resolved it. The smoke also exposed load-generator interval drift. Fixed-clock arrivals replaced that generator before the reported measurements. Pending gates: complete current-head CI, final-head Greptile review, and the real run/checkpoint/recovery journey. The full local test run remains explicitly incomplete. The four concrete review findings were fixed. The prior head received 5/5; final-head review remains pending. Local evidence remains outside Git: `/private/tmp/backup-publication-measurement-20261102.json`, `/private/tmp/native-backup-focused-final.log`, `/private/tmp/native-backup-typecheck.log`, `/private/tmp/native-backup-test.log`, `/private/tmp/native-backup-test-interruption.json`, `/private/tmp/native-backup-build.log`, and `/private/tmp/native-backup-harness-check.json`. The harness-check receipt points to its isolated metadata and worker-failure logs. The final client-failure and baseline-control logs are `/private/tmp/backup-benchmark-check-gprcfiiz/failed-client-cleanup.log` and `/private/tmp/backup-benchmark-check-gprcfiiz/final-probe-baseline-control.log`. These local paths are retained for the author; the commands above let reviewers reproduce the synthetic proof. ## Risks - Asynchronous removal adds filesystem-pool work and used more process CPU in the fixture. Check resource pressure as well as responsiveness before rollout. - Filesystem operations can now interleave with unrelated work. The existing session owner still controls publication. Cleanup remains awaited; no deletion is detached and no scheduler is added. - Errors must still fail publication or restore the prior backup as before. Corruption checks and destructive lease authorization are unchanged. - This synthetic macOS fixture does not represent every Linux volume or backup shape. It does not qualify a deployed fix. - Rollout acceptance should confirm the serving SHA, checkpoint completion, continuation/recovery, error rate, and loop delay under matched backup load. Revert if cleanup or recovery fails. ## Model Used OpenAI Codex, GPT-6 family, with tool use and code execution. The exact served model ID and context-window size were not exposed to this agent. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/performance/native-backup-publication.md | 53 +++++ .../benchmark-native-backup-publication.mjs | 196 ++++++++++++++++++ .../native-harness-backup-cleanup.test.ts | 6 +- .../native-harness-backup-cleanup.ts | 21 +- .../native-harness-backup-publication.test.ts | 97 ++++++++- .../native-harness-backup-publication.ts | 6 +- .../native-runtime/native-session-executor.ts | 2 +- 7 files changed, 364 insertions(+), 17 deletions(-) create mode 100644 doc/performance/native-backup-publication.md create mode 100644 server/scripts/benchmark-native-backup-publication.mjs diff --git a/doc/performance/native-backup-publication.md b/doc/performance/native-backup-publication.md new file mode 100644 index 00000000000..b461ea49d76 --- /dev/null +++ b/doc/performance/native-backup-publication.md @@ -0,0 +1,53 @@ +# Native backup publication benchmark + +Retiring a native harness backup can involve many immutable directories and +small files. Cleanup repairs directory permissions before removing the retired +tree. Publication waits for cleanup, preserves the previous backup when lease +stamping fails, and emits its completion notification after cleanup succeeds. +Cleanup uses asynchronous filesystem operations so unrelated requests can +continue while it runs. Directory preparation is serial and does not follow +symlinks into live trees. + +Run the focused correctness tests from an installed checkout: + +```sh +pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-harness-backup-cleanup.test.ts src/services/native-runtime/native-harness-backup-publication.test.ts +``` + +Run the performance comparison from the repository root with the Node version +used by the target server. Use a quiet machine and an explicit baseline commit: + +```sh +node server/scripts/benchmark-native-backup-publication.mjs \ + --baseline-ref 774ddc72713d6aa7cb516691c71f1fc23bcfa5ff \ + --output /tmp/native-backup-publication.json +``` + +The script compiles the real publication service and its dependencies for both +the baseline commit and the working tree. Each run uses a separate server +process. Fixture preparation occurs before measurement. The default fixture has +100 immutable directories with 100 small files in each of the `current` and +`previous` backups: 20,000 files to remove. Both backups also have a symlink to a +read-only live directory outside the retired trees. The new pending backup has +a distinct manifest. Every completed run checks the new manifest, complete +removal of retired and pending trees, and unchanged live contents and mode. + +A separate process sends requests to a small loopback HTTP endpoint every five +milliseconds. Arrivals use a fixed clock and include overdue requests. The +report includes request completion and error counts, generator lateness, p95 +and maximum foreground latency, requests served during publication, publication +wall time, process CPU time, and maximum event-loop delay. Three pairs alternate +baseline and treatment order. Change the fixture with `--pairs`, `--directories`, +`--files-per-directory`, or `--interval-ms`. + +The result records Node/platform, baseline SHA, treatment HEAD and uncommitted +service diff. Compiled modules and source maps stay in the printed evidence +directory. Successful fixtures are removed; failed fixtures remain for diagnosis. + +This is a service benchmark. The HTTP endpoint is a responsiveness probe, not +the assembled Paperclip API. The lease-stamp callback resolves without database +work. It does not prove a hosted latency or total-CPU improvement. Process CPU +during publication includes foreground requests that the synchronous baseline +cannot serve until cleanup completes. Async cleanup can consume more CPU while +keeping the event loop responsive. Qualify the affected run/checkpoint/recovery +journey and compare matched hosted windows before claiming deployed acceptance. diff --git a/server/scripts/benchmark-native-backup-publication.mjs b/server/scripts/benchmark-native-backup-publication.mjs new file mode 100644 index 00000000000..78d607ed3a8 --- /dev/null +++ b/server/scripts/benchmark-native-backup-publication.mjs @@ -0,0 +1,196 @@ +import assert from "node:assert/strict"; +import { fork, execFileSync } from "node:child_process"; +import { createServer, get, Agent } from "node:http"; +import { promises as fs } from "node:fs"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { fileURLToPath, pathToFileURL } from "node:url"; +import { monitorEventLoopDelay, performance } from "node:perf_hooks"; +import { setTimeout as delay } from "node:timers/promises"; + +const filename = fileURLToPath(import.meta.url); +const checkout = path.resolve(path.dirname(filename), "../.."); +const arg = (name, fallback) => { + const position = process.argv.indexOf(`--${name}`); + return position < 0 ? fallback : process.argv[position + 1]; +}; +const percentile = (values, fraction) => { + const sorted = [...values].sort((a, b) => a - b); + return sorted[Math.min(sorted.length - 1, Math.ceil(sorted.length * fraction) - 1)] ?? null; +}; + +if (process.argv.includes("--worker")) { + const { publishNativeHarnessBackup } = await import(pathToFileURL(arg("module"))); + let publishing = false; + const server = createServer((_request, response) => { + response.setHeader("x-publication-active", publishing ? "1" : "0"); + response.end("ok"); + }); + await new Promise(resolve => server.listen(0, "127.0.0.1", resolve)); + process.send({ kind: "ready", port: server.address().port }); + process.on("message", async message => { + if (message.kind === "close") { + server.closeAllConnections(); + server.close(() => process.disconnect()); + return; + } + if (message.kind !== "publish") return; + const histogram = monitorEventLoopDelay({ resolution: 10 }); + histogram.enable(); + await delay(20); + const cpu = process.cpuUsage(); + const started = performance.now(); + publishing = true; + try { + await publishNativeHarnessBackup({ + db: {}, companyId: "benchmark", scopeDigest: "fixture", + backupRoot: message.backupRoot, pendingRoot: message.pendingRoot, + stamp: async () => {}, + }); + const durationMs = performance.now() - started; + const cpuUsed = process.cpuUsage(cpu); + publishing = false; + await delay(20); + histogram.disable(); + assert.equal(await fs.readFile(path.join(message.backupRoot, "current/manifest.json"), "utf8"), "replacement"); + assert.deepEqual((await fs.readdir(message.backupRoot)).sort(), ["current"]); + assert.equal(await fs.readFile(path.join(message.liveRoot, "untouched.txt"), "utf8"), "live"); + assert.equal((await fs.stat(message.liveRoot)).mode & 0o777, 0o555); + process.send({ kind: "result", durationMs, cpuMs: (cpuUsed.user + cpuUsed.system) / 1000, + eventLoopMaxMs: histogram.max / 1e6, workCompleted: true }); + } catch (error) { + publishing = false; + process.send({ kind: "error", message: error.stack }); + } + }); +} else { + const baselineRef = arg("baseline-ref"); + assert(baselineRef, "--baseline-ref is required"); + const pairs = Number(arg("pairs", "3")); + const directories = Number(arg("directories", "100")); + const filesPerDirectory = Number(arg("files-per-directory", "100")); + const intervalMs = Number(arg("interval-ms", "5")); + for (const value of [pairs, directories, filesPerDirectory, intervalMs]) assert(Number.isSafeInteger(value) && value > 0); + const evidence = await fs.mkdtemp(path.join(tmpdir(), "native-backup-publication-proof-")); + const { build } = await import("esbuild"); + const buildVariant = async variant => { + const outfile = path.join(evidence, `${variant}.mjs`); + await build({ absWorkingDir: checkout, entryPoints: ["server/src/services/native-runtime/native-harness-backup-publication.ts"], + outfile, bundle: true, platform: "node", format: "esm", target: "node24", sourcemap: true, + alias: { "@paperclipai/db/work-signals": path.join(checkout, "packages/db/src/work-signals.ts") }, + plugins: variant === "baseline" ? [{ name: "pinned-source", setup(builder) { + builder.onLoad({ filter: /\.ts$/ }, args => { + const relative = path.relative(checkout, args.path); + if (relative.startsWith("..") || relative.includes("node_modules")) return; + return { contents: execFileSync("git", ["show", `${baselineRef}:${relative}`], { cwd: checkout, encoding: "utf8" }), + loader: "ts", resolveDir: path.dirname(args.path) }; + }); + } }] : [], + }); + return outfile; + }; + const modules = { baseline: await buildVariant("baseline"), treatment: await buildVariant("treatment") }; + const makeFixture = async label => { + const root = await fs.mkdtemp(path.join(tmpdir(), `backup-publication-${label}-`)); + const backupRoot = path.join(root, "backups"); + const liveRoot = path.join(root, "live"); + const pendingRoot = path.join(backupRoot, "pending"); + await fs.mkdir(liveRoot); + await fs.writeFile(path.join(liveRoot, "untouched.txt"), "live"); + await fs.chmod(liveRoot, 0o555); + for (const version of ["current", "previous"]) { + const versionRoot = path.join(backupRoot, version); + for (let directory = 0; directory < directories; directory++) { + const target = path.join(versionRoot, `skill-${directory}`); + await fs.mkdir(target, { recursive: true }); + for (let file = 0; file < filesPerDirectory; file++) { + await fs.writeFile(path.join(target, `file-${file}.txt`), "retired instructions\n".repeat(4), { mode: 0o444 }); + } + await fs.chmod(target, 0o555); + } + await fs.symlink(liveRoot, path.join(versionRoot, "outside")); + await fs.chmod(versionRoot, 0o555); + } + await fs.mkdir(pendingRoot); + await fs.writeFile(path.join(pendingRoot, "manifest.json"), "replacement"); + return { root, backupRoot, liveRoot, pendingRoot }; + }; + const run = async variant => { + const fixture = await makeFixture(variant); + const worker = fork(filename, ["--worker", "--module", modules[variant]], { stdio: ["ignore", "inherit", "inherit", "ipc"] }); + const exited = new Promise(resolve => worker.once("exit", resolve)); + const nextMessage = kind => new Promise((resolve, reject) => { + const onMessage = message => { + if (message.kind !== kind && message.kind !== "error") return; + worker.off("message", onMessage); worker.off("exit", onExit); + if (message.kind === "error") reject(new Error(message.message)); else resolve(message); + }; + const onExit = code => reject(new Error(`worker exited before ${kind}: ${code}`)); + worker.on("message", onMessage); worker.once("exit", onExit); + }); + const { port } = await nextMessage("ready"); + const agent = new Agent({ keepAlive: true, maxSockets: 512 }); + const requests = []; + let sent = 0; + const epoch = performance.now(); + const dispatch = intended => { + const dispatched = performance.now(); + requests.push(new Promise(resolve => { + const request = get({ hostname: "127.0.0.1", port, agent }, response => { + response.resume(); + response.on("end", () => resolve({ intended, dispatchLateMs: dispatched - intended, + latencyMs: performance.now() - intended, active: response.headers["x-publication-active"] === "1", error: false })); + }); + request.setTimeout(10_000, () => request.destroy(new Error("request timeout"))); + request.on("error", () => resolve({ intended, dispatchLateMs: dispatched - intended, + latencyMs: performance.now() - intended, active: false, error: true })); + })); + }; + let ticker; + const pump = () => { + const now = performance.now(); + while (epoch + (sent + 1) * intervalMs <= now) dispatch(epoch + (++sent * intervalMs)); + ticker = setTimeout(pump, Math.max(0, epoch + (sent + 1) * intervalMs - performance.now())); + }; + ticker = setTimeout(pump, intervalMs); + try { + await delay(250); + const pending = nextMessage("result"); + const start = performance.now(); + worker.send({ kind: "publish", ...fixture }); + const publication = await pending; + const end = performance.now(); + await delay(250); + clearTimeout(ticker); + const completed = await Promise.all(requests); + const during = completed.filter(request => request.intended >= start && request.intended <= end); + assert(during.length > 0); + const result = { variant, ...publication, sent, completed: completed.length, + errors: completed.filter(request => request.error).length, duringRequests: during.length, + requestsServedDuringPublication: completed.filter(request => request.active).length, + foregroundP95Ms: percentile(during.map(request => request.latencyMs), 0.95), + foregroundMaxMs: Math.max(...during.map(request => request.latencyMs)), + generatorMaxLateMs: Math.max(...completed.map(request => request.dispatchLateMs)) }; + console.log(JSON.stringify(result)); + return result; + } finally { + clearTimeout(ticker); agent.destroy(); + if (worker.connected) worker.send({ kind: "close" }); + await exited; + await fs.chmod(fixture.liveRoot, 0o700); + if ((await fs.readdir(fixture.backupRoot)).join() === "current") await fs.rm(fixture.root, { recursive: true }); + } + }; + const measurements = []; + for (let pair = 0; pair < pairs; pair++) { + for (const variant of pair % 2 ? ["treatment", "baseline"] : ["baseline", "treatment"]) measurements.push(await run(variant)); + } + const report = { baselineRef: execFileSync("git", ["rev-parse", baselineRef], { cwd: checkout, encoding: "utf8" }).trim(), + treatmentHead: execFileSync("git", ["rev-parse", "HEAD"], { cwd: checkout, encoding: "utf8" }).trim(), + treatmentDiff: execFileSync("git", ["diff", "HEAD", "--", "server/src/services/native-runtime/native-harness-backup-cleanup.ts", "server/src/services/native-runtime/native-harness-backup-publication.ts"], { cwd: checkout, encoding: "utf8" }), + node: process.version, platform: `${process.platform}/${process.arch}`, pairs, directories, filesPerDirectory, + filesRemovedPerPublication: 2 * directories * filesPerDirectory, intervalMs, evidence, measurements }; + const output = arg("output", path.join(evidence, "result.json")); + await fs.writeFile(output, JSON.stringify(report, null, 2)); + console.log(`Evidence: ${output}`); +} diff --git a/server/src/services/native-runtime/native-harness-backup-cleanup.test.ts b/server/src/services/native-runtime/native-harness-backup-cleanup.test.ts index 1c969758ac7..d698855a97f 100644 --- a/server/src/services/native-runtime/native-harness-backup-cleanup.test.ts +++ b/server/src/services/native-runtime/native-harness-backup-cleanup.test.ts @@ -4,7 +4,7 @@ import { join } from "node:path"; import { expect, it } from "vitest"; import { removeNativeHarnessBackup } from "./native-harness-backup-cleanup.js"; -it("removes retired backups containing immutable skills without modifying live files or symlink targets", () => { +it("removes retired backups containing immutable skills without modifying live files or symlink targets", async () => { const base = mkdtempSync(join(tmpdir(), "paperclip-backup-cleanup-")); const retired = join(base, "previous"); const nested = join(retired, "claude", "skills", "first-task"); @@ -19,12 +19,12 @@ it("removes retired backups containing immutable skills without modifying live f chmodSync(retired, 0o555); chmodSync(live, 0o555); try { - removeNativeHarnessBackup(retired); + await removeNativeHarnessBackup(retired); expect(existsSync(retired)).toBe(false); expect(readFileSync(join(live, "SKILL.md"), "utf8")).toBe("live instructions"); expect(lstatSync(live).mode & 0o777).toBe(0o555); expect(lstatSync(join(live, "SKILL.md")).mode & 0o777).toBe(0o444); - expect(() => removeNativeHarnessBackup(retired)).not.toThrow(); + await expect(removeNativeHarnessBackup(retired)).resolves.toBeUndefined(); } finally { for (const directory of [retired, join(retired, "claude", "skills"), nested, live]) { if (existsSync(directory)) chmodSync(directory, 0o700); diff --git a/server/src/services/native-runtime/native-harness-backup-cleanup.ts b/server/src/services/native-runtime/native-harness-backup-cleanup.ts index d52e50ba5c4..7a803704ea7 100644 --- a/server/src/services/native-runtime/native-harness-backup-cleanup.ts +++ b/server/src/services/native-runtime/native-harness-backup-cleanup.ts @@ -1,18 +1,23 @@ -import { chmodSync, lstatSync, readdirSync, rmSync } from "node:fs"; +import { chmod, lstat, readdir, rm } from "node:fs/promises"; import { join } from "node:path"; /** Remove a retired controller-owned backup, never a live harness directory. */ -export function removeNativeHarnessBackup(root: string): void { +export async function removeNativeHarnessBackup(root: string): Promise { // Archives preserve immutable skill directory modes. Deleting their children // requires write permission on those directories, even with rm's force flag. // Only change directories in the backup being discarded. lstat deliberately // avoids following provider-created symlinks into live or unrelated trees. - const prepareDirectory = (path: string): void => { - const metadata = lstatSync(path, { throwIfNoEntry: false }); + const prepareDirectory = async (path: string): Promise => { + const metadata = await lstat(path).catch((error: NodeJS.ErrnoException) => { + if (error.code === "ENOENT") return undefined; + throw error; + }); if (!metadata?.isDirectory()) return; - chmodSync(path, (metadata.mode & 0o777) | 0o700); - for (const name of readdirSync(path)) prepareDirectory(join(path, name)); + await chmod(path, (metadata.mode & 0o777) | 0o700); + for (const entry of await readdir(path, { withFileTypes: true })) { + if (entry.isDirectory()) await prepareDirectory(join(path, entry.name)); + } }; - prepareDirectory(root); - rmSync(root, { recursive: true, force: true }); + await prepareDirectory(root); + await rm(root, { recursive: true, force: true }); } diff --git a/server/src/services/native-runtime/native-harness-backup-publication.test.ts b/server/src/services/native-runtime/native-harness-backup-publication.test.ts index dd9d940d9df..df1029f55f9 100644 --- a/server/src/services/native-runtime/native-harness-backup-publication.test.ts +++ b/server/src/services/native-runtime/native-harness-backup-publication.test.ts @@ -1,4 +1,6 @@ -import { promises as fs } from "node:fs"; +import { existsSync, promises as fs } from "node:fs"; +import { spawn } from "node:child_process"; +import { createServer, type ServerResponse } from "node:http"; import os from "node:os"; import path from "node:path"; import { afterEach, describe, expect, it } from "vitest"; @@ -32,7 +34,17 @@ describe("native harness backup publication", () => { }); it("publishes the completed replacement once and removes its predecessor", async () => { const input = await fixture(); const changes: unknown[] = []; - unsubscribe = onNativeHarnessBackupChange(input.db, change => changes.push(change)); + const previous = path.join(backupRoot, "previous"); + for (const root of [input.current, previous]) { + const skill = path.join(root, "skills", "immutable"); + await fs.mkdir(skill, { recursive: true }); + await fs.writeFile(path.join(skill, "SKILL.md"), "saved instructions", { mode: 0o444 }); + await fs.chmod(skill, 0o555); + } + unsubscribe = onNativeHarnessBackupChange(input.db, change => { + expect(existsSync(previous)).toBe(false); + changes.push(change); + }); await publishNativeHarnessBackup({ ...input, stamp: async current => { expect(changes).toEqual([]); expect(await fs.readFile(path.join(current, "manifest.json"), "utf8")).toBe("replacement"); @@ -41,4 +53,85 @@ describe("native harness backup publication", () => { expect(await fs.readdir(backupRoot)).toEqual(["current"]); expect(changes).toEqual([{ companyId: input.companyId, scopeDigest: input.scopeDigest }]); }); + it("serves an unrelated request while a retired backup is being removed", async () => { + const input = await fixture(); + for (let index = 0; index < 40; index++) { + const directory = path.join(input.current, `skill-${index}`); + await fs.mkdir(directory); + await Promise.all(Array.from({ length: 50 }, (_, file) => + fs.writeFile(path.join(directory, `file-${file}`), "retired instructions", { mode: 0o444 }))); + await fs.chmod(directory, 0o555); + } + let publishing = false; + let probeReceived: (response: ServerResponse | null) => void; + const probeReady = new Promise(resolve => { probeReceived = resolve; }); + const server = createServer((request, response) => { + if (request.url === "/probe") { probeReceived(response); return; } + response.end("ok"); + }); + await new Promise((resolve, reject) => { + server.once("error", reject); + server.listen(0, "127.0.0.1", resolve); + }); + const address = server.address(); + if (!address || typeof address === "string") throw new Error("HTTP probe did not start"); + const client = spawn(process.execPath, ["--input-type=module", "-e", ` + import { Agent, get } from "node:http"; + const agent = new Agent({ keepAlive: true }); + const request = path => new Promise((resolve, reject) => { + get({ hostname: "127.0.0.1", port: Number(process.argv[1]), path, agent }, response => { + response.resume(); + response.on("end", () => resolve(response.headers["x-publication-active"])); + }).on("error", reject); + }); + await request("/"); + process.send({ kind: "ready" }); + process.once("message", async () => { + const active = await request("/probe"); + process.send({ kind: "result", active }, () => { agent.destroy(); process.disconnect(); }); + }); + `, String(address.port)], { stdio: ["ignore", "ignore", "inherit", "ipc"] }); + const message = (kind: string) => new Promise<{ active?: string }>((resolve, reject) => { + const timer = setTimeout(() => { cleanup(); reject(new Error(`HTTP probe ${kind} timed out`)); }, 10_000); + const received = (value: { kind?: string; active?: string }) => { + if (value.kind === kind) { cleanup(); resolve(value); } + }; + const failed = () => { cleanup(); reject(new Error(`HTTP probe exited before ${kind}`)); }; + const cleanup = () => { + clearTimeout(timer); client.off("message", received); client.off("exit", failed); client.off("error", failed); + }; + client.on("message", received); client.once("exit", failed); client.once("error", failed); + }); + let publication: Promise | undefined; + try { + await message("ready"); + const response = message("result"); + publishing = true; + publication = publishNativeHarnessBackup({ + ...input, + stamp: async () => { + client.send({ kind: "probe" }); + const response = await probeReady; + if (!response) return; + setImmediate(() => { + response.setHeader("x-publication-active", publishing ? "1" : "0"); + response.end("ok"); + }); + }, + }).finally(() => { publishing = false; }); + const result = await response; + await publication; + expect(result.active).toBe("1"); + expect(await fs.readdir(backupRoot)).toEqual(["current"]); + expect(await fs.readFile(path.join(input.current, "manifest.json"), "utf8")).toBe("replacement"); + } finally { + probeReceived!(null); + try { await publication; } + finally { + client.kill(); + server.closeAllConnections(); + await new Promise(resolve => server.close(() => resolve())); + } + } + }, 15_000); }); diff --git a/server/src/services/native-runtime/native-harness-backup-publication.ts b/server/src/services/native-runtime/native-harness-backup-publication.ts index 820303584b0..714dafb5f20 100644 --- a/server/src/services/native-runtime/native-harness-backup-publication.ts +++ b/server/src/services/native-runtime/native-harness-backup-publication.ts @@ -9,7 +9,7 @@ export async function publishNativeHarnessBackup(input: { }) { const currentRoot = resolve(input.backupRoot, "current"); const previousRoot = resolve(input.backupRoot, "previous"); - removeNativeHarnessBackup(previousRoot); + await removeNativeHarnessBackup(previousRoot); let movedCurrent = false; if (existsSync(currentRoot)) { renameSync(currentRoot, previousRoot); @@ -19,10 +19,10 @@ export async function publishNativeHarnessBackup(input: { renameSync(input.pendingRoot, currentRoot); await input.stamp?.(currentRoot); } catch (error) { - if (existsSync(currentRoot)) removeNativeHarnessBackup(currentRoot); + if (existsSync(currentRoot)) await removeNativeHarnessBackup(currentRoot); if (movedCurrent && existsSync(previousRoot) && !existsSync(currentRoot)) renameSync(previousRoot, currentRoot); throw error; } - removeNativeHarnessBackup(previousRoot); + await removeNativeHarnessBackup(previousRoot); noteNativeHarnessBackupChange(input.db, { companyId: input.companyId, scopeDigest: input.scopeDigest }); } diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index f3565441c39..93992b4870d 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -13734,7 +13734,7 @@ async function createRunnerdBackendWithinSessionClaim( } : undefined, }); } finally { - removeNativeHarnessBackup(pendingRoot); + await removeNativeHarnessBackup(pendingRoot); } }, { From 97c4dc969e13364464765ffbde74955005bf44d3 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Mon, 2 Nov 2026 11:07:32 -0600 Subject: [PATCH 03/35] test(evals): retire duplicate chat TestOps plans (#18625) ## Thinking Path > - Paperclip manages AI agents and preserves their work history. > - Product E2E cases use TestOps plans to select real AWS evaluations. > - The full chat catalog now includes the former smoke and completion-pilot cases. > - The two old bootstrap workflows can recreate their smaller plans after deletion. > - This pull request removes those workflows and stops their plan-creation functions before API access. > - The full-catalog workflow keeps its shared clients, evidence rules, case identities, and measured results. ## Linked Issues or Issue Description Refs #18542, #18516. The user requested removal of the duplicate smoke and completion-pilot plans and investigation of their recreation paths. Both old workflows accepted manual dispatch only. Their bootstrap operations created plans by name when the plans were absent. They had no schedules in their workflow definitions. The old workflow records were disabled before plan removal. An existing pilot execution was allowed to finish successfully. This change does not cancel executions or delete cases, results, launches, or private evidence. ## What Changed - Remove `chat-feature-testops.yml` and `chat-completion-testops.yml`. - Make the former singleton plan creator reject every call before API reads or mutation markers. - Reject pilot bootstrap before API access. Keep the full-catalog branch and its integration validation. - Remove tests that require the retired workflow files. Keep shared helper and full-catalog controls. Run the trusted-checkout boundary test against the active catalog workflow. - Replace old setup instructions with retirement guidance and the active full-catalog route. The master sync preserves the incoming typed launch-search fix from #18620, including both regression tests. The only conflict was the retired pilot documentation. Its search-format note remains as a retained-helper contract. ## Verification - Two new retirement controls fail against the parent code because it reaches an API or mutation marker. Both pass after the change. - Six owning Node suites after the master sync: 146 passed (144 before the sync). Shared contract: 22 passed. Catalog and retained export controls: 12 passed. Focused workflow and release controls: 7 passed. - Two modified API modules pass Node syntax checks. `git diff --check` passes. - The pure workflow-security file has the same four failures and 13 passes on both the untouched parent and this branch. Two assertions expect eight trusted bootstrap steps where master now has nine. Two lists omit the already-shipped catalog workflow. This change does not repair or relax those unrelated assertions. - A broader release-test invocation had 73 passes and 20 sandbox failures from blocked mock HTTP listeners or npm permissions. Its focused affected workflow control passes. A standard Vitest setup attempt stopped on a missing `acpx/runtime` bundle dependency before tests. No local app, database, provider, or native runtime was qualified. - Current-head PR CI passes: 48 successful jobs and one skipped job, including both required checks. Greptile is 5/5. Independent source review passes. No new model run is needed for this retirement. ## Risks - Existing integrations that point at either retired workflow will stop dispatching. This is the requested behavior; use the full catalog instead. - Historical evidence readers still use shared helpers. Those helpers remain, including the transport and official-context parser used by the active catalog. - Workflow removal is a repository change. TestOps plan deletion and disabling obsolete executors are separate service actions. Neither action should delete shared cases or measurements. ## Model Used OpenAI `gpt-6.1-sol` through Codex, with source editing, shell tools, and independent source review. The context-window size is not asserted. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip --- .github/workflows/chat-completion-testops.yml | 504 ------------------ .github/workflows/chat-feature-testops.yml | 451 ---------------- doc/evals.md | 5 +- .../release-verify-workflow.test.mjs | 37 +- tests/runner-e2e/CHAT-FEATURES.md | 157 +----- .../chat-completion-testops-api.mjs | 26 +- .../chat-completion-testops-api.test.mjs | 32 +- .../chat-completion-testops-contract.test.mjs | 31 +- tests/runner-e2e/chat-feature-testops-api.mjs | 32 +- .../chat-feature-testops-api.test.mjs | 66 +-- .../chat-feature-testops-cli.test.mjs | 9 +- .../runner-e2e/testops-chat-reporter.test.mjs | 12 - .../runner-e2e/testops-chat-workflow.test.ts | 154 ------ tests/runner-e2e/workflow-security.test.ts | 21 +- 14 files changed, 46 insertions(+), 1491 deletions(-) delete mode 100644 .github/workflows/chat-completion-testops.yml delete mode 100644 .github/workflows/chat-feature-testops.yml delete mode 100644 tests/runner-e2e/testops-chat-workflow.test.ts diff --git a/.github/workflows/chat-completion-testops.yml b/.github/workflows/chat-completion-testops.yml deleted file mode 100644 index 7016f76dfec..00000000000 --- a/.github/workflows/chat-completion-testops.yml +++ /dev/null @@ -1,504 +0,0 @@ -name: Chat completion TestOps -run-name: Chat completion TestOps / ${{ inputs.operation || 'evaluate' }} / ${{ inputs.ALLURE_JOB_RUN_ID || github.run_id }} - -on: - workflow_dispatch: - inputs: - operation: - type: choice - options: [evaluate, bootstrap, launch-plan] - default: evaluate - description: Run the exact TestOps selection, register its plan, or launch it once - ALLURE_JOB_RUN_ID: - type: string - required: false - description: TestOps job-run identity; source, model and clocks are fixed - ALLURE_USERNAME: - type: string - required: false - description: Reserved service metadata; never grants authority - -permissions: - contents: read - actions: read - -concurrency: - group: chat-completion-testops - cancel-in-progress: false - -env: - NODE_OPTIONS: '' - NODE_PATH: '' - AWS_EC2_METADATA_DISABLED: 'true' - -jobs: - authorize: - runs-on: ubuntu-latest - timeout-minutes: 5 - outputs: - operation: ${{ steps.guard.outputs.operation }} - target_sha: ${{ steps.guard.outputs.target_sha }} - runner: runs-on/fleet=paperclip-public-pr-x64/env=public-ci - steps: - - &checkout - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 - with: - ref: ${{ github.sha }} - persist-credentials: false - - &node - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7 - with: - node-version: 24 - - &guard - name: Verify actual main-only dispatch and both actors - id: guard - env: - GH_TOKEN: ${{ github.token }} - TESTOPS_ALLOWED_ACTOR_IDS: ${{ vars.RUNNER_E2E_ALLOWED_ACTOR_IDS }} - run: node tests/runner-e2e/chat-completion-testops-cli.mjs guard - - select: - name: Register the dedicated plan or validate its real TestOps selection - needs: authorize - runs-on: ubuntu-latest - timeout-minutes: 15 - environment: runner-e2e-testops-reporting - outputs: - artifact_id: ${{ steps.selection_upload.outputs.artifact-id }} - selection_sha256: ${{ steps.selection.outputs.selection_sha256 }} - execution_ids: ${{ steps.selection.outputs.execution_ids }} - matrix: ${{ steps.selection.outputs.matrix }} - policy: ${{ steps.selection.outputs.policy }} - steps: - - *checkout - - *node - - *guard - - &allure - if: needs.authorize.outputs.operation != 'launch-plan' - uses: allure-framework/setup-allurectl@2d89ee0b7ba2f59dbe9849b6df5763cab78e14db # v1 - with: - allurectl-version: 2.23.1 - allure-endpoint: https://paperclip.testops.cloud - allure-project-id: '1' - allure-token: ${{ secrets.ALLURE_TOKEN }} - - name: Register three exact cases with no synthetic test result - if: needs.authorize.outputs.operation == 'bootstrap' - env: - GH_TOKEN: ${{ github.token }} - TESTOPS_ALLOWED_ACTOR_IDS: ${{ vars.RUNNER_E2E_ALLOWED_ACTOR_IDS }} - ALLURE_TOKEN: ${{ secrets.ALLURE_TOKEN }} - ALLURE_ENDPOINT: https://paperclip.testops.cloud - ALLURE_PROJECT_ID: '1' - ALLURE_RESULTS: ${{ runner.temp }}/chat-completion-bootstrap-results - CHAT_TESTOPS_INTEGRATION_ID: ${{ vars.CHAT_TESTOPS_INTEGRATION_ID }} - run: | - mkdir -m 700 "$ALLURE_RESULTS" - allurectl watch -- node tests/runner-e2e/chat-completion-testops-cli.mjs bootstrap - node tests/runner-e2e/chat-completion-testops-cli.mjs bootstrap-context - - name: Validate case, plan, job, launch and same-run source before provider access - id: selection - if: needs.authorize.outputs.operation == 'evaluate' - env: - GH_TOKEN: ${{ github.token }} - TESTOPS_ALLOWED_ACTOR_IDS: ${{ vars.RUNNER_E2E_ALLOWED_ACTOR_IDS }} - ALLURE_TOKEN: ${{ secrets.ALLURE_TOKEN }} - ALLURE_ENDPOINT: https://paperclip.testops.cloud - ALLURE_PROJECT_ID: '1' - ALLURE_JOB_RUN_ID: ${{ inputs.ALLURE_JOB_RUN_ID }} - ALLURE_RESULTS: ${{ runner.temp }}/chat-completion-selection-results - CHAT_COMPLETION_TESTOPS_BINDING: ${{ vars.CHAT_COMPLETION_TESTOPS_BINDING }} - run: | - mkdir -m 700 "$ALLURE_RESULTS" - allurectl watch -- node tests/runner-e2e/chat-completion-testops-cli.mjs selection - - name: Request the one reviewed plan launch and exit - if: needs.authorize.outputs.operation == 'launch-plan' - env: - GH_TOKEN: ${{ github.token }} - TESTOPS_ALLOWED_ACTOR_IDS: ${{ vars.RUNNER_E2E_ALLOWED_ACTOR_IDS }} - ALLURE_TOKEN: ${{ secrets.ALLURE_TOKEN }} - CHAT_COMPLETION_TESTOPS_BINDING: ${{ vars.CHAT_COMPLETION_TESTOPS_BINDING }} - CHAT_COMPLETION_TESTOPS_LAUNCH_RUN_NUMBER: ${{ vars.CHAT_COMPLETION_TESTOPS_LAUNCH_RUN_NUMBER }} - CHAT_COMPLETION_TESTOPS_LAUNCH_NONCE: ${{ vars.CHAT_COMPLETION_TESTOPS_LAUNCH_NONCE }} - run: node tests/runner-e2e/chat-completion-testops-cli.mjs launch-plan - - name: Retain token-free registration and launch receipts - if: always() && needs.authorize.outputs.operation != 'evaluate' - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 - with: - name: chat-completion-testops-setup-${{ github.run_id }}-1 - path: | - ${{ runner.temp }}/chat-completion-testops-bootstrap/ - ${{ runner.temp }}/chat-completion-testops-launch-plan/ - retention-days: 30 - if-no-files-found: error - - name: Retain the exact authenticated selection - id: selection_upload - if: steps.selection.outcome == 'success' - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 - with: - name: chat-completion-testops-selection-${{ github.run_id }}-1 - path: testops-pilot-selection.json - retention-days: 30 - if-no-files-found: error - - build: - name: Build the immutable catalog source without service or model credentials - needs: [authorize, select] - if: needs.authorize.outputs.operation == 'evaluate' - runs-on: ${{ needs.authorize.outputs.runner }} - timeout-minutes: 45 - outputs: - artifact_id: ${{ steps.bundle.outputs.artifact-id }} - steps: - - *checkout - - *node - - *guard - - &pnpm - uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6 - env: - NPM_CONFIG_AUDIT: 'false' - NPM_CONFIG_FUND: 'false' - NPM_CONFIG_UPDATE_NOTIFIER: 'false' - with: - version: 9.15.4 - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 - with: - ref: ${{ needs.authorize.outputs.target_sha }} - persist-credentials: false - - name: Reject target-owned trusted helper paths - id: target_boundary - run: test -z "$(git ls-tree --name-only HEAD -- .testops-trusted)" - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 - with: - ref: ${{ github.sha }} - path: .testops-trusted - persist-credentials: false - - name: Resolve the fixed candidate graph with lifecycle scripts disabled - run: | - pnpm install --lockfile-only --ignore-scripts - pnpm install --frozen-lockfile --ignore-scripts - - name: Build existing TypeScript and native Runner outputs - run: | - pnpm --filter @paperclipai/paperclip-runner build:typescript - - name: Exercise the real workflow and pinned source host contract before model access - run: node --import ./cli/node_modules/tsx/dist/loader.mjs .testops-trusted/tests/runner-e2e/chat-completion-testops-host-control.mjs - - name: Build native Runner outputs - run: pnpm --filter @paperclipai/paperclip-runner build:runner-binaries - - name: Package the resolved lock and exact source outputs - env: - TARGET_SHA: ${{ needs.authorize.outputs.target_sha }} - run: | - set -euo pipefail - test "$(git rev-parse HEAD)" = "$TARGET_SHA" - tar -czf chat-completion-build.tar.gz pnpm-lock.yaml packages/paperclip-eval-kernel/dist packages/paperclip-runner/dist packages/paperclip-runner/runner/target/debug/paperclip-runnerd packages/paperclip-runner/runner/target/debug/conformance-tracer packages/paperclip-runner/runner/target/debug/fake-harness packages/paperclip-runner/runner/target/debug/fake-codex-app-server packages/paperclip-runner/runner/target/debug/fake-acpx-sidecar - sha256sum chat-completion-build.tar.gz > chat-completion-build.tar.gz.sha256 - - name: Retain credential-free immutable outputs - id: bundle - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 - with: - name: chat-completion-build-${{ github.run_id }}-1 - path: | - chat-completion-build.tar.gz - chat-completion-build.tar.gz.sha256 - retention-days: 1 - if-no-files-found: error - - chat: - name: ${{ matrix.executionId }} - needs: [authorize, select, build] - if: needs.authorize.outputs.operation == 'evaluate' - strategy: - fail-fast: false - max-parallel: 1 - matrix: ${{ fromJSON(needs.select.outputs.matrix) }} - runs-on: ${{ needs.authorize.outputs.runner }} - timeout-minutes: 32 - environment: runner-e2e-private-paid - permissions: - contents: read - actions: read - id-token: write - env: &retention - TARGET_SHA: ${{ needs.authorize.outputs.target_sha }} - MANUAL_PRIVATE_IDS: ${{ needs.select.outputs.execution_ids }} - MANUAL_PRIVATE_APPROVALS_JSON: ${{ needs.select.outputs.policy }} - RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET }} - steps: - - *checkout - - *node - - *guard - - *pnpm - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 - with: - ref: ${{ needs.authorize.outputs.target_sha }} - persist-credentials: false - - name: Reject target-owned trusted helper paths - id: target_boundary - run: test -z "$(git ls-tree --name-only HEAD -- .testops-trusted)" - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 - with: - ref: ${{ github.sha }} - path: .testops-trusted - persist-credentials: false - - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 - with: - artifact-ids: ${{ needs.build.outputs.artifact_id }} - path: chat-completion-build - - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 - with: - artifact-ids: ${{ needs.select.outputs.artifact_id }} - path: chat-completion-testops-selection - - name: Verify source-bound build and authenticated selection before credentials - env: - GH_TOKEN: ${{ github.token }} - TESTOPS_ALLOWED_ACTOR_IDS: ${{ vars.RUNNER_E2E_ALLOWED_ACTOR_IDS }} - EXPECTED_SELECTION_SHA256: ${{ needs.select.outputs.selection_sha256 }} - ALLURE_JOB_RUN_ID: ${{ inputs.ALLURE_JOB_RUN_ID }} - run: | - set -euo pipefail - test "$(git -C .testops-trusted rev-parse HEAD)" = "$GITHUB_SHA" - node .testops-trusted/tests/runner-e2e/chat-completion-testops-cli.mjs verify-selection chat-completion-testops-selection/testops-pilot-selection.json - (cd chat-completion-build && sha256sum -c chat-completion-build.tar.gz.sha256) - tar -xzf chat-completion-build/chat-completion-build.tar.gz -C "$GITHUB_WORKSPACE" - pnpm install --frozen-lockfile --ignore-scripts - (cd .testops-trusted && node tests/runner-e2e/prepare-report-lock.mjs && pnpm install --frozen-lockfile --ignore-scripts --store-dir "$RUNNER_TEMP/chat-completion-retention-store") - - name: Qualify the pinned provider runtime and preinstalled browser - run: | - node packages/paperclip-runner/scripts/materialize-opencode-binary.mjs - node -e 'const fs=require("node:fs");fs.chmodSync(process.execPath,fs.statSync(process.execPath).mode&~0o022);if(fs.statSync(process.execPath).mode&0o022)throw Error("Writable provider Node")' - google-chrome --version - pnpm exec playwright install ffmpeg - - name: Bind exact private evidence under the authenticated TestOps authority - id: private - env: - TARGET_ROOT: ${{ github.workspace }} - SELECTED_EXECUTION_ID: ${{ matrix.executionId }} - run: node .testops-trusted/cli/node_modules/tsx/dist/cli.mjs .testops-trusted/tests/runner-e2e/manual-private-transfer.ts prepare - - name: Obtain only this cell's exact-key storage session - id: cell_credentials - uses: aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd # v6 - with: - role-to-assume: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_ROLE_ARN }} - aws-region: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_REGION }} - inline-session-policy: ${{ steps.private.outputs.session_policy }} - output-env-credentials: false - output-credentials: true - disable-retry: true - action-timeout-s: 60 - - name: Verify private custody before delivering the model key - env: &cell_session - SELECTED_EXECUTION_ID: ${{ matrix.executionId }} - PRIVATE_EVIDENCE_ROOT: ${{ steps.private.outputs.root }} - AWS_ACCESS_KEY_ID: ${{ steps.cell_credentials.outputs.aws-access-key-id }} - AWS_SECRET_ACCESS_KEY: ${{ steps.cell_credentials.outputs.aws-secret-access-key }} - AWS_SESSION_TOKEN: ${{ steps.cell_credentials.outputs.aws-session-token }} - AWS_DEFAULT_REGION: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_REGION }} - run: node .testops-trusted/cli/node_modules/tsx/dist/cli.mjs .testops-trusted/tests/runner-e2e/manual-private-transfer.ts verify - - name: Execute only the selected original case with zero retries - env: - OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }} - PAPERCLIP_CHAT_COMPLETION_TESTOPS: v1 - ALLURE_JOB_RUN_ID: ${{ inputs.ALLURE_JOB_RUN_ID }} - PAPERCLIP_RUNNER_E2E_PRIVATE_PREFLIGHT: ${{ steps.private.outputs.root }}/preflight.json - PRIVATE_EVIDENCE_ROOT: ${{ steps.private.outputs.root }} - PAPERCLIP_E2E_CAMPAIGN_ID: gha-${{ github.run_id }}-1-${{ matrix.executionId }} - PAPERCLIP_E2E_ACCOUNTING_CAMPAIGN_ID: gha-${{ github.run_id }}-1 - PAPERCLIP_RUNNER_E2E_SOURCE_SHA: ${{ needs.authorize.outputs.target_sha }} - PAPERCLIP_RUNNER_E2E_SOURCE_REF: refs/heads/codex/chat-catalog-master-integration - PAPERCLIP_PLAYWRIGHT_CHANNEL: chrome - run: pnpm test:e2e:runner -- --id "${{ matrix.executionId }}" --max-parallel 1 --max-automatic-retries 0 > "$PRIVATE_EVIDENCE_ROOT/launcher-output.log" 2>&1 - - name: Retain original evidence after success or failure - if: always() && steps.cell_credentials.outcome == 'success' - env: *cell_session - run: node .testops-trusted/cli/node_modules/tsx/dist/cli.mjs .testops-trusted/tests/runner-e2e/manual-private-transfer.ts upload-cell - - report: - name: Retain private originals and build the exact safe TestOps projection - needs: [authorize, select, build, chat] - if: always() && !cancelled() && needs.authorize.outputs.operation == 'evaluate' && needs.select.result == 'success' && needs.build.result == 'success' && (needs.chat.result == 'success' || needs.chat.result == 'failure') - runs-on: ubuntu-latest - timeout-minutes: 20 - environment: runner-e2e-private-history - permissions: - contents: read - actions: read - id-token: write - env: *retention - steps: - - *checkout - - *node - - *guard - - *pnpm - - name: Install trusted reporting dependencies with lifecycle scripts disabled - run: | - node tests/runner-e2e/prepare-report-lock.mjs - pnpm install --frozen-lockfile --ignore-scripts --store-dir "$RUNNER_TEMP/chat-completion-report-store" - - name: Prepare unchanged exact-key report and cell policies - id: session - run: node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/manual-private-transfer.ts report-session - - name: Obtain private report writer - id: report_credentials - uses: aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd # v6 - with: - role-to-assume: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_ROLE_ARN }} - aws-region: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_REGION }} - inline-session-policy: ${{ steps.session.outputs.session_policy }} - output-env-credentials: false - output-credentials: true - disable-retry: true - action-timeout-s: 60 - - name: Obtain exact first-cell reader - id: reader_0 - uses: aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd # v6 - with: - role-to-assume: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_ROLE_ARN }} - aws-region: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_REGION }} - inline-session-policy: ${{ steps.session.outputs.read_policy_0 }} - output-env-credentials: false - output-credentials: true - disable-retry: true - action-timeout-s: 60 - - name: Download the first exact original - env: - PRIVATE_REPORT_ROOT: ${{ github.workspace }} - SELECTED_CELL_INDEX: '0' - AWS_ACCESS_KEY_ID: ${{ steps.reader_0.outputs.aws-access-key-id }} - AWS_SECRET_ACCESS_KEY: ${{ steps.reader_0.outputs.aws-secret-access-key }} - AWS_SESSION_TOKEN: ${{ steps.reader_0.outputs.aws-session-token }} - AWS_DEFAULT_REGION: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_REGION }} - run: node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/manual-private-transfer.ts download-cell - - name: Obtain exact second-cell reader - id: reader_1 - if: fromJSON(steps.session.outputs.cell_count) > 1 - uses: aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd # v6 - with: - role-to-assume: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_ROLE_ARN }} - aws-region: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_REGION }} - inline-session-policy: ${{ steps.session.outputs.read_policy_1 }} - output-env-credentials: false - output-credentials: true - disable-retry: true - action-timeout-s: 60 - - name: Download the second exact original - if: steps.reader_1.outcome == 'success' - env: - PRIVATE_REPORT_ROOT: ${{ github.workspace }} - SELECTED_CELL_INDEX: '1' - AWS_ACCESS_KEY_ID: ${{ steps.reader_1.outputs.aws-access-key-id }} - AWS_SECRET_ACCESS_KEY: ${{ steps.reader_1.outputs.aws-secret-access-key }} - AWS_SESSION_TOKEN: ${{ steps.reader_1.outputs.aws-session-token }} - AWS_DEFAULT_REGION: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_REGION }} - run: node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/manual-private-transfer.ts download-cell - - name: Obtain exact third-cell reader - id: reader_2 - if: fromJSON(steps.session.outputs.cell_count) > 2 - uses: aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd # v6 - with: - role-to-assume: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_ROLE_ARN }} - aws-region: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_REGION }} - inline-session-policy: ${{ steps.session.outputs.read_policy_2 }} - output-env-credentials: false - output-credentials: true - disable-retry: true - action-timeout-s: 60 - - name: Download the third exact original - if: steps.reader_2.outcome == 'success' - env: - PRIVATE_REPORT_ROOT: ${{ github.workspace }} - SELECTED_CELL_INDEX: '2' - AWS_ACCESS_KEY_ID: ${{ steps.reader_2.outputs.aws-access-key-id }} - AWS_SECRET_ACCESS_KEY: ${{ steps.reader_2.outputs.aws-secret-access-key }} - AWS_SESSION_TOKEN: ${{ steps.reader_2.outputs.aws-session-token }} - AWS_DEFAULT_REGION: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_REGION }} - run: node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/manual-private-transfer.ts download-cell - - name: Verify every selected original and its completeness receipt - env: - PRIVATE_REPORT_ROOT: ${{ github.workspace }} - run: node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/manual-private-transfer.ts verify-downloads - - name: Build the existing report while preserving failed source grades - id: normalize - continue-on-error: true - env: - PAPERCLIP_RUNNER_E2E_REPORT_ROOT: ${{ github.workspace }}/selected-runner-e2e - PAPERCLIP_RUNNER_E2E_REPORT_OUT: ${{ github.workspace }}/runner-e2e-merged-report/normalized - PAPERCLIP_RUNNER_E2E_EXPECTED_IDS: ${{ needs.select.outputs.execution_ids }} - PAPERCLIP_E2E_CAMPAIGN_ID: gha-${{ github.run_id }}-1 - PAPERCLIP_RUNNER_E2E_SOURCE_SHA: ${{ needs.authorize.outputs.target_sha }} - PAPERCLIP_RUNNER_E2E_SOURCE_REF: refs/heads/codex/chat-catalog-master-integration - run: | - mkdir -p runner-e2e-merged-report - pnpm test:e2e:runner:report > runner-e2e-merged-report/report.log 2>&1 - - name: Retain the complete merged report privately - id: private_report - if: always() && steps.report_credentials.outcome == 'success' - env: - PRIVATE_REPORT_ROOT: ${{ github.workspace }} - PRIVATE_REPORT_SOURCE: ${{ github.workspace }}/runner-e2e-merged-report - AWS_ACCESS_KEY_ID: ${{ steps.report_credentials.outputs.aws-access-key-id }} - AWS_SECRET_ACCESS_KEY: ${{ steps.report_credentials.outputs.aws-secret-access-key }} - AWS_SESSION_TOKEN: ${{ steps.report_credentials.outputs.aws-session-token }} - AWS_DEFAULT_REGION: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_REGION }} - run: node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/manual-private-transfer.ts upload-report - - name: Export only exact reviewed synthetic fixture results - id: export - if: always() && steps.private_report.outcome == 'success' - run: node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/chat-completion-testops-export.ts - - name: Require a complete safe projection even after a failed normalization - if: always() - env: - EXPORT_OUTCOME: ${{ steps.export.outcome }} - run: test "$EXPORT_OUTCOME" = success - - name: Retain the metadata-only TestOps projection - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 - with: - name: chat-completion-testops-projection-${{ github.run_id }}-1 - path: runner-e2e-testops/ - retention-days: 30 - if-no-files-found: error - - upload: - name: Record and independently verify each TestOps result - needs: [authorize, select, chat, report] - if: always() && !cancelled() && needs.authorize.outputs.operation == 'evaluate' && needs.report.result == 'success' - runs-on: ubuntu-latest - timeout-minutes: 10 - environment: runner-e2e-testops-reporting - steps: - - *checkout - - *node - - *guard - - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 - with: - name: chat-completion-testops-projection-${{ github.run_id }}-1 - path: runner-e2e-testops - - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 - with: - artifact-ids: ${{ needs.select.outputs.artifact_id }} - path: chat-completion-testops-selection - - *allure - - name: Upload original outcomes, checks and private-evidence receipts - id: allure_upload - env: &upload_env - GH_TOKEN: ${{ github.token }} - TESTOPS_ALLOWED_ACTOR_IDS: ${{ vars.RUNNER_E2E_ALLOWED_ACTOR_IDS }} - ALLURE_TOKEN: ${{ secrets.ALLURE_TOKEN }} - ALLURE_ENDPOINT: https://paperclip.testops.cloud - ALLURE_PROJECT_ID: '1' - ALLURE_JOB_RUN_ID: ${{ inputs.ALLURE_JOB_RUN_ID }} - EXPECTED_SELECTION_SHA256: ${{ needs.select.outputs.selection_sha256 }} - ALLURE_RESULTS: ${{ runner.temp }}/chat-completion-allure-results - run: allurectl watch -- node tests/runner-e2e/chat-completion-testops-cli.mjs upload runner-e2e-testops - - name: Verify actual selected results and complete result-owned attachments - id: verification - if: always() && !cancelled() && (steps.allure_upload.outcome == 'success' || steps.allure_upload.outcome == 'failure') - env: *upload_env - run: node tests/runner-e2e/chat-completion-testops-cli.mjs verify-upload runner-e2e-testops - - name: Retain closed TestOps verification receipts - if: always() && !cancelled() && (steps.verification.outcome == 'success' || steps.verification.outcome == 'failure') - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 - with: - name: chat-completion-testops-verification-${{ github.run_id }}-1 - path: | - ${{ runner.temp }}/chat-completion-testops-verify-upload/ - ${{ runner.temp }}/chat-completion-testops-upload/exported.json - retention-days: 30 - if-no-files-found: error diff --git a/.github/workflows/chat-feature-testops.yml b/.github/workflows/chat-feature-testops.yml deleted file mode 100644 index 826a340ae91..00000000000 --- a/.github/workflows/chat-feature-testops.yml +++ /dev/null @@ -1,451 +0,0 @@ -name: Chat feature TestOps -run-name: Chat feature TestOps / ${{ inputs.operation || 'evaluate' }} / ${{ inputs.ALLURE_JOB_RUN_ID || github.run_id }} - -on: - workflow_dispatch: - inputs: - operation: - type: choice - options: [evaluate, bootstrap, launch-plan] - default: evaluate - description: Fixed evaluation, zero-result registration, or one TestOps plan launch - ALLURE_JOB_RUN_ID: - type: string - required: false - description: TestOps supplies its job-run identity; no test/source selector is accepted - ALLURE_USERNAME: - type: string - required: false - description: Reserved TestOps service metadata; never used as authorization - -permissions: - contents: read - -concurrency: - group: chat-feature-testops - cancel-in-progress: false - -jobs: - authorize: - runs-on: ubuntu-latest - timeout-minutes: 5 - outputs: - sha: ${{ steps.guard.outputs.sha }} - operation: ${{ steps.guard.outputs.operation }} - triggering_actor_id: ${{ steps.guard.outputs.triggering_actor_id }} - runner: runs-on/fleet=paperclip-public-pr-x64/env=public-ci - steps: - - &checkout - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 - with: - ref: ${{ github.sha }} - persist-credentials: false - - &node - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7 - with: - node-version: 24 - - &guard - name: Verify fixed default-source dispatch and both actor identities - id: guard - env: - GH_TOKEN: ${{ github.token }} - TESTOPS_ALLOWED_ACTOR_IDS: ${{ vars.RUNNER_E2E_ALLOWED_ACTOR_IDS }} - NODE_OPTIONS: '' - NODE_PATH: '' - run: node tests/runner-e2e/testops-chat-dispatch.ts - - select: - name: Validate TestOps selection or register zero-result setup - needs: authorize - runs-on: ubuntu-latest - timeout-minutes: 10 - environment: runner-e2e-testops-reporting - permissions: - contents: read - actions: read - outputs: - artifact_id: ${{ steps.selection_upload.outputs.artifact-id }} - sha256: ${{ steps.selection.outputs.sha256 }} - steps: - - *checkout - - *node - - *guard - - &allure - if: needs.authorize.outputs.operation != 'launch-plan' - uses: allure-framework/setup-allurectl@2d89ee0b7ba2f59dbe9849b6df5763cab78e14db # v1 - with: - allurectl-version: 2.23.1 - allure-endpoint: https://paperclip.testops.cloud - allure-project-id: '1' - allure-token: ${{ secrets.ALLURE_TOKEN }} - - name: Register the public job without an evaluation or a synthetic result - if: needs.authorize.outputs.operation == 'bootstrap' - env: - ALLURE_TOKEN: ${{ secrets.ALLURE_TOKEN }} - ALLURE_ENDPOINT: https://paperclip.testops.cloud - ALLURE_PROJECT_ID: '1' - CHAT_TESTOPS_GITHUB_WORKFLOW_ID: ${{ vars.CHAT_TESTOPS_GITHUB_WORKFLOW_ID }} - ALLURE_RESULTS: ${{ runner.temp }}/chat-testops-bootstrap-results - run: | - mkdir -m 700 "$ALLURE_RESULTS" - allurectl watch -- node tests/runner-e2e/testops-chat-reporter.mjs bootstrap - node tests/runner-e2e/testops-chat-reporter.mjs bootstrap-context - - name: Validate the one real TestOps selection before any build or provider access - id: selection - if: needs.authorize.outputs.operation == 'evaluate' - env: - ALLURE_TOKEN: ${{ secrets.ALLURE_TOKEN }} - ALLURE_ENDPOINT: https://paperclip.testops.cloud - ALLURE_PROJECT_ID: '1' - ALLURE_JOB_RUN_ID: ${{ inputs.ALLURE_JOB_RUN_ID }} - CHAT_TESTOPS_CASE_ID: ${{ vars.CHAT_TESTOPS_CASE_ID }} - CHAT_TESTOPS_PLAN_ID: ${{ vars.CHAT_TESTOPS_PLAN_ID }} - CHAT_TESTOPS_JOB_ID: ${{ vars.CHAT_TESTOPS_JOB_ID }} - CHAT_TESTOPS_GITHUB_WORKFLOW_ID: ${{ vars.CHAT_TESTOPS_GITHUB_WORKFLOW_ID }} - ALLURE_RESULTS: ${{ runner.temp }}/chat-testops-selection-results - run: | - mkdir -m 700 "$ALLURE_RESULTS" - allurectl watch -- node tests/runner-e2e/testops-chat-reporter.mjs selection - - name: Start exactly the validated TestOps plan and exit without waiting for its child - if: needs.authorize.outputs.operation == 'launch-plan' - env: - ALLURE_TOKEN: ${{ secrets.ALLURE_TOKEN }} - ALLURE_ENDPOINT: https://paperclip.testops.cloud - ALLURE_PROJECT_ID: '1' - CHAT_TESTOPS_CASE_ID: ${{ vars.CHAT_TESTOPS_CASE_ID }} - CHAT_TESTOPS_PLAN_ID: ${{ vars.CHAT_TESTOPS_PLAN_ID }} - CHAT_TESTOPS_JOB_ID: ${{ vars.CHAT_TESTOPS_JOB_ID }} - CHAT_TESTOPS_GITHUB_WORKFLOW_ID: ${{ vars.CHAT_TESTOPS_GITHUB_WORKFLOW_ID }} - CHAT_TESTOPS_OPERATION_PARAMETER_ID: ${{ vars.CHAT_TESTOPS_OPERATION_PARAMETER_ID }} - CHAT_TESTOPS_INTEGRATION_ID: ${{ vars.CHAT_TESTOPS_INTEGRATION_ID }} - CHAT_TESTOPS_LAUNCH_NONCE: ${{ vars.CHAT_TESTOPS_LAUNCH_NONCE }} - CHAT_TESTOPS_LAUNCH_RUN_NUMBER: ${{ vars.CHAT_TESTOPS_LAUNCH_RUN_NUMBER }} - TESTOPS_TRIGGERING_ACTOR_ID: ${{ needs.authorize.outputs.triggering_actor_id }} - TESTOPS_ALLOWED_ACTOR_IDS: ${{ vars.RUNNER_E2E_ALLOWED_ACTOR_IDS }} - GH_TOKEN: ${{ github.token }} - run: node tests/runner-e2e/testops-chat-reporter.mjs launch-plan - - name: Retain only closed metadata from zero-result setup - if: always() && needs.authorize.outputs.operation == 'bootstrap' - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 - with: - name: chat-testops-setup-${{ github.run_id }}-1 - path: | - ${{ runner.temp }}/chat-testops-bootstrap-state/metadata.json - ${{ runner.temp }}/chat-testops-bootstrap-state/once.json - ${{ runner.temp }}/chat-testops-bootstrap-state/plan.json - ${{ runner.temp }}/chat-testops-bootstrap-state/job.json - ${{ runner.temp }}/chat-testops-bootstrap-state/failure.json - ${{ runner.temp }}/chat-testops-bootstrap-state/registration.json - ${{ runner.temp }}/chat-testops-bootstrap-state/registration-failure.json - retention-days: 30 - if-no-files-found: error - - name: Retain the one plan-launch receipt and stop state - if: always() && needs.authorize.outputs.operation == 'launch-plan' - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 - with: - name: chat-testops-plan-launch-${{ github.run_id }}-1 - path: | - ${{ runner.temp }}/chat-testops-launch-plan-state/once.json - ${{ runner.temp }}/chat-testops-launch-plan-state/launch.json - ${{ runner.temp }}/chat-testops-launch-plan-state/failure.json - retention-days: 30 - if-no-files-found: error - - name: Retain only the token-free exact selection receipt - id: selection_upload - if: steps.selection.outcome == 'success' - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 - with: - name: chat-testops-selection-${{ github.run_id }}-1 - path: testops-selection.json - retention-days: 30 - if-no-files-found: error - - build: - name: Build the immutable shipped native candidate without model credentials - needs: [authorize, select] - if: needs.authorize.outputs.operation == 'evaluate' - runs-on: ${{ needs.authorize.outputs.runner }} - timeout-minutes: 45 - outputs: - artifact_id: ${{ steps.bundle.outputs.artifact-id }} - steps: - - *checkout - - *node - - *guard - - &pnpm - uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6 - env: - NPM_CONFIG_AUDIT: 'false' - NPM_CONFIG_FUND: 'false' - NPM_CONFIG_UPDATE_NOTIFIER: 'false' - with: - version: 9.15.4 - - name: Install only the source-reviewed graph without lifecycle scripts - run: | - node tests/runner-e2e/testops-chat-lock.ts > chat-testops-lock.json - pnpm install --frozen-lockfile --ignore-scripts - - name: Build the existing TypeScript and native runner owners - run: | - pnpm --filter @paperclipai/paperclip-runner build:typescript - pnpm --filter @paperclipai/paperclip-runner build:runner-binaries - - name: Package source-bound outputs and resolved lock - run: | - set -euo pipefail - test "$(git rev-parse HEAD)" = "$GITHUB_SHA" - node tests/runner-e2e/testops-chat-lock.ts > chat-testops-lock.json - tar -czf chat-testops-build.tar.gz chat-testops-lock.json pnpm-lock.yaml packages/paperclip-eval-kernel/dist packages/paperclip-runner/dist packages/paperclip-runner/runner/target/debug/paperclip-runnerd packages/paperclip-runner/runner/target/debug/conformance-tracer packages/paperclip-runner/runner/target/debug/fake-harness packages/paperclip-runner/runner/target/debug/fake-codex-app-server packages/paperclip-runner/runner/target/debug/fake-acpx-sidecar - sha256sum chat-testops-build.tar.gz > chat-testops-build.tar.gz.sha256 - - name: Retain credential-free immutable outputs - id: bundle - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 - with: - name: chat-testops-build-${{ github.sha }}-${{ github.run_id }}-1 - path: | - chat-testops-build.tar.gz - chat-testops-build.tar.gz.sha256 - retention-days: 1 - if-no-files-found: error - - chat: - name: One real native OpenCode backlog request on AWS - needs: [authorize, select, build] - if: needs.authorize.outputs.operation == 'evaluate' - runs-on: ${{ needs.authorize.outputs.runner }} - timeout-minutes: 32 - environment: runner-e2e-private-paid - permissions: - contents: read - id-token: write - env: &retention - PAPERCLIP_TESTOPS_CHAT_MAIN_ADMISSION: v1 - TESTOPS_ADMITTED_SHA: ${{ needs.authorize.outputs.sha }} - TESTOPS_ALLOWED_ACTOR_IDS: ${{ vars.RUNNER_E2E_ALLOWED_ACTOR_IDS }} - TESTOPS_TRIGGERING_ACTOR_ID: ${{ needs.authorize.outputs.triggering_actor_id }} - TARGET_SHA: ${{ needs.authorize.outputs.sha }} - MANUAL_PRIVATE_IDS: '["chat-features.runner-opencode-deepseek-1017.local.create-backlog"]' - SELECTED_EXECUTION_ID: chat-features.runner-opencode-deepseek-1017.local.create-backlog - RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET }} - NODE_OPTIONS: '' - NODE_PATH: '' - AWS_EC2_METADATA_DISABLED: 'true' - steps: - - *checkout - - *node - - *guard - - *pnpm - - name: Check out the same admitted source for trusted retention - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 - with: - ref: ${{ needs.authorize.outputs.sha }} - path: .testops-trusted - persist-credentials: false - - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 - with: - artifact-ids: ${{ needs.build.outputs.artifact_id }} - path: chat-testops-build - - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 - with: - artifact-ids: ${{ needs.select.outputs.artifact_id }} - path: chat-testops-selection - - name: Verify immutable outputs and token-free TestOps selection - env: - EXPECTED_SELECTION_SHA256: ${{ needs.select.outputs.sha256 }} - ALLURE_JOB_RUN_ID: ${{ inputs.ALLURE_JOB_RUN_ID }} - run: | - set -euo pipefail - (cd chat-testops-build && sha256sum -c chat-testops-build.tar.gz.sha256) - tar -xzf chat-testops-build/chat-testops-build.tar.gz -C "$GITHUB_WORKSPACE" - node tests/runner-e2e/testops-chat-reporter.mjs verify-selection chat-testops-selection/testops-selection.json - node tests/runner-e2e/testops-chat-lock.ts > chat-testops-worker-lock.json - cmp chat-testops-lock.json chat-testops-worker-lock.json - cp pnpm-lock.yaml .testops-trusted/pnpm-lock.yaml - pnpm install --frozen-lockfile --ignore-scripts - (cd .testops-trusted && pnpm install --frozen-lockfile --ignore-scripts --store-dir "$RUNNER_TEMP/chat-retention-store") - - name: Materialize and qualify pinned OpenCode runtime and browser - run: | - node packages/paperclip-runner/scripts/materialize-opencode-binary.mjs - node -e 'const fs=require("node:fs");fs.chmodSync(process.execPath,fs.statSync(process.execPath).mode&~0o022);if(fs.statSync(process.execPath).mode&0o022)throw Error("Writable provider Node")' - google-chrome --version - pnpm exec playwright install ffmpeg - - name: Prepare private evidence with shipped-main authority - id: private - env: - TARGET_ROOT: ${{ github.workspace }} - run: node .testops-trusted/cli/node_modules/tsx/dist/cli.mjs .testops-trusted/tests/runner-e2e/manual-private-transfer.ts prepare - - name: Obtain only the exact private cell storage session - id: cell_credentials - uses: aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd # v6 - with: - role-to-assume: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_ROLE_ARN }} - aws-region: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_REGION }} - inline-session-policy: ${{ steps.private.outputs.session_policy }} - output-env-credentials: false - output-credentials: true - disable-retry: true - action-timeout-s: 60 - - name: Verify strict private custody before the model key - env: &cell_session - PRIVATE_EVIDENCE_ROOT: ${{ steps.private.outputs.root }} - AWS_ACCESS_KEY_ID: ${{ steps.cell_credentials.outputs.aws-access-key-id }} - AWS_SECRET_ACCESS_KEY: ${{ steps.cell_credentials.outputs.aws-secret-access-key }} - AWS_SESSION_TOKEN: ${{ steps.cell_credentials.outputs.aws-session-token }} - AWS_DEFAULT_REGION: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_REGION }} - run: node .testops-trusted/cli/node_modules/tsx/dist/cli.mjs .testops-trusted/tests/runner-e2e/manual-private-transfer.ts verify - - name: Execute only the fixed original request with zero retries - env: - OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }} - PAPERCLIP_RUNNER_E2E_PRIVATE_PREFLIGHT: ${{ steps.private.outputs.root }}/preflight.json - PRIVATE_EVIDENCE_ROOT: ${{ steps.private.outputs.root }} - PAPERCLIP_E2E_CAMPAIGN_ID: gha-${{ github.run_id }}-1-chat-features.runner-opencode-deepseek-1017.local.create-backlog - PAPERCLIP_E2E_ACCOUNTING_CAMPAIGN_ID: gha-${{ github.run_id }}-1 - PAPERCLIP_RUNNER_E2E_SOURCE_SHA: ${{ needs.authorize.outputs.sha }} - PAPERCLIP_RUNNER_E2E_SOURCE_REF: ${{ github.ref }} - PAPERCLIP_PLAYWRIGHT_CHANNEL: chrome - run: pnpm test:e2e:runner -- --id chat-features.runner-opencode-deepseek-1017.local.create-backlog --max-parallel 1 --max-automatic-retries 0 > "$PRIVATE_EVIDENCE_ROOT/launcher-output.log" 2>&1 - - name: Retain exact original grade and private evidence, including failures - if: always() && steps.cell_credentials.outcome == 'success' - env: *cell_session - run: node .testops-trusted/cli/node_modules/tsx/dist/cli.mjs .testops-trusted/tests/runner-e2e/manual-private-transfer.ts upload-cell - - report: - name: Retain private originals and export only reviewed fixture results - needs: [authorize, select, chat] - if: always() && !cancelled() && needs.authorize.outputs.operation == 'evaluate' && needs.select.result == 'success' - runs-on: ubuntu-latest - timeout-minutes: 15 - environment: runner-e2e-private-history - permissions: - contents: read - actions: read - id-token: write - env: *retention - steps: - - *checkout - - *node - - *guard - - *pnpm - - name: Install trusted reporting dependencies only - run: | - node tests/runner-e2e/prepare-report-lock.mjs - pnpm install --frozen-lockfile --ignore-scripts --store-dir "$RUNNER_TEMP/chat-report-store" - - name: Prepare exact private report and cell-read policies - id: session - run: node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/manual-private-transfer.ts report-session - - name: Obtain private report writer - id: report_credentials - uses: aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd # v6 - with: - role-to-assume: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_ROLE_ARN }} - aws-region: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_REGION }} - inline-session-policy: ${{ steps.session.outputs.session_policy }} - output-env-credentials: false - output-credentials: true - disable-retry: true - action-timeout-s: 60 - - name: Obtain exact private cell reader - id: read_credentials - uses: aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd # v6 - with: - role-to-assume: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_ROLE_ARN }} - aws-region: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_REGION }} - inline-session-policy: ${{ steps.session.outputs.read_policy_0 }} - output-env-credentials: false - output-credentials: true - disable-retry: true - action-timeout-s: 60 - - name: Download the exact private original and verify all inventories - env: - PRIVATE_REPORT_ROOT: ${{ github.workspace }} - SELECTED_CELL_INDEX: '0' - AWS_ACCESS_KEY_ID: ${{ steps.read_credentials.outputs.aws-access-key-id }} - AWS_SECRET_ACCESS_KEY: ${{ steps.read_credentials.outputs.aws-secret-access-key }} - AWS_SESSION_TOKEN: ${{ steps.read_credentials.outputs.aws-session-token }} - AWS_DEFAULT_REGION: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_REGION }} - run: | - node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/manual-private-transfer.ts download-cell - node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/manual-private-transfer.ts verify-downloads - - name: Build the existing normalized campaign without exposing original logs - env: - PAPERCLIP_RUNNER_E2E_REPORT_ROOT: ${{ github.workspace }}/selected-runner-e2e - PAPERCLIP_RUNNER_E2E_REPORT_OUT: ${{ github.workspace }}/runner-e2e-merged-report/normalized - PAPERCLIP_RUNNER_E2E_EXPECTED_IDS: '["chat-features.runner-opencode-deepseek-1017.local.create-backlog"]' - PAPERCLIP_E2E_CAMPAIGN_ID: gha-${{ github.run_id }}-1 - PAPERCLIP_RUNNER_E2E_SOURCE_SHA: ${{ needs.authorize.outputs.sha }} - PAPERCLIP_RUNNER_E2E_SOURCE_REF: ${{ github.ref }} - run: | - mkdir -p runner-e2e-merged-report - pnpm test:e2e:runner:report > runner-e2e-merged-report/report.log 2>&1 - - name: Retain the complete merged report privately even when its grade failed - id: private_report - if: always() && steps.report_credentials.outcome == 'success' - env: - PRIVATE_REPORT_ROOT: ${{ github.workspace }} - PRIVATE_REPORT_SOURCE: ${{ github.workspace }}/runner-e2e-merged-report - AWS_ACCESS_KEY_ID: ${{ steps.report_credentials.outputs.aws-access-key-id }} - AWS_SECRET_ACCESS_KEY: ${{ steps.report_credentials.outputs.aws-secret-access-key }} - AWS_SESSION_TOKEN: ${{ steps.report_credentials.outputs.aws-session-token }} - AWS_DEFAULT_REGION: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_REGION }} - run: node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/manual-private-transfer.ts upload-report - - name: Export only the bound normalized result, JUnit and marked PNGs - id: export - if: always() && steps.private_report.outcome == 'success' - env: - TESTOPS_CHAT_TARGET_SHA: ${{ needs.authorize.outputs.sha }} - run: node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/testops-chat-export.ts - - name: Upload only the sanitized TestOps projection - if: always() && steps.export.outcome == 'success' - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 - with: - name: runner-e2e-testops-${{ github.run_id }}-1 - path: runner-e2e-testops/ - retention-days: 30 - if-no-files-found: error - - upload: - name: Report the original outcome to its exact TestOps case - needs: [authorize, select, chat, report] - if: always() && !cancelled() && needs.authorize.outputs.operation == 'evaluate' && needs.select.result == 'success' - runs-on: ubuntu-latest - timeout-minutes: 10 - environment: runner-e2e-testops-reporting - permissions: - contents: read - actions: read - steps: - - *checkout - - *node - - *guard - - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 - with: - name: runner-e2e-testops-${{ github.run_id }}-1 - path: runner-e2e-testops - - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 - with: - artifact-ids: ${{ needs.select.outputs.artifact_id }} - path: chat-testops-selection - - *allure - - name: Verify and upload only the bound safe case outcome - id: allure_upload - env: &testops_reporter - ALLURE_TOKEN: ${{ secrets.ALLURE_TOKEN }} - ALLURE_ENDPOINT: https://paperclip.testops.cloud - ALLURE_PROJECT_ID: '1' - ALLURE_JOB_RUN_ID: ${{ inputs.ALLURE_JOB_RUN_ID }} - EXPECTED_SELECTION_SHA256: ${{ needs.select.outputs.sha256 }} - ALLURE_RESULTS: ${{ runner.temp }}/chat-testops-allure-results - run: allurectl watch -- node tests/runner-e2e/testops-chat-reporter.mjs upload runner-e2e-testops - - name: Verify the actual TestOps result after upload processing - id: verify_upload - if: always() && !cancelled() && (steps.allure_upload.outcome == 'success' || steps.allure_upload.outcome == 'failure') - env: *testops_reporter - run: node tests/runner-e2e/testops-chat-reporter.mjs verify-upload runner-e2e-testops - - name: Retain only closed actual TestOps verification metadata - if: always() && !cancelled() && (steps.verify_upload.outcome == 'success' || steps.verify_upload.outcome == 'failure') - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 - with: - name: chat-testops-verification-${{ github.run_id }}-1 - path: | - ${{ runner.temp }}/chat-testops-verify-upload-state/verification.json - ${{ runner.temp }}/chat-testops-verify-upload-state/failure.json - retention-days: 30 - if-no-files-found: error diff --git a/doc/evals.md b/doc/evals.md index 4071cf01866..d0ea2fa5ec3 100644 --- a/doc/evals.md +++ b/doc/evals.md @@ -57,8 +57,9 @@ OpenCode Product E2E cells. They verify saving an unstarted task and delegating a saved announcement back to chat, using ordinary production instructions, real browser messages and independent state/provenance checks. A separately selected native OpenCode/DeepSeek1017 -backlog cell measures TestOps integration acceptance with the same oracle and -original bounds. Its source registration is not an AWS or TestOps pass. The +backlog cell supplied historical TestOps integration acceptance with the same +oracle and original bounds. Its former smoke plan and the three-case completion +pilot are retired; their cases and measurements remain in history. The [full-catalog TestOps lane](../tests/runner-e2e/CHAT-FEATURES.md#full-catalog-in-testops) maps all 244 YAML scenarios into a separate candidate-scoped plan. Its app and fixtures use a preserved immutable catalog revision; trusted workflow helpers run diff --git a/scripts/__tests__/release-verify-workflow.test.mjs b/scripts/__tests__/release-verify-workflow.test.mjs index 97290f343e1..12dbe906011 100644 --- a/scripts/__tests__/release-verify-workflow.test.mjs +++ b/scripts/__tests__/release-verify-workflow.test.mjs @@ -579,8 +579,6 @@ test("Runner eval workflows pin actions and gate paid live execution", () => { readWorkflow("runner-full-stack-e2e.yml"), readWorkflow("e2e.yml"), readWorkflow("runner-protocol-live-evals.yml"), - readWorkflow("chat-feature-testops.yml"), - readWorkflow("chat-completion-testops.yml"), readWorkflow("chat-catalog-testops.yml"), ]; @@ -621,8 +619,6 @@ test("Runner eval workflows pin actions and gate paid live execution", () => { "runner-full-stack-e2e.yml", "runner-live-evals.yml", "runner-protocol-live-evals.yml", - "chat-feature-testops.yml", - "chat-completion-testops.yml", "chat-catalog-testops.yml", ]; const paidWorkflowNameSet = new Set(paidWorkflowNames); @@ -705,38 +701,7 @@ test("Runner eval workflows pin actions and gate paid live execution", () => { } continue; } - if (name === "chat-feature-testops.yml") { - assert.equal(providerJobs.length, 1); - const assertBoundary = (block) => { - assert.match(block, /^ chat:\n/); - assert.match(block, /\n needs: \[authorize, select, build\]\n/); - assert.match(block, /\n if: needs\.authorize\.outputs\.operation == 'evaluate'\n/); - assert.match(block, /\n environment: runner-e2e-private-paid\n/); - assert.match(block, /\n - \*checkout\n - \*node\n - \*guard\n/); - assert.match(block, /testops-chat-reporter\.mjs verify-selection chat-testops-selection\/testops-selection\.json/); - assert.match(block, /OPENROUTER_API_KEY: \$\{\{ secrets\.OPENROUTER_API_KEY \}\}/); - assert.match(block, /--id chat-features\.runner-opencode-deepseek-1017\.local\.create-backlog --max-parallel 1 --max-automatic-retries 0/); - assert.doesNotMatch(block, /ALLURE_TOKEN/); - assert.equal([...block.matchAll(providerSecretReference)].length, 1); - assert.ok(block.indexOf(" - *guard") < block.search(providerSecretReference)); - assert.ok(block.indexOf("testops-chat-reporter.mjs verify-selection") < block.search(providerSecretReference)); - }; - const block = providerJobs[0]; - assertBoundary(block); - assert.equal([...workflow.matchAll(providerSecretReference)].length, 1); - assert.match(workflow, /ref: \$\{\{ github\.sha \}\}/); - assert.match(workflow, /run: node tests\/runner-e2e\/testops-chat-dispatch\.ts/); - assert.match(workflow, /TESTOPS_ALLOWED_ACTOR_IDS: \$\{\{ vars\.RUNNER_E2E_ALLOWED_ACTOR_IDS \}\}/); - for (const mutant of [ - block.replace("needs: [authorize, select, build]", "needs: [authorize, build]"), - block.replace("environment: runner-e2e-private-paid", "environment: unprotected"), - block.replace(" - *guard\n", ""), - block.replace("testops-chat-reporter.mjs verify-selection", "testops-chat-reporter.mjs unverified-selection"), - block.replace("OPENROUTER_API_KEY:", "ALLURE_TOKEN:"), - block.replace("--id chat-features.runner-opencode-deepseek-1017.local.create-backlog", "--id chat-features.runner-opencode.local.create-backlog"), - ]) assert.throws(() => assertBoundary(mutant)); - continue; - } + for (const block of providerJobs) { if (name === "runner-protocol-live-evals.yml") { assert.ok(block.includes("name: ${{ needs.authorize.outputs.accounting_mode == 'controlled' && 'runner-e2e-private-paid' || 'runner-e2e-paid' }}")); diff --git a/tests/runner-e2e/CHAT-FEATURES.md b/tests/runner-e2e/CHAT-FEATURES.md index a66cf38a374..01d7c354b23 100644 --- a/tests/runner-e2e/CHAT-FEATURES.md +++ b/tests/runner-e2e/CHAT-FEATURES.md @@ -181,82 +181,31 @@ backlog is 5/5 and delegation is 5/6. The local `results/chat-feature-recheck/verification.md` records the follow-up evidence; its standard dashboard is explicitly scoped to the latest campaign. -## TestOps completion pilot +## Retired TestOps plans -The separate `Chat completion TestOps` workflow connects only these preserved -catalog cases to the private Paperclip Evals project: +The one-case **Chat feature evals** plan and three-case +**Chat completion pilot — OpenCode 1017** plan are retired. Their scenarios are +included in the [full catalog](#full-catalog-in-testops). Use that plan or select +cases from it; do not recreate the smoke or pilot plans. -- `projects.project-work` -- `projects.project-defaults` -- `instance-operations.restart-service` +The former `chat-feature-testops.yml` and `chat-completion-testops.yml` workflows +are removed. Their bootstrap functions reject registration before any API call +or mutation marker. Shared transport, evidence, export and verification helpers +remain for the full-catalog workflow and retained historical evidence. Retiring a +plan does not delete its cases, results, launches or private evidence. -It evaluates the immutable catalog source in -`chat-completion-testops-contract.mjs`. This keeps the broader catalog on its -integration branch. The source includes the bounded TestOps host prerequisite -from #18515. Changing the pin requires source review. The original DeepSeek -1017 model, case clocks, independent checks, and zero retries remain fixed. -Source registration does not establish a new live result. - -Run the named plan **Chat completion pilot — OpenCode 1017** in TestOps. The -adapter validates the actual project, plan, job, launch and selected cases before -build or provider access. A selected subset runs only those cases. The AWS matrix -has concurrency one and does not cancel sibling cases after a failure. - -The main-only reporter uses the existing protected `ALLURE_TOKEN`; the evaluated -app and worker never receive it. Original evidence stays in the existing private -AWS storage with the unchanged validators, scanners, byte bounds and exact-key -policies. A separate trusted projection carries verdicts, checks, accounting and -private-evidence receipts. The final PNGs are verified in protected storage; -only their checksums and availability are recorded in TestOps. GitHub artifacts -contain metadata only. Raw logs, screenshots, hidden -reasoning, API snapshots, credentials and provider sessions are not attachments. -Original failed grades and cleanup failures remain failed. Same-run readback -requires matching results and complete result-owned attachment metadata within -the existing bounded processing window; it does not claim remote byte hashes. - -Initial setup uses the existing workflow operations: - -1. Dispatch reviewed `master` once with `operation=bootstrap`. This creates or - reconciles only the three exact automated cases and the dedicated plan. It - registers the GitHub job with an empty Allure results directory. It reuses - only the existing enabled GitHub integration from - `CHAT_TESTOPS_INTEGRATION_ID`. No model or fabricated result is created. - The observation receipt captures a closed parameter-shape inspection before - binding. Only the observed single `Branch/defaultValue` definition for - `refs/heads/master` is accepted. Bootstrap enables the job and assigns cases - while preserving that definition; the plan uses its default branch. - Assignment inspection records unassigned and existing-job counts before - binding. A single unassigned bucket must cover the exact selected cases; - foreign jobs and unknown or incomplete buckets stop before writes. -2. Read the closed registration receipt. Store its observed `binding` JSON as - `CHAT_COMPLETION_TESTOPS_BINDING` in `runner-e2e-testops-reporting`. -3. Use TestOps **Run**. For the initial operator launch, set an exact next - workflow run number in `CHAT_COMPLETION_TESTOPS_LAUNCH_RUN_NUMBER`, set a new - 32-hex `CHAT_COMPLETION_TESTOPS_LAUNCH_NONCE`, then dispatch `launch-plan` - once. Its request produces the TestOps-owned `evaluate` dispatch. Do not - manually dispatch an additional evaluation after an uncertain launch. -4. Verify the fresh TestOps launch, all selected case results, source links, - independent checks, private-evidence receipts and cleanup. Inspect screenshots - through authenticated private-history retrieval. Retain earlier failed attempts. - -Launch preflight searches use TestOps' typed name filter, encoded as Base64 of -UTF-8 JSON in the `search` query parameter. Plain name text returns HTTP 400. -The backlog and completion launchers share this encoding; the full-catalog -launcher uses the completion adapter. Existing nonce and once-only guards still -apply before a plan-run request. - -The runtime accepts only `operation`, `ALLURE_JOB_RUN_ID` and reserved username -metadata. It rejects arbitrary source/model/budget selectors, branch workflows, -replays and foreign cases. All secret-bearing jobs recheck actual GitHub run and -actor identities. The existing one-case backlog plan remains independent. +Retained launch helpers encode TestOps name filters as Base64 of UTF-8 JSON in +`search`. Plain name text returns HTTP 400. Their nonce and once-only guards +remain unchanged; this transport contract does not restore the retired plans. ## Full catalog in TestOps The [Chat catalog TestOps workflow](../../.github/workflows/chat-catalog-testops.yml) registers all 244 scenarios from the 35 YAML areas in a separate **Chat feature catalog — OpenCode 1017 (candidate)** plan. Its source-derived -manifest is `chat-full-catalog-testops-manifest.json`. The existing backlog plan -and three-case completion pilot keep their own jobs, selections and history. +manifest is `chat-full-catalog-testops-manifest.json`. This is the active chat +plan. The former backlog and three-case pilot plans are retired; their cases +and measured history are preserved. Legacy aliases reuse existing TestOps case identities without renaming them. The catalog contains 236 Agent Chat routes, five browser boundaries, one external @@ -322,77 +271,3 @@ and credentials do not enter the public artifact path. Additional credentials reach only cases that declare them. External inputs use the existing selected-case private-input owner, disposable-resource checks and oracle commitments. Unconfigured inputs remain setup blockers. - -## TestOps-backed AWS backlog cell - -The dedicated `Chat feature TestOps` workflow accepts only the shipped default -branch and the fixed OpenCode backlog cell. `evaluate` is the default operation -and requires TestOps' `ALLURE_JOB_RUN_ID`. A protected reporting job verifies the -project, singleton plan, assigned public workflow, selected case and current -GitHub run before any build or provider access. The evaluated process receives -no TestOps token. It retains the original one-query, one-work-run, 120-second -turn, 420-second attempt, 300-second settled-state and zero-retry limits. - -Two metadata operations prepare that route. `bootstrap` registers the actual -public CI job through pinned `allurectl` with an empty results directory and -reconciles only the dedicated case and singleton plan. It produces no passing -test result. After the registered job and protected case/plan/job identifiers -are verified, `launch-plan` sends one TestOps plan-run request and exits. -The TestOps job then dispatches `evaluate`; an operator must not dispatch a -second evaluation manually. Missing or ambiguous registration and selection -metadata stops before inference. - -The workflow reuses the existing AWS fleet and strict private evidence storage. -Its shipped-main authority is separate from manual candidate approval. Original -logs, provider records and failure evidence remain private. The reporting adapter -exports only a source/run-bound normalized result, JUnit and screenshots already -marked by the owned fixture screenshot producer with matching file hashes. -A separate protected reporting job converts that projection to the exact TestOps -case. Failed grades and cleanup failures remain failed. - -Reporting setup uses the master-only `runner-e2e-testops-reporting` environment, -with the existing reporting token delivered through the reviewed sealed-box -bootstrap. `CHAT_TESTOPS_CASE_ID`, `CHAT_TESTOPS_PLAN_ID` and -`CHAT_TESTOPS_JOB_ID` must name the observed dedicated mapping; the numeric -`CHAT_TESTOPS_GITHUB_WORKFLOW_ID` supports TestOps' registered workflow URL. -No arbitrary source, model, case, retry or budget input is accepted. - -The initial `launch-plan` administrator operation also requires a protected -`CHAT_TESTOPS_LAUNCH_RUN_NUMBER`. After registration, the trusted operator creates -this variable once with the reviewed next workflow run number and reads it back. -The launcher compares GitHub's actual run number, attempt, workflow, source and -actors before any TestOps mutation. A later dispatch or replay cannot reuse that -approved number. Missing or uncertain metadata stops; an uncertain request needs -reconciliation rather than a new send. The nonce and launch search provide -additional evidence. Normal TestOps **Run** uses `evaluate` and is independent of -this one-time administrator fence. The launch nonce is protected configuration, -not a dispatch input. Job parameter IDs are opaque. The launcher preserves the -API's exact ID/value pair for `refs/heads/master` and uses the fixed workflow's -`evaluate` default. An operation parameter is used only when the protected -`CHAT_TESTOPS_OPERATION_PARAMETER_ID` matches a returned ID with a `bootstrap` -or `evaluate` value; the launcher then sets it to `evaluate`. Other values, -duplicate IDs and conflicting configuration are rejected. No relationship to -global environment-variable IDs or names is inferred. - -`CHAT_TESTOPS_INTEGRATION_ID` names the reviewed existing project GitHub -integration. The launcher verifies its project, type and enabled state, then binds -only the newly registered public job to it. It never changes integration settings -or credentials. Since the job response does not expose that association, only the -real TestOps-to-GitHub dispatch can qualify it. - -After `allurectl watch` finishes, the same protected reporter checks the actual -TestOps job-run, launch, selected case, result source/status and attachment -metadata. Processing reads are bounded to 12 polls, five-second gaps and a -120-second processing deadline. Unknown or missing results remain unverified. -The check downloads no attachment content and claims no server-side attachment -byte hash. Its closed receipt is retained separately. An upload success alone -is not a passing feature result, and verifying an original failure cannot turn -that failure into a pass. - -Build and worker installs use the existing reviewed report-lock policy. Only its -committed input or output lock is accepted; its authenticated correction is the -only allowed normalization. Dependency manifests, patch inputs and policy must -remain unchanged. Installs are frozen with lifecycle scripts disabled. The -credential-free build rechecks that boundary before packaging, and the worker -matches the source/lock receipt before installing. Arbitrary resolver drift is -rejected before provider access. diff --git a/tests/runner-e2e/chat-completion-testops-api.mjs b/tests/runner-e2e/chat-completion-testops-api.mjs index 5b9f883c935..7d02341af3e 100644 --- a/tests/runner-e2e/chat-completion-testops-api.mjs +++ b/tests/runner-e2e/chat-completion-testops-api.mjs @@ -135,32 +135,12 @@ export async function inspectPilotCatalog(api) { } export async function bootstrapPilotPlan(api, integrationId, markOnce, context = {}) { + // The pilot plan is retired. Reject before API reads or mutation markers. + if (context.catalog !== 'full') reject('plan_retired'); if (!positive(integrationId)) reject('integration'); const integration = await api(`/api/integration/${integrationId}/project/1`); if (integration.id !== integrationId || integration.projectId !== 1 || integration.info?.type !== 'github' || integration.disabled !== false) reject('integration'); - if (context.catalog === 'full') return { ...await reconcileFullCatalog(api, markOnce), integrationId }; - const inspected = await inspectPilotCatalog(api); - if (inspected.plans.length && inspected.cases.length !== cases.length) reject('plan_conflict'); - const selected = [...inspected.cases]; - for (const row of cases) { - if (selected.some(value => value.key === row.key)) continue; - await markOnce(`create_case_${row.key}`); - const created = await api('/api/testcase', 'POST', { projectId: 1, name: row.name, fullName: row.fullName, automated: true }); - if (!positive(created?.id)) reject('created_case'); - assertCase(created, { ...row, caseId: created.id }); - selected.push({ ...row, caseId: created.id }); - } - const ordered = cases.map(row => selected.find(value => value.key === row.key)); - let planId = inspected.plans[0]?.id; - if (!planId) { - await markOnce('create_plan'); - const created = await api('/api/testplan', 'POST', { projectId: 1, name: planName, - treeSelection: { inverted: false, leafsInclude: ordered.map(row => row.caseId) } }); - if (!positive(created?.id)) reject('created_plan'); planId = created.id; - } - const binding = { planId, integrationId, cases: ordered }; - assertPlan(await api(`/api/testplan/${planId}`), binding); - return binding; + return { ...await reconcileFullCatalog(api, markOnce), integrationId }; } export async function completePilotRegistration(api, prepared, official, context, workflowId, markOnce, observe = async () => {}) { diff --git a/tests/runner-e2e/chat-completion-testops-api.test.mjs b/tests/runner-e2e/chat-completion-testops-api.test.mjs index 4ee58ebe003..aee4be2527c 100644 --- a/tests/runner-e2e/chat-completion-testops-api.test.mjs +++ b/tests/runner-e2e/chat-completion-testops-api.test.mjs @@ -180,35 +180,9 @@ test('an uncertain launch response is never automatically resent', async () => { assert.equal(f.calls.filter(row => row.method === 'POST').length, 1); }); -test('registration creates only the three missing cases and the dedicated plan after once markers', async () => { - const calls = []; let nextCase = 100; - const get = async (route, method = 'GET', body) => { - calls.push({ route, method, body }); - if (route === '/api/integration/2/project/1') return { id: 2, projectId: 1, info: { type: 'github' }, disabled: false }; - if (route.startsWith('/api/testcase?')) return { content: [], number: 0, totalPages: 0, totalElements: 0 }; - if (route.startsWith('/api/testplan?')) return { content: [], totalPages: 0 }; - if (route === '/api/testcase' && method === 'POST') return { ...body, id: nextCase++ }; - if (route === '/api/testplan' && method === 'POST') return { ...plan, name: planName }; - if (route === '/api/testplan/10') return plan; - assert.fail(route); - }; - const value = await bootstrapPilotPlan(get, 2, async stage => calls.push({ marker: stage })); - assert.equal(value.cases.length, 3); assert.equal(value.planId, 10); - assert.equal(calls.filter(row => row.method === 'POST').length, 4); - for (const [i, row] of calls.entries()) if (row.method === 'POST') assert.ok(calls[i - 1].marker); -}); - -test('uncertain case creation is never automatically resent', async () => { - let writes = 0, markers = 0; - const get = async (route, method = 'GET') => { - if (method === 'POST') { writes++; throw new Error('uncertain transport'); } - if (route === '/api/integration/2/project/1') return { id: 2, projectId: 1, info: { type: 'github' }, disabled: false }; - if (route.startsWith('/api/testcase?')) return { content: [], number: 0, totalPages: 0, totalElements: 0 }; - if (route.startsWith('/api/testplan?')) return { content: [], totalPages: 0 }; - assert.fail(route); - }; - await assert.rejects(bootstrapPilotPlan(get, 2, async () => { markers++; }), /uncertain transport/); - assert.equal(writes, 1); assert.equal(markers, 1); +test('retired pilot bootstrap cannot recreate a plan or call the integration API', async () => { + await assert.rejects(bootstrapPilotPlan(() => assert.fail('retired route must not call the API'), 2, + () => assert.fail('retired route must not mark a mutation')), /chat_completion_testops_plan_retired/); }); function api(replace = {}) { diff --git a/tests/runner-e2e/chat-completion-testops-contract.test.mjs b/tests/runner-e2e/chat-completion-testops-contract.test.mjs index c77d89031a8..c1199bbeb40 100644 --- a/tests/runner-e2e/chat-completion-testops-contract.test.mjs +++ b/tests/runner-e2e/chat-completion-testops-contract.test.mjs @@ -96,37 +96,8 @@ test('the executable guard binds the actual original GitHub run before service o } finally { await rm(root, { recursive: true, force: true }); } }); -test('the real workflow separates metadata authority, provider execution and result reporting', async () => { - const text = await readFile(new URL('../../.github/workflows/chat-completion-testops.yml', import.meta.url), 'utf8'); - const w = parse(text); - assert.deepEqual(Object.keys(w.on.workflow_dispatch.inputs).sort(), ['ALLURE_JOB_RUN_ID', 'ALLURE_USERNAME', 'operation']); - assert.equal(w.concurrency['cancel-in-progress'], false); - assert.equal(w.jobs.chat.strategy['max-parallel'], 1); assert.equal(w.jobs.chat.strategy['fail-fast'], false); - assert.equal(w.jobs.select.environment, 'runner-e2e-testops-reporting'); - assert.equal(w.jobs.chat.environment, 'runner-e2e-private-paid'); - assert.equal(w.jobs.report.environment, 'runner-e2e-private-history'); - assert.equal(w.jobs.upload.environment, 'runner-e2e-testops-reporting'); - for (const [name, job] of Object.entries(w.jobs)) { - assert.ok(job.steps.slice(0, 3).some(step => step.run?.endsWith('chat-completion-testops-cli.mjs guard')), name); - if (name !== 'chat') assert.ok(!JSON.stringify(job).includes('secrets.OPENROUTER_API_KEY'), name); - if (!['select', 'upload'].includes(name)) assert.ok(!JSON.stringify(job).includes('secrets.ALLURE_TOKEN'), name); - } - assert.ok(w.jobs.upload.if.includes("needs.report.result == 'success'")); - assert.ok(w.jobs.report.needs.includes('build')); - assert.ok(w.jobs.report.if.includes("needs.build.result == 'success'")); - assert.ok(w.jobs.report.if.includes("needs.chat.result == 'failure'")); - assert.ok(!text.includes('<<:')); assert.ok(!text.includes('target_branch:')); - assert.equal(w.jobs.build.steps.find(step => step.name?.startsWith('Resolve the fixed')).run.includes('--ignore-scripts'), true); - const paid = w.jobs.chat.steps.find(step => step.env?.OPENROUTER_API_KEY); - assert.equal(paid.env.PAPERCLIP_CHAT_COMPLETION_TESTOPS, 'v1'); - assert.equal(paid.env.ALLURE_JOB_RUN_ID, '${{ inputs.ALLURE_JOB_RUN_ID }}'); - assert.ok(paid.run.includes('--max-automatic-retries 0')); - assert.ok(w.jobs.build.steps.some(step => step.run?.includes('chat-completion-testops-host-control.mjs'))); - assert.ok(w.jobs.chat.steps.indexOf(paid) > w.jobs.chat.steps.findIndex(step => step.name?.startsWith('Verify source-bound'))); -}); - test('target checkout cannot populate or overwrite the trusted helper path', async () => { - const definition = parse(await readFile(new URL('../../.github/workflows/chat-completion-testops.yml', import.meta.url), 'utf8')); + const definition = parse(await readFile(new URL('../../.github/workflows/chat-catalog-testops.yml', import.meta.url), 'utf8')); const root = await realpath(await mkdtemp(path.join(os.tmpdir(), 'testops-checkout-boundary-'))); try { const git = (...args) => execFileSync('git', ['-c', 'core.hooksPath=/dev/null', '-c', 'commit.gpgsign=false', ...args], { cwd: root, stdio: 'pipe' }); diff --git a/tests/runner-e2e/chat-feature-testops-api.mjs b/tests/runner-e2e/chat-feature-testops-api.mjs index 2b4a4f1d049..47fc4939da8 100644 --- a/tests/runner-e2e/chat-feature-testops-api.mjs +++ b/tests/runner-e2e/chat-feature-testops-api.mjs @@ -53,7 +53,7 @@ const diagnosticCodes = new Set(['untrusted_chat_testops_context', 'testops_toke 'testops_case_alias_conflict', 'testops_plan_name_conflict', 'testops_plan_identity_invalid', 'testops_integration_identity_invalid', 'testops_bootstrap_identity_invalid', 'testops_plan_without_matching_case', 'testops_created_case_identity_invalid', 'testops_created_plan_identity_invalid', 'testops_plan_scope_mismatch', - 'testops_operation_invalid', 'testops_output_invalid', 'untrusted_public_testops_context', 'chat_testops_identity_missing', + 'testops_plan_retired', 'testops_operation_invalid', 'testops_output_invalid', 'untrusted_public_testops_context', 'chat_testops_identity_missing', 'chat_testops_case_mismatch', 'chat_testops_plan_mismatch', 'chat_testops_plan_job_mismatch', 'chat_testops_job_mismatch', 'chat_testops_job_id_mismatch', 'chat_testops_job_type_mismatch', 'chat_testops_job_parameters_shape_mismatch', 'chat_testops_job_parameters_limit', 'chat_testops_job_url_mismatch', 'chat_testops_jobrun_mismatch', 'chat_testops_launch_mismatch', 'chat_testops_selection_mismatch', 'chat_testops_parameter_mismatch', 'chat_testops_nonce_invalid', 'chat_testops_nonce_conflict', 'chat_testops_catalog_incomplete', 'chat_testops_launch_admission', 'chat_testops_github_http', 'chat_testops_integration_mismatch', 'chat_testops_upload_mismatch', 'chat_testops_upload_unverified']); @@ -139,32 +139,10 @@ export async function inspectChatMetadata(api) { return { projectId: 1, caseConflict: cases.size > 1, cases: [...cases.values()], plans: plans.map(value => ({ id: value.id, name: planName, count: value.testCasesCount })), integrations }; } -export async function ensureChatPlan(api, inspected, markOnce) { - if (inspected.projectId !== 1 || inspected.plans.length > 1) fail('testops_bootstrap_identity_invalid'); - if (inspected.caseConflict === true || inspected.cases.length > 1) fail('testops_case_alias_conflict'); - let selected = inspected.cases[0], plan = inspected.plans[0]; - if (plan && !selected) fail('testops_plan_without_matching_case'); - if (!selected || !plan) await markOnce(); - if (!selected) { - const created = await api('/api/testcase', 'POST', { projectId: 1, name: 'Create a task backlog through chat', fullName, - automated: true, description: 'Canonical scenario: tasks.create-backlog. Readiness and recorded outcomes remain separate.', - expectedResult: 'The real agent creates the requested backlog; public Product E2E assertions and cleanup must pass.' }); - if (!id(created?.id) || created.projectId !== 1 || created.fullName !== fullName) fail('testops_created_case_identity_invalid'); - selected = { id: created.id, fullName, automated: created.automated === true }; - } - if (!plan) { - const created = await api('/api/testplan', 'POST', { projectId: 1, name: planName, - treeSelection: { inverted: false, leafsInclude: [selected.id] } }); - if (!id(created?.id) || created.projectId !== 1 || created.name !== planName) fail('testops_created_plan_identity_invalid'); - plan = { id: created.id, name: planName }; - } - const actual = await api(`/api/testplan/${plan.id}`); - if (actual.projectId !== 1 || actual.name !== planName || actual.testCasesCount !== 1 - || actual.treeSelection?.inverted === true || !same(actual.treeSelection?.leafsInclude, [selected.id]) - || (actual.treeSelection?.groupsInclude?.length ?? 0) > 0 || (actual.treeSelection?.leafsExclude?.length ?? 0) > 0) fail('testops_plan_scope_mismatch'); - return { projectId: 1, case: selected, plan: { id: plan.id, name: planName, count: 1 }, - caseUrl: `${endpoint}/project/1/test-cases/${selected.id}`, planUrl: `${endpoint}/testplan/${plan.id}`, - actualResult: null, note: 'Plan preparation is not an executed measurement or a passing result.' }; +// Kept as an explicit retirement boundary for callers of the former smoke lane. +// Catalog registration owns the shared cases; deleting the old plan must persist. +export async function ensureChatPlan() { + fail('testops_plan_retired'); } diff --git a/tests/runner-e2e/chat-feature-testops-api.test.mjs b/tests/runner-e2e/chat-feature-testops-api.test.mjs index 318c616208c..68f689beee6 100644 --- a/tests/runner-e2e/chat-feature-testops-api.test.mjs +++ b/tests/runner-e2e/chat-feature-testops-api.test.mjs @@ -31,64 +31,14 @@ test('multiple alias identities, foreign project and incomplete metadata stop be await assert.rejects(inspectChatMetadata(async () => ({ content: [], totalPages: 11, totalElements: 1100, number: 0 })), /catalog_incomplete/); }); -test('existing exact plan reuses case and plan without a write', async () => { - const result = await ensureChatPlan(async route => { assert.equal(route, '/api/testplan/8'); return plan; }, - await inspectChatMetadata(inspectApi()), () => assert.fail('must not mark or write')); - assert.equal(result.case.id, 91); - assert.equal(result.plan.id, 8); - assert.equal(result.actualResult, null); -}); - -test('bootstrap creates only the missing exact plan after exclusive marker and reads back scope', async () => { - const calls = []; - const result = await ensureChatPlan(async (route, method, body) => { - calls.push({ route, method, body }); - if (method === 'POST') { assert.equal(calls[0].route, 'MARK'); assert.deepEqual(body, { projectId: 1, name: planName, treeSelection: { inverted: false, leafsInclude: [91] } }); } - return plan; - }, { projectId: 1, cases: [caseRow], plans: [] }, async () => calls.push({ route: 'MARK' })); - assert.equal(calls.filter(row => row.method === 'POST').length, 1); - assert.equal(result.plan.count, 1); -}); - -test('empty project bootstrap links the genuinely created case into only the new plan', async () => { - const calls = []; - const createdCase = { ...caseRow, id: 92 }; - const createdPlan = { ...plan, treeSelection: { inverted: false, leafsInclude: [92] } }; - const result = await ensureChatPlan(async (route, method, body) => { - calls.push({ route, method, body }); - assert.equal(calls[0].route, 'MARK'); - if (route === '/api/testcase') { - assert.equal(body.projectId, 1); assert.equal(body.fullName, fullName); assert.equal(body.automated, true); - return createdCase; - } - if (route === '/api/testplan') { - assert.deepEqual(body, { projectId: 1, name: planName, treeSelection: { inverted: false, leafsInclude: [92] } }); - return createdPlan; - } - assert.equal(route, '/api/testplan/8'); assert.equal(method, undefined); return createdPlan; - }, { projectId: 1, cases: [], plans: [] }, async () => calls.push({ route: 'MARK' })); - assert.equal(result.case.id, 92); assert.equal(result.plan.id, 8); - assert.deepEqual(calls.map(call => call.route), ['MARK', '/api/testcase', '/api/testplan', '/api/testplan/8']); -}); - -test('known conflicting plan stops before creating an otherwise missing case or marker', async () => { - await assert.rejects(ensureChatPlan(() => assert.fail('no API write or read needed'), - { projectId: 1, cases: [], plans: [plan] }, () => assert.fail('must not mark')), /plan_without_matching_case/); -}); - -test('new case is created only after alias reconciliation; an uncertain mutation is never resent', async () => { - let writes = 0, markers = 0; - await assert.rejects(ensureChatPlan(async (_route, method) => { - assert.equal(method, 'POST'); writes++; throw new Error('uncertain transport'); - }, { projectId: 1, cases: [], plans: [] }, async () => { markers++; }), /uncertain/); - assert.equal(writes, 1); assert.equal(markers, 1); -}); - -test('existing broad or conflicting plan never expands or changes its scope', async () => { - for (const actual of [{ ...plan, testCasesCount: 2 }, { ...plan, treeSelection: { leafsInclude: [92] } }, - { ...plan, treeSelection: { leafsInclude: [91], inverted: true } }, { ...plan, projectId: 2 }]) { - await assert.rejects(ensureChatPlan(async (_route, method) => { assert.equal(method, undefined); return actual; }, - { projectId: 1, cases: [caseRow], plans: [plan] }, () => assert.fail('no writes')), /scope_mismatch/); +test('retired singleton bootstrap cannot recreate an absent plan or mutate retained metadata', async () => { + for (const inspected of [ + { projectId: 1, cases: [], plans: [] }, + { projectId: 1, cases: [caseRow], plans: [] }, + { projectId: 1, cases: [caseRow], plans: [plan] }, + ]) { + await assert.rejects(ensureChatPlan(() => assert.fail('retired route must not call the API'), inspected, + () => assert.fail('retired route must not mark a mutation')), /testops_plan_retired/); } }); diff --git a/tests/runner-e2e/chat-feature-testops-cli.test.mjs b/tests/runner-e2e/chat-feature-testops-cli.test.mjs index 107f046b109..70238eb15f0 100644 --- a/tests/runner-e2e/chat-feature-testops-cli.test.mjs +++ b/tests/runner-e2e/chat-feature-testops-cli.test.mjs @@ -9,7 +9,7 @@ const context = { GITHUB_REPOSITORY: 'paperclipai/paperclip', GITHUB_REF: 'refs/ GITHUB_RUN_ATTEMPT: '1', GITHUB_SHA: 'a'.repeat(40), GITHUB_RUN_ID: '123', GITHUB_WORKFLOW_REF: 'paperclipai/paperclip/.github/workflows/chat-feature-testops.yml@refs/heads/master' }; -test('actual metadata bootstrap caller creates no result or agent and reuses exact case/plan', async t => { +test('actual retired singleton caller refuses plan recreation and creates no result', async t => { const temporary = await realpath(await mkdtemp(path.join(tmpdir(), 'testops-bootstrap-'))); t.after(() => rm(temporary, { recursive: true })); const state = path.join(temporary, 'state'), requests = []; @@ -30,9 +30,10 @@ test('actual metadata bootstrap caller creates no result or agent and reuses exa return response({ id: 8, projectId: 1, name: planName, testCasesCount: 1, treeSelection: { leafsInclude: [91], inverted: false } }); }; assert.equal(await run({ ...context, RUNNER_TEMP: temporary, CHAT_TESTOPS_STATE_ROOT: state, - CHAT_TESTOPS_OPERATION: 'bootstrap', GITHUB_EVENT_PATH: eventPath, ALLURE_JOB_RUN_ID: '9', ALLURE_TOKEN: 'SYNTHETIC_API_TOKEN' }, fetchImpl), 0); - const plan = JSON.parse(await readFile(path.join(state, 'plan.json'))); - assert.equal(plan.actualResult, null); + CHAT_TESTOPS_OPERATION: 'bootstrap', GITHUB_EVENT_PATH: eventPath, ALLURE_JOB_RUN_ID: '9', ALLURE_TOKEN: 'SYNTHETIC_API_TOKEN' }, fetchImpl), 1); + const failure = JSON.parse(await readFile(path.join(state, 'failure.json'))); + assert.equal(failure.code, 'testops_plan_retired'); + await assert.rejects(readFile(path.join(state, 'plan.json')), { code: 'ENOENT' }); assert.equal(requests.filter(value => value.method === 'POST').length, 1, 'only auth token exchange; no case/plan write'); await assert.rejects(readFile(path.join(state, 'allure-results.json')), { code: 'ENOENT' }); }); diff --git a/tests/runner-e2e/testops-chat-reporter.test.mjs b/tests/runner-e2e/testops-chat-reporter.test.mjs index 917d6912f90..65c32e7a7e9 100644 --- a/tests/runner-e2e/testops-chat-reporter.test.mjs +++ b/tests/runner-e2e/testops-chat-reporter.test.mjs @@ -244,18 +244,6 @@ test('post-watch foreign backend binding returns safe nonzero and preserves orig assert.deepEqual(await readFile(path.join(f.state, 'job.json')), f.originalJob); }); -test('bootstrap workflow captures official context only after watch and retains exactly named closed receipts', async () => { - const workflow = await readFile(new URL('../../.github/workflows/chat-feature-testops.yml', import.meta.url), 'utf8'); - const start = workflow.indexOf(' - name: Register the public job without an evaluation or a synthetic result'); - const end = workflow.indexOf(' - name: Validate the one real TestOps selection', start); - const step = workflow.slice(start, end); - assert.ok(step.includes("if: needs.authorize.outputs.operation == 'bootstrap'")); - assert.match(step, /allurectl watch -- node tests\/runner-e2e\/testops-chat-reporter\.mjs bootstrap\n\s+node tests\/runner-e2e\/testops-chat-reporter\.mjs bootstrap-context/); - assert.equal((step.match(/allurectl watch/g) ?? []).length, 1); assert.ok(!step.includes('continue-on-error')); - assert.ok(workflow.includes('${{ runner.temp }}/chat-testops-bootstrap-state/registration.json')); - assert.ok(workflow.includes('${{ runner.temp }}/chat-testops-bootstrap-state/registration-failure.json')); -}); - test('post-watch refuses nonprivate state custody before any command auth or receipt write', async t => { const f = await bootstrapContextFixture(t); await chmod(f.state, 0o755); const result = await captureSafeErrors(() => reporter('bootstrap-context', undefined, f.env, () => assert.fail('no API'), () => assert.fail('no CLI'))); diff --git a/tests/runner-e2e/testops-chat-workflow.test.ts b/tests/runner-e2e/testops-chat-workflow.test.ts deleted file mode 100644 index 14484cbd30e..00000000000 --- a/tests/runner-e2e/testops-chat-workflow.test.ts +++ /dev/null @@ -1,154 +0,0 @@ -import { readFileSync } from "node:fs"; -import { parse } from "yaml"; -import { expect, it } from "vitest"; -import { validateChatTestOpsDispatch } from "./testops-chat-dispatch.js"; - -const workflow = parse(readFileSync(new URL("../../.github/workflows/chat-feature-testops.yml", import.meta.url), "utf8")); -function context() { - return { env: { GITHUB_ACTIONS: "true", GITHUB_EVENT_NAME: "workflow_dispatch", GITHUB_REPOSITORY: "paperclipai/paperclip", GITHUB_REF: "refs/heads/master", - GITHUB_WORKFLOW_REF: "paperclipai/paperclip/.github/workflows/chat-feature-testops.yml@refs/heads/master", GITHUB_SHA: "a".repeat(40), GITHUB_RUN_ID: "12345", GITHUB_RUN_ATTEMPT: "1", GITHUB_ACTOR_ID: "123", TESTOPS_ALLOWED_ACTOR_IDS: "[123]" }, - event: { repository: { full_name: "paperclipai/paperclip", private: false, default_branch: "master" }, ref: "refs/heads/master", sender: { id: 123 }, inputs: { ALLURE_JOB_RUN_ID: "456" } as Record } }; -} -it("admits the actual TestOps service input and only the closed metadata bootstrap alternative", () => { - const f = context(); expect(validateChatTestOpsDispatch(f.env, f.event, 123).operation).toBe("evaluate"); - f.event.inputs = { operation: "bootstrap" }; expect(validateChatTestOpsDispatch(f.env, f.event, 123).operation).toBe("bootstrap"); - f.event.inputs.operation = "launch-plan"; expect(validateChatTestOpsDispatch(f.env, f.event, 123).operation).toBe("launch-plan"); - f.event.inputs.ALLURE_JOB_RUN_ID = "456"; expect(() => validateChatTestOpsDispatch(f.env, f.event, 123)).toThrow(); - delete f.event.inputs.ALLURE_JOB_RUN_ID; - f.event.inputs.operation = "other"; expect(() => validateChatTestOpsDispatch(f.env, f.event, 123)).toThrow(); -}); -it.each(["unlisted-trigger", "wrong-source", "replay", "extra-input", "missing-jobrun", "foreign-repo"])("rejects %s before any service credential operation", kind => { - const f = context(); let trigger = 123; - if (kind === "unlisted-trigger") trigger = 456; - if (kind === "wrong-source") f.env.GITHUB_REF = "refs/heads/unreviewed"; - if (kind === "replay") f.env.GITHUB_RUN_ATTEMPT = "2"; - if (kind === "extra-input") f.event.inputs.target_branch = "other"; - if (kind === "missing-jobrun") f.event.inputs = {}; - if (kind === "foreign-repo") f.event.repository.full_name = "foreign/repo"; - expect(() => validateChatTestOpsDispatch(f.env, f.event, trigger)).toThrow(); -}); -it("keeps provider and TestOps service credentials in separate jobs", () => { - const jobs = workflow.jobs; - for (const [id, job] of Object.entries(jobs) as Array<[string, any]>) { - const text = JSON.stringify(job); - if (text.includes("secrets.OPENROUTER_API_KEY")) expect(id).toBe("chat"); - if (text.includes("secrets.ALLURE_TOKEN")) expect(["select", "upload"]).toContain(id); - if (id !== "chat") expect(text).not.toContain("pnpm test:e2e:runner --"); - expect(job.steps.some((step: any) => step.run === "node tests/runner-e2e/testops-chat-dispatch.ts")).toBe(true); - } - expect(jobs.chat.environment).toBe("runner-e2e-private-paid"); - expect(jobs.select.environment).toBe("runner-e2e-testops-reporting"); - expect(jobs.upload.environment).toBe("runner-e2e-testops-reporting"); -}); -it("requires token-isolated selection before the fixed AWS run and keeps bootstrap outside all model jobs", () => { - expect(workflow.on).toHaveProperty("workflow_dispatch"); expect(Object.keys(workflow.on)).toEqual(["workflow_dispatch"]); - expect(Object.keys(workflow.on.workflow_dispatch.inputs)).toEqual(["operation", "ALLURE_JOB_RUN_ID", "ALLURE_USERNAME"]); - expect(workflow.on.workflow_dispatch.inputs.operation.default).toBe("evaluate"); - expect(workflow.jobs.chat.needs).toContain("select"); expect(workflow.jobs.build.needs).toContain("select"); - expect(workflow.jobs.chat.if).toBe("needs.authorize.outputs.operation == 'evaluate'"); - expect(workflow.jobs.build.if).toBe("needs.authorize.outputs.operation == 'evaluate'"); - const paid = workflow.jobs.chat.steps.find((s: any) => s.env?.OPENROUTER_API_KEY); - expect(paid.run).toContain("--id chat-features.runner-opencode-deepseek-1017.local.create-backlog --max-parallel 1 --max-automatic-retries 0"); - expect(workflow.jobs.chat.steps.findIndex((s: any) => s.name === "Verify immutable outputs and token-free TestOps selection")).toBeLessThan(workflow.jobs.chat.steps.indexOf(paid)); - expect(workflow.jobs.authorize.outputs.runner).toBe("runs-on/fleet=paperclip-public-pr-x64/env=public-ci"); - expect(workflow.jobs.chat["runs-on"]).toBe("${{ needs.authorize.outputs.runner }}"); -}); -it("uploads only the trusted safe projection after successful strict private report retention", () => { - const steps = workflow.jobs.report.steps; - expect(steps.find((s: any) => s.id === "export").if).toBe("always() && steps.private_report.outcome == 'success'"); - expect(steps.find((s: any) => s.with?.name?.startsWith("runner-e2e-testops-")).with.path).toBe("runner-e2e-testops/"); - expect(JSON.stringify(workflow)).not.toContain("MANUAL_PRIVATE_APPROVALS_JSON"); - expect(Object.values(workflow.jobs).flatMap((j: any) => j.steps).filter((s: any) => s.uses?.startsWith("actions/upload-artifact")).some((s: any) => s.with.path === "tests/runner-e2e/results/")).toBe(false); -}); - -it("initializes the actual TestOps association and never fabricates setup results", () => { - const steps = workflow.jobs.select.steps; - const installer = steps.find((s: any) => s.uses?.startsWith("allure-framework/setup-allurectl@")); - expect(installer.uses).toBe("allure-framework/setup-allurectl@2d89ee0b7ba2f59dbe9849b6df5763cab78e14db"); - expect(installer.with["allurectl-version"]).toBe("2.23.1"); - const bootstrap = steps.find((s: any) => s.run?.includes("testops-chat-reporter.mjs bootstrap")); - expect(bootstrap.run).toContain("mkdir -m 700"); - expect(bootstrap.run).toContain("allurectl watch -- node"); - expect(bootstrap.env.ALLURE_RESULTS).toBe("${{ runner.temp }}/chat-testops-bootstrap-results"); - const select = steps.find((s: any) => s.id === "selection"); - expect(select.run).toContain("allurectl watch -- node"); - for (const name of ["CHAT_TESTOPS_CASE_ID", "CHAT_TESTOPS_PLAN_ID", "CHAT_TESTOPS_JOB_ID"]) - expect(select.env[name]).toBe(`\${{ vars.${name} }}`); - const metadata = steps.find((s: any) => s.with?.name?.startsWith("chat-testops-setup-")); - expect(metadata.if).toBe("always() && needs.authorize.outputs.operation == 'bootstrap'"); - expect(metadata.with.path).not.toContain("results"); -}); -it("authenticates the same-run selection again before converting and uploading", () => { - const steps = workflow.jobs.upload.steps; - const receipt = steps.find((s: any) => s.with?.path === "chat-testops-selection"); - expect(receipt.with["artifact-ids"]).toBe("${{ needs.select.outputs.artifact_id }}"); - const upload = steps.find((s: any) => s.run?.includes("testops-chat-reporter.mjs upload")); - expect(upload.run).toContain("allurectl watch -- node"); - expect(upload.env.EXPECTED_SELECTION_SHA256).toBe("${{ needs.select.outputs.sha256 }}"); - expect(steps.indexOf(receipt)).toBeLessThan(steps.indexOf(upload)); - expect(JSON.stringify(workflow.jobs.select)).not.toContain("OPENROUTER_API_KEY"); - expect(JSON.stringify(workflow.jobs.upload)).not.toContain("OPENROUTER_API_KEY"); -}); - -it("keeps the one plan launch outside Allure watch and every AWS/model job", () => { - const steps = workflow.jobs.select.steps; - const launch = steps.find((s: any) => s.run?.endsWith("testops-chat-reporter.mjs launch-plan")); - expect(launch.if).toBe("needs.authorize.outputs.operation == 'launch-plan'"); - expect(launch.run).not.toContain("watch"); - expect(launch.env.CHAT_TESTOPS_OPERATION_PARAMETER_ID).toBe("${{ vars.CHAT_TESTOPS_OPERATION_PARAMETER_ID }}"); - expect(launch.env.CHAT_TESTOPS_INTEGRATION_ID).toBe("${{ vars.CHAT_TESTOPS_INTEGRATION_ID }}"); - expect(launch.env.CHAT_TESTOPS_LAUNCH_NONCE).toBe("${{ vars.CHAT_TESTOPS_LAUNCH_NONCE }}"); - expect(launch.env.CHAT_TESTOPS_LAUNCH_RUN_NUMBER).toBe("${{ vars.CHAT_TESTOPS_LAUNCH_RUN_NUMBER }}"); - expect(launch.env.TESTOPS_TRIGGERING_ACTOR_ID).toBe("${{ needs.authorize.outputs.triggering_actor_id }}"); - expect(launch.env.TESTOPS_ALLOWED_ACTOR_IDS).toBe("${{ vars.RUNNER_E2E_ALLOWED_ACTOR_IDS }}"); - expect(launch.env.GH_TOKEN).toBe("${{ github.token }}"); - expect(launch.run).not.toContain("sleep"); - expect(steps.find((s: any) => s.uses?.startsWith("allure-framework/setup-allurectl@")).if).toBe("needs.authorize.outputs.operation != 'launch-plan'"); - for (const name of ["build", "chat"]) expect(workflow.jobs[name].if).toBe("needs.authorize.outputs.operation == 'evaluate'"); - expect(workflow.concurrency["cancel-in-progress"]).toBe(false); -}); - -it("accepts the documented TestOps username metadata without treating it as an actor", () => { - const f = context(); f.event.inputs.ALLURE_USERNAME = "service-metadata"; - expect(validateChatTestOpsDispatch(f.env, f.event, 123).operation).toBe("evaluate"); - expect(() => validateChatTestOpsDispatch(f.env, f.event, 456)).toThrow(); - expect(JSON.stringify(workflow.jobs)).not.toContain("inputs.ALLURE_USERNAME"); -}); - -it("includes every native Node helper control in the normal unit owner", () => { - const launcher = readFileSync(new URL("./run-unit.mjs", import.meta.url), "utf8"); - const config = readFileSync(new URL("./vitest.config.ts", import.meta.url), "utf8"); - for (const file of ["chat-feature-testops-api.test.mjs", "chat-feature-testops-cli.test.mjs", "chat-feature-testops-results.test.mjs", "testops-chat-reporter.test.mjs"]) { - expect(launcher).toContain(`"tests/runner-e2e/${file}"`); - expect(config).toContain(`"${file}"`); - } -}); - -it("verifies the actual server outcome after watch without masking an original failed upload", () => { - const steps = workflow.jobs.upload.steps; - const watch = steps.find((s: any) => s.id === "allure_upload"); - const verify = steps.find((s: any) => s.id === "verify_upload"); - expect(watch.run).toContain("allurectl watch -- node"); - expect(watch["continue-on-error"]).toBeUndefined(); - expect(verify.run).toBe("node tests/runner-e2e/testops-chat-reporter.mjs verify-upload runner-e2e-testops"); - expect(verify.if).toBe("always() && !cancelled() && (steps.allure_upload.outcome == 'success' || steps.allure_upload.outcome == 'failure')"); - expect(verify.env).toEqual(watch.env); - expect(steps.indexOf(verify)).toBeGreaterThan(steps.indexOf(watch)); - const proof = steps.find((s: any) => s.with?.name?.startsWith("chat-testops-verification-")); - expect(proof.with.path.split("\n").filter(Boolean)).toHaveLength(2); - expect(proof.with.path).not.toContain("allure-results"); -}); - -it("carries only the existing source-reviewed lock through build and worker install", () => { - const build = workflow.jobs.build.steps; - const install = build.find((s: any) => s.run?.includes("pnpm install --frozen-lockfile")); - expect(install.run).toContain("node tests/runner-e2e/testops-chat-lock.ts > chat-testops-lock.json"); - expect(install.run).not.toContain("--lockfile-only"); - const archive = build.find((s: any) => s.run?.includes("tar -czf chat-testops-build")); - expect(archive.run).toContain("node tests/runner-e2e/testops-chat-lock.ts > chat-testops-lock.json"); - expect(archive.run).toContain("chat-testops-lock.json pnpm-lock.yaml"); - const worker = workflow.jobs.chat.steps.find((s: any) => s.name === "Verify immutable outputs and token-free TestOps selection"); - expect(worker.run).toContain("node tests/runner-e2e/testops-chat-lock.ts > chat-testops-worker-lock.json"); - expect(worker.run).toContain("cmp chat-testops-lock.json chat-testops-worker-lock.json"); - expect(worker.run.indexOf("cmp chat-testops-lock.json")).toBeLessThan(worker.run.indexOf("pnpm install --frozen-lockfile")); -}); diff --git a/tests/runner-e2e/workflow-security.test.ts b/tests/runner-e2e/workflow-security.test.ts index 91c0e3e9f29..ca84b9de5ab 100644 --- a/tests/runner-e2e/workflow-security.test.ts +++ b/tests/runner-e2e/workflow-security.test.ts @@ -606,8 +606,6 @@ describe("public repository paid workflow security", () => { "runner-full-stack-e2e.yml", "runner-live-evals.yml", "runner-protocol-live-evals.yml", - "chat-feature-testops.yml", - "chat-completion-testops.yml", ]); const names = (await readdir(workflowDirectory)).filter((name) => /\.ya?ml$/.test(name), @@ -893,24 +891,7 @@ describe("public repository paid workflow security", () => { for (const filename of await readdir(workflows)) { if (!/\.ya?ml$/u.test(filename) || ["runner-full-stack-e2e.yml", "runner-protocol-live-evals.yml"].includes(filename)) continue; const other = await readFile(path.join(workflows, filename), "utf8"); - if (["chat-feature-testops.yml", "chat-completion-testops.yml"].includes(filename)) { - const guard = filename === "chat-feature-testops.yml" ? "testops-chat-dispatch.ts" : "chat-completion-testops-cli.mjs guard"; - expect(other).toContain(`run: node tests/runner-e2e/${guard}`); - expect(other).toContain("ref: ${{ github.sha }}"); - expect(other).toContain("TESTOPS_ALLOWED_ACTOR_IDS: ${{ vars.RUNNER_E2E_ALLOWED_ACTOR_IDS }}"); - const chat = other.split("\n chat:")[1]!.split("\n report:")[0]!; - expect(chat).toContain("environment: runner-e2e-private-paid"); - expect(chat).toContain("needs: [authorize, select, build]"); - expect(chat).toContain(" - *guard"); - expect(chat).toContain("verify-selection"); - expect(chat).not.toContain("ALLURE_TOKEN"); - expect(chat.indexOf("verify-selection")).toBeLessThan(chat.indexOf("secrets.OPENROUTER_API_KEY")); - const report = other.split("\n report:")[1]!.split("\n upload:")[0]!; - expect(report).toContain("environment: runner-e2e-private-history"); - expect(report).toContain("manual-private-transfer.ts report-session"); - expect(report).not.toContain("ALLURE_TOKEN"); - continue; - } + for (const environment of environments) expect(other, filename).not.toContain(environment); } const protocol = await readFile(path.join(workflows, "runner-protocol-live-evals.yml"), "utf8"); From b9e94b85a69bf679818eeef9ecf4655554a89c6b Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Mon, 2 Nov 2026 11:32:03 -0600 Subject: [PATCH 04/35] feat(observability): trace Daytona lease acquisition and resume (#18626) ## Thinking Path > - Paperclip manages work performed by AI agents. > - The server prepares a remote execution environment before the agent can respond. > - OpenTelemetry traces show the time spent in each preparation phase. > - A retained Daytona sandbox can take minutes to resume from an archive. > - The provider worker previously exposed no child timings for this operation. > - This pull request adds bounded lease phase spans under the existing worker trace. > - Operators can separate provider startup from lookup, teardown waits, and workspace verification. ## Linked Issues or Issue Description **What existing behavior does this improve?** Daytona environment acquisition and resume tracing. A long worker invocation currently hides the operation that consumed its time. Related tracing foundation: #10999. No duplicate open PR was found. **Subsystem affected** The Daytona sandbox provider and the server's existing plugin span validation boundary. **Proposed behavior** Show lookup, start or recovery, teardown waits, workspace verification, and fresh acquisition setup as timed child spans. Group verification steps beneath one parent. Record the bounded provider state before start, including archived. **Reason and benefit** An operator can identify which step delayed the first agent response. SDK startup includes its internal readiness wait; this change does not claim to measure provider queue time separately. **Breaking changes** The opt-in trace schema gains fixed span names and one closed state attribute. Execution, authorization, resource ownership, deadlines, and retry policies keep their existing contracts. ## What Changed - Add lease phase spans through the existing invocation tracer. - Add the fixed span names and state values to host validation. Define the state key in the shared startup attribute contract. - Preserve the exact work result or error if tracing fails. Never repeat work through the tracing wrapper. - Test worker receipts for archived and warm resume, recovery, concurrency, failure, disabled tracing, and cleanup. - Document the waterfall and its attribution limits. ## Verification - Daytona plugin suite: 525 passed, 7 skipped. The expanded tracing suite separately passes all 12 cases. - Host span validation and instrumentation tests: 46 passed, 3 skipped. - Standalone Daytona TypeScript check passes. - Module boundary and whitespace checks pass. - Full recursive typecheck and full build pass. - The full local Vitest attempt at b007a8e64f20f9fdadb457cc3ce1940208359d65 stopped in chat shard 1: 337 passed, 2 failed, 1478 skipped. One retry-notice case exceeded 15 seconds. The next case then reported that fake timers were no longer installed. A focused rerun of all four retry-notice cases passed. This is not a full local suite pass. - Final head 8e1f81523b3869b9877b44d22c2cb7c938abf014 passes CI and Greptile 5/5, with no unresolved review threads. The CI chat shards pass despite the preserved local timeout attempt. - The expanded lease tracing suite passes 15 tests; host/shared tracing suites pass 66 tests. Disabling the wait flag makes the new teardown test fail, and restoring it passes. Standalone plugin typecheck passes. - The tests drive the real plugin worker RPC and invocation tracer with a controlled provider fixture. They validate receipts and host parent IDs. They do not establish deployment or Honeycomb ingestion. Inspect a new deployed run to verify the exported waterfall. ## Risks - A new provider phase must be added to the closed host allowlist or it appears as other. - SDK-internal start and readiness work remains one measured operation. - Older workers without withSpan run normally without these new detail spans. - Receipt budgets and clock validation can omit detail from very large or invalid receipts. ## Model Used OpenAI Codex, GPT-6, with repository tools and code execution. The exact runtime model ID and context window were not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/observability.md | 44 +++ .../src/acpx-engine/startup-timing.ts | 2 + .../daytona/src/lease-tracing.test.ts | 273 ++++++++++++++++++ .../daytona/src/lease-tracing.ts | 49 ++++ .../sandbox-providers/daytona/src/plugin.ts | 117 ++++---- .../plugin-host-services-span.test.ts | 30 ++ server/src/services/plugin-host-services.ts | 8 + 7 files changed, 471 insertions(+), 52 deletions(-) create mode 100644 packages/plugins/sandbox-providers/daytona/src/lease-tracing.test.ts create mode 100644 packages/plugins/sandbox-providers/daytona/src/lease-tracing.ts diff --git a/doc/observability.md b/doc/observability.md index a96ae96058b..abadb0e6538 100644 --- a/doc/observability.md +++ b/doc/observability.md @@ -1900,6 +1900,49 @@ name to `sandbox.daytona.`. The host maps any other value to daytona provider emits these spans today, so the segment is the literal `daytona`. +### Daytona lease acquisition and resume + +The worker invocation carries nested, monotonic timing receipts for lease work. +The host also admits the following fixed `lease.*` names and prefixes them with +`sandbox.daytona.`. These use the existing opt-in endpoint and host-owned parent +context; they do not create an exporter or first-party Telemetry events. + +```text +execution.plugin_dispatch +└── plugin.worker + ├── sandbox.daytona.lease.wait_for_teardown (only when teardown blocks resume) + ├── sandbox.daytona.lease.lookup + ├── sandbox.daytona.lease.refresh (missing-container confirmation) + ├── sandbox.daytona.lease.channel_cleanup (before restart) + ├── sandbox.daytona.lease.start (or lease.recover) + └── sandbox.daytona.lease.verify + ├── sandbox.daytona.lease.workspace + ├── sandbox.daytona.lease.sentinel + └── sandbox.daytona.lease.shell +``` + +`lease.lookup` includes cached-handle lookup and any required provider refresh. +`lease.start` and `lease.recover` measure the actual SDK operation, including its +internal readiness wait. They also cover starts triggered by execution or sync. +They do not separate SDK-internal requests or provider-side queue time. +Their `paperclip.sandbox.startup.state` attribute records the state before the +call: `started`, `stopped`, `archived`, `error`, or `other`. Both the plugin and +host restrict this value; no provider error text or resource identity is sent. + +Fresh acquisition emits `lease.create`, `lease.workspace`, `lease.shell`, +`lease.expiry`, and `lease.sentinel`. `lease.create` covers the existing creation +helper, including qualification-only snapshot resolution and creation-failure +cleanup. Setup-failure deletion is a separate `lease.cleanup` sibling, so it +does not inherit a failed workspace or verification phase. Existing deadlines, +ownership, cleanup and retry behavior are unchanged. Failed awaited steps close +with error status; the receipt contains no exception message. + +Children nest beneath the current `plugin.worker` receipt and verification +children nest beneath `lease.verify`. Never add parent and child durations. +Receipt budgets and clock validation still apply. Older workers without +`tracer.withSpan` keep the original behavior without these detail spans. Tracing +failures preserve the operation's result or error and never repeat it. + The `sandbox.daytona.*` spans use this closed attribute allowlist. The host drops every other key, so a command, an argument, a path, an id, a standard output, or a standard error never rides a provider span. The host records only @@ -1908,6 +1951,7 @@ the attributes that the producer sends for one span. | Attribute | Type | Optional | Meaning | | ------------------------------------------------ | ------ | -------- | --------------------------------------------------------------------------------------------------------- | | `paperclip.sandbox.startup.provider` | string | no | The normalized provider family. | +| `paperclip.sandbox.startup.state` | string | yes | State before start/recovery: `started`, `stopped`, `archived`, `error`, or `other`. | | `paperclip.sandbox.startup.outcome` | string | yes | The step outcome (`ok`, `skipped`, or `failed`). | | `paperclip.sandbox.startup.pack.wall_ms` | number | yes | The host-local wall time of the pack step. It rides the `sandbox.daytona.pack` span. | | `paperclip.sandbox.startup.transfer.wall_ms` | number | yes | The wall time of the transfer step. It rides the `sandbox.daytona.transfer` span. | diff --git a/packages/adapter-utils/src/acpx-engine/startup-timing.ts b/packages/adapter-utils/src/acpx-engine/startup-timing.ts index 74f1977c5c3..5ce26855ef2 100644 --- a/packages/adapter-utils/src/acpx-engine/startup-timing.ts +++ b/packages/adapter-utils/src/acpx-engine/startup-timing.ts @@ -67,6 +67,8 @@ export const EXECUTION_PREPARATION_SPAN_ATTRS = { * command, a path, an argument, or an environment value can never ride a span. */ export const SANDBOX_STARTUP_SPAN_ATTRS = { + /** Provider state before a lease start or recovery; the host clamps values. */ + state: `${SANDBOX_STARTUP_SPAN_ATTR_PREFIX}state`, /** The low-cardinality provider family (through `normalizeProviderFamily`). */ provider: `${SANDBOX_STARTUP_SPAN_ATTR_PREFIX}provider`, /** The step or execution outcome: `ok`, `skipped`, or `failed`. */ diff --git a/packages/plugins/sandbox-providers/daytona/src/lease-tracing.test.ts b/packages/plugins/sandbox-providers/daytona/src/lease-tracing.test.ts new file mode 100644 index 00000000000..dbb8b9eac18 --- /dev/null +++ b/packages/plugins/sandbox-providers/daytona/src/lease-tracing.test.ts @@ -0,0 +1,273 @@ +import { createInterface } from "node:readline"; +import { PassThrough } from "node:stream"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { NOOP_PLUGIN_TRACER, startWorkerRpcHost, type PluginInvocationContext, type PluginTracer } from "@paperclipai/plugin-sdk"; +import plugin, { __resetDaytonaSandboxHandleCacheForTest, __setDaytonaPluginContextForTest } from "./plugin.js"; +import manifest from "./manifest.js"; +import { leaseStateAttributes, withLeaseSpan } from "./lease-tracing.js"; + +const { get, create } = vi.hoisted(() => ({ get: vi.fn(), create: vi.fn() })); +vi.mock("@daytonaio/sdk", () => ({ + Daytona: class { get = get; create = create; }, + DaytonaNotFoundError: class extends Error {}, + DaytonaTimeoutError: class extends Error {}, +})); + +type Receipt = { + name: string; + startTimeMs: number; + endTimeMs: number; + attributes?: Record; + status?: { code: number }; + children: Receipt[]; +}; + +function sandbox(id = "fixture-sandbox", state = "archived") { + return { + id, state, target: "us", recoverable: true, errorReason: null as string | null, + start: vi.fn().mockResolvedValue(undefined), + stop: vi.fn().mockResolvedValue(undefined), + refreshData: vi.fn().mockResolvedValue(undefined), + recover: vi.fn().mockResolvedValue(undefined), + delete: vi.fn().mockResolvedValue(undefined), + getWorkDir: vi.fn().mockResolvedValue("/home/daytona"), + fs: { createFolder: vi.fn().mockResolvedValue(undefined), uploadFile: vi.fn().mockResolvedValue(undefined) }, + process: { executeCommand: vi.fn(async (command: string) => ({ + exitCode: 0, result: command.startsWith("cat ") ? '{"token":"private-token"}' : "bash", + })) }, + }; +} + +const params = { + driverKey: "daytona", companyId: "fixture-company", environmentId: "fixture-environment", + providerLeaseId: "fixture-sandbox", config: { apiKey: "private-api-key", timeoutMs: 300000, reuseLease: true }, + leaseMetadata: { workspaceSentinel: { token: "private-token", result: "written" } }, +}; + +function workerHarness() { + const stdin = new PassThrough(), stdout = new PassThrough(); + const lines = createInterface({ input: stdout }); + const pending = new Map(); + const receipts: Array<{ invocationId: string; span: Receipt }> = []; + let sequence = 0; + lines.on("line", line => { + const message = JSON.parse(line); + if (message.method) { + if (message.method === "span.record") receipts.push({ invocationId: message.paperclipInvocationId, span: message.params }); + if (message.id !== undefined) stdin.write(JSON.stringify({ jsonrpc: "2.0", id: message.id, result: null }) + "\n"); + } else { + const waiter = pending.get(message.id); + pending.delete(message.id); + if (message.error) waiter?.reject(new Error(message.error.message)); + else waiter?.resolve(message.result); + } + }); + const worker = startWorkerRpcHost({ plugin, stdin, stdout }); + function call(method: string, input: unknown, invocation?: PluginInvocationContext): Promise { + const id = String(++sequence); + return new Promise((resolve, reject) => { + pending.set(id, { resolve, reject }); + stdin.write(JSON.stringify({ jsonrpc: "2.0", id, method, params: input, + ...(invocation ? { paperclipInvocation: invocation } : {}) }) + "\n"); + }); + } + return { + call, receipts, + initialize: () => call("initialize", { manifest, config: {}, apiVersion: 1, + instanceInfo: { instanceId: "fixture", hostVersion: "0.0.0" } }), + close() { worker.stop(); lines.close(); stdin.destroy(); stdout.destroy(); }, + }; +} + +function invocation(id: string, enabled = true): PluginInvocationContext { + return { id, scope: { companyId: "fixture-company" }, + ...(enabled ? { traceparent: "00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01" } : {}) }; +} + +describe("Daytona lease worker tracing", () => { + beforeEach(() => { get.mockReset(); create.mockReset(); __resetDaytonaSandboxHandleCacheForTest(); }); + afterEach(() => { __setDaytonaPluginContextForTest(null); __resetDaytonaSandboxHandleCacheForTest(); }); + + it.each([true, false])("records the archived start and nested verification only with trace context: %s", async enabled => { + const resource = sandbox(); get.mockResolvedValue(resource); + const worker = workerHarness(); + try { + await worker.initialize(); + expect(await worker.call("environmentResumeLease", params, invocation("resume", enabled))) + .toMatchObject({ providerLeaseId: resource.id, metadata: { resumedFromState: "archived" } }); + expect(resource.start).toHaveBeenCalledExactlyOnceWith(300); + expect(get).toHaveBeenCalledExactlyOnceWith(resource.id); + if (!enabled) { expect(worker.receipts).toEqual([]); return; } + expect(worker.receipts).toHaveLength(1); + const root = worker.receipts[0].span; + expect(root.name).toBe("plugin.worker"); + expect(root.children.map(span => span.name)).toEqual(["lease.lookup", "lease.channel_cleanup", "lease.start", "lease.verify"]); + expect(root.children[2].attributes).toMatchObject({ "paperclip.sandbox.startup.state": "archived" }); + expect(root.children[3].children.map(span => span.name)).toEqual(["lease.workspace", "lease.sentinel", "lease.shell"]); + function checkBounds(parent: Receipt) { + for (const child of parent.children) { + expect(child.startTimeMs).toBeGreaterThanOrEqual(parent.startTimeMs); + expect(child.endTimeMs).toBeGreaterThanOrEqual(child.startTimeMs); + expect(child.endTimeMs).toBeLessThanOrEqual(parent.endTimeMs); + checkBounds(child); + } + } + checkBounds(root); + expect(JSON.stringify(root)).not.toMatch(/private-|fixture-sandbox|fixture-company|\/home\//); + } finally { worker.close(); } + }); + + it("keeps simultaneous resume receipts separate and closes failed starts", async () => { + const slow = sandbox(), failed = sandbox("failed-sandbox", "stopped"); + let release!: () => void, started!: () => void; + const waiting = new Promise(resolve => { release = resolve; }); + const entered = new Promise(resolve => { started = resolve; }); + slow.start.mockImplementation(async () => { started(); await waiting; }); + failed.start.mockRejectedValue(new Error("private-provider-error")); + get.mockImplementation(async id => id === slow.id ? slow : failed); + const worker = workerHarness(); + try { + await worker.initialize(); + const first = worker.call("environmentResumeLease", params, invocation("slow")); + await entered; + await expect(worker.call("environmentResumeLease", { ...params, providerLeaseId: failed.id }, invocation("failed"))) + .rejects.toThrow("private-provider-error"); + release(); await first; + expect(worker.receipts.map(record => record.invocationId)).toEqual(["failed", "slow"]); + const failure = worker.receipts[0].span; + expect(failure.status).toEqual({ code: 2 }); + expect(failure.children.find(span => span.name === "lease.start")?.status).toEqual({ code: 2 }); + expect(failure.children.some(span => span.name === "lease.verify")).toBe(false); + expect(JSON.stringify(worker.receipts)).not.toContain("private-provider-error"); + expect(slow.start).toHaveBeenCalledTimes(1); + expect(failed.start).toHaveBeenCalledTimes(1); + } finally { release(); worker.close(); } + }); + + it("waits for a pending teardown and records the wait under the resume worker", async () => { + const resource = sandbox("fixture-sandbox", "started"); get.mockResolvedValue(resource); + let release!: () => void, stopping!: () => void; + const waiting = new Promise(resolve => { release = resolve; }); + const entered = new Promise(resolve => { stopping = resolve; }); + resource.stop.mockImplementation(async () => { stopping(); await waiting; resource.state = "stopped"; }); + const worker = workerHarness(); + try { + await worker.initialize(); + const teardown = worker.call("environmentReleaseLease", params, invocation("release")); + await entered; + let settled = false; + const resume = worker.call("environmentResumeLease", params, invocation("resume")) + .then(result => { settled = true; return result; }); + // Process the resume request while the actual provider stop is held open. + await new Promise(resolve => setImmediate(resolve)); + expect(settled).toBe(false); + expect(get).toHaveBeenCalledTimes(1); + expect(resource.start).not.toHaveBeenCalled(); + release(); + await teardown; + expect(await resume).toMatchObject({ providerLeaseId: resource.id }); + const root = worker.receipts.find(receipt => receipt.invocationId === "resume")!.span; + expect(root.name).toBe("plugin.worker"); + expect(root.children.filter(span => span.name === "lease.wait_for_teardown")).toHaveLength(1); + expect(root.children[0].name).toBe("lease.wait_for_teardown"); + expect(root.children[0].endTimeMs).toBeLessThanOrEqual(root.children[1].startTimeMs); + expect(get).toHaveBeenCalledTimes(2); + expect(resource.start).toHaveBeenCalledTimes(1); + } finally { release(); worker.close(); } + }); + + it.each([false, true])("records missing-container confirmation and preserves refresh failures: %s", async fails => { + const resource = sandbox("fixture-sandbox", "error"); + resource.recoverable = false; + resource.errorReason = `not found: failed to inspect sandbox container ${resource.id}: Error response from daemon: No such container: ${resource.id}`; + get.mockResolvedValue(resource); + resource.refreshData.mockImplementation(async () => { + if (fails) throw new Error("private-refresh-failure"); + resource.state = "started"; + }); + const worker = workerHarness(); + try { + await worker.initialize(); + const result = worker.call("environmentResumeLease", params, invocation("resume")); + if (fails) await expect(result).rejects.toThrow("private-refresh-failure"); + else expect(await result).toMatchObject({ providerLeaseId: resource.id }); + expect(resource.refreshData).toHaveBeenCalledTimes(1); + const root = worker.receipts[0].span; + expect(root.children.map(span => span.name)).toEqual(fails + ? ["lease.lookup", "lease.refresh"] : ["lease.lookup", "lease.refresh", "lease.verify"]); + expect(root.children[1].status).toEqual(fails ? { code: 2 } : undefined); + expect(resource.start).not.toHaveBeenCalled(); + expect(resource.delete).not.toHaveBeenCalled(); + expect(create).not.toHaveBeenCalled(); + expect(JSON.stringify(worker.receipts)).not.toContain("private-refresh-failure"); + } finally { worker.close(); } + }); + + it.each(["started", "error"])("traces a %s sandbox without adding a provider start", async state => { + const resource = sandbox("fixture-sandbox", state); get.mockResolvedValue(resource); + const worker = workerHarness(); + try { + await worker.initialize(); + await worker.call("environmentResumeLease", params, invocation("resume")); + expect(resource.start).not.toHaveBeenCalled(); + expect(resource.recover).toHaveBeenCalledTimes(state === "error" ? 1 : 0); + const phases = worker.receipts[0].span.children; + expect(phases.some(span => span.name === "lease.start")).toBe(false); + expect(phases.some(span => span.name === "lease.recover")).toBe(state === "error"); + } finally { worker.close(); } + }); + + it("closes the verification group and failing workspace child", async () => { + const resource = sandbox(); get.mockResolvedValue(resource); + resource.getWorkDir.mockRejectedValue(new Error("private-workspace-error")); + const worker = workerHarness(); + try { + await worker.initialize(); + await expect(worker.call("environmentResumeLease", params, invocation("resume"))) + .rejects.toThrow("private-workspace-error"); + const verify = worker.receipts[0].span.children.find(span => span.name === "lease.verify")!; + expect(verify.status).toEqual({ code: 2 }); + expect(verify.children.map(span => span.name)).toEqual(["lease.workspace"]); + expect(verify.children[0].status).toEqual({ code: 2 }); + expect(resource.delete).not.toHaveBeenCalled(); + } finally { worker.close(); } + }); + + it("records setup failure and compensating cleanup as siblings", async () => { + const resource = sandbox(); create.mockResolvedValue(resource); + resource.getWorkDir.mockRejectedValue(new Error("private-workspace-error")); + const worker = workerHarness(); + try { + await worker.initialize(); + await expect(worker.call("environmentAcquireLease", { ...params, runId: "fixture-run" }, invocation("acquire"))) + .rejects.toThrow("private-workspace-error"); + expect(resource.delete).toHaveBeenCalledTimes(1); + const root = worker.receipts[0].span; + expect(root.children.map(span => span.name)).toEqual(["lease.create", "lease.workspace", "lease.cleanup"]); + expect(root.children[1].status).toEqual({ code: 2 }); + expect(root.children[2].status).toBeUndefined(); + } finally { worker.close(); } + }); +}); + +describe("lease tracing cannot change provider work", () => { + it.each(["missing", "before", "after", "during"])("preserves the result and error with a %s tracer failure", async mode => { + const tracer: PluginTracer = { ...NOOP_PLUGIN_TRACER, + ...(mode === "missing" ? {} : { async withSpan(_name: string, work: () => T | Promise): Promise { + if (mode === "before") throw new Error("trace failure"); + const result = work(); + if (mode === "during") throw new Error("trace failure"); + await result; + throw new Error("trace failure"); + } }), + }; + const success = vi.fn(async () => 42), error = new Error("provider failure"), failure = vi.fn(async () => { throw error; }); + expect(await withLeaseSpan(tracer, "lease.lookup", success)).toBe(42); + await expect(withLeaseSpan(tracer, "lease.lookup", failure)).rejects.toBe(error); + expect(success).toHaveBeenCalledTimes(1); expect(failure).toHaveBeenCalledTimes(1); + }); + + it("normalizes unknown provider states before recording", () => { + expect(leaseStateAttributes("provider-private-state")).toEqual({ "paperclip.sandbox.startup.state": "other" }); + }); +}); diff --git a/packages/plugins/sandbox-providers/daytona/src/lease-tracing.ts b/packages/plugins/sandbox-providers/daytona/src/lease-tracing.ts new file mode 100644 index 00000000000..8ad09499f64 --- /dev/null +++ b/packages/plugins/sandbox-providers/daytona/src/lease-tracing.ts @@ -0,0 +1,49 @@ +import type { PluginTracer } from "@paperclipai/plugin-sdk"; + +export type LeaseSpanName = + | "lease.wait_for_teardown" + | "lease.lookup" + | "lease.refresh" + | "lease.channel_cleanup" + | "lease.start" + | "lease.recover" + | "lease.verify" + | "lease.workspace" + | "lease.sentinel" + | "lease.shell" + | "lease.create" + | "lease.expiry" + | "lease.cleanup"; + +/** Keep provider state useful without exporting a new SDK value as free text. */ +export function leaseStateAttributes(state: unknown): Record { + return { + "paperclip.sandbox.startup.state": + state === "started" || state === "stopped" || state === "archived" || state === "error" + ? state : "other", + }; +} + +/** Use the invocation's async parent, while keeping tracer faults out of work. */ +export async function withLeaseSpan( + tracer: PluginTracer, + name: LeaseSpanName, + work: () => Promise, + attributes?: Record, +): Promise { + let result: Promise | undefined; + // A tracer can fail before or after invoking work. Preserve its exact result + // or error and never repeat provider operations in either case. + const runOnce = () => result ??= Promise.resolve().then(work); + try { + if (tracer.withSpan) { + await tracer.withSpan(name, runOnce, { + "paperclip.sandbox.startup.provider": "daytona", + ...attributes, + }); + } + } catch { + // The operation's error is still held by result and is rethrown below. + } + return runOnce(); +} diff --git a/packages/plugins/sandbox-providers/daytona/src/plugin.ts b/packages/plugins/sandbox-providers/daytona/src/plugin.ts index 74dd8029c15..55f1aa8eaf0 100644 --- a/packages/plugins/sandbox-providers/daytona/src/plugin.ts +++ b/packages/plugins/sandbox-providers/daytona/src/plugin.ts @@ -50,6 +50,7 @@ import type { } from "@paperclipai/plugin-sdk"; import { performSyncIn, performSyncOut, withProviderSpan } from "./file-sync.js"; import { DEFAULT_DAYTONA_OPERATION_TIMEOUT_MS } from "./manifest.js"; +import { withLeaseSpan, leaseStateAttributes, type LeaseSpanName } from "./lease-tracing.js"; import { observeSessionStreamExit } from "./session-stream-exit.js"; import { DaytonaIdleActivity, trackDaytonaHooks, type DaytonaIdleProof } from "./idle-activity.js"; @@ -126,6 +127,10 @@ export function getPluginTracer(): PluginTracer { return pluginContext?.tracer ?? NOOP_PLUGIN_TRACER; } +function traceLeaseStep(name: LeaseSpanName, work: () => Promise, attributes?: Record): Promise { + return withLeaseSpan(getPluginTracer(), name, work, attributes); +} + /** * Test seam: set the module-level plugin context, and return a restore function. * `plugin.test.ts` uses it to inject a recording tracer without running `setup`. @@ -528,12 +533,16 @@ async function ensureSandboxStarted(sandbox: Sandbox, timeoutSeconds: number): P const startBoundMs = timeoutSeconds * 1_000 + LIVENESS_START_TIMEOUT_MARGIN_MS; if (sandbox.state === "error") { if (sandbox.recoverable) { - await withLivenessTimeout("sandbox.recover", startBoundMs, () => sandbox.recover(timeoutSeconds)); + await traceLeaseStep("lease.recover", () => + withLivenessTimeout("sandbox.recover", startBoundMs, () => sandbox.recover(timeoutSeconds)), + leaseStateAttributes(sandbox.state)); return; } throw new Error(`Daytona sandbox ${sandbox.id} is in an unrecoverable error state: ${sandbox.errorReason ?? "unknown error"}`); } - await withLivenessTimeout("sandbox.start", startBoundMs, () => sandbox.start(timeoutSeconds)); + await traceLeaseStep("lease.start", () => + withLivenessTimeout("sandbox.start", startBoundMs, () => sandbox.start(timeoutSeconds)), + leaseStateAttributes(sandbox.state)); } function hasMissingSandboxContainer(sandbox: Sandbox): boolean { @@ -1362,6 +1371,7 @@ const sandboxHandleActivityGates = (() => { type SandboxLeaseAdmissionOptions = { allowClosed?: boolean; + traceLeaseWait?: boolean; }; const sandboxHandleLeaseAdmissionStates = (() => { @@ -1402,7 +1412,8 @@ async function withSandboxActivityGate( const teardownGate = sandboxHandleTeardownGates.current(scope); if (teardownGate) { - await teardownGate.promise; + if (options.traceLeaseWait) await traceLeaseStep("lease.wait_for_teardown", () => teardownGate.promise); + else await teardownGate.promise; if (!options.allowClosed && sandboxHandleLeaseAdmissionStates.isClosed(scope)) { throw new Error(`Daytona sandbox lease ${scope.providerLeaseId} is no longer active.`); } @@ -2682,39 +2693,39 @@ const plugin = definePlugin(trackDaytonaHooks({ return timeoutFailure(); }; const acquire = async (): Promise => { - const sandbox = await createSandbox(params, config, { + const sandbox = await traceLeaseStep("lease.create", () => createSandbox(params, config, { onCreateAttempt: (attempt) => { assertActive(); cleanup = attempt; }, onCreateFailure: observeFailure, onSnapshotIdentity: identity => { snapshotIdentity = identity; }, - }); + })); try { assertActive(); if (cleanup) cleanup.observedProviderLeaseId = sandbox.id; phase = "workspace"; - const remoteCwd = await resolveSandboxWorkingDirectory(sandbox); + const remoteCwd = await traceLeaseStep("lease.workspace", () => resolveSandboxWorkingDirectory(sandbox)); assertActive(); phase = "shell"; - const shellCommand = await detectSandboxShellCommand(sandbox, toTimeoutSeconds(Math.max(1, deadline - Date.now()))); + const shellCommand = await traceLeaseStep("lease.shell", () => detectSandboxShellCommand(sandbox, toTimeoutSeconds(Math.max(1, deadline - Date.now())))); assertActive(); // Configure a provider-side destroy time at or before a caller deadline, so // an abandoned sandbox self-destroys even if Paperclip is down. The lease // carries the real provider expiry (or none) as evidence of the bound. phase = "expiry"; - const expiresAt = await configureSandboxExpiry({ + const expiresAt = await traceLeaseStep("lease.expiry", () => configureSandboxExpiry({ sandbox, requestedExpiresAt: params.requestedExpiresAt, nowMs: Date.now(), preserveCreationExpiry: config.ttlMinutes != null, - }); + })); assertActive(); phase = "sentinel"; - const workspaceSentinel = await writeWorkspaceSentinel({ + const workspaceSentinel = await traceLeaseStep("lease.sentinel", () => writeWorkspaceSentinel({ sandbox, remoteCwd, params, config, timeoutSeconds: toTimeoutSeconds(Math.max(1, deadline - Date.now())), - }); + })); assertActive(); sandboxHandleLeaseAdmissionStates.open({ driverKey: params.driverKey, @@ -2757,7 +2768,7 @@ const plugin = definePlugin(trackDaytonaHooks({ cleanupInProgress = true; try { const proof = idleActivity.proof(providerActivityKey(params, config, sandbox.id)); - await sandbox.delete(toTimeoutSeconds(Math.max(1, deadline - Date.now())), true); + await traceLeaseStep("lease.cleanup", () => sandbox.delete(toTimeoutSeconds(Math.max(1, deadline - Date.now())), true)); idleActivity.confirm(proof); } catch (cleanupError) { if (cleanup) { @@ -2821,7 +2832,7 @@ const plugin = definePlugin(trackDaytonaHooks({ throw new Error("The stopped Daytona sandbox is still settling cancelled work. Retry shortly."); } return await withSandboxActivityGate(scope, async () => { - const sandbox = await getSandboxOrNull(scope, { bypassTeardownGate: true }); + const sandbox = await traceLeaseStep("lease.lookup", () => getSandboxOrNull(scope, { bypassTeardownGate: true })); if (!sandbox) { return { providerLeaseId: null, metadata: { expired: true } }; } @@ -2831,7 +2842,7 @@ const plugin = definePlugin(trackDaytonaHooks({ // backup-guarded replacement path. Unknown errors preserve the lease. if (hasMissingSandboxContainer(sandbox)) { try { - await withLivenessTimeout("sandbox.refreshData", config.livenessTimeoutMs, () => sandbox.refreshData()); + await traceLeaseStep("lease.refresh", () => withLivenessTimeout("sandbox.refreshData", config.livenessTimeoutMs, () => sandbox.refreshData())); assertHandleMatchesLease(sandbox, params.providerLeaseId); } catch (error) { evictSandboxHandle(scope); @@ -2859,48 +2870,50 @@ const plugin = definePlugin(trackDaytonaHooks({ // A stopped sandbox loses its pseudo-terminals, so a stored duplex channel // is dead after a real restart. Close and drop every channel on this lease // before the restart, so no stale channel id survives the resume. - await closeDaytonaDuplexChannelsForLease(params.providerLeaseId); + await traceLeaseStep("lease.channel_cleanup", () => closeDaytonaDuplexChannelsForLease(params.providerLeaseId)); } const resumedFromState = sandbox.state ?? null; await ensureSandboxStarted(sandbox, toTimeoutSeconds(config.timeoutMs)); try { - const remoteCwd = await resolveSandboxWorkingDirectory(sandbox, false); - // C3: a resumed lease must clear the workspace sentinel before it is - // trusted, even when the handle came from the cache. On any non-match we - // evict the cached handle and expire the lease so a stale/foreign sandbox - // is never reused on the subsequent (sentinel-skipping) exec path. - const workspaceSentinel = await verifyWorkspaceSentinel({ - sandbox, - remoteCwd, - leaseMetadata: params.leaseMetadata, - timeoutSeconds: toTimeoutSeconds(config.timeoutMs), - }); - if (workspaceSentinel.result !== "matched") { - evictSandboxHandle(scope); - return { - providerLeaseId: null, - metadata: { expired: true, workspaceSentinel }, - }; - } - const shellCommand = await detectSandboxShellCommand( - sandbox, - toTimeoutSeconds(config.timeoutMs), - ); - sandboxHandleCache.markFresh(scope); - sandboxHandleLeaseAdmissionStates.open(scope); - return { - providerLeaseId: sandbox.id, - metadata: leaseMetadata({ - config, + return await traceLeaseStep("lease.verify", async () => { + const remoteCwd = await traceLeaseStep("lease.workspace", () => resolveSandboxWorkingDirectory(sandbox, false)); + // C3: a resumed lease must clear the workspace sentinel before it is + // trusted, even when the handle came from the cache. On any non-match we + // evict the cached handle and expire the lease so a stale/foreign sandbox + // is never reused on the subsequent (sentinel-skipping) exec path. + const workspaceSentinel = await traceLeaseStep("lease.sentinel", () => verifyWorkspaceSentinel({ sandbox, - shellCommand, - remoteCwd: requestedCwd ?? remoteCwd, - resumedLease: true, - resumedFromState, - sandboxState: "started", - workspaceSentinel, - }), - }; + remoteCwd, + leaseMetadata: params.leaseMetadata, + timeoutSeconds: toTimeoutSeconds(config.timeoutMs), + })); + if (workspaceSentinel.result !== "matched") { + evictSandboxHandle(scope); + return { + providerLeaseId: null, + metadata: { expired: true, workspaceSentinel }, + }; + } + const shellCommand = await traceLeaseStep("lease.shell", () => detectSandboxShellCommand( + sandbox, + toTimeoutSeconds(config.timeoutMs), + )); + sandboxHandleCache.markFresh(scope); + sandboxHandleLeaseAdmissionStates.open(scope); + return { + providerLeaseId: sandbox.id, + metadata: leaseMetadata({ + config, + sandbox, + shellCommand, + remoteCwd: requestedCwd ?? remoteCwd, + resumedLease: true, + resumedFromState, + sandboxState: "started", + workspaceSentinel, + }), + }; + }); } catch (error) { evictSandboxHandle(scope); // A timeout, rate limit, or provider 5xx does not prove this sandbox is @@ -2910,7 +2923,7 @@ const plugin = definePlugin(trackDaytonaHooks({ throw error; } }, - { allowClosed: true }, + { allowClosed: true, traceLeaseWait: true }, ); }, diff --git a/server/src/__tests__/plugin-host-services-span.test.ts b/server/src/__tests__/plugin-host-services-span.test.ts index 01b7eb06c7a..a91e6c4e626 100644 --- a/server/src/__tests__/plugin-host-services-span.test.ts +++ b/server/src/__tests__/plugin-host-services-span.test.ts @@ -332,6 +332,36 @@ describe("clampProviderSpanAttributes", () => { }); describe("nested worker timing receipts",()=>{ + it("keeps lease verification beneath its worker and clamps provider state", () => { + mockRecordSpan.mockReset(); + const rootParent = { traceId: "0af7651916cd43dd8448eb211c80319c", spanId: "1111111111111111", traceFlags: 1 }; + const verifyParent = { ...rootParent, spanId: "2222222222222222" }; + mockRecordSpan.mockReturnValueOnce(rootParent).mockReturnValueOnce(verifyParent); + const now = Date.now(); + recordWorkerProviderSpan({ name: "plugin.worker", startTimeMs: now - 100, endTimeMs: now, children: [ + { name: "lease.verify", startTimeMs: now - 90, endTimeMs: now - 10, children: [ + { name: "lease.sentinel", startTimeMs: now - 80, endTimeMs: now - 20, + attributes: { "paperclip.sandbox.startup.state": "archived", token: "private-token" } }, + ] }, + ] }, { traceparent: VALID_TRACEPARENT }); + expect(mockRecordSpan.mock.calls[1][0]).toMatchObject({ name: "sandbox.daytona.lease.verify", parent: rootParent }); + expect(mockRecordSpan.mock.calls[2][0]).toMatchObject({ name: "sandbox.daytona.lease.sentinel", parent: verifyParent, + startTimeMs: now - 80, endTimeMs: now - 20, attributes: { "paperclip.sandbox.startup.state": "archived" } }); + expect(JSON.stringify(mockRecordSpan.mock.calls)).not.toContain("private-token"); + expect(clampProviderSpanAttributes({ "paperclip.sandbox.startup.state": "private-state" })).toEqual({}); + }); + + it("admits every closed lease phase and rejects dynamic phase names", () => { + mockRecordSpan.mockReset(); + const names = ["lease.wait_for_teardown", "lease.lookup", "lease.refresh", "lease.channel_cleanup", + "lease.start", "lease.recover", "lease.verify", "lease.workspace", "lease.sentinel", "lease.shell", + "lease.create", "lease.expiry", "lease.cleanup", "lease.private-resource-id"]; + for (const name of names) recordWorkerProviderSpan({ name }, { traceparent: VALID_TRACEPARENT }); + expect(mockRecordSpan.mock.calls.map(call => call[0].name)).toEqual([ + ...names.slice(0, -1).map(name => `sandbox.daytona.${name}`), "sandbox.daytona.other", + ]); + }); + it("uses host-generated parents, clamps content and bounds child clocks",()=>{ mockRecordSpan.mockReset(); const childParent={traceId:"0af7651916cd43dd8448eb211c80319c",spanId:"1111111111111111",traceFlags:1}; diff --git a/server/src/services/plugin-host-services.ts b/server/src/services/plugin-host-services.ts index 2a9a845d01a..eeff010abbe 100644 --- a/server/src/services/plugin-host-services.ts +++ b/server/src/services/plugin-host-services.ts @@ -537,6 +537,9 @@ const KNOWN_PROVIDER_SPAN_NAMES: ReadonlySet = new Set([ "postUploadCommand", "session.open", "session.close", + "lease.wait_for_teardown", "lease.lookup", "lease.refresh", "lease.channel_cleanup", + "lease.start", "lease.recover", "lease.verify", "lease.workspace", "lease.sentinel", + "lease.shell", "lease.create", "lease.expiry", "lease.cleanup", "plugin.worker", "plugin.host.identity.read", "plugin.host.config.read", "plugin.host.progress.read", "plugin.host.progress.write", "plugin.workspace.prepare", "plugin.storage.queue", "plugin.storage.execute", "plugin.runtime.queue", "plugin.runtime.prepare", "plugin.runtime.reconcile", "plugin.runtime.verify", @@ -564,6 +567,7 @@ const PROVIDER_SPAN_ATTR_ALLOWLIST: ReadonlySet = new Set([ SPAN_ATTRS.transferWallMs, SPAN_ATTRS.transferGuardCount, SPAN_ATTRS.transferDirection, + SPAN_ATTRS.state, "plugin.operation", "plugin.reused", "plugin.item_count", "plugin.bytes", "plugin.call_count", ]); @@ -595,6 +599,10 @@ export function clampProviderSpanAttributes( if (!raw) return clamped; for (const [key, value] of Object.entries(raw)) { if (!PROVIDER_SPAN_ATTR_ALLOWLIST.has(key)) continue; + if (key === SPAN_ATTRS.state) { + if (typeof value === "string" && ["started", "stopped", "archived", "error", "other"].includes(value)) clamped[key] = value; + continue; + } if (key === "plugin.operation") { if (typeof value === "string" && ["request", "queue.wait", "launch.wait", "mount.refresh", "guest.setup", "process.execute", "script.workspace", "script.git", "script.settings", "script.publication", "cleanup", "process.start", "hello", "checkpoint.load"].includes(value)) clamped[key] = value; continue; From 40b807591618c0291369ada779c93da6ad4bfc7f Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Mon, 2 Nov 2026 11:35:30 -0600 Subject: [PATCH 05/35] feat(ui): show scheduled task waits as Monitoring (#18624) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Task indicators tell an operator whether work is active, waiting, or complete. > - An agent can yield until a saved monitor checks CI or another external result. > - The task currently keeps the In Progress icon between those runs. > - Changing the stored status would affect monitor admission and recovery. > - This pull request adds a read-only Monitoring display state with a blue clock. > - Operators can see and filter scheduled waits while execution keeps its existing rules. ## Linked Issues or Issue Description **What existing behavior does this improve?** Task status indicators, task collection filters, board columns, and search facets. **Current behavior** A task that waits for a scheduled monitor shows the same In Progress icon as active work. **Proposed behavior** Show Monitoring with a static blue clock when an agent-owned task has a saved ordinary monitor and no active run, execution lock, review participant, or active recovery action. Return to the normal task indicator when the check starts. Preserve review, blocked, terminal, and quota-recovery states. **Reason and benefit** Operators can distinguish scheduled waits from current agent work without reading the task history. **Breaking changes** None to stored task statuses or execution rules. Task reads gain optional displayStatus. Collections and search gain a separate read filter. Existing status filters keep their lifecycle meaning. Related historical work: #4396 covered a monitoring label exemption. This change uses the saved monitor and a read projection. ## What Changed - Add one SQL projection for task detail, compact lists, summaries, relations, search results, and display-status counts. - Show the blue clock in shared task icons and chips, task detail and breadcrumbs, properties, references, list rows, recent navigation, and agent task cards. - Add Monitoring to task filtering, grouping, sorting, board columns, and search facets. Prevent a drag from creating this derived state. - Keep stored lifecycle statuses, monitor scheduling, checkout, recovery, permissions, and agent status filters unchanged. - Add database and UI regression coverage, full-page Storybook examples, and product and access-map documentation. ## Verification - Passed `pnpm -r typecheck`. - Passed `pnpm build`. - Passed `pnpm check:token-gates` and the map/catalog structural checks. - Ran the new database suite on a disposable PostgreSQL 17 database. It covers read surfaces, filter pagination/counts, queued/running checks, rescheduling, completion, retained monitors, review, and recovery holds. - Ran existing search, list, board, sidebar, task status, and filter tests. Corrected board queries use displayStatus and eight columns. - The full local `pnpm test:run` stopped in a chat integration shard: 12 failures (four timeouts and eight FIFO publication assertions). That run spanned a rebase. A focused rerun on the current source passed all 21 selected cases without assertion or timeout changes. The original cause is not established; this does not claim the full local suite passed. Final-head required CI passed (ci / verify and ci / e2e, run 41488267209). - Passed 82 tests for task API summaries, recovery precedence, search, and optimistic updates; 54 tests for status cards, database reads, and board moves; and 27 tests for blocker relations and recent-task cache updates. One intermediate database setup hit its 20-second hook timeout; the serial rerun passed. - Greptile is 5/5 on d003ea672d9e9faac68e8c4ef054d34272037340. All review threads are resolved. The separate optional Paperclip Review ended early and returned neutral; GitHub rejected a direct rerun request with HTTP 404. It is not counted as a completed review. - Review fixes cover escalated recovery, is:open, optimistic status changes, blocker-summary projection, and recent-task dedupe. - Reviewed 15 screenshots from production pages and components with isolated Storybook API fixtures: task detail/properties/breadcrumb, list, groups, status filter, board, search/results/filter counts, reference pills, sibling navigation, agent cards, picker, dashboard, inbox, profile, watched status-card tasks, light theme, and mobile. This does not claim deployed staging verification. ## Risks - Read projections must stay consistent across full, compact, summary, and search responses. Tests cover those paths. - Monitoring is not a writable lifecycle status. The board does not accept drops into its Monitoring lane or onto a monitored card. This prevents a drag from implying a scheduled check. Move work into In Progress through that lane instead. - The read projection adds run/recovery existence checks. Existing heartbeat_runs_company_issue_created_idx and issue_recovery_actions_company_source_status_idx indexes cover the company/task lookup. It adds no worker, timer, recurring query, migration, or provider call. No production load benchmark is claimed. - Older clients can ignore displayStatus. Existing lifecycle status filters keep their behavior. ## Model Used OpenAI Codex, GPT-6, with reasoning, repository tools, code execution, and browser tools. The exact backend model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/PRODUCT.md | 5 + doc/SPEC-implementation.md | 1 + doc/execution-semantics.md | 16 +++ maps/agent-access-map/tasks.md | 6 + maps/feature-map/tasks.md | 6 + packages/shared/src/constants.ts | 6 + packages/shared/src/index.ts | 2 + packages/shared/src/types/issue.ts | 7 +- packages/shared/src/types/search.ts | 4 +- packages/shared/src/types/user-profile.ts | 3 +- packages/shared/src/validators/search.ts | 5 +- .../__tests__/issue-display-status.test.ts | 127 ++++++++++++++++++ .../__tests__/issue-summaries-routes.test.ts | 2 +- server/src/routes/issues.ts | 3 + server/src/routes/user-profiles.ts | 2 + server/src/services/company-search.ts | 21 ++- server/src/services/issue-display-status.ts | 37 +++++ server/src/services/issue-summaries.ts | 2 + server/src/services/issues.ts | 25 +++- server/src/services/status-cards.ts | 2 + server/src/services/task-search.ts | 3 +- ui/src/api/issues.ts | 2 + ui/src/api/search.test.ts | 14 ++ ui/src/api/search.ts | 6 +- ui/src/components/BlockedInboxView.tsx | 6 +- ui/src/components/CoordinatorSidebar.tsx | 4 +- ui/src/components/IssueColumns.tsx | 2 +- ui/src/components/IssueMonitorBanner.tsx | 2 +- ui/src/components/IssueReferencePill.tsx | 4 +- ui/src/components/IssueRow.tsx | 8 +- ui/src/components/IssueSiblingNavigation.tsx | 2 +- ui/src/components/IssuesList.test.tsx | 16 +-- ui/src/components/IssuesList.tsx | 28 ++-- ui/src/components/KanbanBoard.test.tsx | 15 +++ ui/src/components/KanbanBoard.tsx | 27 ++-- ui/src/components/LegacyIssuesList.tsx | 26 ++-- ui/src/components/SidebarRecentTasks.tsx | 2 + ui/src/components/StatusBadge.tsx | 4 +- ui/src/components/StatusGlyph.test.tsx | 1 + ui/src/components/StatusGlyph.tsx | 5 +- ui/src/components/StatusIcon.test.tsx | 17 +++ ui/src/components/StatusIcon.tsx | 12 +- ui/src/components/chat/AgentWorkPanels.tsx | 2 +- .../issue-properties/IssueProperties.tsx | 8 +- .../issue-properties/relation-controls.tsx | 2 +- ui/src/components/search/SearchFilterBar.tsx | 10 +- ui/src/components/search/SearchResultRow.tsx | 2 +- .../task-detail/TaskDetailRelationsPanel.tsx | 3 +- ui/src/index.css | 2 + ui/src/lib/issue-detail-subissues.ts | 9 +- ui/src/lib/issue-filters.test.ts | 8 ++ ui/src/lib/issue-filters.ts | 6 +- ui/src/lib/issue-properties-panel-key.ts | 4 + ui/src/lib/issueDetailBreadcrumb.ts | 3 + ui/src/lib/optimistic-issue-comments.test.ts | 6 + ui/src/lib/optimistic-issue-comments.ts | 2 + ui/src/lib/recent-tasks.test.ts | 16 +++ ui/src/lib/recent-tasks.ts | 12 +- ui/src/lib/search-query-parser.test.ts | 2 +- ui/src/lib/search-query-parser.ts | 14 +- ui/src/lib/status-colors.ts | 2 + ui/src/lib/task-views.test.ts | 2 +- ui/src/lib/task-views.ts | 2 +- ui/src/pages/Dashboard.tsx | 2 +- ui/src/pages/DesignGuide.tsx | 4 + ui/src/pages/Inbox.tsx | 4 +- ui/src/pages/IssueDetail.tsx | 7 +- ui/src/pages/LegacyInbox.tsx | 4 +- ui/src/pages/MyIssues.tsx | 2 +- .../StatusCards/StatusCardDetailDrawer.tsx | 2 +- ui/src/pages/UserProfile.tsx | 2 +- .../tasks/monitoring-status.stories.tsx | 109 +++++++++++++++ 72 files changed, 614 insertions(+), 127 deletions(-) create mode 100644 server/src/__tests__/issue-display-status.test.ts create mode 100644 server/src/services/issue-display-status.ts create mode 100644 ui/src/api/search.test.ts create mode 100644 ui/storybook/stories/tasks/monitoring-status.stories.tsx diff --git a/doc/PRODUCT.md b/doc/PRODUCT.md index cba3bec4ae2..230d7e38cb3 100644 --- a/doc/PRODUCT.md +++ b/doc/PRODUCT.md @@ -97,6 +97,11 @@ When an agent creates a subtask or follow-up task, its source task's feed shows a linked card at creation time with the current title, task number, and status. The Tasks tab keeps the full relationship list. +Tasks waiting between automatic checks show **Monitoring** with a blue clock. +Task lists, board columns, filters, search, recent navigation and task properties +use the same read-only display state. The task retains its execution lifecycle +status, and the monitor banner shows when the next check is due. + Automatic monitor checks pause when a task is blocked. A person can use Check now to request one check of a retained monitor through normal execution admission. Dependencies, recovery, pause, budgets and ownership still apply. Provider-usage diff --git a/doc/SPEC-implementation.md b/doc/SPEC-implementation.md index 59874d54e8b..135b70dcf47 100644 --- a/doc/SPEC-implementation.md +++ b/doc/SPEC-implementation.md @@ -423,6 +423,7 @@ retain their titles and default to no generation request. - `title` text not null - `description` text null - `status` enum: `backlog | todo | in_progress | in_review | done | blocked | cancelled` +- Task reads additionally expose read-only `displayStatus`: the lifecycle status, or `monitoring` for an agent-owned scheduled ordinary monitor wait with no active execution, locks, review participant or active recovery action. Board collection/search `displayStatus` filters and counts use the same projection before pagination. This field is not persisted and cannot authorize a status transition; existing `status` filters retain their lifecycle semantics. - `priority` enum: `critical | high | medium | low` - `review_policy` nullable enum: `anyone | not_creator | human_only`; null is equivalent to `anyone` - `assignee_agent_id` uuid fk `agents.id` null diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index dd0378688f5..03d2d05ebbc 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -112,6 +112,22 @@ The issue is actively owned work. For agent-owned issues, `in_progress` should not be allowed to become a silent dead state. +### Monitoring presentation + +An agent-owned `in_progress` task displays **Monitoring** with a blue clock while +it has a persisted ordinary monitor deadline and no active run, execution or +checkout lock, review participant, or active recovery action. The persisted +status remains `in_progress`. Monitor admission, checkout, finalization, recovery, +and agent lifecycle filters keep their existing contracts. Review, blocked, +terminal, and provider-quota recovery states keep their own presentation. + +Task reads expose optional, read-only `displayStatus`. Board task collections and +search can filter by `displayStatus`; filtering happens before pagination and +counts. Existing `status` filters retain their lifecycle meaning. Monitoring is +not a manually selectable task status or a board drag target. An overdue check +retains its visible deadline warning; the label does not attest that a provider +is running or that a check succeeded. + ### `blocked` The issue cannot proceed until something external changes. diff --git a/maps/agent-access-map/tasks.md b/maps/agent-access-map/tasks.md index 7f80a1314c0..6ba136f9091 100644 --- a/maps/agent-access-map/tasks.md +++ b/maps/agent-access-map/tasks.md @@ -522,3 +522,9 @@ Evidence: `source-only` — reviewed at `3bbc2104b65caf9f7763938a01260a5673e12f7 Verification: `not-run` — Compare eligible owner/admin ordinary denied with explicit read/audit+notice; ordinary operator/agent/foreign denied, grants unchanged. Gaps: Local implicit/instance admin bypass cannot establish ordinary two-user private isolation. + +Task reads expose optional read-only `displayStatus`. `monitoring` describes a +scheduled ordinary monitor wait; it is not accepted as a writable task `status`. +Existing lifecycle `status` filters retain their meaning. Board collection and +search readers can use the separate `displayStatus` filter before pagination. +Agents keep using the existing monitor and semantic completion tools. diff --git a/maps/feature-map/tasks.md b/maps/feature-map/tasks.md index a9afa296295..2f335db00d6 100644 --- a/maps/feature-map/tasks.md +++ b/maps/feature-map/tasks.md @@ -56,6 +56,12 @@ Use New task from the global launcher, task list, or project task view. ### `task-list` +Scheduled ordinary monitor waits display **Monitoring** with a blue clock in +current task indicators, groups, board columns, and filters. Compare Monitoring +with In Progress, then start the check and verify the status returns to its +execution presentation. Retained monitors must not override blocked/review or +terminal states. Inspect due/overdue copy separately; no runner is implied. + Open `/issues` or a project’s Issues view; legacy `/tasks` links redirect. Filters → **Routine tasks** selects tasks created by routine runs. The `/issues?view=all&originKind=routine_execution` link opens that selection. diff --git a/packages/shared/src/constants.ts b/packages/shared/src/constants.ts index 2d6d93331d5..28f9fe42585 100644 --- a/packages/shared/src/constants.ts +++ b/packages/shared/src/constants.ts @@ -198,6 +198,12 @@ export const ISSUE_STATUSES = [ ] as const; export type IssueStatus = (typeof ISSUE_STATUSES)[number]; +/** Read-only task presentation. Monitoring never grants execution authority. */ +export const ISSUE_DISPLAY_STATUSES = [ + "backlog", "todo", "in_progress", "monitoring", "in_review", "done", "blocked", "cancelled", +] as const; +export type IssueDisplayStatus = (typeof ISSUE_DISPLAY_STATUSES)[number]; + export const INBOX_MINE_ISSUE_STATUSES = [ "backlog", "todo", diff --git a/packages/shared/src/index.ts b/packages/shared/src/index.ts index d6a9e5cbd02..da43d193d30 100644 --- a/packages/shared/src/index.ts +++ b/packages/shared/src/index.ts @@ -364,6 +364,7 @@ export { AGENT_ICON_NAMES, PROJECT_ICON_NAMES, ISSUE_STATUSES, + ISSUE_DISPLAY_STATUSES, ISSUE_SUMMARIES_MAX_IDS, INBOX_MINE_ISSUE_STATUSES, INBOX_MINE_ISSUE_STATUS_FILTER, @@ -562,6 +563,7 @@ export { type AgentIconName, type ProjectIconName, type IssueStatus, + type IssueDisplayStatus, type IssuePriority, type IssueReviewPolicy, type IssueWorkMode, diff --git a/packages/shared/src/types/issue.ts b/packages/shared/src/types/issue.ts index 29a8008ec80..451f745db97 100644 --- a/packages/shared/src/types/issue.ts +++ b/packages/shared/src/types/issue.ts @@ -20,6 +20,7 @@ import type { IssueHarnessKind, IssueOriginKind, IssuePriority, + IssueDisplayStatus, IssueReviewPolicy, IssueRecoveryActionKind, IssueRecoveryActionOutcome, @@ -270,6 +271,7 @@ export interface AcceptedPlanDecompositionSummary extends AcceptedPlanDecomposit } export interface IssueRelationIssueSummary { + displayStatus?: IssueDisplayStatus; id: string; identifier: string | null; title: string; @@ -850,12 +852,14 @@ export type IssueChanges = Record; export type IssueSummary = Pick< Issue, - "id" | "companyId" | "title" | "identifier" | "status" | "updatedAt" + "id" | "companyId" | "title" | "identifier" | "status" | "displayStatus" | "updatedAt" > & { externalConversationState: "active" | "waiting" | null; }; export interface Issue { + /** Read-only status for task presentation; never use it for execution admission. */ + displayStatus?: IssueDisplayStatus; /** Collection projections omit task bodies and cannot seed the detail cache. */ listProjection?: "index" | "rows"; statusVersion?: number; @@ -985,6 +989,7 @@ export type CompactIssue = Pick< | "privacyRootIssueId" | "title" | "description" + | "displayStatus" | "status" | "workMode" | "priority" diff --git a/packages/shared/src/types/search.ts b/packages/shared/src/types/search.ts index 825d4d4d1c4..bc9ea562d53 100644 --- a/packages/shared/src/types/search.ts +++ b/packages/shared/src/types/search.ts @@ -1,4 +1,4 @@ -import type { IssuePriority, IssueStatus } from "../constants.js"; +import type { IssueDisplayStatus, IssuePriority, IssueStatus } from "../constants.js"; export const COMPANY_SEARCH_SCOPES = ["all", "issues", "comments", "documents", "artifacts", "agents", "projects"] as const; export type CompanySearchScope = (typeof COMPANY_SEARCH_SCOPES)[number]; @@ -38,6 +38,7 @@ export interface CompanySearchIssueSummary { identifier: string | null; title: string; status: IssueStatus; + displayStatus?: IssueDisplayStatus; externalConversationState?: "active" | "waiting" | null; priority: IssuePriority; assigneeAgentId: string | null; @@ -76,6 +77,7 @@ export interface CompanySearchResult { export interface CompanySearchFilterOptionCounts { status: Partial>; + displayStatus?: Partial>; priority: Partial>; assigneeAgentId: Record; assigneeUserId: Record; diff --git a/packages/shared/src/types/user-profile.ts b/packages/shared/src/types/user-profile.ts index d961a407485..1ac26e033a7 100644 --- a/packages/shared/src/types/user-profile.ts +++ b/packages/shared/src/types/user-profile.ts @@ -1,4 +1,4 @@ -import type { IssuePriority, IssueStatus } from "../constants.js"; +import type { IssuePriority, IssueStatus, IssueDisplayStatus } from "../constants.js"; export interface UserProfileIdentity { id: string; @@ -42,6 +42,7 @@ export interface UserProfileIssueSummary { identifier: string | null; title: string; status: IssueStatus; + displayStatus?: IssueDisplayStatus; priority: IssuePriority; assigneeAgentId: string | null; assigneeUserId: string | null; diff --git a/packages/shared/src/validators/search.ts b/packages/shared/src/validators/search.ts index 839f4cb5296..3b81dae3836 100644 --- a/packages/shared/src/validators/search.ts +++ b/packages/shared/src/validators/search.ts @@ -1,5 +1,5 @@ import { z } from "zod"; -import { ISSUE_PRIORITIES, ISSUE_STATUSES } from "../constants.js"; +import { ISSUE_DISPLAY_STATUSES, ISSUE_PRIORITIES, ISSUE_STATUSES } from "../constants.js"; import { isUuidLike } from "../agent-url-key.js"; import { COMPANY_SEARCH_EXTRACT_KINDS, @@ -156,6 +156,9 @@ export const companySearchQuerySchema = z.object({ status: z.unknown() .optional() .transform((value, ctx) => parseEnumList(value, ctx, "status", ISSUE_STATUSES)), + displayStatus: z.unknown() + .optional() + .transform((value, ctx) => parseEnumList(value, ctx, "displayStatus", ISSUE_DISPLAY_STATUSES)), priority: z.unknown() .optional() .transform((value, ctx) => parseEnumList(value, ctx, "priority", ISSUE_PRIORITIES)), diff --git a/server/src/__tests__/issue-display-status.test.ts b/server/src/__tests__/issue-display-status.test.ts new file mode 100644 index 00000000000..e2f05c26720 --- /dev/null +++ b/server/src/__tests__/issue-display-status.test.ts @@ -0,0 +1,127 @@ +import { randomUUID } from "node:crypto"; +import { eq, sql } from "drizzle-orm"; +import { expect, it } from "vitest"; +import request from "supertest"; +import { agents, heartbeatRuns, issueRecoveryActions, issueRelations, issues, statusCards } from "@paperclipai/db"; +import { companySearchQuerySchema, createIssueSchema, PROVIDER_QUOTA_MONITOR_SERVICE_NAME } from "@paperclipai/shared"; +import { issueDisplayStatusSql } from "../services/issue-display-status.js"; +import { issueService } from "../services/issues.js"; +import { companySearchService } from "../services/company-search.js"; +import { statusCardService } from "../services/status-cards.js"; +import { issueRoutes } from "../routes/issues.js"; +import { + describeEmbeddedPostgres, routeApp, + seedCompanyWithBoardAccess, useEmbeddedPostgres, +} from "./helpers/route-test-harness.js"; + +describeEmbeddedPostgres("Monitoring task presentation", () => { + const ctx = useEmbeddedPostgres("paperclip-monitoring-status-", { + resetEach: async (db) => { await db.execute(sql`SET client_min_messages = warning`); await db.execute(sql`TRUNCATE TABLE companies CASCADE`); }, + }); + + async function fixture() { + const company = await seedCompanyWithBoardAccess(ctx.db, "Monitoring"); + const agentId = randomUUID(); + await ctx.db.insert(agents).values({ id: agentId, companyId: company.companyId, name: "Monitor owner", role: "engineer", status: "paused", adapterType: "process", adapterConfig: {} }); + const nextCheckAt = new Date(Date.now() + 60 * 60_000); + const issueId = randomUUID(); + await ctx.db.insert(issues).values({ + id: issueId, companyId: company.companyId, title: "Check deployment", status: "in_progress", + assigneeAgentId: agentId, createdByUserId: company.actor.userId, + monitorNextCheckAt: nextCheckAt, + executionPolicy: { mode: "normal", stages: [], commentRequired: false, monitor: { nextCheckAt: nextCheckAt.toISOString(), notes: "Check deployment" } }, + }); + return { ...company, agentId, issueId }; + } + + async function displayStatus(id: string) { + const [row] = await ctx.db.select({ status: issues.status, displayStatus: issueDisplayStatusSql() }).from(issues).where(eq(issues.id, id)); + return row; + } + + it("projects a scheduled wait consistently across detail, compact rows, summaries, search and counts", async () => { + const f = await fixture(); + const svc = issueService(ctx.db); + expect(await svc.getById(f.issueId)).toMatchObject({ status: "in_progress", displayStatus: "monitoring" }); + expect(await svc.list(f.companyId, { displayStatus: "monitoring", listProjection: "index" })).toEqual([expect.objectContaining({ id: f.issueId, status: "in_progress", displayStatus: "monitoring" })]); + expect(await svc.count(f.companyId, { displayStatus: "monitoring" })).toBe(1); + const app = routeApp(ctx.db, f.actor, issueRoutes); + const summary = await request(app).get(`/api/companies/${f.companyId}/issues/summaries?ids=${f.issueId}`).expect(200); + expect(summary.body[0]).toMatchObject({ status: "in_progress", displayStatus: "monitoring" }); + const result = await companySearchService(ctx.db).search(f.companyId, companySearchQuerySchema.parse({ q: "deployment", displayStatus: "monitoring" })); + expect(result.results[0]?.issue).toMatchObject({ id: f.issueId, status: "in_progress", displayStatus: "monitoring" }); + expect(result.filterOptionCounts.displayStatus?.monitoring).toBe(1); + expect(result.filterOptionCounts.status.in_progress).toBe(1); + const [card] = await ctx.db.insert(statusCards).values({ companyId: f.companyId, interestPrompt: "Watch deployment", mentionedIssueIds: [f.issueId], refreshPolicy: { mode: "manual" } }).returning(); + expect(await statusCardService(ctx.db).listMentionedIssues(card!)).toEqual([expect.objectContaining({ id: f.issueId, status: "in_progress", displayStatus: "monitoring" })]); + // Existing agent/API lifecycle filters keep their original meaning. + expect(await svc.count(f.companyId, { status: "in_progress" })).toBe(1); + expect(createIssueSchema.safeParse({ title: "Manual monitoring", status: "monitoring" }).success).toBe(false); + }); + + it("filters before pagination and keeps In Progress separate from Monitoring", async () => { + const f = await fixture(); + await ctx.db.insert(issues).values({ companyId: f.companyId, title: "Ordinary work", status: "in_progress", assigneeAgentId: f.agentId }); + const svc = issueService(ctx.db); + expect(await svc.list(f.companyId, { displayStatus: "monitoring", limit: 1, offset: 0 })).toEqual([expect.objectContaining({ id: f.issueId })]); + expect(await svc.list(f.companyId, { displayStatus: "monitoring", limit: 1, offset: 1 })).toEqual([]); + expect(await svc.count(f.companyId, { displayStatus: "in_progress" })).toBe(1); + expect(await svc.count(f.companyId, { status: "in_progress" })).toBe(2); + }); + + it("shows the same wait indicator in list blocker summaries and task detail relations", async () => { + const f = await fixture(); + const [dependent] = await ctx.db.insert(issues).values({ companyId: f.companyId, title: "Ship after deployment", status: "blocked" }).returning(); + await ctx.db.insert(issueRelations).values({ companyId: f.companyId, issueId: f.issueId, relatedIssueId: dependent!.id, type: "blocks" }); + const svc = issueService(ctx.db); + const list = await svc.list(f.companyId, { includeBlockedBy: true }); + expect(list.find(row => row.id === dependent!.id)?.blockedBy).toEqual([expect.objectContaining({ id: f.issueId, displayStatus: "monitoring" })]); + const detail = await request(routeApp(ctx.db, f.actor, issueRoutes)).get(`/api/issues/${dependent!.id}`).expect(200); + expect(detail.body.blockedBy).toEqual([expect.objectContaining({ id: f.issueId, displayStatus: "monitoring" })]); + }); + + it("returns to In Progress during the check and Monitoring after another yield, then Done", async () => { + const f = await fixture(); + const runId = randomUUID(); + await ctx.db.insert(heartbeatRuns).values({ id: runId, companyId: f.companyId, agentId: f.agentId, issueId: f.issueId, status: "queued" }); + expect((await displayStatus(f.issueId)).displayStatus).toBe("in_progress"); + await ctx.db.update(heartbeatRuns).set({ status: "running" }).where(eq(heartbeatRuns.id, runId)); + await ctx.db.update(issues).set({ executionRunId: runId, checkoutRunId: runId }).where(eq(issues.id, f.issueId)); + expect((await displayStatus(f.issueId)).displayStatus).toBe("in_progress"); + await ctx.db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, runId)); + await ctx.db.update(issues).set({ executionRunId: null, checkoutRunId: null }).where(eq(issues.id, f.issueId)); + expect((await displayStatus(f.issueId)).displayStatus).toBe("monitoring"); + await ctx.db.update(issues).set({ status: "done" }).where(eq(issues.id, f.issueId)); + expect(await displayStatus(f.issueId)).toEqual({ status: "done", displayStatus: "done" }); + }); + + it.each(["blocked", "in_review", "todo", "cancelled"])("preserves %s even with a retained monitor", async (status) => { + const f = await fixture(); + await ctx.db.update(issues).set({ status }).where(eq(issues.id, f.issueId)); + expect((await displayStatus(f.issueId)).displayStatus).toBe(status); + }); + + it("does not label cleared, consumed, orphaned or quota-recovery schedules Monitoring", async () => { + const f = await fixture(); + await ctx.db.update(issues).set({ executionState: { monitor: { status: "cleared" } } }).where(eq(issues.id, f.issueId)); + expect((await displayStatus(f.issueId)).displayStatus).toBe("in_progress"); + await ctx.db.update(issues).set({ executionState: null, monitorNextCheckAt: null }).where(eq(issues.id, f.issueId)); + expect((await displayStatus(f.issueId)).displayStatus).toBe("in_progress"); + await ctx.db.update(issues).set({ monitorNextCheckAt: new Date(), executionPolicy: { monitor: { serviceName: PROVIDER_QUOTA_MONITOR_SERVICE_NAME } } }).where(eq(issues.id, f.issueId)); + expect((await displayStatus(f.issueId)).displayStatus).toBe("in_progress"); + await ctx.db.update(issues).set({ executionPolicy: null }).where(eq(issues.id, f.issueId)); + expect((await displayStatus(f.issueId)).displayStatus).toBe("in_progress"); + }); + + it("keeps recovery and review holds visible and never writes a new lifecycle state", async () => { + const f = await fixture(); + await ctx.db.insert(issueRecoveryActions).values({ companyId: f.companyId, sourceIssueId: f.issueId, kind: "source", ownerType: "board", cause: "process_lost", fingerprint: randomUUID(), nextAction: "Inspect stopped run" }); + expect((await displayStatus(f.issueId)).displayStatus).toBe("in_progress"); + await ctx.db.execute(sql`UPDATE issue_recovery_actions SET status = 'escalated' WHERE source_issue_id = ${f.issueId}`); + expect((await displayStatus(f.issueId)).displayStatus).toBe("in_progress"); + await ctx.db.execute(sql`UPDATE issue_recovery_actions SET status = 'resolved' WHERE source_issue_id = ${f.issueId}`); + expect((await displayStatus(f.issueId)).displayStatus).toBe("monitoring"); + await ctx.db.update(issues).set({ executionState: { currentParticipant: { type: "user", userId: "reviewer" } } }).where(eq(issues.id, f.issueId)); + expect((await displayStatus(f.issueId)).displayStatus).toBe("in_progress"); + }); +}); diff --git a/server/src/__tests__/issue-summaries-routes.test.ts b/server/src/__tests__/issue-summaries-routes.test.ts index c832d92af03..f82446cd20c 100644 --- a/server/src/__tests__/issue-summaries-routes.test.ts +++ b/server/src/__tests__/issue-summaries-routes.test.ts @@ -50,7 +50,7 @@ describeEmbeddedPostgres("bounded issue summaries", () => { expect(res.headers["cache-control"]).toBe("private, no-store"); expect(res.body.map((row: { id: string }) => row.id)).toEqual([f.rows[1]!.id, f.rows[0]!.id]); expect(Object.keys(res.body[0]).sort()).toEqual([ - "companyId", "externalConversationState", "id", "identifier", "status", "title", "updatedAt", + "companyId", "displayStatus", "externalConversationState", "id", "identifier", "status", "title", "updatedAt", ]); expect(res.body[0].externalConversationState).toBeNull(); expect(forbiddenRead).not.toHaveBeenCalled(); diff --git a/server/src/routes/issues.ts b/server/src/routes/issues.ts index 3d6af2376f9..b63d1135afa 100644 --- a/server/src/routes/issues.ts +++ b/server/src/routes/issues.ts @@ -8379,6 +8379,7 @@ export function issueRoutes( title: row.title, identifier: row.identifier, status: row.status as IssueSummary["status"], + displayStatus: row.displayStatus, updatedAt: row.updatedAt, externalConversationState: row.externalConversationState, })); @@ -8619,6 +8620,7 @@ export function issueRoutes( readCondition: await issueReadSqlCondition(db, req.actor), attention: attention === "blocked" ? "blocked" : undefined, status: req.query.status as string | string[] | undefined, + displayStatus: req.query.displayStatus as string | string[] | undefined, assigneeAgentId, participantAgentId: uuidFilters.participantAgentId, assigneeUserId, @@ -8869,6 +8871,7 @@ export function issueRoutes( readCondition: await issueReadSqlCondition(db, req.actor), attention: "blocked", status: req.query.status as string | string[] | undefined, + displayStatus: req.query.displayStatus as string | string[] | undefined, assigneeAgentId: req.query.assigneeAgentId as string | undefined, participantAgentId: countUuidFilters.participantAgentId, assigneeUserId: req.query.assigneeUserId as string | undefined, diff --git a/server/src/routes/user-profiles.ts b/server/src/routes/user-profiles.ts index 24ed9f9e007..4dc8a9e0109 100644 --- a/server/src/routes/user-profiles.ts +++ b/server/src/routes/user-profiles.ts @@ -1,3 +1,4 @@ +import { issueDisplayStatusSql } from "../services/issue-display-status.js"; import { Router } from "express"; import { and, desc, eq, gte, inArray, isNull, or, sql, type SQLWrapper } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; @@ -347,6 +348,7 @@ export function userProfileRoutes(db: Db) { identifier: issues.identifier, title: issues.title, status: issues.status, + displayStatus: issueDisplayStatusSql(), priority: issues.priority, assigneeAgentId: issues.assigneeAgentId, assigneeUserId: issues.assigneeUserId, diff --git a/server/src/services/company-search.ts b/server/src/services/company-search.ts index fb27743e0d2..17055aa3146 100644 --- a/server/src/services/company-search.ts +++ b/server/src/services/company-search.ts @@ -1,3 +1,4 @@ +import { issueDisplayStatusSql } from "./issue-display-status.js"; import { externalConversationStateSql } from "./slack-conversation-state.js"; import { and, desc, eq, gte, inArray, isNotNull, isNull, notInArray, or, sql } from "drizzle-orm"; import type { SQL } from "drizzle-orm"; @@ -21,6 +22,7 @@ import { COMPANY_ARTIFACTS_MAX_QUERY_LENGTH, ISSUE_PRIORITIES, ISSUE_STATUSES, + ISSUE_DISPLAY_STATUSES, SYSTEM_ISSUE_DOCUMENT_KEYS, type CompanyArtifact, type CompanySearchArtifactSummary, @@ -50,6 +52,7 @@ type IssueSearchRow = { title: string; description: string | null; status: string; + displayStatus: string; priority: string; assigneeAgentId: string | null; assigneeUserId: string | null; @@ -193,6 +196,7 @@ function emptySearchCounts(): Record { function emptyFilterOptionCounts(): CompanySearchFilterOptionCounts { return { status: {}, + displayStatus: {}, priority: {}, assigneeAgentId: {}, assigneeUserId: {}, @@ -224,7 +228,7 @@ function updatedWithinStart(value: string | undefined, now = new Date()): Date | } function issueOnlyFiltersActive(query: CompanySearchQuery) { - return query.status.length > 0 + return (query.status.length > 0 || query.displayStatus.length > 0) || query.priority.length > 0 || query.assigneeAgentId !== undefined || Boolean(query.assigneeUserId) @@ -236,7 +240,7 @@ function issueOnlyFiltersActive(query: CompanySearchQuery) { function activeIssueFilters(query: CompanySearchQuery): Array<{ key: CompanySearchIssueFilterKey; values: string[] }> { const filters: Array<{ key: CompanySearchIssueFilterKey; values: string[] }> = []; - if (query.status.length > 0) filters.push({ key: "status", values: query.status }); + if (query.status.length > 0 || query.displayStatus.length > 0) filters.push({ key: "status", values: [...query.status, ...query.displayStatus] }); if (query.assigneeAgentId !== undefined) filters.push({ key: "assigneeAgentId", values: [query.assigneeAgentId ?? "null"] }); if (query.assigneeUserId) filters.push({ key: "assigneeUserId", values: [query.assigneeUserId] }); if (query.projectId) filters.push({ key: "projectId", values: [query.projectId] }); @@ -251,6 +255,7 @@ function queryWithoutFilter(query: CompanySearchQuery, key: CompanySearchIssueFi return { ...query, status: key === "status" ? [] : query.status, + displayStatus: key === "status" ? [] : query.displayStatus, priority: key === "priority" ? [] : query.priority, assigneeAgentId: key === "assigneeAgentId" ? undefined : query.assigneeAgentId, assigneeUserId: key === "assigneeUserId" ? undefined : query.assigneeUserId, @@ -265,6 +270,7 @@ function queryWithoutIssueFilters(query: CompanySearchQuery): CompanySearchQuery return { ...query, status: [], + displayStatus: [], priority: [], assigneeAgentId: undefined, assigneeUserId: undefined, @@ -277,6 +283,7 @@ function queryWithoutIssueFilters(query: CompanySearchQuery): CompanySearchQuery function issueFilterConditions(companyId: string, query: CompanySearchQuery, omit?: CompanySearchIssueFilterKey): SQL[] { const conditions: SQL[] = []; + if (omit !== "status" && query.displayStatus.length) conditions.push(inArray(issueDisplayStatusSql(), query.displayStatus)); if (omit !== "status" && query.status.length > 0) { conditions.push(query.status.length === 1 ? eq(issues.status, query.status[0]!) : inArray(issues.status, query.status)); } @@ -315,6 +322,7 @@ function issueFilterConditions(companyId: string, query: CompanySearchQuery, omi // matched-issues CTE (plain columns, no drizzle table references). function matchedFacetConditions(companyId: string, query: CompanySearchQuery, omit?: CompanySearchIssueFilterKey): SQL[] { const conditions: SQL[] = []; + if (omit !== "status" && query.displayStatus.length) conditions.push(sql`m.display_status = ANY(${sqlTextArray(query.displayStatus)})`); if (omit !== "status" && query.status.length > 0) { conditions.push(sql`m.status = ANY(${sqlTextArray(query.status)})`); } @@ -447,6 +455,7 @@ function issueResult(row: IssueSearchRow, prefix: string, normalizedQuery: strin identifier: row.identifier, title: row.title, status: row.status as CompanySearchIssueSummary["status"], + displayStatus: row.displayStatus as CompanySearchIssueSummary["displayStatus"], priority: row.priority as CompanySearchIssueSummary["priority"], assigneeAgentId: row.assigneeAgentId, assigneeUserId: row.assigneeUserId, @@ -626,7 +635,7 @@ export function companySearchService(db: Db) { conditions.length > 0 ? sql`WHERE ${sql.join(conditions, sql` AND `)}` : sql``; // Count branches must match the result branch's column list; the // trailing NULLs pad the issue data columns. - const countTail = sql`, NULL::uuid, NULL::text, NULL::text, NULL::text, NULL::text, NULL::text, NULL::uuid, NULL::text, NULL::uuid, NULL::timestamptz, NULL::timestamptz, NULL::double precision, NULL::text[]`; + const countTail = sql`, NULL::uuid, NULL::text, NULL::text, NULL::text, NULL::text, NULL::text, NULL::text, NULL::uuid, NULL::text, NULL::uuid, NULL::timestamptz, NULL::timestamptz, NULL::double precision, NULL::text[]`; const branches: SQL[] = []; const wantResultRows = scopeIncludesIssues(scope) @@ -652,6 +661,7 @@ export function companySearchService(db: Db) { (SELECT issue_text.description FROM issues issue_text WHERE issue_text.id = m.id AND issue_text.company_id = ${companyId}) AS description, m.status, + m.display_status AS "displayStatus", m.priority, m.assignee_agent_id AS "assigneeAgentId", m.assignee_user_id AS "assigneeUserId", @@ -690,6 +700,7 @@ export function companySearchService(db: Db) { GROUP BY 2 `; branches.push(facetBranch("facet:status", sql`m.status`, "status")); + branches.push(facetBranch("facet:displayStatus", sql`m.display_status`, "status")); branches.push(facetBranch("facet:priority", sql`m.priority`, "priority")); branches.push(facetBranch("facet:assigneeAgentId", sql`m.assignee_agent_id`, "assigneeAgentId", [sql`m.assignee_agent_id IS NOT NULL`])); branches.push(facetBranch("facet:assigneeUserId", sql`m.assignee_user_id`, "assigneeUserId", [sql`m.assignee_user_id IS NOT NULL`])); @@ -757,6 +768,7 @@ export function companySearchService(db: Db) { title: row.title, description: row.description, status: row.status, + displayStatus: row.displayStatus, priority: row.priority, assigneeAgentId: row.assigneeAgentId, assigneeUserId: row.assigneeUserId, @@ -780,6 +792,9 @@ export function companySearchService(db: Db) { if (row.kind === "type:issue") aggregates.typeCounts.issue = count; else if (row.kind === "type:comment") aggregates.typeCounts.comment = count; else if (row.kind === "type:document") aggregates.typeCounts.document = count; + else if (row.kind === "facet:displayStatus" && row.value && (ISSUE_DISPLAY_STATUSES as readonly string[]).includes(row.value)) { + aggregates.filterOptionCounts.displayStatus![row.value as keyof NonNullable] = count; + } else if (row.kind === "facet:status" && row.value && (ISSUE_STATUSES as readonly string[]).includes(row.value)) { aggregates.filterOptionCounts.status[row.value as keyof CompanySearchFilterOptionCounts["status"]] = count; } else if (row.kind === "facet:priority" && row.value && (ISSUE_PRIORITIES as readonly string[]).includes(row.value)) { diff --git a/server/src/services/issue-display-status.ts b/server/src/services/issue-display-status.ts new file mode 100644 index 00000000000..5c9bdefb54b --- /dev/null +++ b/server/src/services/issue-display-status.ts @@ -0,0 +1,37 @@ +import { sql, type SQL } from "drizzle-orm"; +import { issues } from "@paperclipai/db"; +import { PROVIDER_QUOTA_MONITOR_SERVICE_NAME, type IssueDisplayStatus } from "@paperclipai/shared"; + +/** Read-only projection, shared by task reads, search, filters and counts. + * The persisted deadline is the scheduler's source of truth. Locks and recovery + * take precedence; a retained monitor alone must not disguise active work. + */ +export function issueDisplayStatusSql(): SQL { + // Drizzle removes column qualifiers in a single-table SELECT. Keep these + // outer references explicitly qualified so correlated reads cannot bind to + // the run/recovery table's own id or company_id instead. + const issueId = sql`"issues"."id"`; + const companyId = sql`"issues"."company_id"`; + return sql`CASE WHEN + ${issues.status} = 'in_progress' + AND ${issues.assigneeAgentId} IS NOT NULL AND ${issues.assigneeUserId} IS NULL + AND ${issues.monitorNextCheckAt} IS NOT NULL + AND ${issues.executionRunId} IS NULL AND ${issues.checkoutRunId} IS NULL + AND ${issues.executionState}->>'currentParticipant' IS NULL + AND COALESCE(${issues.executionState}#>>'{monitor,status}', 'scheduled') = 'scheduled' + AND ${issues.executionPolicy}->'monitor' IS NOT NULL + AND COALESCE(${issues.executionPolicy}#>>'{monitor,serviceName}', '') <> ${PROVIDER_QUOTA_MONITOR_SERVICE_NAME} + AND NOT EXISTS ( + SELECT 1 FROM heartbeat_runs display_run + WHERE display_run.company_id = ${companyId} + AND display_run.issue_id = ${issueId} + AND display_run.status IN ('queued', 'running') + ) + AND NOT EXISTS ( + SELECT 1 FROM issue_recovery_actions display_recovery + WHERE display_recovery.company_id = ${companyId} + AND display_recovery.source_issue_id = ${issueId} + AND display_recovery.status IN ('active', 'escalated') + ) + THEN 'monitoring' ELSE ${issues.status} END`; +} diff --git a/server/src/services/issue-summaries.ts b/server/src/services/issue-summaries.ts index 485bb2250db..645ff8a86c2 100644 --- a/server/src/services/issue-summaries.ts +++ b/server/src/services/issue-summaries.ts @@ -1,3 +1,4 @@ +import { issueDisplayStatusSql } from "./issue-display-status.js"; import { and, eq, inArray, isNull, type SQL } from "drizzle-orm"; import { issues, type Db } from "@paperclipai/db"; import { ISSUE_SUMMARIES_MAX_IDS } from "@paperclipai/shared"; @@ -18,6 +19,7 @@ export async function listIssueSummaries( title: issues.title, identifier: issues.identifier, status: issues.status, + displayStatus: issueDisplayStatusSql(), updatedAt: issues.updatedAt, externalConversationState: externalConversationStateSql(), projectId: issues.projectId, diff --git a/server/src/services/issues.ts b/server/src/services/issues.ts index ecdb45c43a4..d390be0475d 100644 --- a/server/src/services/issues.ts +++ b/server/src/services/issues.ts @@ -1,3 +1,5 @@ +import type { IssueDisplayStatus } from "@paperclipai/shared"; +import { issueDisplayStatusSql } from "./issue-display-status.js"; import { notifyStrandedAgentStatus, notifyActiveRunSilence } from "./live-run-recovery-notifications.js"; import { finishedIdleWake } from "./idle-work-proof.js"; import { notifyDeferredExecutionEligibility } from "./deferred-execution-notifications.js"; @@ -1900,6 +1902,7 @@ export function parseStatusFilter( export interface IssueFilters { attention?: "blocked"; status?: string | readonly string[]; + displayStatus?: string | readonly string[]; /** * Filter by assignee agent ID. * - `string` (UUID): match issues assigned to that agent. @@ -1951,7 +1954,7 @@ export interface IssueFilters { updatedSince?: string; } -type IssueRow = typeof issues.$inferSelect & { externalConversationState?: "active" | "waiting" | null }; +type IssueRow = typeof issues.$inferSelect & { displayStatus?: IssueDisplayStatus; externalConversationState?: "active" | "waiting" | null }; type IssueLabelRow = typeof labels.$inferSelect; type IssuePlanDecompositionRow = typeof issuePlanDecompositions.$inferSelect; type IssueActiveRunRow = { @@ -3603,6 +3606,7 @@ function appendBlockerAttentionEdges( } type IssueRelationSummaryRow = { + displayStatus?: IssueDisplayStatus; relatedId: string; identifier: string | null; title: string; @@ -3620,6 +3624,7 @@ function summarizeIssueRelationRow( identifier: row.identifier, title: row.title, status: row.status as IssueRelationIssueSummary["status"], + displayStatus: row.displayStatus, priority: row.priority as IssueRelationIssueSummary["priority"], assigneeAgentId: row.assigneeAgentId, assigneeUserId: row.assigneeUserId, @@ -3735,6 +3740,7 @@ async function terminalExplicitBlockersByRoot( identifier: issues.identifier, title: issues.title, status: issues.status, + displayStatus: issueDisplayStatusSql(), priority: issues.priority, assigneeAgentId: issues.assigneeAgentId, assigneeUserId: issues.assigneeUserId, @@ -5039,6 +5045,7 @@ const issueListSelect = { END `, status: issues.status, + displayStatus: issueDisplayStatusSql(), statusVersion: issues.statusVersion, lastStatusDecisionId: issues.lastStatusDecisionId, workMode: issues.workMode, @@ -5279,6 +5286,7 @@ async function blockedByMapForIssues( identifier: issues.identifier, title: issues.title, status: issues.status, + displayStatus: issueDisplayStatusSql(), priority: issues.priority, assigneeAgentId: issues.assigneeAgentId, assigneeUserId: issues.assigneeUserId, @@ -5301,6 +5309,7 @@ async function blockedByMapForIssues( identifier: row.identifier, title: row.title, status: row.status as IssueRelationIssueSummary["status"], + displayStatus: row.displayStatus, priority: row.priority as IssueRelationIssueSummary["priority"], assigneeAgentId: row.assigneeAgentId, assigneeUserId: row.assigneeUserId, @@ -6420,6 +6429,8 @@ async function blockedInboxIssueConditions( filters?.lowTrustBoundary, ); if (lowTrustCondition) conditions.push(lowTrustCondition); + const displayStatuses = parseStatusFilter(filters?.displayStatus); + if (displayStatuses.length) conditions.push(inArray(issueDisplayStatusSql(), displayStatuses)); const statuses = parseStatusFilter(filters?.status); if (statuses.length > 0) { conditions.push( @@ -6872,7 +6883,7 @@ export function issueService(db: Db) { async function getIssueByUuid(id: string) { const row = await retryIdempotentDatabaseOperation(() => db - .select({ ...getTableColumns(issues), externalConversationState: externalConversationStateSql() }) + .select({ ...getTableColumns(issues), displayStatus: issueDisplayStatusSql(), externalConversationState: externalConversationStateSql() }) .from(issues) .where(eq(issues.id, id)) .then((rows) => rows[0] ?? null)); @@ -6883,7 +6894,7 @@ export function issueService(db: Db) { async function getIssueByIdentifier(identifier: string) { const row = await retryIdempotentDatabaseOperation(() => db - .select({ ...getTableColumns(issues), externalConversationState: externalConversationStateSql() }) + .select({ ...getTableColumns(issues), displayStatus: issueDisplayStatusSql(), externalConversationState: externalConversationStateSql() }) .from(issues) .where(eq(issues.identifier, identifier.toUpperCase())) .then((rows) => rows[0] ?? null)); @@ -7482,6 +7493,7 @@ export function issueService(db: Db) { identifier: issues.identifier, title: issues.title, status: issues.status, + displayStatus: issueDisplayStatusSql(), priority: issues.priority, assigneeAgentId: issues.assigneeAgentId, assigneeUserId: issues.assigneeUserId, @@ -7502,6 +7514,7 @@ export function issueService(db: Db) { identifier: issues.identifier, title: issues.title, status: issues.status, + displayStatus: issueDisplayStatusSql(), priority: issues.priority, assigneeAgentId: issues.assigneeAgentId, assigneeUserId: issues.assigneeUserId, @@ -8281,6 +8294,8 @@ export function issueService(db: Db) { filters?.lowTrustBoundary, ); if (lowTrustCondition) conditions.push(lowTrustCondition); + const displayStatuses = parseStatusFilter(filters?.displayStatus); + if (displayStatuses.length) conditions.push(inArray(issueDisplayStatusSql(), displayStatuses)); const statuses = parseStatusFilter(filters?.status); if (statuses.length === 1) { conditions.push(eq(issues.status, statuses[0])); @@ -8574,6 +8589,8 @@ export function issueService(db: Db) { conditions.push(nonIdleSlackIssueCondition()); } } + const displayStatuses = parseStatusFilter(filters?.displayStatus); + if (displayStatuses.length) conditions.push(inArray(issueDisplayStatusSql(), displayStatuses)); const statuses = parseStatusFilter(filters?.status); if (statuses.length === 1) conditions.push(eq(issues.status, statuses[0]!)); @@ -8895,6 +8912,7 @@ export function issueService(db: Db) { identifier: issues.identifier, title: issues.title, status: issues.status, + displayStatus: issueDisplayStatusSql(), priority: issues.priority, assigneeAgentId: issues.assigneeAgentId, assigneeUserId: issues.assigneeUserId, @@ -9522,6 +9540,7 @@ export function issueService(db: Db) { identifier: issues.identifier, title: issues.title, status: issues.status, + displayStatus: issueDisplayStatusSql(), priority: issues.priority, assigneeAgentId: issues.assigneeAgentId, assigneeUserId: issues.assigneeUserId, diff --git a/server/src/services/status-cards.ts b/server/src/services/status-cards.ts index d4fe9d7ce80..669e097c532 100644 --- a/server/src/services/status-cards.ts +++ b/server/src/services/status-cards.ts @@ -1,4 +1,5 @@ import { notifyStatusCardEvaluation } from "./status-card-notifications.js"; +import { issueDisplayStatusSql } from "./issue-display-status.js"; import { createHash } from "node:crypto"; import { and, desc, eq, gt, gte, inArray, isNotNull, isNull, lte, ne, or, sql, type SQL } from "drizzle-orm"; import { @@ -552,6 +553,7 @@ export function statusCardService( identifier: issues.identifier, title: issues.title, status: issues.status, + displayStatus: issueDisplayStatusSql(), priority: issues.priority, assigneeAgentId: issues.assigneeAgentId, assigneeUserId: issues.assigneeUserId, diff --git a/server/src/services/task-search.ts b/server/src/services/task-search.ts index 600e355217e..9f56b5a7996 100644 --- a/server/src/services/task-search.ts +++ b/server/src/services/task-search.ts @@ -1,3 +1,4 @@ +import { issueDisplayStatusSql } from "./issue-display-status.js"; import { sql, type SQL } from "drizzle-orm"; import { COMPANY_SEARCH_MAX_QUERY_LENGTH, COMPANY_SEARCH_MAX_TOKENS } from "@paperclipai/shared"; import { visibleIssueCondition } from "./issue-visibility.js"; @@ -134,7 +135,7 @@ export function taskSearchCtes(companyId: string, search: TaskSearch, includeCon // The search page fetches descriptions only for its result window. const flags = (fuzzyMatch: SQL) => sql` SELECT issues.id, issues.identifier, issues.title, - issues.status, issues.priority, issues.assignee_agent_id, issues.assignee_user_id, + issues.status, ${issueDisplayStatusSql()} AS display_status, issues.priority, issues.assignee_agent_id, issues.assignee_user_id, issues.project_id, issues.created_at, issues.updated_at, ${identExact} AS ident_exact, ${identStarts} AS ident_starts, ${phraseMatch(sql`issues.identifier`)} AS ident_phrase, diff --git a/ui/src/api/issues.ts b/ui/src/api/issues.ts index abfc428b39a..afe23d44d09 100644 --- a/ui/src/api/issues.ts +++ b/ui/src/api/issues.ts @@ -194,6 +194,7 @@ export type IssueListFilters = { projection?: "index" | "rows"; attention?: "blocked"; status?: string; + displayStatus?: string; projectId?: string; parentId?: string; assigneeAgentId?: string; @@ -229,6 +230,7 @@ function issueListSearchParams(filters?: IssueListFilters) { const params = new URLSearchParams(); if (filters?.attention) params.set("attention", filters.attention); if (filters?.status) params.set("status", filters.status); + if (filters?.displayStatus) params.set("displayStatus", filters.displayStatus); if (filters?.projectId) params.set("projectId", filters.projectId); if (filters?.parentId) params.set("parentId", filters.parentId); if (filters?.parentIds) params.set("parentIds", filters.parentIds.join(",")); diff --git a/ui/src/api/search.test.ts b/ui/src/api/search.test.ts new file mode 100644 index 00000000000..9f2d39324e5 --- /dev/null +++ b/ui/src/api/search.test.ts @@ -0,0 +1,14 @@ +import { expect, it, vi } from "vitest"; +import { parseSearchQuery } from "../lib/search-query-parser"; +import { searchApi } from "./search"; +import { api } from "./client"; + +vi.mock("./client", () => ({ api: { get: vi.fn() } })); + +it("keeps scheduled monitor waits in is:open search requests", async () => { + const parsed = parseSearchQuery("is:open deployment", {}); + await searchApi.search("company-1", { q: parsed.query, ...parsed.filters }); + const url = new URL(vi.mocked(api.get).mock.calls[0]![0], "http://localhost"); + expect(url.searchParams.getAll("displayStatus")).toEqual(["backlog", "todo", "in_progress", "monitoring", "in_review", "blocked"]); + expect(url.searchParams.get("q")).toBe("deployment"); +}); diff --git a/ui/src/api/search.ts b/ui/src/api/search.ts index 5715d8685b8..b9e97435685 100644 --- a/ui/src/api/search.ts +++ b/ui/src/api/search.ts @@ -1,4 +1,4 @@ -import type { CompanySearchResponse, CompanySearchScope, CompanySearchSort, IssuePriority, IssueStatus } from "@paperclipai/shared"; +import type { CompanySearchResponse, CompanySearchScope, CompanySearchSort, IssuePriority, IssueDisplayStatus } from "@paperclipai/shared"; import { api } from "./client"; export interface CompanySearchParams { @@ -6,7 +6,7 @@ export interface CompanySearchParams { scope?: CompanySearchScope; limit?: number; offset?: number; - status?: IssueStatus[]; + status?: IssueDisplayStatus[]; priority?: IssuePriority[]; assigneeAgentId?: string | null; assigneeUserId?: string; @@ -28,7 +28,7 @@ export const searchApi = { if (params.scope) search.set("scope", params.scope); if (params.limit !== undefined) search.set("limit", String(params.limit)); if (params.offset !== undefined) search.set("offset", String(params.offset)); - appendMulti(search, "status", params.status); + appendMulti(search, "displayStatus", params.status); appendMulti(search, "priority", params.priority); if (params.assigneeAgentId !== undefined) search.set("assigneeAgentId", params.assigneeAgentId ?? "null"); if (params.assigneeUserId !== undefined) search.set("assigneeUserId", params.assigneeUserId); diff --git a/ui/src/components/BlockedInboxView.tsx b/ui/src/components/BlockedInboxView.tsx index 0cb145a359f..fb854d377df 100644 --- a/ui/src/components/BlockedInboxView.tsx +++ b/ui/src/components/BlockedInboxView.tsx @@ -376,7 +376,7 @@ function BlockedInboxRow({ showDivider={presentation === "legacy"} statusSlot={presentation === "task" ? showStatusColumn - ? + ? :