mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-12 08:19:54 +02:00
Dotta
5458940a6e
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Runner needs repeatable evaluation contracts. > - Evaluation code must stay separate from provider launch and production orchestration. > - Offline fixtures need stable compatibility, scoring, traceability, and report rules. > - Published Runner consumers need only the supported evaluation contract surface. > - This pull request adds offline evaluation tooling and a workspace-private matrix kernel. > - The benefit is deterministic evaluation without credentials or paid provider calls. ## Linked Issues or Issue Description Refs #11297 This pull request extracts the offline evaluation unit from the earlier aggregate Runner work. ## What Changed - Add a workspace-private, provider-neutral evaluation matrix kernel. - Add the public `@paperclipai/paperclip-runner/evals` compatibility and native execution contracts. - Add fail-closed runnerd artifact and protocol compatibility checks. - Add deterministic workflow catalogs, scoring, traceability, and report generation. - Add sanitized Codex, OpenCode, and ACPX fixtures. - Add package-boundary and clean-consumer checks. - Add the eval package manifest to the Docker dependency stage. - Add the generated protocol fixture digest without changing the lockfile. ## Verification GitHub Actions must run: - Runner TypeScript and Rust type checks. - Runner unit and protocol tests. - Evaluation kernel tests. - Workflow traceability checks. - Clean-consumer and package-boundary checks. - Repository test, type-check, build, policy, and security gates. No local test command was run. The repository owner requested GitHub-only verification. ## Risks This is a large greenfield review surface with 51 files. The code does not launch a live provider or load credentials. Package and protocol drift fail closed. The workspace lockfile remains under the existing CI-owned process. ## Model Used OpenAI Codex with the GPT-5 agent model. The work used high reasoning, repository inspection, tool use, and parallel code review. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge
66 lines
8.7 KiB
JSON
66 lines
8.7 KiB
JSON
{
|
|
"schema": "paperclip.runner.stress-eval-traceability.v1",
|
|
"campaign": "2026-08-23-paperclip-runner-stress-campaign",
|
|
"expectedFindings": 44,
|
|
"workflows": [
|
|
"final-response",
|
|
"rich-activity",
|
|
"verification-policy",
|
|
"governed-interaction",
|
|
"steering-causality",
|
|
"planning-lifecycle",
|
|
"review-lifecycle",
|
|
"delegation-return",
|
|
"completion-robustness",
|
|
"restart-recovery",
|
|
"cancellation-permissions",
|
|
"trace-lineage"
|
|
],
|
|
"findings": [
|
|
{ "id": "STRESS-001", "classification": "workflow_eval", "workflowIds": ["governed-interaction"], "regressionTests": ["src/contracts/native-execution.test.ts"] },
|
|
{ "id": "STRESS-002", "classification": "workflow_eval", "workflowIds": ["restart-recovery"], "regressionTests": ["src/backends/harness-driver-backend.test.ts"] },
|
|
{ "id": "STRESS-003", "classification": "workflow_eval", "workflowIds": ["governed-interaction"], "regressionTests": ["src/contracts/native-execution.test.ts"] },
|
|
{ "id": "STRESS-004", "classification": "workflow_eval", "workflowIds": ["restart-recovery"], "regressionTests": ["../../server/src/__tests__/heartbeat-workspace-session.test.ts"] },
|
|
{ "id": "STRESS-005", "classification": "workflow_eval", "workflowIds": ["cancellation-permissions"], "regressionTests": ["src/drivers/acpx/codex-acpx-driver.test.ts"] },
|
|
{ "id": "STRESS-006", "classification": "workflow_eval", "workflowIds": ["final-response", "verification-policy"], "regressionTests": ["src/drivers/opencode/opencode-server-driver.test.ts"] },
|
|
{ "id": "STRESS-007", "classification": "workflow_eval", "workflowIds": ["completion-robustness"], "regressionTests": ["src/protocol/result-normalization.test.ts"] },
|
|
{ "id": "STRESS-008", "classification": "workflow_eval", "workflowIds": ["restart-recovery"], "regressionTests": ["src/native-session-runtime.test.ts"] },
|
|
{ "id": "STRESS-009", "classification": "workflow_eval", "workflowIds": ["completion-robustness"], "regressionTests": ["src/drivers/opencode/mcp-bridge.test.ts"] },
|
|
{ "id": "STRESS-010", "classification": "workflow_eval", "workflowIds": ["completion-robustness"], "regressionTests": ["../../server/src/__tests__/native-finalization-recovery.test.ts"] },
|
|
{ "id": "STRESS-011", "classification": "workflow_eval", "workflowIds": ["rich-activity"], "regressionTests": ["src/drivers/opencode/opencode-server-driver.test.ts"] },
|
|
{ "id": "STRESS-012", "classification": "workflow_eval", "workflowIds": ["final-response"], "regressionTests": ["src/drivers/opencode/opencode-server-driver.test.ts"] },
|
|
{ "id": "STRESS-013", "classification": "workflow_eval", "workflowIds": ["final-response", "restart-recovery"], "regressionTests": ["../../server/src/__tests__/heartbeat-run-summary.test.ts"] },
|
|
{ "id": "STRESS-014", "classification": "workflow_eval", "workflowIds": ["completion-robustness"], "regressionTests": ["../../server/src/__tests__/issue-recovery-actions.test.ts"] },
|
|
{ "id": "STRESS-015", "classification": "workflow_eval", "workflowIds": ["steering-causality"], "regressionTests": ["../../ui/src/components/TaskChatThread.test.tsx"] },
|
|
{ "id": "STRESS-016", "classification": "workflow_eval", "workflowIds": ["review-lifecycle"], "regressionTests": ["../../ui/src/components/IssueThreadInteractionCard.test.tsx"] },
|
|
{ "id": "STRESS-017", "classification": "workflow_eval", "workflowIds": ["cancellation-permissions"], "regressionTests": ["../../ui/src/components/task-chat/transcript-adapter.test.ts"] },
|
|
{ "id": "STRESS-018", "classification": "workflow_eval", "workflowIds": ["restart-recovery", "completion-robustness"], "regressionTests": ["../../server/src/services/native-runtime/native-session-executor.test.ts"] },
|
|
{ "id": "STRESS-019", "classification": "workflow_eval", "workflowIds": ["restart-recovery", "governed-interaction"], "regressionTests": ["src/backends/harness-driver-backend.test.ts"] },
|
|
{ "id": "STRESS-020", "classification": "workflow_eval", "workflowIds": ["completion-robustness"], "regressionTests": ["../../server/src/__tests__/heartbeat-process-recovery.test.ts", "../../server/src/__tests__/heartbeat-comment-wake-batching.test.ts"] },
|
|
{ "id": "STRESS-021", "classification": "workflow_eval", "workflowIds": ["governed-interaction"], "regressionTests": ["../../server/src/services/native-runtime/native-session-executor.test.ts"] },
|
|
{ "id": "STRESS-022", "classification": "workflow_eval", "workflowIds": ["governed-interaction", "restart-recovery"], "regressionTests": ["src/drivers/acpx/codex-acpx-driver.test.ts"] },
|
|
{ "id": "STRESS-023", "classification": "workflow_eval", "workflowIds": ["cancellation-permissions"], "regressionTests": ["../../server/src/services/native-runtime/native-session-executor.test.ts"] },
|
|
{ "id": "STRESS-024", "classification": "workflow_eval", "workflowIds": ["cancellation-permissions"], "regressionTests": ["src/drivers/acpx/codex-acpx-driver.test.ts"] },
|
|
{ "id": "STRESS-025", "classification": "workflow_eval", "workflowIds": ["review-lifecycle"], "regressionTests": ["../../ui/src/context/LiveUpdatesProvider.test.ts"] },
|
|
{ "id": "STRESS-026", "classification": "workflow_eval", "workflowIds": ["planning-lifecycle"], "regressionTests": ["../../server/src/__tests__/issue-thread-interaction-routes.test.ts"] },
|
|
{ "id": "STRESS-027", "classification": "workflow_eval", "workflowIds": ["delegation-return"], "regressionTests": ["../../server/src/__tests__/heartbeat-dependency-scheduling.test.ts"] },
|
|
{ "id": "STRESS-028", "classification": "workflow_eval", "workflowIds": ["delegation-return"], "regressionTests": ["../../server/src/services/native-runtime/native-session-resume.test.ts"] },
|
|
{ "id": "STRESS-029", "classification": "workflow_eval", "workflowIds": ["restart-recovery"], "regressionTests": ["../../server/src/__tests__/native-finalization-recovery.test.ts"] },
|
|
{ "id": "STRESS-030", "classification": "workflow_eval", "workflowIds": ["review-lifecycle"], "regressionTests": ["../../server/src/services/native-runtime/evidence-classifier.test.ts"] },
|
|
{ "id": "STRESS-031", "classification": "workflow_eval", "workflowIds": ["governed-interaction"], "regressionTests": ["../../server/src/__tests__/native-interaction-bridge.test.ts"] },
|
|
{ "id": "STRESS-032", "classification": "workflow_eval", "workflowIds": ["governed-interaction"], "regressionTests": ["../../server/src/services/native-runtime/native-session-executor.test.ts"] },
|
|
{ "id": "STRESS-033", "classification": "workflow_eval", "workflowIds": ["governed-interaction", "final-response"], "regressionTests": ["../../server/src/__tests__/heartbeat-run-summary.test.ts"] },
|
|
{ "id": "STRESS-034", "classification": "workflow_eval", "workflowIds": ["governed-interaction"], "regressionTests": ["src/drivers/opencode/opencode-server-driver.test.ts"] },
|
|
{ "id": "STRESS-035", "classification": "workflow_eval", "workflowIds": ["cancellation-permissions"], "regressionTests": ["src/drivers/opencode/opencode-server-driver.test.ts"] },
|
|
{ "id": "STRESS-036", "classification": "workflow_eval", "workflowIds": ["final-response"], "regressionTests": ["src/drivers/acpx/codex-acpx-driver.test.ts"] },
|
|
{ "id": "STRESS-037", "classification": "workflow_eval", "workflowIds": ["cancellation-permissions", "completion-robustness"], "regressionTests": ["src/drivers/acpx/runtime-host.test.ts"] },
|
|
{ "id": "STRESS-038", "classification": "workflow_eval", "workflowIds": ["trace-lineage"], "regressionTests": ["../../server/src/__tests__/provider-trace-store.test.ts"] },
|
|
{ "id": "STRESS-039", "classification": "workflow_eval", "workflowIds": ["trace-lineage"], "regressionTests": ["src/drivers/acpx/runtime-host.test.ts"] },
|
|
{ "id": "STRESS-040", "classification": "explicit_exclusion", "workflowIds": [], "regressionTests": [], "reason": "Campaign control cleanup is an operational hygiene check, not candidate behavior." },
|
|
{ "id": "STRESS-041", "classification": "workflow_eval", "workflowIds": ["final-response", "completion-robustness"], "regressionTests": ["../../server/src/__tests__/heartbeat-running-followup.test.ts"] },
|
|
{ "id": "STRESS-042", "classification": "regression_test", "workflowIds": [], "regressionTests": ["../../server/src/__tests__/openapi-routes.test.ts"], "reason": "Mounted-route documentation parity is deterministic API inventory." },
|
|
{ "id": "STRESS-043", "classification": "regression_test", "workflowIds": [], "regressionTests": ["../../server/src/__tests__/server-startup-feedback-export.test.ts"], "reason": "The HTTP listener test double is ordinary startup integration behavior." },
|
|
{ "id": "STRESS-044", "classification": "regression_test", "workflowIds": [], "regressionTests": ["../../server/src/__tests__/company-skills-service.test.ts"], "reason": "Filesystem path portability is deterministic service behavior." }
|
|
]
|
|
}
|