Jannik/PaperClipAI
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-10-27 11:20:42 +01:00
Code Issues Packages Projects Releases Wiki Activity

Update private task enforcement with master

Browse Source 
Preserve the reviewed privacy rules and existing PR history.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip committed 2026-10-06 20:36:10 -05:00
2 parents c7c9bf3510 dd05c14e87
commit 25e92218db
152 files changed
+12263 -5564
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

doc/ISSUE-PRIVACY.md
+2 -2
View File
@@ -130,7 +130,7 @@ are read from the database without a cross-request positive cache.
## Migration and current runtime integration
Migration `0311_private_task_access.sql` adds the task/project ACL tables and
Migration `0312_private_task_access.sql` adds the task/project ACL tables and
backfills privacy parent edges with indexed keyset batches. Run binding is derived
from native task identity, explicit issue identity, or legacy `issueId`/`taskId` context. Missing
source tasks remain issue-scoped tombstones. Context changes cannot turn those
@@ -138,7 +138,7 @@ runs into company-wide history. The migration is idempotent for preview installs
existing explicit grants are preserved. Operators upgrading an unreleased preview
should inspect root grants created under its former whole-tree sharing semantics.
Migration `0312_private_task_draft_assets.sql` accompanies enforcement and binds
Migration `0313_private_task_draft_assets.sql` accompanies enforcement and binds
historical inline images to their first owned task. Unbound drafts remain
uploader-only under the new asset-content guard.
doc/connections/AI-CONNECTIONS.md
+48 -3
View File
@@ -193,6 +193,17 @@ and contain neither credentials nor raw provider errors. Expired credentials
report `authentication_required`; the probe does not exchange refresh tokens
or change connection health.
The Costs dashboard uses the company quota endpoint. It reads each authorized
managed subscription separately, retaining the last successful observation on
transient errors. For managed Codex accounts, a 401 can trigger one persisted
OAuth refresh; it defers while potentially competing OpenAI work is active.
Refresh takes the company secret-mutation lock before database row locks and
requires those row locks immediately, before exchanging a single-use token.
Reconnect and runtime credential write-back follow the same lock order.
Credential resolution checks its version after any database wait and reloads a
rotated value before giving it to a new run. Unknown utilization has no progress
bar, but any provider-reported reset time remains visible.
Verification:
```sh
@@ -498,9 +509,10 @@ existing account name. Connecting installs access for that agent and resumes the
work automatically. Explicit incompatible bindings and shared-account permission
denials still fail; hiring never expands a restricted shared account's audience.
Concurrent runs of one subscription do not wait for each other. No credential
lease exists to hold them, so a fresh task execution cannot enter a contention
wait. A run that already entered this wait keeps its scheduled retries. It does
Concurrent runs of one subscription do not hold a credential lease. A fresh
task can briefly wait when a credential rotation holds the company file lock or
a grant/secret database row lock. Lock timeouts become `ai_connection_busy`,
keeping the task on its automatic pre-provider scheduled retry path. It does
not request new credentials, and it does not consume the provider-failure retry
allowance. Each retry revalidates the account, and existing run-dispatch rules
still suppress cancelled, reassigned, or otherwise ineligible work. An assignee
@@ -524,6 +536,39 @@ requires a fresh session. The metadata is removed before passing session params
to an adapter. Temporary authentication-home paths do not change the configuration
fingerprint. These checks do not relax current connection authorization.
Quota polling has a 20-second response deadline. Once a managed OAuth refresh
starts, it has its own 60-second request lifetime so the replacement token body
can still be read and committed after the dashboard stops waiting. A successful
refreshed observation uses the saved grant/secret revision as its cache identity.
A reconnect during credential resolution defers the poll instead of associating
one account's quota with another revision. Both Costs surfaces retain matching
successful observations on transient failures and clear them on authentication
failure or credential rotation.
Managed runtime token write-back retries a company-lock timeout twice (three
30-second acquisition attempts), so a slow quota exchange does not discard a
different account's replacement tokens. Any remaining write-back error preserves
the private runtime home for retry; cleanup deletes it only after a successful
transaction or an intentional freshness/authorization discard. These retained
homes are recovery evidence, not a background replay queue; persistent database
or lock failures still require operator intervention.
Quota OAuth replacement tokens are encrypted with the instance secrets master
key and fsynced under `<instance-root>/quota-credential-recovery/<company-id>/`
before vault, grant, or activity writes. Storage and encryption are checked before
exchanging a single-use refresh token. Failed saves retry without another OAuth
exchange; persistent failures keep the encrypted record for the next quota poll,
including after a restart. New OpenAI subscription runtimes serialize credential
reads with quota exchanges and recover a matching pending replacement before
materializing an auth home. Authentication-failure handling also recovers a
matching replacement before marking the grant invalid. A failed recovery save
defers these actions and leaves both the active grant and journal intact.
The journal is removed only after database commit (or
when an authorized newer credential makes it obsolete). Replay checks the grant,
connection, secret, and original credential fingerprint and cannot reactivate a
revoked grant or overwrite a reconnect. Back up this directory and the instance
secrets key with the instance data. Loss of durable storage while receiving a
provider token can still require reconnecting the account.
## Advanced provider routing (2026-10-02)
doc/plans/2026-10-06-expanded-assistant-mcp-eval-evidence.jsonl
+106
View File
@@ -0,0 +1,106 @@
{"billing": {"agentRunDurationMs": 18353, "assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0, "inputTokens": 0, "outputTokens": 0, "requests": 0}, "estimatedRuntimeCostUsd": 0, "leaseDurationMs": 0, "llm": {"cachedInputTokens": 68608, "costStatus": "unpriced", "inputTokens": 87888, "outputTokens": 874, "reportedCostUsd": 0, "runCount": 1, "runsWithReportedCost": 0, "runsWithTokenUsage": 1, "totalTokens": 157370}, "observedAndEstimatedCostUsd": 0, "reportedLlmCostUsd": 0, "testCount": 1, "testsWithCompleteBilling": 0}, "campaignId": "local-2026-10-06T19-42-13-410Z", "cleanupPassed": true, "complete": false, "executed": 1, "expectedSelectedCases": ["public-mcp.assistant-codex-mini.local.expanded-task-edit"], "fullSuiteCoverage": [{"billing": {"agentRunDurationMs": 18353, "assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0, "inputTokens": 0, "outputTokens": 0, "requests": 0}, "estimatedRuntimeCostUsd": 0, "leaseDurationMs": 0, "llm": {"cachedInputTokens": 68608, "costStatus": "unpriced", "inputTokens": 87888, "outputTokens": 874, "reportedCostUsd": 0, "runCount": 1, "runsWithReportedCost": 0, "runsWithTokenUsage": 1, "totalTokens": 157370}, "observedAndEstimatedCostUsd": 0, "reportedLlmCostUsd": 0, "testCount": 1, "testsWithCompleteBilling": 0}, "cleanupPassed": true, "complete": false, "durationMs": 51610, "executed": 1, "expected": 63, "failed": 1, "incomplete": 0, "passed": 0, "retries": 0, "selected": 1, "suiteDefinitionHash": "a0f2d073f6fdb204b6bb965f7013cac2069b0b0b752bad51d46589b36f232376", "suiteId": "public-mcp"}], "kind": "campaign", "passed": 0, "selected": 1, "source": {"eventName": null, "ref": null, "sha": null, "workflowRunUrl": null}}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0, "inputTokens": 0, "model": "gpt-5.4-mini", "observedModels": [], "outputTokens": 0, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 0}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0, "inputTokens": 0, "model": "gpt-5.4-mini", "observedModels": [], "outputTokens": 0, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 0}, "complete": false, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 68608, "costStatus": "unpriced", "inputTokens": 87888, "outputTokens": 874, "reportedCostUsd": 0, "runCount": 1, "runsWithReportedCost": 0, "runsWithTokenUsage": 1, "totalTokens": 157370}, "observedAndEstimatedCostUsd": 0, "reportedCostUsd": 0, "runtime": {"agentRunDurationMs": 18353, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T19-42-13-410Z", "caseId": "expanded-task-edit", "durationMs": 51610, "evidence": "tests/runner-e2e/results/local-2026-10-06T19-42-13-410Z/public-mcp/assistant-codex-mini/local/expanded-task-edit/attempt-1/result.json", "executionId": "public-mcp.assistant-codex-mini.local.expanded-task-edit", "failedMatchers": [], "finishedAt": "2026-10-06T19:44:10.891Z", "kind": "case", "model": "gpt-5.4-mini", "profileId": "assistant-codex-mini", "provider": "codex", "source": {"ref": null, "sha": null, "workflowRunUrl": null}, "startedAt": "2026-10-06T19:43:19.281Z", "status": "failed", "suiteDefinitionHash": "a0f2d073f6fdb204b6bb965f7013cac2069b0b0b752bad51d46589b36f232376"}
{"billing": {"agentRunDurationMs": 44623, "assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.14893699999999999, "inputTokens": 141327, "outputTokens": 1522, "requests": 5}, "estimatedRuntimeCostUsd": 0, "leaseDurationMs": 0, "llm": {"cachedInputTokens": 297727, "costStatus": "partial", "inputTokens": 102242, "outputTokens": 3998, "reportedCostUsd": 0.08013395, "runCount": 2, "runsWithReportedCost": 1, "runsWithTokenUsage": 2, "totalTokens": 403967}, "observedAndEstimatedCostUsd": 0.22907095, "reportedLlmCostUsd": 0.08013395, "testCount": 2, "testsWithCompleteBilling": 1}, "campaignId": "local-2026-10-06T19-46-03-353Z", "cleanupPassed": true, "complete": false, "executed": 2, "expectedSelectedCases": ["public-mcp.assistant-codex-mini.local.expanded-task-edit", "public-mcp.assistant-claude-haiku.local.expanded-task-edit"], "fullSuiteCoverage": [{"billing": {"agentRunDurationMs": 44623, "assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.14893699999999999, "inputTokens": 141327, "outputTokens": 1522, "requests": 5}, "estimatedRuntimeCostUsd": 0, "leaseDurationMs": 0, "llm": {"cachedInputTokens": 297727, "costStatus": "partial", "inputTokens": 102242, "outputTokens": 3998, "reportedCostUsd": 0.08013395, "runCount": 2, "runsWithReportedCost": 1, "runsWithTokenUsage": 2, "totalTokens": 403967}, "observedAndEstimatedCostUsd": 0.22907095, "reportedLlmCostUsd": 0.08013395, "testCount": 2, "testsWithCompleteBilling": 1}, "cleanupPassed": true, "complete": false, "durationMs": 97012, "executed": 2, "expected": 63, "failed": 2, "incomplete": 0, "passed": 0, "retries": 0, "selected": 2, "suiteDefinitionHash": "a0f2d073f6fdb204b6bb965f7013cac2069b0b0b752bad51d46589b36f232376", "suiteId": "public-mcp"}], "kind": "campaign", "passed": 0, "selected": 2, "source": {"eventName": null, "ref": "codex/expanded-assistant-mcp", "sha": "442b7d6d9750537c373f22ca4fffffcb94fe4a0b", "workflowRunUrl": null}}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.14893699999999999, "inputTokens": 141327, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 1522, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 5}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.14893699999999999, "inputTokens": 141327, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 1522, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 5}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 245887, "costStatus": "reported", "inputTokens": 31716, "outputTokens": 3183, "reportedCostUsd": 0.08013395, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 280786}, "observedAndEstimatedCostUsd": 0.22907095, "reportedCostUsd": 0.08013395, "runtime": {"agentRunDurationMs": 34614, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T19-46-03-353Z", "caseId": "expanded-task-edit", "durationMs": 73204, "evidence": "tests/runner-e2e/results/local-2026-10-06T19-46-03-353Z/public-mcp/assistant-claude-haiku/local/expanded-task-edit/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-haiku.local.expanded-task-edit", "failedMatchers": [], "finishedAt": "2026-10-06T19:48:34.176Z", "kind": "case", "model": "claude-haiku-4-5-20251001", "profileId": "assistant-claude-haiku", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "442b7d6d9750537c373f22ca4fffffcb94fe4a0b", "workflowRunUrl": null}, "startedAt": "2026-10-06T19:47:20.972Z", "status": "failed", "suiteDefinitionHash": "a0f2d073f6fdb204b6bb965f7013cac2069b0b0b752bad51d46589b36f232376"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0, "inputTokens": 0, "model": "gpt-5.4-mini", "observedModels": [], "outputTokens": 0, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 0}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0, "inputTokens": 0, "model": "gpt-5.4-mini", "observedModels": [], "outputTokens": 0, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 0}, "complete": false, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 51840, "costStatus": "unpriced", "inputTokens": 70526, "outputTokens": 815, "reportedCostUsd": 0, "runCount": 1, "runsWithReportedCost": 0, "runsWithTokenUsage": 1, "totalTokens": 123181}, "observedAndEstimatedCostUsd": 0, "reportedCostUsd": 0, "runtime": {"agentRunDurationMs": 10009, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T19-46-03-353Z", "caseId": "expanded-task-edit", "durationMs": 23808, "evidence": "tests/runner-e2e/results/local-2026-10-06T19-46-03-353Z/public-mcp/assistant-codex-mini/local/expanded-task-edit/attempt-1/result.json", "executionId": "public-mcp.assistant-codex-mini.local.expanded-task-edit", "failedMatchers": [], "finishedAt": "2026-10-06T19:46:53.272Z", "kind": "case", "model": "gpt-5.4-mini", "profileId": "assistant-codex-mini", "provider": "codex", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "442b7d6d9750537c373f22ca4fffffcb94fe4a0b", "workflowRunUrl": null}, "startedAt": "2026-10-06T19:46:29.464Z", "status": "failed", "suiteDefinitionHash": "a0f2d073f6fdb204b6bb965f7013cac2069b0b0b752bad51d46589b36f232376"}
{"billing": null, "campaignId": "local-2026-10-06T19-53-49-845Z", "cleanupPassed": null, "complete": false, "executed": 13, "expectedSelectedCases": null, "fullSuiteCoverage": null, "kind": "campaign", "passed": 2, "selected": null, "source": null}
{"assistantUsage": null, "billing": {}, "campaignId": "local-2026-10-06T19-53-49-845Z", "caseId": "expanded-agent-config", "durationMs": 50521, "evidence": "tests/runner-e2e/results/local-2026-10-06T19-53-49-845Z/public-mcp/assistant-claude-haiku/local/expanded-agent-config/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-haiku.local.expanded-agent-config", "failedMatchers": [], "finishedAt": "2026-10-06T19:59:08.770Z", "kind": "case", "model": "claude-haiku-4-5-20251001", "profileId": "assistant-claude-haiku", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "f1cee3d2f8b820032f8829633847fcdfd6a4f8dd", "workflowRunUrl": null}, "startedAt": "2026-10-06T19:58:18.249Z", "status": "failed", "suiteDefinitionHash": "7e6b793d3fbf8c98faf9e935bbe8f9482d9e9db30491a7d8e1b894542a40609e"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.258942, "inputTokens": 251702, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 1448, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 9}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.258942, "inputTokens": 251702, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 1448, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 9}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 110314, "costStatus": "reported", "inputTokens": 12863, "outputTokens": 2171, "reportedCostUsd": 0.03795465, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 125348}, "observedAndEstimatedCostUsd": 0.29689665, "reportedCostUsd": 0.03795465, "runtime": {"agentRunDurationMs": 23483, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T19-53-49-845Z", "caseId": "expanded-documents", "durationMs": 52690, "evidence": "tests/runner-e2e/results/local-2026-10-06T19-53-49-845Z/public-mcp/assistant-claude-haiku/local/expanded-documents/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-haiku.local.expanded-documents", "failedMatchers": [], "finishedAt": "2026-10-06T19:58:16.874Z", "kind": "case", "model": "claude-haiku-4-5-20251001", "profileId": "assistant-claude-haiku", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "f1cee3d2f8b820032f8829633847fcdfd6a4f8dd", "workflowRunUrl": null}, "startedAt": "2026-10-06T19:57:24.184Z", "status": "passed", "suiteDefinitionHash": "7e6b793d3fbf8c98faf9e935bbe8f9482d9e9db30491a7d8e1b894542a40609e"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.085577, "inputTokens": 82587, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 598, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 3}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.085577, "inputTokens": 82587, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 598, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 3}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 287443, "costStatus": "reported", "inputTokens": 29691, "outputTokens": 2556, "reportedCostUsd": 0.0786213, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 319690}, "observedAndEstimatedCostUsd": 0.16419830000000002, "reportedCostUsd": 0.0786213, "runtime": {"agentRunDurationMs": 28435, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T19-53-49-845Z", "caseId": "expanded-files", "durationMs": 44143, "evidence": "tests/runner-e2e/results/local-2026-10-06T19-53-49-845Z/public-mcp/assistant-claude-haiku/local/expanded-files/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-haiku.local.expanded-files", "failedMatchers": [], "finishedAt": "2026-10-06T19:58:26.447Z", "kind": "case", "model": "claude-haiku-4-5-20251001", "profileId": "assistant-claude-haiku", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "f1cee3d2f8b820032f8829633847fcdfd6a4f8dd", "workflowRunUrl": null}, "startedAt": "2026-10-06T19:57:42.304Z", "status": "failed", "suiteDefinitionHash": "7e6b793d3fbf8c98faf9e935bbe8f9482d9e9db30491a7d8e1b894542a40609e"}
{"assistantUsage": null, "billing": {}, "campaignId": "local-2026-10-06T19-53-49-845Z", "caseId": "expanded-projects", "durationMs": 41315, "evidence": "tests/runner-e2e/results/local-2026-10-06T19-53-49-845Z/public-mcp/assistant-claude-haiku/local/expanded-projects/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-haiku.local.expanded-projects", "failedMatchers": [], "finishedAt": "2026-10-06T19:59:08.986Z", "kind": "case", "model": "claude-haiku-4-5-20251001", "profileId": "assistant-claude-haiku", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "f1cee3d2f8b820032f8829633847fcdfd6a4f8dd", "workflowRunUrl": null}, "startedAt": "2026-10-06T19:58:27.671Z", "status": "failed", "suiteDefinitionHash": "7e6b793d3fbf8c98faf9e935bbe8f9482d9e9db30491a7d8e1b894542a40609e"}
{"assistantUsage": null, "billing": {}, "campaignId": "local-2026-10-06T19-53-49-845Z", "caseId": "expanded-task-edit", "durationMs": 12334, "evidence": "tests/runner-e2e/results/local-2026-10-06T19-53-49-845Z/public-mcp/assistant-claude-haiku/local/expanded-task-edit/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-haiku.local.expanded-task-edit", "failedMatchers": [], "finishedAt": "2026-10-06T19:57:07.423Z", "kind": "case", "model": "claude-haiku-4-5-20251001", "profileId": "assistant-claude-haiku", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "f1cee3d2f8b820032f8829633847fcdfd6a4f8dd", "workflowRunUrl": null}, "startedAt": "2026-10-06T19:56:55.089Z", "status": "failed", "suiteDefinitionHash": "7e6b793d3fbf8c98faf9e935bbe8f9482d9e9db30491a7d8e1b894542a40609e"}
{"assistantUsage": null, "billing": {}, "campaignId": "local-2026-10-06T19-53-49-845Z", "caseId": "expanded-agent-config", "durationMs": 20214, "evidence": "tests/runner-e2e/results/local-2026-10-06T19-53-49-845Z/public-mcp/assistant-codex-mini/local/expanded-agent-config/attempt-1/result.json", "executionId": "public-mcp.assistant-codex-mini.local.expanded-agent-config", "failedMatchers": [], "finishedAt": "2026-10-06T19:56:21.690Z", "kind": "case", "model": "gpt-5.4-mini", "profileId": "assistant-codex-mini", "provider": "codex", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "f1cee3d2f8b820032f8829633847fcdfd6a4f8dd", "workflowRunUrl": null}, "startedAt": "2026-10-06T19:56:01.476Z", "status": "failed", "suiteDefinitionHash": "7e6b793d3fbf8c98faf9e935bbe8f9482d9e9db30491a7d8e1b894542a40609e"}
{"assistantUsage": null, "billing": {}, "campaignId": "local-2026-10-06T19-53-49-845Z", "caseId": "expanded-api", "durationMs": 6617, "evidence": "tests/runner-e2e/results/local-2026-10-06T19-53-49-845Z/public-mcp/assistant-codex-mini/local/expanded-api/attempt-1/result.json", "executionId": "public-mcp.assistant-codex-mini.local.expanded-api", "failedMatchers": [], "finishedAt": "2026-10-06T19:56:52.115Z", "kind": "case", "model": "gpt-5.4-mini", "profileId": "assistant-codex-mini", "provider": "codex", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "f1cee3d2f8b820032f8829633847fcdfd6a4f8dd", "workflowRunUrl": null}, "startedAt": "2026-10-06T19:56:45.498Z", "status": "failed", "suiteDefinitionHash": "7e6b793d3fbf8c98faf9e935bbe8f9482d9e9db30491a7d8e1b894542a40609e"}
{"assistantUsage": {"cachedInputTokens": 149504, "estimatedCostUsd": 0.03298905, "inputTokens": 24487, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 758, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 9}, "billing": {"assistant": {"cachedInputTokens": 149504, "estimatedCostUsd": 0.03298905, "inputTokens": 24487, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 758, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 9}, "complete": false, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 52352, "costStatus": "unpriced", "inputTokens": 71673, "outputTokens": 785, "reportedCostUsd": 0, "runCount": 1, "runsWithReportedCost": 0, "runsWithTokenUsage": 1, "totalTokens": 124810}, "observedAndEstimatedCostUsd": 0.03298905, "reportedCostUsd": 0, "runtime": {"agentRunDurationMs": 11426, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T19-53-49-845Z", "caseId": "expanded-documents", "durationMs": 64793, "evidence": "tests/runner-e2e/results/local-2026-10-06T19-53-49-845Z/public-mcp/assistant-codex-mini/local/expanded-documents/attempt-1/result.json", "executionId": "public-mcp.assistant-codex-mini.local.expanded-documents", "failedMatchers": [], "finishedAt": "2026-10-06T19:55:25.374Z", "kind": "case", "model": "gpt-5.4-mini", "profileId": "assistant-codex-mini", "provider": "codex", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "f1cee3d2f8b820032f8829633847fcdfd6a4f8dd", "workflowRunUrl": null}, "startedAt": "2026-10-06T19:54:20.581Z", "status": "passed", "suiteDefinitionHash": "7e6b793d3fbf8c98faf9e935bbe8f9482d9e9db30491a7d8e1b894542a40609e"}
{"assistantUsage": {"cachedInputTokens": 53248, "estimatedCostUsd": 0.00863985, "inputTokens": 3849, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 391, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 3}, "billing": {"assistant": {"cachedInputTokens": 53248, "estimatedCostUsd": 0.00863985, "inputTokens": 3849, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 391, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 3}, "complete": false, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 53376, "costStatus": "unpriced", "inputTokens": 72251, "outputTokens": 1431, "reportedCostUsd": 0, "runCount": 1, "runsWithReportedCost": 0, "runsWithTokenUsage": 1, "totalTokens": 127058}, "observedAndEstimatedCostUsd": 0.00863985, "reportedCostUsd": 0, "runtime": {"agentRunDurationMs": 10522, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T19-53-49-845Z", "caseId": "expanded-files", "durationMs": 43214, "evidence": "tests/runner-e2e/results/local-2026-10-06T19-53-49-845Z/public-mcp/assistant-codex-mini/local/expanded-files/attempt-1/result.json", "executionId": "public-mcp.assistant-codex-mini.local.expanded-files", "failedMatchers": [], "finishedAt": "2026-10-06T19:56:53.471Z", "kind": "case", "model": "gpt-5.4-mini", "profileId": "assistant-codex-mini", "provider": "codex", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "f1cee3d2f8b820032f8829633847fcdfd6a4f8dd", "workflowRunUrl": null}, "startedAt": "2026-10-06T19:56:10.257Z", "status": "failed", "suiteDefinitionHash": "7e6b793d3fbf8c98faf9e935bbe8f9482d9e9db30491a7d8e1b894542a40609e"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0, "inputTokens": 0, "model": "gpt-5.4-mini", "observedModels": [], "outputTokens": 0, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 0}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0, "inputTokens": 0, "model": "gpt-5.4-mini", "observedModels": [], "outputTokens": 0, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 0}, "complete": false, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 47232, "costStatus": "unpriced", "inputTokens": 52304, "outputTokens": 653, "reportedCostUsd": 0, "runCount": 2, "runsWithReportedCost": 0, "runsWithTokenUsage": 1, "totalTokens": 100189}, "observedAndEstimatedCostUsd": 0, "reportedCostUsd": 0, "runtime": {"agentRunDurationMs": 8459, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T19-53-49-845Z", "caseId": "expanded-permissions", "durationMs": 18646, "evidence": "tests/runner-e2e/results/local-2026-10-06T19-53-49-845Z/public-mcp/assistant-codex-mini/local/expanded-permissions/attempt-1/result.json", "executionId": "public-mcp.assistant-codex-mini.local.expanded-permissions", "failedMatchers": [], "finishedAt": "2026-10-06T19:57:24.501Z", "kind": "case", "model": "gpt-5.4-mini", "profileId": "assistant-codex-mini", "provider": "codex", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "f1cee3d2f8b820032f8829633847fcdfd6a4f8dd", "workflowRunUrl": null}, "startedAt": "2026-10-06T19:57:05.855Z", "status": "failed", "suiteDefinitionHash": "7e6b793d3fbf8c98faf9e935bbe8f9482d9e9db30491a7d8e1b894542a40609e"}
{"assistantUsage": null, "billing": {}, "campaignId": "local-2026-10-06T19-53-49-845Z", "caseId": "expanded-projects", "durationMs": 16511, "evidence": "tests/runner-e2e/results/local-2026-10-06T19-53-49-845Z/public-mcp/assistant-codex-mini/local/expanded-projects/attempt-1/result.json", "executionId": "public-mcp.assistant-codex-mini.local.expanded-projects", "failedMatchers": [], "finishedAt": "2026-10-06T19:56:38.314Z", "kind": "case", "model": "gpt-5.4-mini", "profileId": "assistant-codex-mini", "provider": "codex", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "f1cee3d2f8b820032f8829633847fcdfd6a4f8dd", "workflowRunUrl": null}, "startedAt": "2026-10-06T19:56:21.803Z", "status": "failed", "suiteDefinitionHash": "7e6b793d3fbf8c98faf9e935bbe8f9482d9e9db30491a7d8e1b894542a40609e"}
{"assistantUsage": null, "billing": {}, "campaignId": "local-2026-10-06T19-53-49-845Z", "caseId": "expanded-skills", "durationMs": 6615, "evidence": "tests/runner-e2e/results/local-2026-10-06T19-53-49-845Z/public-mcp/assistant-codex-mini/local/expanded-skills/attempt-1/result.json", "executionId": "public-mcp.assistant-codex-mini.local.expanded-skills", "failedMatchers": [], "finishedAt": "2026-10-06T19:56:45.279Z", "kind": "case", "model": "gpt-5.4-mini", "profileId": "assistant-codex-mini", "provider": "codex", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "f1cee3d2f8b820032f8829633847fcdfd6a4f8dd", "workflowRunUrl": null}, "startedAt": "2026-10-06T19:56:38.664Z", "status": "failed", "suiteDefinitionHash": "7e6b793d3fbf8c98faf9e935bbe8f9482d9e9db30491a7d8e1b894542a40609e"}
{"assistantUsage": {"cachedInputTokens": 114176, "estimatedCostUsd": 0.0151512, "inputTokens": 4812, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 662, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 6}, "billing": {"assistant": {"cachedInputTokens": 114176, "estimatedCostUsd": 0.0151512, "inputTokens": 4812, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 662, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 6}, "complete": false, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 71168, "costStatus": "unpriced", "inputTokens": 110757, "outputTokens": 1487, "reportedCostUsd": 0, "runCount": 1, "runsWithReportedCost": 0, "runsWithTokenUsage": 1, "totalTokens": 183412}, "observedAndEstimatedCostUsd": 0.0151512, "reportedCostUsd": 0, "runtime": {"agentRunDurationMs": 16327, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T19-53-49-845Z", "caseId": "expanded-task-edit", "durationMs": 95793, "evidence": "tests/runner-e2e/results/local-2026-10-06T19-53-49-845Z/public-mcp/assistant-codex-mini/local/expanded-task-edit/attempt-1/result.json", "executionId": "public-mcp.assistant-codex-mini.local.expanded-task-edit", "failedMatchers": [], "finishedAt": "2026-10-06T19:55:56.375Z", "kind": "case", "model": "gpt-5.4-mini", "profileId": "assistant-codex-mini", "provider": "codex", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "f1cee3d2f8b820032f8829633847fcdfd6a4f8dd", "workflowRunUrl": null}, "startedAt": "2026-10-06T19:54:20.582Z", "status": "failed", "suiteDefinitionHash": "7e6b793d3fbf8c98faf9e935bbe8f9482d9e9db30491a7d8e1b894542a40609e"}
{"billing": {"agentRunDurationMs": 254619, "assistant": {"cachedInputTokens": 650752, "estimatedCostUsd": 1.5112224, "inputTokens": 1414341, "outputTokens": 13955, "requests": 78}, "estimatedRuntimeCostUsd": 0, "leaseDurationMs": 0, "llm": {"cachedInputTokens": 1358615, "costStatus": "partial", "inputTokens": 780401, "outputTokens": 25547, "reportedCostUsd": 0.2931575, "runCount": 19, "runsWithReportedCost": 8, "runsWithTokenUsage": 16, "totalTokens": 2164563}, "observedAndEstimatedCostUsd": 1.8043799000000003, "reportedLlmCostUsd": 0.2931575, "testCount": 16, "testsWithCompleteBilling": 8}, "campaignId": "local-2026-10-06T20-04-15-928Z", "cleanupPassed": true, "complete": false, "executed": 16, "expectedSelectedCases": ["public-mcp.assistant-codex-mini.local.expanded-task-edit", "public-mcp.assistant-codex-mini.local.expanded-documents", "public-mcp.assistant-codex-mini.local.expanded-files", "public-mcp.assistant-codex-mini.local.expanded-agent-config", "public-mcp.assistant-codex-mini.local.expanded-projects", "public-mcp.assistant-codex-mini.local.expanded-skills", "public-mcp.assistant-codex-mini.local.expanded-api", "public-mcp.assistant-codex-mini.local.expanded-permissions", "public-mcp.assistant-claude-haiku.local.expanded-task-edit", "public-mcp.assistant-claude-haiku.local.expanded-documents", "public-mcp.assistant-claude-haiku.local.expanded-files", "public-mcp.assistant-claude-haiku.local.expanded-agent-config", "public-mcp.assistant-claude-haiku.local.expanded-projects", "public-mcp.assistant-claude-haiku.local.expanded-skills", "public-mcp.assistant-claude-haiku.local.expanded-api", "public-mcp.assistant-claude-haiku.local.expanded-permissions"], "fullSuiteCoverage": [{"billing": {"agentRunDurationMs": 254619, "assistant": {"cachedInputTokens": 650752, "estimatedCostUsd": 1.5112224, "inputTokens": 1414341, "outputTokens": 13955, "requests": 78}, "estimatedRuntimeCostUsd": 0, "leaseDurationMs": 0, "llm": {"cachedInputTokens": 1358615, "costStatus": "partial", "inputTokens": 780401, "outputTokens": 25547, "reportedCostUsd": 0.2931575, "runCount": 19, "runsWithReportedCost": 8, "runsWithTokenUsage": 16, "totalTokens": 2164563}, "observedAndEstimatedCostUsd": 1.8043799000000003, "reportedLlmCostUsd": 0.2931575, "testCount": 16, "testsWithCompleteBilling": 8}, "cleanupPassed": true, "complete": false, "durationMs": 624179, "executed": 16, "expected": 63, "failed": 4, "incomplete": 0, "passed": 12, "retries": 0, "selected": 16, "suiteDefinitionHash": "6051fbfd0ab10cf4365f5b44de02e9ae6496300037c266210986e010fdccca6e", "suiteId": "public-mcp"}], "kind": "campaign", "passed": 12, "selected": 16, "source": {"eventName": null, "ref": "codex/expanded-assistant-mcp", "sha": "c95cb72a988dcc02f81e6cbb8b0c4619dcf199d3", "workflowRunUrl": null}}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.14960400000000001, "inputTokens": 142854, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 1350, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 5}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.14960400000000001, "inputTokens": 142854, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 1350, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 5}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 110725, "costStatus": "reported", "inputTokens": 13244, "outputTokens": 1909, "reportedCostUsd": 0.037162, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 125878}, "observedAndEstimatedCostUsd": 0.18676600000000002, "reportedCostUsd": 0.037162, "runtime": {"agentRunDurationMs": 21435, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T20-04-15-928Z", "caseId": "expanded-agent-config", "durationMs": 46538, "evidence": "tests/runner-e2e/results/local-2026-10-06T20-04-15-928Z/public-mcp/assistant-claude-haiku/local/expanded-agent-config/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-haiku.local.expanded-agent-config", "failedMatchers": [], "finishedAt": "2026-10-06T20:09:40.405Z", "kind": "case", "model": "claude-haiku-4-5-20251001", "profileId": "assistant-claude-haiku", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "c95cb72a988dcc02f81e6cbb8b0c4619dcf199d3", "workflowRunUrl": null}, "startedAt": "2026-10-06T20:08:53.867Z", "status": "passed", "suiteDefinitionHash": "6051fbfd0ab10cf4365f5b44de02e9ae6496300037c266210986e010fdccca6e"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.151702, "inputTokens": 146177, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 1105, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 5}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.151702, "inputTokens": 146177, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 1105, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 5}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 138794, "costStatus": "reported", "inputTokens": 15125, "outputTokens": 2241, "reportedCostUsd": 0.04397815, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 156160}, "observedAndEstimatedCostUsd": 0.19568015, "reportedCostUsd": 0.04397815, "runtime": {"agentRunDurationMs": 25339, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T20-04-15-928Z", "caseId": "expanded-api", "durationMs": 50680, "evidence": "tests/runner-e2e/results/local-2026-10-06T20-04-15-928Z/public-mcp/assistant-claude-haiku/local/expanded-api/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-haiku.local.expanded-api", "failedMatchers": [], "finishedAt": "2026-10-06T20:11:35.446Z", "kind": "case", "model": "claude-haiku-4-5-20251001", "profileId": "assistant-claude-haiku", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "c95cb72a988dcc02f81e6cbb8b0c4619dcf199d3", "workflowRunUrl": null}, "startedAt": "2026-10-06T20:10:44.766Z", "status": "passed", "suiteDefinitionHash": "6051fbfd0ab10cf4365f5b44de02e9ae6496300037c266210986e010fdccca6e"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.29096900000000003, "inputTokens": 281474, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 1899, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 10}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.29096900000000003, "inputTokens": 281474, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 1899, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 10}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 110153, "costStatus": "reported", "inputTokens": 12930, "outputTokens": 2009, "reportedCostUsd": 0.037212300000000004, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 125092}, "observedAndEstimatedCostUsd": 0.3281813, "reportedCostUsd": 0.037212300000000004, "runtime": {"agentRunDurationMs": 21508, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T20-04-15-928Z", "caseId": "expanded-documents", "durationMs": 52781, "evidence": "tests/runner-e2e/results/local-2026-10-06T20-04-15-928Z/public-mcp/assistant-claude-haiku/local/expanded-documents/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-haiku.local.expanded-documents", "failedMatchers": [], "finishedAt": "2026-10-06T20:08:36.637Z", "kind": "case", "model": "claude-haiku-4-5-20251001", "profileId": "assistant-claude-haiku", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "c95cb72a988dcc02f81e6cbb8b0c4619dcf199d3", "workflowRunUrl": null}, "startedAt": "2026-10-06T20:07:43.856Z", "status": "passed", "suiteDefinitionHash": "6051fbfd0ab10cf4365f5b44de02e9ae6496300037c266210986e010fdccca6e"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.207943, "inputTokens": 201398, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 1309, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 7}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.207943, "inputTokens": 201398, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 1309, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 7}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 110954, "costStatus": "reported", "inputTokens": 13216, "outputTokens": 1946, "reportedCostUsd": 0.0373349, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 126116}, "observedAndEstimatedCostUsd": 0.2452779, "reportedCostUsd": 0.0373349, "runtime": {"agentRunDurationMs": 21031, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T20-04-15-928Z", "caseId": "expanded-files", "durationMs": 46360, "evidence": "tests/runner-e2e/results/local-2026-10-06T20-04-15-928Z/public-mcp/assistant-claude-haiku/local/expanded-files/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-haiku.local.expanded-files", "failedMatchers": [], "finishedAt": "2026-10-06T20:09:29.434Z", "kind": "case", "model": "claude-haiku-4-5-20251001", "profileId": "assistant-claude-haiku", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "c95cb72a988dcc02f81e6cbb8b0c4619dcf199d3", "workflowRunUrl": null}, "startedAt": "2026-10-06T20:08:43.074Z", "status": "passed", "suiteDefinitionHash": "6051fbfd0ab10cf4365f5b44de02e9ae6496300037c266210986e010fdccca6e"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.11301, "inputTokens": 109685, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 665, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 4}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.11301, "inputTokens": 109685, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 665, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 4}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 110328, "costStatus": "reported", "inputTokens": 12789, "outputTokens": 1738, "reportedCostUsd": 0.035698549999999996, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 124855}, "observedAndEstimatedCostUsd": 0.14870855, "reportedCostUsd": 0.035698549999999996, "runtime": {"agentRunDurationMs": 19276, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T20-04-15-928Z", "caseId": "expanded-permissions", "durationMs": 39191, "evidence": "tests/runner-e2e/results/local-2026-10-06T20-04-15-928Z/public-mcp/assistant-claude-haiku/local/expanded-permissions/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-haiku.local.expanded-permissions", "failedMatchers": [], "finishedAt": "2026-10-06T20:11:40.949Z", "kind": "case", "model": "claude-haiku-4-5-20251001", "profileId": "assistant-claude-haiku", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "c95cb72a988dcc02f81e6cbb8b0c4619dcf199d3", "workflowRunUrl": null}, "startedAt": "2026-10-06T20:11:01.758Z", "status": "passed", "suiteDefinitionHash": "6051fbfd0ab10cf4365f5b44de02e9ae6496300037c266210986e010fdccca6e"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.14163900000000001, "inputTokens": 137869, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 754, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 5}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.14163900000000001, "inputTokens": 137869, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 754, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 5}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 86317, "costStatus": "reported", "inputTokens": 12732, "outputTokens": 1761, "reportedCostUsd": 0.0333432, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 100810}, "observedAndEstimatedCostUsd": 0.17498220000000003, "reportedCostUsd": 0.0333432, "runtime": {"agentRunDurationMs": 18613, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T20-04-15-928Z", "caseId": "expanded-projects", "durationMs": 38092, "evidence": "tests/runner-e2e/results/local-2026-10-06T20-04-15-928Z/public-mcp/assistant-claude-haiku/local/expanded-projects/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-haiku.local.expanded-projects", "failedMatchers": [], "finishedAt": "2026-10-06T20:10:24.393Z", "kind": "case", "model": "claude-haiku-4-5-20251001", "profileId": "assistant-claude-haiku", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "c95cb72a988dcc02f81e6cbb8b0c4619dcf199d3", "workflowRunUrl": null}, "startedAt": "2026-10-06T20:09:46.301Z", "status": "passed", "suiteDefinitionHash": "6051fbfd0ab10cf4365f5b44de02e9ae6496300037c266210986e010fdccca6e"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.180478, "inputTokens": 172198, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 1656, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 6}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.180478, "inputTokens": 172198, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 1656, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 6}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 86260, "costStatus": "reported", "inputTokens": 12826, "outputTokens": 1776, "reportedCostUsd": 0.03353, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 100862}, "observedAndEstimatedCostUsd": 0.214008, "reportedCostUsd": 0.03353, "runtime": {"agentRunDurationMs": 18770, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T20-04-15-928Z", "caseId": "expanded-skills", "durationMs": 47323, "evidence": "tests/runner-e2e/results/local-2026-10-06T20-04-15-928Z/public-mcp/assistant-claude-haiku/local/expanded-skills/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-haiku.local.expanded-skills", "failedMatchers": [], "finishedAt": "2026-10-06T20:10:44.604Z", "kind": "case", "model": "claude-haiku-4-5-20251001", "profileId": "assistant-claude-haiku", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "c95cb72a988dcc02f81e6cbb8b0c4619dcf199d3", "workflowRunUrl": null}, "startedAt": "2026-10-06T20:09:57.281Z", "status": "passed", "suiteDefinitionHash": "6051fbfd0ab10cf4365f5b44de02e9ae6496300037c266210986e010fdccca6e"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.151114, "inputTokens": 143124, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 1598, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 5}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.151114, "inputTokens": 143124, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 1598, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 5}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 86044, "costStatus": "reported", "inputTokens": 12934, "outputTokens": 2027, "reportedCostUsd": 0.034898399999999996, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 101005}, "observedAndEstimatedCostUsd": 0.1860124, "reportedCostUsd": 0.034898399999999996, "runtime": {"agentRunDurationMs": 20947, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T20-04-15-928Z", "caseId": "expanded-task-edit", "durationMs": 45896, "evidence": "tests/runner-e2e/results/local-2026-10-06T20-04-15-928Z/public-mcp/assistant-claude-haiku/local/expanded-task-edit/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-haiku.local.expanded-task-edit", "failedMatchers": [], "finishedAt": "2026-10-06T20:08:24.725Z", "kind": "case", "model": "claude-haiku-4-5-20251001", "profileId": "assistant-claude-haiku", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "c95cb72a988dcc02f81e6cbb8b0c4619dcf199d3", "workflowRunUrl": null}, "startedAt": "2026-10-06T20:07:38.829Z", "status": "failed", "suiteDefinitionHash": "6051fbfd0ab10cf4365f5b44de02e9ae6496300037c266210986e010fdccca6e"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0, "inputTokens": 0, "model": "gpt-5.4-mini", "observedModels": [], "outputTokens": 0, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 0}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0, "inputTokens": 0, "model": "gpt-5.4-mini", "observedModels": [], "outputTokens": 0, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 0}, "complete": false, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 91008, "costStatus": "unpriced", "inputTokens": 110454, "outputTokens": 1672, "reportedCostUsd": 0, "runCount": 2, "runsWithReportedCost": 0, "runsWithTokenUsage": 1, "totalTokens": 203134}, "observedAndEstimatedCostUsd": 0, "reportedCostUsd": 0, "runtime": {"agentRunDurationMs": 13571, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T20-04-15-928Z", "caseId": "expanded-agent-config", "durationMs": 27558, "evidence": "tests/runner-e2e/results/local-2026-10-06T20-04-15-928Z/public-mcp/assistant-codex-mini/local/expanded-agent-config/attempt-1/result.json", "executionId": "public-mcp.assistant-codex-mini.local.expanded-agent-config", "failedMatchers": [], "finishedAt": "2026-10-06T20:05:48.601Z", "kind": "case", "model": "gpt-5.4-mini", "profileId": "assistant-codex-mini", "provider": "codex", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "c95cb72a988dcc02f81e6cbb8b0c4619dcf199d3", "workflowRunUrl": null}, "startedAt": "2026-10-06T20:05:21.043Z", "status": "failed", "suiteDefinitionHash": "6051fbfd0ab10cf4365f5b44de02e9ae6496300037c266210986e010fdccca6e"}
{"assistantUsage": {"cachedInputTokens": 170496, "estimatedCostUsd": 0.06121619999999999, "inputTokens": 61416, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 526, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 6}, "billing": {"assistant": {"cachedInputTokens": 170496, "estimatedCostUsd": 0.06121619999999999, "inputTokens": 61416, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 526, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 6}, "complete": false, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 74240, "costStatus": "unpriced", "inputTokens": 94471, "outputTokens": 1856, "reportedCostUsd": 0, "runCount": 1, "runsWithReportedCost": 0, "runsWithTokenUsage": 1, "totalTokens": 170567}, "observedAndEstimatedCostUsd": 0.06121619999999999, "reportedCostUsd": 0, "runtime": {"agentRunDurationMs": 11989, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T20-04-15-928Z", "caseId": "expanded-api", "durationMs": 36731, "evidence": "tests/runner-e2e/results/local-2026-10-06T20-04-15-928Z/public-mcp/assistant-codex-mini/local/expanded-api/attempt-1/result.json", "executionId": "public-mcp.assistant-codex-mini.local.expanded-api", "failedMatchers": [], "finishedAt": "2026-10-06T20:07:16.291Z", "kind": "case", "model": "gpt-5.4-mini", "profileId": "assistant-codex-mini", "provider": "codex", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "c95cb72a988dcc02f81e6cbb8b0c4619dcf199d3", "workflowRunUrl": null}, "startedAt": "2026-10-06T20:06:39.560Z", "status": "passed", "suiteDefinitionHash": "6051fbfd0ab10cf4365f5b44de02e9ae6496300037c266210986e010fdccca6e"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0, "inputTokens": 0, "model": "gpt-5.4-mini", "observedModels": [], "outputTokens": 0, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 0}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0, "inputTokens": 0, "model": "gpt-5.4-mini", "observedModels": [], "outputTokens": 0, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 0}, "complete": false, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 35072, "costStatus": "unpriced", "inputTokens": 52877, "outputTokens": 732, "reportedCostUsd": 0, "runCount": 2, "runsWithReportedCost": 0, "runsWithTokenUsage": 1, "totalTokens": 88681}, "observedAndEstimatedCostUsd": 0, "reportedCostUsd": 0, "runtime": {"agentRunDurationMs": 9515, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T20-04-15-928Z", "caseId": "expanded-documents", "durationMs": 21184, "evidence": "tests/runner-e2e/results/local-2026-10-06T20-04-15-928Z/public-mcp/assistant-codex-mini/local/expanded-documents/attempt-1/result.json", "executionId": "public-mcp.assistant-codex-mini.local.expanded-documents", "failedMatchers": [], "finishedAt": "2026-10-06T20:04:46.237Z", "kind": "case", "model": "gpt-5.4-mini", "profileId": "assistant-codex-mini", "provider": "codex", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "c95cb72a988dcc02f81e6cbb8b0c4619dcf199d3", "workflowRunUrl": null}, "startedAt": "2026-10-06T20:04:25.053Z", "status": "failed", "suiteDefinitionHash": "6051fbfd0ab10cf4365f5b44de02e9ae6496300037c266210986e010fdccca6e"}
{"assistantUsage": {"cachedInputTokens": 157696, "estimatedCostUsd": 0.0206202, "inputTokens": 5802, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 987, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 8}, "billing": {"assistant": {"cachedInputTokens": 157696, "estimatedCostUsd": 0.0206202, "inputTokens": 5802, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 987, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 8}, "complete": false, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 52352, "costStatus": "unpriced", "inputTokens": 71382, "outputTokens": 1005, "reportedCostUsd": 0, "runCount": 1, "runsWithReportedCost": 0, "runsWithTokenUsage": 1, "totalTokens": 124739}, "observedAndEstimatedCostUsd": 0.0206202, "reportedCostUsd": 0, "runtime": {"agentRunDurationMs": 8469, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T20-04-15-928Z", "caseId": "expanded-files", "durationMs": 43555, "evidence": "tests/runner-e2e/results/local-2026-10-06T20-04-15-928Z/public-mcp/assistant-codex-mini/local/expanded-files/attempt-1/result.json", "executionId": "public-mcp.assistant-codex-mini.local.expanded-files", "failedMatchers": [], "finishedAt": "2026-10-06T20:05:43.592Z", "kind": "case", "model": "gpt-5.4-mini", "profileId": "assistant-codex-mini", "provider": "codex", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "c95cb72a988dcc02f81e6cbb8b0c4619dcf199d3", "workflowRunUrl": null}, "startedAt": "2026-10-06T20:05:00.037Z", "status": "passed", "suiteDefinitionHash": "6051fbfd0ab10cf4365f5b44de02e9ae6496300037c266210986e010fdccca6e"}
{"assistantUsage": {"cachedInputTokens": 73216, "estimatedCostUsd": 0.0092847, "inputTokens": 2640, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 403, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 4}, "billing": {"assistant": {"cachedInputTokens": 73216, "estimatedCostUsd": 0.0092847, "inputTokens": 2640, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 403, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 4}, "complete": false, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 71680, "costStatus": "unpriced", "inputTokens": 92168, "outputTokens": 1331, "reportedCostUsd": 0, "runCount": 1, "runsWithReportedCost": 0, "runsWithTokenUsage": 1, "totalTokens": 165179}, "observedAndEstimatedCostUsd": 0.0092847, "reportedCostUsd": 0, "runtime": {"agentRunDurationMs": 12165, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T20-04-15-928Z", "caseId": "expanded-permissions", "durationMs": 31698, "evidence": "tests/runner-e2e/results/local-2026-10-06T20-04-15-928Z/public-mcp/assistant-codex-mini/local/expanded-permissions/attempt-1/result.json", "executionId": "public-mcp.assistant-codex-mini.local.expanded-permissions", "failedMatchers": [], "finishedAt": "2026-10-06T20:07:27.277Z", "kind": "case", "model": "gpt-5.4-mini", "profileId": "assistant-codex-mini", "provider": "codex", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "c95cb72a988dcc02f81e6cbb8b0c4619dcf199d3", "workflowRunUrl": null}, "startedAt": "2026-10-06T20:06:55.579Z", "status": "passed", "suiteDefinitionHash": "6051fbfd0ab10cf4365f5b44de02e9ae6496300037c266210986e010fdccca6e"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0, "inputTokens": 0, "model": "gpt-5.4-mini", "observedModels": [], "outputTokens": 0, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 0}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0, "inputTokens": 0, "model": "gpt-5.4-mini", "observedModels": [], "outputTokens": 0, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 0}, "complete": false, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 52352, "costStatus": "unpriced", "inputTokens": 70673, "outputTokens": 1059, "reportedCostUsd": 0, "runCount": 2, "runsWithReportedCost": 0, "runsWithTokenUsage": 1, "totalTokens": 124084}, "observedAndEstimatedCostUsd": 0, "reportedCostUsd": 0, "runtime": {"agentRunDurationMs": 11458, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T20-04-15-928Z", "caseId": "expanded-projects", "durationMs": 22336, "evidence": "tests/runner-e2e/results/local-2026-10-06T20-04-15-928Z/public-mcp/assistant-codex-mini/local/expanded-projects/attempt-1/result.json", "executionId": "public-mcp.assistant-codex-mini.local.expanded-projects", "failedMatchers": [], "finishedAt": "2026-10-06T20:06:25.408Z", "kind": "case", "model": "gpt-5.4-mini", "profileId": "assistant-codex-mini", "provider": "codex", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "c95cb72a988dcc02f81e6cbb8b0c4619dcf199d3", "workflowRunUrl": null}, "startedAt": "2026-10-06T20:06:03.072Z", "status": "failed", "suiteDefinitionHash": "6051fbfd0ab10cf4365f5b44de02e9ae6496300037c266210986e010fdccca6e"}
{"assistantUsage": {"cachedInputTokens": 135168, "estimatedCostUsd": 0.01840485, "inputTokens": 4861, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 1027, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 7}, "billing": {"assistant": {"cachedInputTokens": 135168, "estimatedCostUsd": 0.01840485, "inputTokens": 4861, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 1027, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 7}, "complete": false, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 72192, "costStatus": "unpriced", "inputTokens": 92023, "outputTokens": 1539, "reportedCostUsd": 0, "runCount": 1, "runsWithReportedCost": 0, "runsWithTokenUsage": 1, "totalTokens": 165754}, "observedAndEstimatedCostUsd": 0.01840485, "reportedCostUsd": 0, "runtime": {"agentRunDurationMs": 10573, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T20-04-15-928Z", "caseId": "expanded-skills", "durationMs": 37882, "evidence": "tests/runner-e2e/results/local-2026-10-06T20-04-15-928Z/public-mcp/assistant-codex-mini/local/expanded-skills/attempt-1/result.json", "executionId": "public-mcp.assistant-codex-mini.local.expanded-skills", "failedMatchers": [], "finishedAt": "2026-10-06T20:06:43.382Z", "kind": "case", "model": "gpt-5.4-mini", "profileId": "assistant-codex-mini", "provider": "codex", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "c95cb72a988dcc02f81e6cbb8b0c4619dcf199d3", "workflowRunUrl": null}, "startedAt": "2026-10-06T20:06:05.500Z", "status": "passed", "suiteDefinitionHash": "6051fbfd0ab10cf4365f5b44de02e9ae6496300037c266210986e010fdccca6e"}
{"assistantUsage": {"cachedInputTokens": 114176, "estimatedCostUsd": 0.01523745, "inputTokens": 4843, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 676, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 6}, "billing": {"assistant": {"cachedInputTokens": 114176, "estimatedCostUsd": 0.01523745, "inputTokens": 4843, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 676, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 6}, "complete": false, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 70144, "costStatus": "unpriced", "inputTokens": 90557, "outputTokens": 946, "reportedCostUsd": 0, "runCount": 1, "runsWithReportedCost": 0, "runsWithTokenUsage": 1, "totalTokens": 161647}, "observedAndEstimatedCostUsd": 0.01523745, "reportedCostUsd": 0, "runtime": {"agentRunDurationMs": 9960, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T20-04-15-928Z", "caseId": "expanded-task-edit", "durationMs": 36374, "evidence": "tests/runner-e2e/results/local-2026-10-06T20-04-15-928Z/public-mcp/assistant-codex-mini/local/expanded-task-edit/attempt-1/result.json", "executionId": "public-mcp.assistant-codex-mini.local.expanded-task-edit", "failedMatchers": [], "finishedAt": "2026-10-06T20:05:01.428Z", "kind": "case", "model": "gpt-5.4-mini", "profileId": "assistant-codex-mini", "provider": "codex", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "c95cb72a988dcc02f81e6cbb8b0c4619dcf199d3", "workflowRunUrl": null}, "startedAt": "2026-10-06T20:04:25.054Z", "status": "passed", "suiteDefinitionHash": "6051fbfd0ab10cf4365f5b44de02e9ae6496300037c266210986e010fdccca6e"}
{"billing": {"agentRunDurationMs": 380435, "assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 4.18869, "inputTokens": 1344430, "outputTokens": 10360, "requests": 47}, "estimatedRuntimeCostUsd": 0, "leaseDurationMs": 0, "llm": {"cachedInputTokens": 2137072, "costStatus": "reported", "inputTokens": 237178, "outputTokens": 16983, "reportedCostUsd": 1.78522035, "runCount": 8, "runsWithReportedCost": 8, "runsWithTokenUsage": 8, "totalTokens": 2391233}, "observedAndEstimatedCostUsd": 5.97391035, "reportedLlmCostUsd": 1.78522035, "testCount": 8, "testsWithCompleteBilling": 8}, "campaignId": "local-2026-10-06T20-13-15-458Z", "cleanupPassed": true, "complete": false, "executed": 8, "expectedSelectedCases": ["public-mcp.assistant-claude-sonnet.local.expanded-task-edit", "public-mcp.assistant-claude-sonnet.local.expanded-documents", "public-mcp.assistant-claude-sonnet.local.expanded-files", "public-mcp.assistant-claude-sonnet.local.expanded-agent-config", "public-mcp.assistant-claude-sonnet.local.expanded-projects", "public-mcp.assistant-claude-sonnet.local.expanded-skills", "public-mcp.assistant-claude-sonnet.local.expanded-api", "public-mcp.assistant-claude-sonnet.local.expanded-permissions"], "fullSuiteCoverage": [{"billing": {"agentRunDurationMs": 380435, "assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 4.18869, "inputTokens": 1344430, "outputTokens": 10360, "requests": 47}, "estimatedRuntimeCostUsd": 0, "leaseDurationMs": 0, "llm": {"cachedInputTokens": 2137072, "costStatus": "reported", "inputTokens": 237178, "outputTokens": 16983, "reportedCostUsd": 1.78522035, "runCount": 8, "runsWithReportedCost": 8, "runsWithTokenUsage": 8, "totalTokens": 2391233}, "observedAndEstimatedCostUsd": 5.97391035, "reportedLlmCostUsd": 1.78522035, "testCount": 8, "testsWithCompleteBilling": 8}, "cleanupPassed": true, "complete": false, "durationMs": 697798, "executed": 8, "expected": 63, "failed": 0, "incomplete": 0, "passed": 8, "retries": 0, "selected": 8, "suiteDefinitionHash": "de7afa8ff696f1db948627374b7396fa6ed38c9f5effe53970e20423c6e90a8c", "suiteId": "public-mcp"}], "kind": "campaign", "passed": 8, "selected": 8, "source": {"eventName": null, "ref": "codex/expanded-assistant-mcp", "sha": "32b3a162a5e712c31687562be08076f30409eb38", "workflowRunUrl": null}}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.44495700000000005, "inputTokens": 142564, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 1151, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 5}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.44495700000000005, "inputTokens": 142564, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 1151, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 5}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 239423, "costStatus": "reported", "inputTokens": 30051, "outputTokens": 1815, "reportedCostUsd": 0.21173564999999997, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 271289}, "observedAndEstimatedCostUsd": 0.65669265, "reportedCostUsd": 0.21173564999999997, "runtime": {"agentRunDurationMs": 38143, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T20-13-15-458Z", "caseId": "expanded-agent-config", "durationMs": 73838, "evidence": "tests/runner-e2e/results/local-2026-10-06T20-13-15-458Z/public-mcp/assistant-claude-sonnet/local/expanded-agent-config/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-sonnet.local.expanded-agent-config", "failedMatchers": [], "finishedAt": "2026-10-06T20:16:55.614Z", "kind": "case", "model": "claude-sonnet-4-6", "profileId": "assistant-claude-sonnet", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "32b3a162a5e712c31687562be08076f30409eb38", "workflowRunUrl": null}, "startedAt": "2026-10-06T20:15:41.776Z", "status": "passed", "suiteDefinitionHash": "de7afa8ff696f1db948627374b7396fa6ed38c9f5effe53970e20423c6e90a8c"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.37275600000000003, "inputTokens": 118297, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 1191, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 4}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.37275600000000003, "inputTokens": 118297, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 1191, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 4}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 240849, "costStatus": "reported", "inputTokens": 28940, "outputTokens": 1739, "reportedCostUsd": 0.2068572, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 271528}, "observedAndEstimatedCostUsd": 0.5796132, "reportedCostUsd": 0.2068572, "runtime": {"agentRunDurationMs": 40435, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T20-13-15-458Z", "caseId": "expanded-api", "durationMs": 77780, "evidence": "tests/runner-e2e/results/local-2026-10-06T20-13-15-458Z/public-mcp/assistant-claude-sonnet/local/expanded-api/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-sonnet.local.expanded-api", "failedMatchers": [], "finishedAt": "2026-10-06T20:20:19.697Z", "kind": "case", "model": "claude-sonnet-4-6", "profileId": "assistant-claude-sonnet", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "32b3a162a5e712c31687562be08076f30409eb38", "workflowRunUrl": null}, "startedAt": "2026-10-06T20:19:01.917Z", "status": "passed", "suiteDefinitionHash": "de7afa8ff696f1db948627374b7396fa6ed38c9f5effe53970e20423c6e90a8c"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.780546, "inputTokens": 252362, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 1564, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 9}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.780546, "inputTokens": 252362, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 1564, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 9}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 239583, "costStatus": "reported", "inputTokens": 28354, "outputTokens": 1730, "reportedCostUsd": 0.20414490000000005, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 269667}, "observedAndEstimatedCostUsd": 0.9846909, "reportedCostUsd": 0.20414490000000005, "runtime": {"agentRunDurationMs": 39894, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T20-13-15-458Z", "caseId": "expanded-documents", "durationMs": 84844, "evidence": "tests/runner-e2e/results/local-2026-10-06T20-13-15-458Z/public-mcp/assistant-claude-sonnet/local/expanded-documents/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-sonnet.local.expanded-documents", "failedMatchers": [], "finishedAt": "2026-10-06T20:15:04.883Z", "kind": "case", "model": "claude-sonnet-4-6", "profileId": "assistant-claude-sonnet", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "32b3a162a5e712c31687562be08076f30409eb38", "workflowRunUrl": null}, "startedAt": "2026-10-06T20:13:40.039Z", "status": "passed", "suiteDefinitionHash": "de7afa8ff696f1db948627374b7396fa6ed38c9f5effe53970e20423c6e90a8c"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.639774, "inputTokens": 204813, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 1689, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 7}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.639774, "inputTokens": 204813, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 1689, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 7}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 371030, "costStatus": "reported", "inputTokens": 32976, "outputTokens": 3627, "reportedCostUsd": 0.28936425000000005, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 407633}, "observedAndEstimatedCostUsd": 0.92913825, "reportedCostUsd": 0.28936425000000005, "runtime": {"agentRunDurationMs": 68403, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T20-13-15-458Z", "caseId": "expanded-files", "durationMs": 115085, "evidence": "tests/runner-e2e/results/local-2026-10-06T20-13-15-458Z/public-mcp/assistant-claude-sonnet/local/expanded-files/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-sonnet.local.expanded-files", "failedMatchers": [], "finishedAt": "2026-10-06T20:17:33.675Z", "kind": "case", "model": "claude-sonnet-4-6", "profileId": "assistant-claude-sonnet", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "32b3a162a5e712c31687562be08076f30409eb38", "workflowRunUrl": null}, "startedAt": "2026-10-06T20:15:38.590Z", "status": "passed", "suiteDefinitionHash": "de7afa8ff696f1db948627374b7396fa6ed38c9f5effe53970e20423c6e90a8c"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.33981300000000003, "inputTokens": 109706, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 713, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 4}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.33981300000000003, "inputTokens": 109706, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 713, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 4}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 239121, "costStatus": "reported", "inputTokens": 28797, "outputTokens": 1917, "reportedCostUsd": 0.20847254999999998, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 269835}, "observedAndEstimatedCostUsd": 0.54828555, "reportedCostUsd": 0.20847254999999998, "runtime": {"agentRunDurationMs": 62358, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T20-13-15-458Z", "caseId": "expanded-permissions", "durationMs": 97722, "evidence": "tests/runner-e2e/results/local-2026-10-06T20-13-15-458Z/public-mcp/assistant-claude-sonnet/local/expanded-permissions/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-sonnet.local.expanded-permissions", "failedMatchers": [], "finishedAt": "2026-10-06T20:21:54.616Z", "kind": "case", "model": "claude-sonnet-4-6", "profileId": "assistant-claude-sonnet", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "32b3a162a5e712c31687562be08076f30409eb38", "workflowRunUrl": null}, "startedAt": "2026-10-06T20:20:16.894Z", "status": "passed", "suiteDefinitionHash": "de7afa8ff696f1db948627374b7396fa6ed38c9f5effe53970e20423c6e90a8c"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.42672299999999996, "inputTokens": 138336, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 781, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 5}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.42672299999999996, "inputTokens": 138336, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 781, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 5}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 239704, "costStatus": "reported", "inputTokens": 29165, "outputTokens": 1530, "reportedCostUsd": 0.20422245, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 270399}, "observedAndEstimatedCostUsd": 0.63094545, "reportedCostUsd": 0.20422245, "runtime": {"agentRunDurationMs": 33436, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T20-13-15-458Z", "caseId": "expanded-projects", "durationMs": 63306, "evidence": "tests/runner-e2e/results/local-2026-10-06T20-13-15-458Z/public-mcp/assistant-claude-sonnet/local/expanded-projects/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-sonnet.local.expanded-projects", "failedMatchers": [], "finishedAt": "2026-10-06T20:18:33.837Z", "kind": "case", "model": "claude-sonnet-4-6", "profileId": "assistant-claude-sonnet", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "32b3a162a5e712c31687562be08076f30409eb38", "workflowRunUrl": null}, "startedAt": "2026-10-06T20:17:30.531Z", "status": "passed", "suiteDefinitionHash": "de7afa8ff696f1db948627374b7396fa6ed38c9f5effe53970e20423c6e90a8c"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.5553389999999999, "inputTokens": 175533, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 1916, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 6}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.5553389999999999, "inputTokens": 175533, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 1916, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 6}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 240157, "costStatus": "reported", "inputTokens": 28683, "outputTokens": 1960, "reportedCostUsd": 0.20900085000000002, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 270800}, "observedAndEstimatedCostUsd": 0.7643398499999999, "reportedCostUsd": 0.20900085000000002, "runtime": {"agentRunDurationMs": 43780, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T20-13-15-458Z", "caseId": "expanded-skills", "durationMs": 90168, "evidence": "tests/runner-e2e/results/local-2026-10-06T20-13-15-458Z/public-mcp/assistant-claude-sonnet/local/expanded-skills/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-sonnet.local.expanded-skills", "failedMatchers": [], "finishedAt": "2026-10-06T20:19:38.206Z", "kind": "case", "model": "claude-sonnet-4-6", "profileId": "assistant-claude-sonnet", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "32b3a162a5e712c31687562be08076f30409eb38", "workflowRunUrl": null}, "startedAt": "2026-10-06T20:18:08.038Z", "status": "passed", "suiteDefinitionHash": "de7afa8ff696f1db948627374b7396fa6ed38c9f5effe53970e20423c6e90a8c"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.628782, "inputTokens": 202819, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 1355, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 7}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.628782, "inputTokens": 202819, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 1355, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 7}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 327205, "costStatus": "reported", "inputTokens": 30212, "outputTokens": 2665, "reportedCostUsd": 0.2514225, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 360082}, "observedAndEstimatedCostUsd": 0.8802045, "reportedCostUsd": 0.2514225, "runtime": {"agentRunDurationMs": 53986, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T20-13-15-458Z", "caseId": "expanded-task-edit", "durationMs": 95055, "evidence": "tests/runner-e2e/results/local-2026-10-06T20-13-15-458Z/public-mcp/assistant-claude-sonnet/local/expanded-task-edit/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-sonnet.local.expanded-task-edit", "failedMatchers": [], "finishedAt": "2026-10-06T20:15:15.094Z", "kind": "case", "model": "claude-sonnet-4-6", "profileId": "assistant-claude-sonnet", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "32b3a162a5e712c31687562be08076f30409eb38", "workflowRunUrl": null}, "startedAt": "2026-10-06T20:13:40.039Z", "status": "passed", "suiteDefinitionHash": "de7afa8ff696f1db948627374b7396fa6ed38c9f5effe53970e20423c6e90a8c"}
{"billing": {"agentRunDurationMs": 340801, "assistant": {"cachedInputTokens": 962048, "estimatedCostUsd": 1.5474926000000002, "inputTokens": 1428238, "outputTokens": 14573, "requests": 95}, "estimatedRuntimeCostUsd": 0, "leaseDurationMs": 0, "llm": {"cachedInputTokens": 1752988, "costStatus": "partial", "inputTokens": 848787, "outputTokens": 30856, "reportedCostUsd": 0.45213139999999996, "runCount": 16, "runsWithReportedCost": 8, "runsWithTokenUsage": 16, "totalTokens": 2632631}, "observedAndEstimatedCostUsd": 1.9996240000000003, "reportedLlmCostUsd": 0.45213139999999996, "testCount": 16, "testsWithCompleteBilling": 8}, "campaignId": "local-2026-10-06T20-28-55-414Z", "cleanupPassed": true, "complete": false, "executed": 16, "expectedSelectedCases": ["public-mcp.assistant-codex-mini.local.expanded-task-edit", "public-mcp.assistant-codex-mini.local.expanded-documents", "public-mcp.assistant-codex-mini.local.expanded-files", "public-mcp.assistant-codex-mini.local.expanded-agent-config", "public-mcp.assistant-codex-mini.local.expanded-projects", "public-mcp.assistant-codex-mini.local.expanded-skills", "public-mcp.assistant-codex-mini.local.expanded-api", "public-mcp.assistant-codex-mini.local.expanded-permissions", "public-mcp.assistant-claude-haiku.local.expanded-task-edit", "public-mcp.assistant-claude-haiku.local.expanded-documents", "public-mcp.assistant-claude-haiku.local.expanded-files", "public-mcp.assistant-claude-haiku.local.expanded-agent-config", "public-mcp.assistant-claude-haiku.local.expanded-projects", "public-mcp.assistant-claude-haiku.local.expanded-skills", "public-mcp.assistant-claude-haiku.local.expanded-api", "public-mcp.assistant-claude-haiku.local.expanded-permissions"], "fullSuiteCoverage": [{"billing": {"agentRunDurationMs": 340801, "assistant": {"cachedInputTokens": 962048, "estimatedCostUsd": 1.5474926000000002, "inputTokens": 1428238, "outputTokens": 14573, "requests": 95}, "estimatedRuntimeCostUsd": 0, "leaseDurationMs": 0, "llm": {"cachedInputTokens": 1752988, "costStatus": "partial", "inputTokens": 848787, "outputTokens": 30856, "reportedCostUsd": 0.45213139999999996, "runCount": 16, "runsWithReportedCost": 8, "runsWithTokenUsage": 16, "totalTokens": 2632631}, "observedAndEstimatedCostUsd": 1.9996240000000003, "reportedLlmCostUsd": 0.45213139999999996, "testCount": 16, "testsWithCompleteBilling": 8}, "cleanupPassed": true, "complete": false, "durationMs": 913542, "executed": 16, "expected": 63, "failed": 0, "incomplete": 0, "passed": 16, "retries": 0, "selected": 16, "suiteDefinitionHash": "d86dc8451519467b8680e92b7d6c3bee975a4bff419e6beaace2d8eeb5ce43f5", "suiteId": "public-mcp"}], "kind": "campaign", "passed": 16, "selected": 16, "source": {"eventName": null, "ref": "codex/expanded-assistant-mcp", "sha": "9089870a4da4b03b12f7eab0afd134311b840a02", "workflowRunUrl": null}}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.17982199999999998, "inputTokens": 172967, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 1371, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 6}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.17982199999999998, "inputTokens": 172967, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 1371, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 6}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 171125, "costStatus": "reported", "inputTokens": 15937, "outputTokens": 3233, "reportedCostUsd": 0.05318425, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 190295}, "observedAndEstimatedCostUsd": 0.23300624999999997, "reportedCostUsd": 0.05318425, "runtime": {"agentRunDurationMs": 34624, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T20-28-55-414Z", "caseId": "expanded-agent-config", "durationMs": 83888, "evidence": "tests/runner-e2e/results/local-2026-10-06T20-28-55-414Z/public-mcp/assistant-claude-haiku/local/expanded-agent-config/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-haiku.local.expanded-agent-config", "failedMatchers": [], "finishedAt": "2026-10-06T20:37:42.822Z", "kind": "case", "model": "claude-haiku-4-5-20251001", "profileId": "assistant-claude-haiku", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "9089870a4da4b03b12f7eab0afd134311b840a02", "workflowRunUrl": null}, "startedAt": "2026-10-06T20:36:18.934Z", "status": "passed", "suiteDefinitionHash": "d86dc8451519467b8680e92b7d6c3bee975a4bff419e6beaace2d8eeb5ce43f5"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.153384, "inputTokens": 148474, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 982, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 5}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.153384, "inputTokens": 148474, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 982, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 5}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 111429, "costStatus": "reported", "inputTokens": 13179, "outputTokens": 2010, "reportedCostUsd": 0.03765615, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 126618}, "observedAndEstimatedCostUsd": 0.19104015, "reportedCostUsd": 0.03765615, "runtime": {"agentRunDurationMs": 24177, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T20-28-55-414Z", "caseId": "expanded-api", "durationMs": 55624, "evidence": "tests/runner-e2e/results/local-2026-10-06T20-28-55-414Z/public-mcp/assistant-claude-haiku/local/expanded-api/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-haiku.local.expanded-api", "failedMatchers": [], "finishedAt": "2026-10-06T20:40:29.442Z", "kind": "case", "model": "claude-haiku-4-5-20251001", "profileId": "assistant-claude-haiku", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "9089870a4da4b03b12f7eab0afd134311b840a02", "workflowRunUrl": null}, "startedAt": "2026-10-06T20:39:33.818Z", "status": "passed", "suiteDefinitionHash": "d86dc8451519467b8680e92b7d6c3bee975a4bff419e6beaace2d8eeb5ce43f5"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.25866, "inputTokens": 251525, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 1427, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 9}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.25866, "inputTokens": 251525, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 1427, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 9}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 119295, "costStatus": "reported", "inputTokens": 30005, "outputTokens": 2205, "reportedCostUsd": 0.060452000000000006, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 151505}, "observedAndEstimatedCostUsd": 0.319112, "reportedCostUsd": 0.060452000000000006, "runtime": {"agentRunDurationMs": 24248, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T20-28-55-414Z", "caseId": "expanded-documents", "durationMs": 81090, "evidence": "tests/runner-e2e/results/local-2026-10-06T20-28-55-414Z/public-mcp/assistant-claude-haiku/local/expanded-documents/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-haiku.local.expanded-documents", "failedMatchers": [], "finishedAt": "2026-10-06T20:35:36.809Z", "kind": "case", "model": "claude-haiku-4-5-20251001", "profileId": "assistant-claude-haiku", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "9089870a4da4b03b12f7eab0afd134311b840a02", "workflowRunUrl": null}, "startedAt": "2026-10-06T20:34:15.719Z", "status": "passed", "suiteDefinitionHash": "d86dc8451519467b8680e92b7d6c3bee975a4bff419e6beaace2d8eeb5ce43f5"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.207377, "inputTokens": 201287, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 1218, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 7}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.207377, "inputTokens": 201287, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 1218, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 7}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 115327, "costStatus": "reported", "inputTokens": 14737, "outputTokens": 2022, "reportedCostUsd": 0.04005345, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 132086}, "observedAndEstimatedCostUsd": 0.24743045, "reportedCostUsd": 0.04005345, "runtime": {"agentRunDurationMs": 25753, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T20-28-55-414Z", "caseId": "expanded-files", "durationMs": 62803, "evidence": "tests/runner-e2e/results/local-2026-10-06T20-28-55-414Z/public-mcp/assistant-claude-haiku/local/expanded-files/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-haiku.local.expanded-files", "failedMatchers": [], "finishedAt": "2026-10-06T20:37:10.326Z", "kind": "case", "model": "claude-haiku-4-5-20251001", "profileId": "assistant-claude-haiku", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "9089870a4da4b03b12f7eab0afd134311b840a02", "workflowRunUrl": null}, "startedAt": "2026-10-06T20:36:07.523Z", "status": "passed", "suiteDefinitionHash": "d86dc8451519467b8680e92b7d6c3bee975a4bff419e6beaace2d8eeb5ce43f5"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.143877, "inputTokens": 138782, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 1019, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 5}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.143877, "inputTokens": 138782, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 1019, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 5}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 293778, "costStatus": "reported", "inputTokens": 35600, "outputTokens": 5099, "reportedCostUsd": 0.09935405, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 334477}, "observedAndEstimatedCostUsd": 0.24323105, "reportedCostUsd": 0.09935405, "runtime": {"agentRunDurationMs": 56053, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T20-28-55-414Z", "caseId": "expanded-permissions", "durationMs": 85030, "evidence": "tests/runner-e2e/results/local-2026-10-06T20-28-55-414Z/public-mcp/assistant-claude-haiku/local/expanded-permissions/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-haiku.local.expanded-permissions", "failedMatchers": [], "finishedAt": "2026-10-06T20:41:37.683Z", "kind": "case", "model": "claude-haiku-4-5-20251001", "profileId": "assistant-claude-haiku", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "9089870a4da4b03b12f7eab0afd134311b840a02", "workflowRunUrl": null}, "startedAt": "2026-10-06T20:40:12.653Z", "status": "passed", "suiteDefinitionHash": "d86dc8451519467b8680e92b7d6c3bee975a4bff419e6beaace2d8eeb5ce43f5"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.141822, "inputTokens": 137902, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 784, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 5}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.141822, "inputTokens": 137902, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 784, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 5}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 161048, "costStatus": "reported", "inputTokens": 30451, "outputTokens": 2206, "reportedCostUsd": 0.0651878, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 193705}, "observedAndEstimatedCostUsd": 0.20700980000000002, "reportedCostUsd": 0.0651878, "runtime": {"agentRunDurationMs": 26275, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T20-28-55-414Z", "caseId": "expanded-projects", "durationMs": 51060, "evidence": "tests/runner-e2e/results/local-2026-10-06T20-28-55-414Z/public-mcp/assistant-claude-haiku/local/expanded-projects/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-haiku.local.expanded-projects", "failedMatchers": [], "finishedAt": "2026-10-06T20:38:42.882Z", "kind": "case", "model": "claude-haiku-4-5-20251001", "profileId": "assistant-claude-haiku", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "9089870a4da4b03b12f7eab0afd134311b840a02", "workflowRunUrl": null}, "startedAt": "2026-10-06T20:37:51.822Z", "status": "passed", "suiteDefinitionHash": "d86dc8451519467b8680e92b7d6c3bee975a4bff419e6beaace2d8eeb5ce43f5"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.18043399999999998, "inputTokens": 172044, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 1678, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 6}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.18043399999999998, "inputTokens": 172044, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 1678, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 6}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 160912, "costStatus": "reported", "inputTokens": 30592, "outputTokens": 2032, "reportedCostUsd": 0.06448045, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 193536}, "observedAndEstimatedCostUsd": 0.24491444999999998, "reportedCostUsd": 0.06448045, "runtime": {"agentRunDurationMs": 25746, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T20-28-55-414Z", "caseId": "expanded-skills", "durationMs": 64197, "evidence": "tests/runner-e2e/results/local-2026-10-06T20-28-55-414Z/public-mcp/assistant-claude-haiku/local/expanded-skills/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-haiku.local.expanded-skills", "failedMatchers": [], "finishedAt": "2026-10-06T20:39:17.992Z", "kind": "case", "model": "claude-haiku-4-5-20251001", "profileId": "assistant-claude-haiku", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "9089870a4da4b03b12f7eab0afd134311b840a02", "workflowRunUrl": null}, "startedAt": "2026-10-06T20:38:13.795Z", "status": "passed", "suiteDefinitionHash": "d86dc8451519467b8680e92b7d6c3bee975a4bff419e6beaace2d8eeb5ce43f5"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.117392, "inputTokens": 112387, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 1001, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 4}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.117392, "inputTokens": 112387, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 1001, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 4}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 85930, "costStatus": "reported", "inputTokens": 12715, "outputTokens": 1457, "reportedCostUsd": 0.03176324999999999, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 100102}, "observedAndEstimatedCostUsd": 0.14915525, "reportedCostUsd": 0.03176324999999999, "runtime": {"agentRunDurationMs": 19445, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T20-28-55-414Z", "caseId": "expanded-task-edit", "durationMs": 70516, "evidence": "tests/runner-e2e/results/local-2026-10-06T20-28-55-414Z/public-mcp/assistant-claude-haiku/local/expanded-task-edit/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-haiku.local.expanded-task-edit", "failedMatchers": [], "finishedAt": "2026-10-06T20:35:26.235Z", "kind": "case", "model": "claude-haiku-4-5-20251001", "profileId": "assistant-claude-haiku", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "9089870a4da4b03b12f7eab0afd134311b840a02", "workflowRunUrl": null}, "startedAt": "2026-10-06T20:34:15.719Z", "status": "passed", "suiteDefinitionHash": "d86dc8451519467b8680e92b7d6c3bee975a4bff419e6beaace2d8eeb5ce43f5"}
{"assistantUsage": {"cachedInputTokens": 93184, "estimatedCostUsd": 0.0144108, "inputTokens": 5582, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 719, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 5}, "billing": {"assistant": {"cachedInputTokens": 93184, "estimatedCostUsd": 0.0144108, "inputTokens": 5582, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 719, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 5}, "complete": false, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 72192, "costStatus": "unpriced", "inputTokens": 93361, "outputTokens": 1354, "reportedCostUsd": 0, "runCount": 1, "runsWithReportedCost": 0, "runsWithTokenUsage": 1, "totalTokens": 166907}, "observedAndEstimatedCostUsd": 0.0144108, "reportedCostUsd": 0, "runtime": {"agentRunDurationMs": 12191, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T20-28-55-414Z", "caseId": "expanded-agent-config", "durationMs": 41025, "evidence": "tests/runner-e2e/results/local-2026-10-06T20-28-55-414Z/public-mcp/assistant-codex-mini/local/expanded-agent-config/attempt-1/result.json", "executionId": "public-mcp.assistant-codex-mini.local.expanded-agent-config", "failedMatchers": [], "finishedAt": "2026-10-06T20:31:05.540Z", "kind": "case", "model": "gpt-5.4-mini", "profileId": "assistant-codex-mini", "provider": "codex", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "9089870a4da4b03b12f7eab0afd134311b840a02", "workflowRunUrl": null}, "startedAt": "2026-10-06T20:30:24.515Z", "status": "passed", "suiteDefinitionHash": "d86dc8451519467b8680e92b7d6c3bee975a4bff419e6beaace2d8eeb5ce43f5"}
{"assistantUsage": {"cachedInputTokens": 190464, "estimatedCostUsd": 0.0490683, "inputTokens": 42214, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 694, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 6}, "billing": {"assistant": {"cachedInputTokens": 190464, "estimatedCostUsd": 0.0490683, "inputTokens": 42214, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 694, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 6}, "complete": false, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 66560, "costStatus": "unpriced", "inputTokens": 72466, "outputTokens": 1678, "reportedCostUsd": 0, "runCount": 1, "runsWithReportedCost": 0, "runsWithTokenUsage": 1, "totalTokens": 140704}, "observedAndEstimatedCostUsd": 0.0490683, "reportedCostUsd": 0, "runtime": {"agentRunDurationMs": 13293, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T20-28-55-414Z", "caseId": "expanded-api", "durationMs": 51230, "evidence": "tests/runner-e2e/results/local-2026-10-06T20-28-55-414Z/public-mcp/assistant-codex-mini/local/expanded-api/attempt-1/result.json", "executionId": "public-mcp.assistant-codex-mini.local.expanded-api", "failedMatchers": [], "finishedAt": "2026-10-06T20:33:26.435Z", "kind": "case", "model": "gpt-5.4-mini", "profileId": "assistant-codex-mini", "provider": "codex", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "9089870a4da4b03b12f7eab0afd134311b840a02", "workflowRunUrl": null}, "startedAt": "2026-10-06T20:32:35.205Z", "status": "passed", "suiteDefinitionHash": "d86dc8451519467b8680e92b7d6c3bee975a4bff419e6beaace2d8eeb5ce43f5"}
{"assistantUsage": {"cachedInputTokens": 149504, "estimatedCostUsd": 0.032437799999999996, "inputTokens": 24274, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 671, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 9}, "billing": {"assistant": {"cachedInputTokens": 149504, "estimatedCostUsd": 0.032437799999999996, "inputTokens": 24274, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 671, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 9}, "complete": false, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 52352, "costStatus": "unpriced", "inputTokens": 71209, "outputTokens": 1391, "reportedCostUsd": 0, "runCount": 1, "runsWithReportedCost": 0, "runsWithTokenUsage": 1, "totalTokens": 124952}, "observedAndEstimatedCostUsd": 0.032437799999999996, "reportedCostUsd": 0, "runtime": {"agentRunDurationMs": 13304, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T20-28-55-414Z", "caseId": "expanded-documents", "durationMs": 49688, "evidence": "tests/runner-e2e/results/local-2026-10-06T20-28-55-414Z/public-mcp/assistant-codex-mini/local/expanded-documents/attempt-1/result.json", "executionId": "public-mcp.assistant-codex-mini.local.expanded-documents", "failedMatchers": [], "finishedAt": "2026-10-06T20:30:08.340Z", "kind": "case", "model": "gpt-5.4-mini", "profileId": "assistant-codex-mini", "provider": "codex", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "9089870a4da4b03b12f7eab0afd134311b840a02", "workflowRunUrl": null}, "startedAt": "2026-10-06T20:29:18.652Z", "status": "passed", "suiteDefinitionHash": "d86dc8451519467b8680e92b7d6c3bee975a4bff419e6beaace2d8eeb5ce43f5"}
{"assistantUsage": {"cachedInputTokens": 134144, "estimatedCostUsd": 0.017674799999999997, "inputTokens": 5982, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 695, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 7}, "billing": {"assistant": {"cachedInputTokens": 134144, "estimatedCostUsd": 0.017674799999999997, "inputTokens": 5982, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 695, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 7}, "complete": false, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 51840, "costStatus": "unpriced", "inputTokens": 71124, "outputTokens": 717, "reportedCostUsd": 0, "runCount": 1, "runsWithReportedCost": 0, "runsWithTokenUsage": 1, "totalTokens": 123681}, "observedAndEstimatedCostUsd": 0.017674799999999997, "reportedCostUsd": 0, "runtime": {"agentRunDurationMs": 9758, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T20-28-55-414Z", "caseId": "expanded-files", "durationMs": 40581, "evidence": "tests/runner-e2e/results/local-2026-10-06T20-28-55-414Z/public-mcp/assistant-codex-mini/local/expanded-files/attempt-1/result.json", "executionId": "public-mcp.assistant-codex-mini.local.expanded-files", "failedMatchers": [], "finishedAt": "2026-10-06T20:31:05.096Z", "kind": "case", "model": "gpt-5.4-mini", "profileId": "assistant-codex-mini", "provider": "codex", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "9089870a4da4b03b12f7eab0afd134311b840a02", "workflowRunUrl": null}, "startedAt": "2026-10-06T20:30:24.515Z", "status": "passed", "suiteDefinitionHash": "d86dc8451519467b8680e92b7d6c3bee975a4bff419e6beaace2d8eeb5ce43f5"}
{"assistantUsage": {"cachedInputTokens": 73216, "estimatedCostUsd": 0.0091272, "inputTokens": 2622, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 371, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 4}, "billing": {"assistant": {"cachedInputTokens": 73216, "estimatedCostUsd": 0.0091272, "inputTokens": 2622, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 371, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 4}, "complete": false, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 89472, "costStatus": "unpriced", "inputTokens": 121527, "outputTokens": 1936, "reportedCostUsd": 0, "runCount": 1, "runsWithReportedCost": 0, "runsWithTokenUsage": 1, "totalTokens": 212935}, "observedAndEstimatedCostUsd": 0.0091272, "reportedCostUsd": 0, "runtime": {"agentRunDurationMs": 16618, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T20-28-55-414Z", "caseId": "expanded-permissions", "durationMs": 47813, "evidence": "tests/runner-e2e/results/local-2026-10-06T20-28-55-414Z/public-mcp/assistant-codex-mini/local/expanded-permissions/attempt-1/result.json", "executionId": "public-mcp.assistant-codex-mini.local.expanded-permissions", "failedMatchers": [], "finishedAt": "2026-10-06T20:33:23.017Z", "kind": "case", "model": "gpt-5.4-mini", "profileId": "assistant-codex-mini", "provider": "codex", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "9089870a4da4b03b12f7eab0afd134311b840a02", "workflowRunUrl": null}, "startedAt": "2026-10-06T20:32:35.204Z", "status": "passed", "suiteDefinitionHash": "d86dc8451519467b8680e92b7d6c3bee975a4bff419e6beaace2d8eeb5ce43f5"}
{"assistantUsage": {"cachedInputTokens": 92160, "estimatedCostUsd": 0.011332500000000002, "inputTokens": 3206, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 448, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 5}, "billing": {"assistant": {"cachedInputTokens": 92160, "estimatedCostUsd": 0.011332500000000002, "inputTokens": 3206, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 448, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 5}, "complete": false, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 66048, "costStatus": "unpriced", "inputTokens": 72879, "outputTokens": 1377, "reportedCostUsd": 0, "runCount": 1, "runsWithReportedCost": 0, "runsWithTokenUsage": 1, "totalTokens": 140304}, "observedAndEstimatedCostUsd": 0.011332500000000002, "reportedCostUsd": 0, "runtime": {"agentRunDurationMs": 13830, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T20-28-55-414Z", "caseId": "expanded-projects", "durationMs": 40612, "evidence": "tests/runner-e2e/results/local-2026-10-06T20-28-55-414Z/public-mcp/assistant-codex-mini/local/expanded-projects/attempt-1/result.json", "executionId": "public-mcp.assistant-codex-mini.local.expanded-projects", "failedMatchers": [], "finishedAt": "2026-10-06T20:32:11.669Z", "kind": "case", "model": "gpt-5.4-mini", "profileId": "assistant-codex-mini", "provider": "codex", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "9089870a4da4b03b12f7eab0afd134311b840a02", "workflowRunUrl": null}, "startedAt": "2026-10-06T20:31:31.057Z", "status": "passed", "suiteDefinitionHash": "d86dc8451519467b8680e92b7d6c3bee975a4bff419e6beaace2d8eeb5ce43f5"}
{"assistantUsage": {"cachedInputTokens": 115200, "estimatedCostUsd": 0.015942, "inputTokens": 4390, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 891, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 6}, "billing": {"assistant": {"cachedInputTokens": 115200, "estimatedCostUsd": 0.015942, "inputTokens": 4390, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 891, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 6}, "complete": false, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 65024, "costStatus": "unpriced", "inputTokens": 71970, "outputTokens": 1241, "reportedCostUsd": 0, "runCount": 1, "runsWithReportedCost": 0, "runsWithTokenUsage": 1, "totalTokens": 138235}, "observedAndEstimatedCostUsd": 0.015942, "reportedCostUsd": 0, "runtime": {"agentRunDurationMs": 11202, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T20-28-55-414Z", "caseId": "expanded-skills", "durationMs": 42479, "evidence": "tests/runner-e2e/results/local-2026-10-06T20-28-55-414Z/public-mcp/assistant-codex-mini/local/expanded-skills/attempt-1/result.json", "executionId": "public-mcp.assistant-codex-mini.local.expanded-skills", "failedMatchers": [], "finishedAt": "2026-10-06T20:32:13.537Z", "kind": "case", "model": "gpt-5.4-mini", "profileId": "assistant-codex-mini", "provider": "codex", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "9089870a4da4b03b12f7eab0afd134311b840a02", "workflowRunUrl": null}, "startedAt": "2026-10-06T20:31:31.058Z", "status": "passed", "suiteDefinitionHash": "d86dc8451519467b8680e92b7d6c3bee975a4bff419e6beaace2d8eeb5ce43f5"}
{"assistantUsage": {"cachedInputTokens": 114176, "estimatedCostUsd": 0.0147312, "inputTokens": 4600, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 604, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 6}, "billing": {"assistant": {"cachedInputTokens": 114176, "estimatedCostUsd": 0.0147312, "inputTokens": 4600, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 604, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 6}, "complete": false, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 70656, "costStatus": "unpriced", "inputTokens": 91035, "outputTokens": 898, "reportedCostUsd": 0, "runCount": 1, "runsWithReportedCost": 0, "runsWithTokenUsage": 1, "totalTokens": 162589}, "observedAndEstimatedCostUsd": 0.0147312, "reportedCostUsd": 0, "runtime": {"agentRunDurationMs": 14284, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T20-28-55-414Z", "caseId": "expanded-task-edit", "durationMs": 45906, "evidence": "tests/runner-e2e/results/local-2026-10-06T20-28-55-414Z/public-mcp/assistant-codex-mini/local/expanded-task-edit/attempt-1/result.json", "executionId": "public-mcp.assistant-codex-mini.local.expanded-task-edit", "failedMatchers": [], "finishedAt": "2026-10-06T20:30:04.468Z", "kind": "case", "model": "gpt-5.4-mini", "profileId": "assistant-codex-mini", "provider": "codex", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "9089870a4da4b03b12f7eab0afd134311b840a02", "workflowRunUrl": null}, "startedAt": "2026-10-06T20:29:18.562Z", "status": "passed", "suiteDefinitionHash": "d86dc8451519467b8680e92b7d6c3bee975a4bff419e6beaace2d8eeb5ce43f5"}
{"billing": {"agentRunDurationMs": 355090, "assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 4.175256, "inputTokens": 1341982, "outputTokens": 9954, "requests": 47}, "estimatedRuntimeCostUsd": 0, "leaseDurationMs": 0, "llm": {"cachedInputTokens": 2045883, "costStatus": "reported", "inputTokens": 233285, "outputTokens": 13999, "reportedCostUsd": 1.6985064, "runCount": 8, "runsWithReportedCost": 8, "runsWithTokenUsage": 8, "totalTokens": 2293167}, "observedAndEstimatedCostUsd": 5.8737623999999995, "reportedLlmCostUsd": 1.6985064, "testCount": 8, "testsWithCompleteBilling": 8}, "campaignId": "local-2026-10-06T20-44-06-234Z", "cleanupPassed": true, "complete": false, "executed": 8, "expectedSelectedCases": ["public-mcp.assistant-claude-sonnet.local.expanded-task-edit", "public-mcp.assistant-claude-sonnet.local.expanded-documents", "public-mcp.assistant-claude-sonnet.local.expanded-files", "public-mcp.assistant-claude-sonnet.local.expanded-agent-config", "public-mcp.assistant-claude-sonnet.local.expanded-projects", "public-mcp.assistant-claude-sonnet.local.expanded-skills", "public-mcp.assistant-claude-sonnet.local.expanded-api", "public-mcp.assistant-claude-sonnet.local.expanded-permissions"], "fullSuiteCoverage": [{"billing": {"agentRunDurationMs": 355090, "assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 4.175256, "inputTokens": 1341982, "outputTokens": 9954, "requests": 47}, "estimatedRuntimeCostUsd": 0, "leaseDurationMs": 0, "llm": {"cachedInputTokens": 2045883, "costStatus": "reported", "inputTokens": 233285, "outputTokens": 13999, "reportedCostUsd": 1.6985064, "runCount": 8, "runsWithReportedCost": 8, "runsWithTokenUsage": 8, "totalTokens": 2293167}, "observedAndEstimatedCostUsd": 5.8737623999999995, "reportedLlmCostUsd": 1.6985064, "testCount": 8, "testsWithCompleteBilling": 8}, "cleanupPassed": true, "complete": false, "durationMs": 722226, "executed": 8, "expected": 63, "failed": 0, "incomplete": 0, "passed": 8, "retries": 0, "selected": 8, "suiteDefinitionHash": "d86dc8451519467b8680e92b7d6c3bee975a4bff419e6beaace2d8eeb5ce43f5", "suiteId": "public-mcp"}], "kind": "campaign", "passed": 8, "selected": 8, "source": {"eventName": null, "ref": "codex/expanded-assistant-mcp", "sha": "b9c4be5f0f8a3f1143f83edf3e8a047d02888eea", "workflowRunUrl": null}}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.445821, "inputTokens": 142837, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 1154, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 5}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.445821, "inputTokens": 142837, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 1154, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 5}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 281235, "costStatus": "reported", "inputTokens": 28808, "outputTokens": 1655, "reportedCostUsd": 0.21721725, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 311698}, "observedAndEstimatedCostUsd": 0.66303825, "reportedCostUsd": 0.21721725, "runtime": {"agentRunDurationMs": 60726, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T20-44-06-234Z", "caseId": "expanded-agent-config", "durationMs": 114430, "evidence": "tests/runner-e2e/results/local-2026-10-06T20-44-06-234Z/public-mcp/assistant-claude-sonnet/local/expanded-agent-config/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-sonnet.local.expanded-agent-config", "failedMatchers": [], "finishedAt": "2026-10-06T20:48:46.088Z", "kind": "case", "model": "claude-sonnet-4-6", "profileId": "assistant-claude-sonnet", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "b9c4be5f0f8a3f1143f83edf3e8a047d02888eea", "workflowRunUrl": null}, "startedAt": "2026-10-06T20:46:51.658Z", "status": "passed", "suiteDefinitionHash": "d86dc8451519467b8680e92b7d6c3bee975a4bff419e6beaace2d8eeb5ce43f5"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.372834, "inputTokens": 118388, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 1178, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 4}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.372834, "inputTokens": 118388, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 1178, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 4}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 280702, "costStatus": "reported", "inputTokens": 28680, "outputTokens": 1713, "reportedCostUsd": 0.21744734999999998, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 311095}, "observedAndEstimatedCostUsd": 0.5902813499999999, "reportedCostUsd": 0.21744734999999998, "runtime": {"agentRunDurationMs": 37420, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T20-44-06-234Z", "caseId": "expanded-api", "durationMs": 85823, "evidence": "tests/runner-e2e/results/local-2026-10-06T20-44-06-234Z/public-mcp/assistant-claude-sonnet/local/expanded-api/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-sonnet.local.expanded-api", "failedMatchers": [], "finishedAt": "2026-10-06T20:52:41.920Z", "kind": "case", "model": "claude-sonnet-4-6", "profileId": "assistant-claude-sonnet", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "b9c4be5f0f8a3f1143f83edf3e8a047d02888eea", "workflowRunUrl": null}, "startedAt": "2026-10-06T20:51:16.097Z", "status": "passed", "suiteDefinitionHash": "d86dc8451519467b8680e92b7d6c3bee975a4bff419e6beaace2d8eeb5ce43f5"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.782931, "inputTokens": 252632, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 1669, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 9}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.782931, "inputTokens": 252632, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 1669, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 9}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 238892, "costStatus": "reported", "inputTokens": 29940, "outputTokens": 1496, "reportedCostUsd": 0.2063751, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 270328}, "observedAndEstimatedCostUsd": 0.9893061000000001, "reportedCostUsd": 0.2063751, "runtime": {"agentRunDurationMs": 32248, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T20-44-06-234Z", "caseId": "expanded-documents", "durationMs": 90205, "evidence": "tests/runner-e2e/results/local-2026-10-06T20-44-06-234Z/public-mcp/assistant-claude-sonnet/local/expanded-documents/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-sonnet.local.expanded-documents", "failedMatchers": [], "finishedAt": "2026-10-06T20:46:09.860Z", "kind": "case", "model": "claude-sonnet-4-6", "profileId": "assistant-claude-sonnet", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "b9c4be5f0f8a3f1143f83edf3e8a047d02888eea", "workflowRunUrl": null}, "startedAt": "2026-10-06T20:44:39.655Z", "status": "passed", "suiteDefinitionHash": "d86dc8451519467b8680e92b7d6c3bee975a4bff419e6beaace2d8eeb5ce43f5"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.6220439999999999, "inputTokens": 201168, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 1236, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 7}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.6220439999999999, "inputTokens": 201168, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 1236, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 7}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 328148, "costStatus": "reported", "inputTokens": 30862, "outputTokens": 2804, "reportedCostUsd": 0.2562279, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 361814}, "observedAndEstimatedCostUsd": 0.8782718999999999, "reportedCostUsd": 0.2562279, "runtime": {"agentRunDurationMs": 82159, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T20-44-06-234Z", "caseId": "expanded-files", "durationMs": 128085, "evidence": "tests/runner-e2e/results/local-2026-10-06T20-44-06-234Z/public-mcp/assistant-claude-sonnet/local/expanded-files/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-sonnet.local.expanded-files", "failedMatchers": [], "finishedAt": "2026-10-06T20:48:55.626Z", "kind": "case", "model": "claude-sonnet-4-6", "profileId": "assistant-claude-sonnet", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "b9c4be5f0f8a3f1143f83edf3e8a047d02888eea", "workflowRunUrl": null}, "startedAt": "2026-10-06T20:46:47.541Z", "status": "passed", "suiteDefinitionHash": "d86dc8451519467b8680e92b7d6c3bee975a4bff419e6beaace2d8eeb5ce43f5"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.338097, "inputTokens": 109654, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 609, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 4}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.338097, "inputTokens": 109654, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 609, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 4}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 157229, "costStatus": "reported", "inputTokens": 27815, "outputTokens": 1275, "reportedCostUsd": 0.17059395000000002, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 186319}, "observedAndEstimatedCostUsd": 0.50869095, "reportedCostUsd": 0.17059395000000002, "runtime": {"agentRunDurationMs": 33886, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T20-44-06-234Z", "caseId": "expanded-permissions", "durationMs": 66505, "evidence": "tests/runner-e2e/results/local-2026-10-06T20-44-06-234Z/public-mcp/assistant-claude-sonnet/local/expanded-permissions/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-sonnet.local.expanded-permissions", "failedMatchers": [], "finishedAt": "2026-10-06T20:53:03.940Z", "kind": "case", "model": "claude-sonnet-4-6", "profileId": "assistant-claude-sonnet", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "b9c4be5f0f8a3f1143f83edf3e8a047d02888eea", "workflowRunUrl": null}, "startedAt": "2026-10-06T20:51:57.435Z", "status": "passed", "suiteDefinitionHash": "d86dc8451519467b8680e92b7d6c3bee975a4bff419e6beaace2d8eeb5ce43f5"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.42749099999999995, "inputTokens": 138497, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 800, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 5}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.42749099999999995, "inputTokens": 138497, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 800, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 5}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 239517, "costStatus": "reported", "inputTokens": 28326, "outputTokens": 1553, "reportedCostUsd": 0.20136510000000002, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 269396}, "observedAndEstimatedCostUsd": 0.6288560999999999, "reportedCostUsd": 0.20136510000000002, "runtime": {"agentRunDurationMs": 33371, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T20-44-06-234Z", "caseId": "expanded-projects", "durationMs": 66480, "evidence": "tests/runner-e2e/results/local-2026-10-06T20-44-06-234Z/public-mcp/assistant-claude-sonnet/local/expanded-projects/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-sonnet.local.expanded-projects", "failedMatchers": [], "finishedAt": "2026-10-06T20:50:32.596Z", "kind": "case", "model": "claude-sonnet-4-6", "profileId": "assistant-claude-sonnet", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "b9c4be5f0f8a3f1143f83edf3e8a047d02888eea", "workflowRunUrl": null}, "startedAt": "2026-10-06T20:49:26.116Z", "status": "passed", "suiteDefinitionHash": "d86dc8451519467b8680e92b7d6c3bee975a4bff419e6beaace2d8eeb5ce43f5"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.55593, "inputTokens": 175630, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 1936, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 6}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.55593, "inputTokens": 175630, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 1936, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 6}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 280406, "costStatus": "reported", "inputTokens": 28614, "outputTokens": 1763, "reportedCostUsd": 0.21786104999999997, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 310783}, "observedAndEstimatedCostUsd": 0.77379105, "reportedCostUsd": 0.21786104999999997, "runtime": {"agentRunDurationMs": 37755, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T20-44-06-234Z", "caseId": "expanded-skills", "durationMs": 91116, "evidence": "tests/runner-e2e/results/local-2026-10-06T20-44-06-234Z/public-mcp/assistant-claude-sonnet/local/expanded-skills/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-sonnet.local.expanded-skills", "failedMatchers": [], "finishedAt": "2026-10-06T20:51:09.431Z", "kind": "case", "model": "claude-sonnet-4-6", "profileId": "assistant-claude-sonnet", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "b9c4be5f0f8a3f1143f83edf3e8a047d02888eea", "workflowRunUrl": null}, "startedAt": "2026-10-06T20:49:38.315Z", "status": "passed", "suiteDefinitionHash": "d86dc8451519467b8680e92b7d6c3bee975a4bff419e6beaace2d8eeb5ce43f5"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.630108, "inputTokens": 203176, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 1372, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 7}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.630108, "inputTokens": 203176, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 1372, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 7}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 239754, "costStatus": "reported", "inputTokens": 30240, "outputTokens": 1740, "reportedCostUsd": 0.21141870000000001, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 271734}, "observedAndEstimatedCostUsd": 0.8415267, "reportedCostUsd": 0.21141870000000001, "runtime": {"agentRunDurationMs": 37525, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T20-44-06-234Z", "caseId": "expanded-task-edit", "durationMs": 79582, "evidence": "tests/runner-e2e/results/local-2026-10-06T20-44-06-234Z/public-mcp/assistant-claude-sonnet/local/expanded-task-edit/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-sonnet.local.expanded-task-edit", "failedMatchers": [], "finishedAt": "2026-10-06T20:45:59.237Z", "kind": "case", "model": "claude-sonnet-4-6", "profileId": "assistant-claude-sonnet", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "b9c4be5f0f8a3f1143f83edf3e8a047d02888eea", "workflowRunUrl": null}, "startedAt": "2026-10-06T20:44:39.655Z", "status": "passed", "suiteDefinitionHash": "d86dc8451519467b8680e92b7d6c3bee975a4bff419e6beaace2d8eeb5ce43f5"}
{"billing": {"agentRunDurationMs": 772086, "assistant": {"cachedInputTokens": 910336, "estimatedCostUsd": 5.5999777, "inputTokens": 2638535, "outputTokens": 25228, "requests": 138}, "estimatedRuntimeCostUsd": 0, "leaseDurationMs": 0, "llm": {"cachedInputTokens": 3452553, "costStatus": "partial", "inputTokens": 997303, "outputTokens": 40655, "reportedCostUsd": 1.9669619500000002, "runCount": 24, "runsWithReportedCost": 15, "runsWithTokenUsage": 23, "totalTokens": 4490511}, "observedAndEstimatedCostUsd": 7.56693965, "reportedLlmCostUsd": 1.9669619500000002, "testCount": 24, "testsWithCompleteBilling": 15}, "campaignId": "local-2026-10-06T21-17-35-704Z", "cleanupPassed": false, "complete": false, "executed": 24, "expectedSelectedCases": ["public-mcp.assistant-codex-mini.local.expanded-task-edit", "public-mcp.assistant-codex-mini.local.expanded-documents", "public-mcp.assistant-codex-mini.local.expanded-files", "public-mcp.assistant-codex-mini.local.expanded-agent-config", "public-mcp.assistant-codex-mini.local.expanded-projects", "public-mcp.assistant-codex-mini.local.expanded-skills", "public-mcp.assistant-codex-mini.local.expanded-api", "public-mcp.assistant-codex-mini.local.expanded-permissions", "public-mcp.assistant-claude-haiku.local.expanded-task-edit", "public-mcp.assistant-claude-haiku.local.expanded-documents", "public-mcp.assistant-claude-haiku.local.expanded-files", "public-mcp.assistant-claude-haiku.local.expanded-agent-config", "public-mcp.assistant-claude-haiku.local.expanded-projects", "public-mcp.assistant-claude-haiku.local.expanded-skills", "public-mcp.assistant-claude-haiku.local.expanded-api", "public-mcp.assistant-claude-haiku.local.expanded-permissions", "public-mcp.assistant-claude-sonnet.local.expanded-task-edit", "public-mcp.assistant-claude-sonnet.local.expanded-documents", "public-mcp.assistant-claude-sonnet.local.expanded-files", "public-mcp.assistant-claude-sonnet.local.expanded-agent-config", "public-mcp.assistant-claude-sonnet.local.expanded-projects", "public-mcp.assistant-claude-sonnet.local.expanded-skills", "public-mcp.assistant-claude-sonnet.local.expanded-api", "public-mcp.assistant-claude-sonnet.local.expanded-permissions"], "fullSuiteCoverage": [{"billing": {"agentRunDurationMs": 772086, "assistant": {"cachedInputTokens": 910336, "estimatedCostUsd": 5.5999777, "inputTokens": 2638535, "outputTokens": 25228, "requests": 138}, "estimatedRuntimeCostUsd": 0, "leaseDurationMs": 0, "llm": {"cachedInputTokens": 3452553, "costStatus": "partial", "inputTokens": 997303, "outputTokens": 40655, "reportedCostUsd": 1.9669619500000002, "runCount": 24, "runsWithReportedCost": 15, "runsWithTokenUsage": 23, "totalTokens": 4490511}, "observedAndEstimatedCostUsd": 7.56693965, "reportedLlmCostUsd": 1.9669619500000002, "testCount": 24, "testsWithCompleteBilling": 15}, "cleanupPassed": false, "complete": false, "durationMs": 1650611, "executed": 24, "expected": 63, "failed": 1, "incomplete": 0, "passed": 23, "retries": 0, "selected": 24, "suiteDefinitionHash": "6c2b454cf8700fe2345f8072dc4a7ccde066ae58a8afd150bf1f72150bef1f04", "suiteId": "public-mcp"}], "kind": "campaign", "passed": 23, "selected": 24, "source": {"eventName": null, "ref": "codex/expanded-assistant-mcp", "sha": "e81f114bc6c813e7784cde0d8199538880aebe1f", "workflowRunUrl": null}}
{"assistantUsage": null, "billing": {}, "campaignId": "local-2026-10-06T21-17-35-704Z", "caseId": "expanded-agent-config", "durationMs": 184062, "evidence": "tests/runner-e2e/results/local-2026-10-06T21-17-35-704Z/public-mcp/assistant-claude-haiku/local/expanded-agent-config/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-haiku.local.expanded-agent-config", "failedMatchers": [], "finishedAt": "2026-10-06T21:29:50.671Z", "kind": "case", "model": "claude-haiku-4-5-20251001", "profileId": "assistant-claude-haiku", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "e81f114bc6c813e7784cde0d8199538880aebe1f", "workflowRunUrl": null}, "startedAt": "2026-10-06T21:26:46.609Z", "status": "failed", "suiteDefinitionHash": "6c2b454cf8700fe2345f8072dc4a7ccde066ae58a8afd150bf1f72150bef1f04"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.15365099999999998, "inputTokens": 148436, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 1043, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 5}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.15365099999999998, "inputTokens": 148436, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 1043, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 5}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 163826, "costStatus": "reported", "inputTokens": 13644, "outputTokens": 2043, "reportedCostUsd": 0.0436381, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 179513}, "observedAndEstimatedCostUsd": 0.1972891, "reportedCostUsd": 0.0436381, "runtime": {"agentRunDurationMs": 21735, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T21-17-35-704Z", "caseId": "expanded-api", "durationMs": 49083, "evidence": "tests/runner-e2e/results/local-2026-10-06T21-17-35-704Z/public-mcp/assistant-claude-haiku/local/expanded-api/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-haiku.local.expanded-api", "failedMatchers": [], "finishedAt": "2026-10-06T21:34:21.132Z", "kind": "case", "model": "claude-haiku-4-5-20251001", "profileId": "assistant-claude-haiku", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "e81f114bc6c813e7784cde0d8199538880aebe1f", "workflowRunUrl": null}, "startedAt": "2026-10-06T21:33:32.049Z", "status": "passed", "suiteDefinitionHash": "6c2b454cf8700fe2345f8072dc4a7ccde066ae58a8afd150bf1f72150bef1f04"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.35932000000000003, "inputTokens": 345825, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 2699, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 12}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.35932000000000003, "inputTokens": 345825, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 2699, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 12}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 138715, "costStatus": "reported", "inputTokens": 13470, "outputTokens": 1987, "reportedCostUsd": 0.04063150000000001, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 154172}, "observedAndEstimatedCostUsd": 0.3999515, "reportedCostUsd": 0.04063150000000001, "runtime": {"agentRunDurationMs": 32206, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T21-17-35-704Z", "caseId": "expanded-documents", "durationMs": 98288, "evidence": "tests/runner-e2e/results/local-2026-10-06T21-17-35-704Z/public-mcp/assistant-claude-haiku/local/expanded-documents/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-haiku.local.expanded-documents", "failedMatchers": [], "finishedAt": "2026-10-06T21:26:35.522Z", "kind": "case", "model": "claude-haiku-4-5-20251001", "profileId": "assistant-claude-haiku", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "e81f114bc6c813e7784cde0d8199538880aebe1f", "workflowRunUrl": null}, "startedAt": "2026-10-06T21:24:57.234Z", "status": "passed", "suiteDefinitionHash": "6c2b454cf8700fe2345f8072dc4a7ccde066ae58a8afd150bf1f72150bef1f04"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.20797500000000002, "inputTokens": 201720, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 1251, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 7}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.20797500000000002, "inputTokens": 201720, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 1251, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 7}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 111457, "costStatus": "reported", "inputTokens": 13103, "outputTokens": 1752, "reportedCostUsd": 0.03627395, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 126312}, "observedAndEstimatedCostUsd": 0.24424895000000002, "reportedCostUsd": 0.03627395, "runtime": {"agentRunDurationMs": 25614, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T21-17-35-704Z", "caseId": "expanded-files", "durationMs": 84982, "evidence": "tests/runner-e2e/results/local-2026-10-06T21-17-35-704Z/public-mcp/assistant-claude-haiku/local/expanded-files/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-haiku.local.expanded-files", "failedMatchers": [], "finishedAt": "2026-10-06T21:30:38.639Z", "kind": "case", "model": "claude-haiku-4-5-20251001", "profileId": "assistant-claude-haiku", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "e81f114bc6c813e7784cde0d8199538880aebe1f", "workflowRunUrl": null}, "startedAt": "2026-10-06T21:29:13.657Z", "status": "passed", "suiteDefinitionHash": "6c2b454cf8700fe2345f8072dc4a7ccde066ae58a8afd150bf1f72150bef1f04"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.11337599999999999, "inputTokens": 109791, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 717, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 4}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.11337599999999999, "inputTokens": 109791, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 717, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 4}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 112359, "costStatus": "reported", "inputTokens": 13295, "outputTokens": 1931, "reportedCostUsd": 0.03749915, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 127585}, "observedAndEstimatedCostUsd": 0.15087514999999999, "reportedCostUsd": 0.03749915, "runtime": {"agentRunDurationMs": 21384, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T21-17-35-704Z", "caseId": "expanded-permissions", "durationMs": 45236, "evidence": "tests/runner-e2e/results/local-2026-10-06T21-17-35-704Z/public-mcp/assistant-claude-haiku/local/expanded-permissions/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-haiku.local.expanded-permissions", "failedMatchers": [], "finishedAt": "2026-10-06T21:34:49.702Z", "kind": "case", "model": "claude-haiku-4-5-20251001", "profileId": "assistant-claude-haiku", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "e81f114bc6c813e7784cde0d8199538880aebe1f", "workflowRunUrl": null}, "startedAt": "2026-10-06T21:34:04.466Z", "status": "passed", "suiteDefinitionHash": "6c2b454cf8700fe2345f8072dc4a7ccde066ae58a8afd150bf1f72150bef1f04"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.141537, "inputTokens": 137807, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 746, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 5}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.141537, "inputTokens": 137807, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 746, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 5}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 164292, "costStatus": "reported", "inputTokens": 14393, "outputTokens": 2500, "reportedCostUsd": 0.046905949999999995, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 181185}, "observedAndEstimatedCostUsd": 0.18844295, "reportedCostUsd": 0.046905949999999995, "runtime": {"agentRunDurationMs": 29120, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T21-17-35-704Z", "caseId": "expanded-projects", "durationMs": 59089, "evidence": "tests/runner-e2e/results/local-2026-10-06T21-17-35-704Z/public-mcp/assistant-claude-haiku/local/expanded-projects/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-haiku.local.expanded-projects", "failedMatchers": [], "finishedAt": "2026-10-06T21:32:48.196Z", "kind": "case", "model": "claude-haiku-4-5-20251001", "profileId": "assistant-claude-haiku", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "e81f114bc6c813e7784cde0d8199538880aebe1f", "workflowRunUrl": null}, "startedAt": "2026-10-06T21:31:49.107Z", "status": "passed", "suiteDefinitionHash": "6c2b454cf8700fe2345f8072dc4a7ccde066ae58a8afd150bf1f72150bef1f04"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.18001499999999998, "inputTokens": 172010, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 1601, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 6}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.18001499999999998, "inputTokens": 172010, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 1601, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 6}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 85989, "costStatus": "reported", "inputTokens": 12468, "outputTokens": 1343, "reportedCostUsd": 0.0308904, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 99800}, "observedAndEstimatedCostUsd": 0.21090539999999997, "reportedCostUsd": 0.0308904, "runtime": {"agentRunDurationMs": 17022, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T21-17-35-704Z", "caseId": "expanded-skills", "durationMs": 53124, "evidence": "tests/runner-e2e/results/local-2026-10-06T21-17-35-704Z/public-mcp/assistant-claude-haiku/local/expanded-skills/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-haiku.local.expanded-skills", "failedMatchers": [], "finishedAt": "2026-10-06T21:33:36.319Z", "kind": "case", "model": "claude-haiku-4-5-20251001", "profileId": "assistant-claude-haiku", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "e81f114bc6c813e7784cde0d8199538880aebe1f", "workflowRunUrl": null}, "startedAt": "2026-10-06T21:32:43.195Z", "status": "passed", "suiteDefinitionHash": "6c2b454cf8700fe2345f8072dc4a7ccde066ae58a8afd150bf1f72150bef1f04"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.11823900000000001, "inputTokens": 112524, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 1143, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 4}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.11823900000000001, "inputTokens": 112524, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 1143, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 4}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 202792, "costStatus": "reported", "inputTokens": 18931, "outputTokens": 2650, "reportedCostUsd": 0.05717645000000001, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 224373}, "observedAndEstimatedCostUsd": 0.17541545000000003, "reportedCostUsd": 0.05717645000000001, "runtime": {"agentRunDurationMs": 32081, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T21-17-35-704Z", "caseId": "expanded-task-edit", "durationMs": 81076, "evidence": "tests/runner-e2e/results/local-2026-10-06T21-17-35-704Z/public-mcp/assistant-claude-haiku/local/expanded-task-edit/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-haiku.local.expanded-task-edit", "failedMatchers": [], "finishedAt": "2026-10-06T21:25:41.520Z", "kind": "case", "model": "claude-haiku-4-5-20251001", "profileId": "assistant-claude-haiku", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "e81f114bc6c813e7784cde0d8199538880aebe1f", "workflowRunUrl": null}, "startedAt": "2026-10-06T21:24:20.444Z", "status": "passed", "suiteDefinitionHash": "6c2b454cf8700fe2345f8072dc4a7ccde066ae58a8afd150bf1f72150bef1f04"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.44577300000000003, "inputTokens": 142851, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 1148, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 5}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.44577300000000003, "inputTokens": 142851, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 1148, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 5}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 241325, "costStatus": "reported", "inputTokens": 29014, "outputTokens": 1686, "reportedCostUsd": 0.20648249999999999, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 272025}, "observedAndEstimatedCostUsd": 0.6522555, "reportedCostUsd": 0.20648249999999999, "runtime": {"agentRunDurationMs": 35202, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T21-17-35-704Z", "caseId": "expanded-agent-config", "durationMs": 71298, "evidence": "tests/runner-e2e/results/local-2026-10-06T21-17-35-704Z/public-mcp/assistant-claude-sonnet/local/expanded-agent-config/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-sonnet.local.expanded-agent-config", "failedMatchers": [], "finishedAt": "2026-10-06T21:38:20.681Z", "kind": "case", "model": "claude-sonnet-4-6", "profileId": "assistant-claude-sonnet", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "e81f114bc6c813e7784cde0d8199538880aebe1f", "workflowRunUrl": null}, "startedAt": "2026-10-06T21:37:09.383Z", "status": "passed", "suiteDefinitionHash": "6c2b454cf8700fe2345f8072dc4a7ccde066ae58a8afd150bf1f72150bef1f04"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.37098, "inputTokens": 118110, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 1110, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 4}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.37098, "inputTokens": 118110, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 1110, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 4}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 323786, "costStatus": "reported", "inputTokens": 28944, "outputTokens": 1755, "reportedCostUsd": 0.23199180000000003, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 354485}, "observedAndEstimatedCostUsd": 0.6029718, "reportedCostUsd": 0.23199180000000003, "runtime": {"agentRunDurationMs": 36340, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T21-17-35-704Z", "caseId": "expanded-api", "durationMs": 70190, "evidence": "tests/runner-e2e/results/local-2026-10-06T21-17-35-704Z/public-mcp/assistant-claude-sonnet/local/expanded-api/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-sonnet.local.expanded-api", "failedMatchers": [], "finishedAt": "2026-10-06T21:40:58.966Z", "kind": "case", "model": "claude-sonnet-4-6", "profileId": "assistant-claude-sonnet", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "e81f114bc6c813e7784cde0d8199538880aebe1f", "workflowRunUrl": null}, "startedAt": "2026-10-06T21:39:48.776Z", "status": "passed", "suiteDefinitionHash": "6c2b454cf8700fe2345f8072dc4a7ccde066ae58a8afd150bf1f72150bef1f04"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.782082, "inputTokens": 252459, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 1647, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 9}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.782082, "inputTokens": 252459, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 1647, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 9}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 198920, "costStatus": "reported", "inputTokens": 28385, "outputTokens": 1600, "reportedCostUsd": 0.190113, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 228905}, "observedAndEstimatedCostUsd": 0.972195, "reportedCostUsd": 0.190113, "runtime": {"agentRunDurationMs": 33031, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T21-17-35-704Z", "caseId": "expanded-documents", "durationMs": 79503, "evidence": "tests/runner-e2e/results/local-2026-10-06T21-17-35-704Z/public-mcp/assistant-claude-sonnet/local/expanded-documents/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-sonnet.local.expanded-documents", "failedMatchers": [], "finishedAt": "2026-10-06T21:36:40.901Z", "kind": "case", "model": "claude-sonnet-4-6", "profileId": "assistant-claude-sonnet", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "e81f114bc6c813e7784cde0d8199538880aebe1f", "workflowRunUrl": null}, "startedAt": "2026-10-06T21:35:21.398Z", "status": "passed", "suiteDefinitionHash": "6c2b454cf8700fe2345f8072dc4a7ccde066ae58a8afd150bf1f72150bef1f04"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.6290610000000001, "inputTokens": 202682, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 1401, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 7}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.6290610000000001, "inputTokens": 202682, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 1401, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 7}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 284171, "costStatus": "reported", "inputTokens": 30384, "outputTokens": 1780, "reportedCostUsd": 0.22588304999999995, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 316335}, "observedAndEstimatedCostUsd": 0.8549440500000001, "reportedCostUsd": 0.22588304999999995, "runtime": {"agentRunDurationMs": 36261, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T21-17-35-704Z", "caseId": "expanded-files", "durationMs": 76675, "evidence": "tests/runner-e2e/results/local-2026-10-06T21-17-35-704Z/public-mcp/assistant-claude-sonnet/local/expanded-files/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-sonnet.local.expanded-files", "failedMatchers": [], "finishedAt": "2026-10-06T21:37:55.686Z", "kind": "case", "model": "claude-sonnet-4-6", "profileId": "assistant-claude-sonnet", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "e81f114bc6c813e7784cde0d8199538880aebe1f", "workflowRunUrl": null}, "startedAt": "2026-10-06T21:36:39.011Z", "status": "passed", "suiteDefinitionHash": "6c2b454cf8700fe2345f8072dc4a7ccde066ae58a8afd150bf1f72150bef1f04"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.339963, "inputTokens": 109741, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 716, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 4}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.339963, "inputTokens": 109741, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 716, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 4}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 158513, "costStatus": "reported", "inputTokens": 28531, "outputTokens": 1919, "reportedCostUsd": 0.18332415, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 188963}, "observedAndEstimatedCostUsd": 0.52328715, "reportedCostUsd": 0.18332415, "runtime": {"agentRunDurationMs": 35103, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T21-17-35-704Z", "caseId": "expanded-permissions", "durationMs": 65924, "evidence": "tests/runner-e2e/results/local-2026-10-06T21-17-35-704Z/public-mcp/assistant-claude-sonnet/local/expanded-permissions/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-sonnet.local.expanded-permissions", "failedMatchers": [], "finishedAt": "2026-10-06T21:41:29.427Z", "kind": "case", "model": "claude-sonnet-4-6", "profileId": "assistant-claude-sonnet", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "e81f114bc6c813e7784cde0d8199538880aebe1f", "workflowRunUrl": null}, "startedAt": "2026-10-06T21:40:23.503Z", "status": "passed", "suiteDefinitionHash": "6c2b454cf8700fe2345f8072dc4a7ccde066ae58a8afd150bf1f72150bef1f04"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.42745799999999995, "inputTokens": 138506, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 796, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 5}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.42745799999999995, "inputTokens": 138506, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 796, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 5}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 283094, "costStatus": "reported", "inputTokens": 29160, "outputTokens": 1677, "reportedCostUsd": 0.21942494999999998, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 313931}, "observedAndEstimatedCostUsd": 0.64688295, "reportedCostUsd": 0.21942494999999998, "runtime": {"agentRunDurationMs": 32516, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T21-17-35-704Z", "caseId": "expanded-projects", "durationMs": 64765, "evidence": "tests/runner-e2e/results/local-2026-10-06T21-17-35-704Z/public-mcp/assistant-claude-sonnet/local/expanded-projects/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-sonnet.local.expanded-projects", "failedMatchers": [], "finishedAt": "2026-10-06T21:39:26.337Z", "kind": "case", "model": "claude-sonnet-4-6", "profileId": "assistant-claude-sonnet", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "e81f114bc6c813e7784cde0d8199538880aebe1f", "workflowRunUrl": null}, "startedAt": "2026-10-06T21:38:21.572Z", "status": "passed", "suiteDefinitionHash": "6c2b454cf8700fe2345f8072dc4a7ccde066ae58a8afd150bf1f72150bef1f04"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.557067, "inputTokens": 175934, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 1951, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 6}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.557067, "inputTokens": 175934, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 1951, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 6}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 199571, "costStatus": "reported", "inputTokens": 28549, "outputTokens": 1481, "reportedCostUsd": 0.18913830000000004, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 229601}, "observedAndEstimatedCostUsd": 0.7462053, "reportedCostUsd": 0.18913830000000004, "runtime": {"agentRunDurationMs": 29635, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T21-17-35-704Z", "caseId": "expanded-skills", "durationMs": 74868, "evidence": "tests/runner-e2e/results/local-2026-10-06T21-17-35-704Z/public-mcp/assistant-claude-sonnet/local/expanded-skills/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-sonnet.local.expanded-skills", "failedMatchers": [], "finishedAt": "2026-10-06T21:40:01.732Z", "kind": "case", "model": "claude-sonnet-4-6", "profileId": "assistant-claude-sonnet", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "e81f114bc6c813e7784cde0d8199538880aebe1f", "workflowRunUrl": null}, "startedAt": "2026-10-06T21:38:46.864Z", "status": "passed", "suiteDefinitionHash": "6c2b454cf8700fe2345f8072dc4a7ccde066ae58a8afd150bf1f72150bef1f04"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.627936, "inputTokens": 202537, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 1355, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 7}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.627936, "inputTokens": 202537, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 1355, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 7}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 283519, "costStatus": "reported", "inputTokens": 29219, "outputTokens": 2198, "reportedCostUsd": 0.2275887, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 314936}, "observedAndEstimatedCostUsd": 0.8555247, "reportedCostUsd": 0.2275887, "runtime": {"agentRunDurationMs": 46835, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T21-17-35-704Z", "caseId": "expanded-task-edit", "durationMs": 86199, "evidence": "tests/runner-e2e/results/local-2026-10-06T21-17-35-704Z/public-mcp/assistant-claude-sonnet/local/expanded-task-edit/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-sonnet.local.expanded-task-edit", "failedMatchers": [], "finishedAt": "2026-10-06T21:36:15.478Z", "kind": "case", "model": "claude-sonnet-4-6", "profileId": "assistant-claude-sonnet", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "e81f114bc6c813e7784cde0d8199538880aebe1f", "workflowRunUrl": null}, "startedAt": "2026-10-06T21:34:49.279Z", "status": "passed", "suiteDefinitionHash": "6c2b454cf8700fe2345f8072dc4a7ccde066ae58a8afd150bf1f72150bef1f04"}
{"assistantUsage": {"cachedInputTokens": 114688, "estimatedCostUsd": 0.0169806, "inputTokens": 5550, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 937, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 6}, "billing": {"assistant": {"cachedInputTokens": 114688, "estimatedCostUsd": 0.0169806, "inputTokens": 5550, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 937, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 6}, "complete": false, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 72704, "costStatus": "unpriced", "inputTokens": 92511, "outputTokens": 1836, "reportedCostUsd": 0, "runCount": 1, "runsWithReportedCost": 0, "runsWithTokenUsage": 1, "totalTokens": 167051}, "observedAndEstimatedCostUsd": 0.0169806, "reportedCostUsd": 0, "runtime": {"agentRunDurationMs": 15163, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T21-17-35-704Z", "caseId": "expanded-agent-config", "durationMs": 45595, "evidence": "tests/runner-e2e/results/local-2026-10-06T21-17-35-704Z/public-mcp/assistant-codex-mini/local/expanded-agent-config/attempt-1/result.json", "executionId": "public-mcp.assistant-codex-mini.local.expanded-agent-config", "failedMatchers": [], "finishedAt": "2026-10-06T21:20:09.373Z", "kind": "case", "model": "gpt-5.4-mini", "profileId": "assistant-codex-mini", "provider": "codex", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "e81f114bc6c813e7784cde0d8199538880aebe1f", "workflowRunUrl": null}, "startedAt": "2026-10-06T21:19:23.778Z", "status": "passed", "suiteDefinitionHash": "6c2b454cf8700fe2345f8072dc4a7ccde066ae58a8afd150bf1f72150bef1f04"}
{"assistantUsage": {"cachedInputTokens": 134656, "estimatedCostUsd": 0.02458245, "inputTokens": 15447, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 644, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 6}, "billing": {"assistant": {"cachedInputTokens": 134656, "estimatedCostUsd": 0.02458245, "inputTokens": 15447, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 644, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 6}, "complete": false, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 53888, "costStatus": "unpriced", "inputTokens": 73003, "outputTokens": 1467, "reportedCostUsd": 0, "runCount": 1, "runsWithReportedCost": 0, "runsWithTokenUsage": 1, "totalTokens": 128358}, "observedAndEstimatedCostUsd": 0.02458245, "reportedCostUsd": 0, "runtime": {"agentRunDurationMs": 17763, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T21-17-35-704Z", "caseId": "expanded-api", "durationMs": 52708, "evidence": "tests/runner-e2e/results/local-2026-10-06T21-17-35-704Z/public-mcp/assistant-codex-mini/local/expanded-api/attempt-1/result.json", "executionId": "public-mcp.assistant-codex-mini.local.expanded-api", "failedMatchers": [], "finishedAt": "2026-10-06T21:23:08.334Z", "kind": "case", "model": "gpt-5.4-mini", "profileId": "assistant-codex-mini", "provider": "codex", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "e81f114bc6c813e7784cde0d8199538880aebe1f", "workflowRunUrl": null}, "startedAt": "2026-10-06T21:22:15.626Z", "status": "passed", "suiteDefinitionHash": "6c2b454cf8700fe2345f8072dc4a7ccde066ae58a8afd150bf1f72150bef1f04"}
{"assistantUsage": {"cachedInputTokens": 149504, "estimatedCostUsd": 0.03270255, "inputTokens": 24225, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 738, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 9}, "billing": {"assistant": {"cachedInputTokens": 149504, "estimatedCostUsd": 0.03270255, "inputTokens": 24225, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 738, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 9}, "complete": false, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 53376, "costStatus": "unpriced", "inputTokens": 73455, "outputTokens": 1267, "reportedCostUsd": 0, "runCount": 1, "runsWithReportedCost": 0, "runsWithTokenUsage": 1, "totalTokens": 128098}, "observedAndEstimatedCostUsd": 0.03270255, "reportedCostUsd": 0, "runtime": {"agentRunDurationMs": 12892, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T21-17-35-704Z", "caseId": "expanded-documents", "durationMs": 54572, "evidence": "tests/runner-e2e/results/local-2026-10-06T21-17-35-704Z/public-mcp/assistant-codex-mini/local/expanded-documents/attempt-1/result.json", "executionId": "public-mcp.assistant-codex-mini.local.expanded-documents", "failedMatchers": [], "finishedAt": "2026-10-06T21:18:54.733Z", "kind": "case", "model": "gpt-5.4-mini", "profileId": "assistant-codex-mini", "provider": "codex", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "e81f114bc6c813e7784cde0d8199538880aebe1f", "workflowRunUrl": null}, "startedAt": "2026-10-06T21:18:00.161Z", "status": "passed", "suiteDefinitionHash": "6c2b454cf8700fe2345f8072dc4a7ccde066ae58a8afd150bf1f72150bef1f04"}
{"assistantUsage": {"cachedInputTokens": 155648, "estimatedCostUsd": 0.0220881, "inputTokens": 7688, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 1033, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 8}, "billing": {"assistant": {"cachedInputTokens": 155648, "estimatedCostUsd": 0.0220881, "inputTokens": 7688, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 1033, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 8}, "complete": false, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 68096, "costStatus": "unpriced", "inputTokens": 77200, "outputTokens": 1238, "reportedCostUsd": 0, "runCount": 1, "runsWithReportedCost": 0, "runsWithTokenUsage": 1, "totalTokens": 146534}, "observedAndEstimatedCostUsd": 0.0220881, "reportedCostUsd": 0, "runtime": {"agentRunDurationMs": 11685, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T21-17-35-704Z", "caseId": "expanded-files", "durationMs": 44267, "evidence": "tests/runner-e2e/results/local-2026-10-06T21-17-35-704Z/public-mcp/assistant-codex-mini/local/expanded-files/attempt-1/result.json", "executionId": "public-mcp.assistant-codex-mini.local.expanded-files", "failedMatchers": [], "finishedAt": "2026-10-06T21:20:08.046Z", "kind": "case", "model": "gpt-5.4-mini", "profileId": "assistant-codex-mini", "provider": "codex", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "e81f114bc6c813e7784cde0d8199538880aebe1f", "workflowRunUrl": null}, "startedAt": "2026-10-06T21:19:23.779Z", "status": "passed", "suiteDefinitionHash": "6c2b454cf8700fe2345f8072dc4a7ccde066ae58a8afd150bf1f72150bef1f04"}
{"assistantUsage": {"cachedInputTokens": 73216, "estimatedCostUsd": 0.00943245, "inputTokens": 2675, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 430, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 4}, "billing": {"assistant": {"cachedInputTokens": 73216, "estimatedCostUsd": 0.00943245, "inputTokens": 2675, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 430, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 4}, "complete": false, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 36096, "costStatus": "unpriced", "inputTokens": 53755, "outputTokens": 1743, "reportedCostUsd": 0, "runCount": 1, "runsWithReportedCost": 0, "runsWithTokenUsage": 1, "totalTokens": 91594}, "observedAndEstimatedCostUsd": 0.00943245, "reportedCostUsd": 0, "runtime": {"agentRunDurationMs": 18373, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T21-17-35-704Z", "caseId": "expanded-permissions", "durationMs": 59169, "evidence": "tests/runner-e2e/results/local-2026-10-06T21-17-35-704Z/public-mcp/assistant-codex-mini/local/expanded-permissions/attempt-1/result.json", "executionId": "public-mcp.assistant-codex-mini.local.expanded-permissions", "failedMatchers": [], "finishedAt": "2026-10-06T21:23:54.867Z", "kind": "case", "model": "gpt-5.4-mini", "profileId": "assistant-codex-mini", "provider": "codex", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "e81f114bc6c813e7784cde0d8199538880aebe1f", "workflowRunUrl": null}, "startedAt": "2026-10-06T21:22:55.698Z", "status": "passed", "suiteDefinitionHash": "6c2b454cf8700fe2345f8072dc4a7ccde066ae58a8afd150bf1f72150bef1f04"}
{"assistantUsage": {"cachedInputTokens": 73728, "estimatedCostUsd": 0.009462600000000002, "inputTokens": 2664, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 430, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 4}, "billing": {"assistant": {"cachedInputTokens": 73728, "estimatedCostUsd": 0.009462600000000002, "inputTokens": 2664, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 430, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 4}, "complete": false, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 70656, "costStatus": "unpriced", "inputTokens": 90565, "outputTokens": 1170, "reportedCostUsd": 0, "runCount": 1, "runsWithReportedCost": 0, "runsWithTokenUsage": 1, "totalTokens": 162391}, "observedAndEstimatedCostUsd": 0.009462600000000002, "reportedCostUsd": 0, "runtime": {"agentRunDurationMs": 14833, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T21-17-35-704Z", "caseId": "expanded-projects", "durationMs": 45088, "evidence": "tests/runner-e2e/results/local-2026-10-06T21-17-35-704Z/public-mcp/assistant-codex-mini/local/expanded-projects/attempt-1/result.json", "executionId": "public-mcp.assistant-codex-mini.local.expanded-projects", "failedMatchers": [], "finishedAt": "2026-10-06T21:21:31.260Z", "kind": "case", "model": "gpt-5.4-mini", "profileId": "assistant-codex-mini", "provider": "codex", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "e81f114bc6c813e7784cde0d8199538880aebe1f", "workflowRunUrl": null}, "startedAt": "2026-10-06T21:20:46.172Z", "status": "passed", "suiteDefinitionHash": "6c2b454cf8700fe2345f8072dc4a7ccde066ae58a8afd150bf1f72150bef1f04"}
{"assistantUsage": {"cachedInputTokens": 94720, "estimatedCostUsd": 0.01497375, "inputTokens": 4439, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 1009, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 5}, "billing": {"assistant": {"cachedInputTokens": 94720, "estimatedCostUsd": 0.01497375, "inputTokens": 4439, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 1009, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 5}, "complete": false, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 72192, "costStatus": "unpriced", "inputTokens": 110227, "outputTokens": 2054, "reportedCostUsd": 0, "runCount": 1, "runsWithReportedCost": 0, "runsWithTokenUsage": 1, "totalTokens": 184473}, "observedAndEstimatedCostUsd": 0.01497375, "reportedCostUsd": 0, "runtime": {"agentRunDurationMs": 17848, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T21-17-35-704Z", "caseId": "expanded-skills", "durationMs": 57073, "evidence": "tests/runner-e2e/results/local-2026-10-06T21-17-35-704Z/public-mcp/assistant-codex-mini/local/expanded-skills/attempt-1/result.json", "executionId": "public-mcp.assistant-codex-mini.local.expanded-skills", "failedMatchers": [], "finishedAt": "2026-10-06T21:21:43.246Z", "kind": "case", "model": "gpt-5.4-mini", "profileId": "assistant-codex-mini", "provider": "codex", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "e81f114bc6c813e7784cde0d8199538880aebe1f", "workflowRunUrl": null}, "startedAt": "2026-10-06T21:20:46.173Z", "status": "passed", "suiteDefinitionHash": "6c2b454cf8700fe2345f8072dc4a7ccde066ae58a8afd150bf1f72150bef1f04"}
{"assistantUsage": {"cachedInputTokens": 114176, "estimatedCostUsd": 0.015322200000000001, "inputTokens": 4914, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 683, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 6}, "billing": {"assistant": {"cachedInputTokens": 114176, "estimatedCostUsd": 0.015322200000000001, "inputTokens": 4914, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 683, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 6}, "complete": false, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 73216, "costStatus": "unpriced", "inputTokens": 95097, "outputTokens": 1578, "reportedCostUsd": 0, "runCount": 1, "runsWithReportedCost": 0, "runsWithTokenUsage": 1, "totalTokens": 169891}, "observedAndEstimatedCostUsd": 0.015322200000000001, "reportedCostUsd": 0, "runtime": {"agentRunDurationMs": 15382, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T21-17-35-704Z", "caseId": "expanded-task-edit", "durationMs": 47777, "evidence": "tests/runner-e2e/results/local-2026-10-06T21-17-35-704Z/public-mcp/assistant-codex-mini/local/expanded-task-edit/attempt-1/result.json", "executionId": "public-mcp.assistant-codex-mini.local.expanded-task-edit", "failedMatchers": [], "finishedAt": "2026-10-06T21:18:47.938Z", "kind": "case", "model": "gpt-5.4-mini", "profileId": "assistant-codex-mini", "provider": "codex", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "e81f114bc6c813e7784cde0d8199538880aebe1f", "workflowRunUrl": null}, "startedAt": "2026-10-06T21:18:00.161Z", "status": "passed", "suiteDefinitionHash": "6c2b454cf8700fe2345f8072dc4a7ccde066ae58a8afd150bf1f72150bef1f04"}
{"billing": {"agentRunDurationMs": 225546, "assistant": {"cachedInputTokens": 399360, "estimatedCostUsd": 2.5403012499999997, "inputTokens": 1231801, "outputTokens": 10500, "requests": 64}, "estimatedRuntimeCostUsd": 0, "leaseDurationMs": 0, "llm": {"cachedInputTokens": 1448794, "costStatus": "partial", "inputTokens": 483274, "outputTokens": 17230, "reportedCostUsd": 0.7709905499999999, "runCount": 9, "runsWithReportedCost": 6, "runsWithTokenUsage": 9, "totalTokens": 1949298}, "observedAndEstimatedCostUsd": 3.3112918000000002, "reportedLlmCostUsd": 0.7709905499999999, "testCount": 9, "testsWithCompleteBilling": 6}, "campaignId": "local-2026-10-06T21-43-35-026Z", "cleanupPassed": true, "complete": false, "executed": 9, "expectedSelectedCases": ["public-mcp.assistant-codex-mini.local.expanded-documents", "public-mcp.assistant-codex-mini.local.expanded-files", "public-mcp.assistant-codex-mini.local.expanded-agent-config", "public-mcp.assistant-claude-haiku.local.expanded-documents", "public-mcp.assistant-claude-haiku.local.expanded-files", "public-mcp.assistant-claude-haiku.local.expanded-agent-config", "public-mcp.assistant-claude-sonnet.local.expanded-documents", "public-mcp.assistant-claude-sonnet.local.expanded-files", "public-mcp.assistant-claude-sonnet.local.expanded-agent-config"], "fullSuiteCoverage": [{"billing": {"agentRunDurationMs": 225546, "assistant": {"cachedInputTokens": 399360, "estimatedCostUsd": 2.5403012499999997, "inputTokens": 1231801, "outputTokens": 10500, "requests": 64}, "estimatedRuntimeCostUsd": 0, "leaseDurationMs": 0, "llm": {"cachedInputTokens": 1448794, "costStatus": "partial", "inputTokens": 483274, "outputTokens": 17230, "reportedCostUsd": 0.7709905499999999, "runCount": 9, "runsWithReportedCost": 6, "runsWithTokenUsage": 9, "totalTokens": 1949298}, "observedAndEstimatedCostUsd": 3.3112918000000002, "reportedLlmCostUsd": 0.7709905499999999, "testCount": 9, "testsWithCompleteBilling": 6}, "cleanupPassed": true, "complete": false, "durationMs": 527426, "executed": 9, "expected": 63, "failed": 0, "incomplete": 0, "passed": 9, "retries": 0, "selected": 9, "suiteDefinitionHash": "0d49f11ce0ebed41e74866616a1e5f03b1e52ead86131f3bbd239cbd36a81dba", "suiteId": "public-mcp"}], "kind": "campaign", "passed": 9, "selected": 9, "source": {"eventName": null, "ref": "codex/expanded-assistant-mcp", "sha": "b2196fae1a60077eb40dfe7f67e95f65f81ed8b2", "workflowRunUrl": null}}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.14882299999999998, "inputTokens": 142893, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 1186, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 5}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.14882299999999998, "inputTokens": 142893, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 1186, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 5}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 140897, "costStatus": "reported", "inputTokens": 14539, "outputTokens": 2708, "reportedCostUsd": 0.04579095, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 158144}, "observedAndEstimatedCostUsd": 0.19461394999999998, "reportedCostUsd": 0.04579095, "runtime": {"agentRunDurationMs": 30203, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T21-43-35-026Z", "caseId": "expanded-agent-config", "durationMs": 57257, "evidence": "tests/runner-e2e/results/local-2026-10-06T21-43-35-026Z/public-mcp/assistant-claude-haiku/local/expanded-agent-config/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-haiku.local.expanded-agent-config", "failedMatchers": [], "finishedAt": "2026-10-06T21:47:52.886Z", "kind": "case", "model": "claude-haiku-4-5-20251001", "profileId": "assistant-claude-haiku", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "b2196fae1a60077eb40dfe7f67e95f65f81ed8b2", "workflowRunUrl": null}, "startedAt": "2026-10-06T21:46:55.629Z", "status": "passed", "suiteDefinitionHash": "0d49f11ce0ebed41e74866616a1e5f03b1e52ead86131f3bbd239cbd36a81dba"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.26001799999999997, "inputTokens": 252188, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 1566, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 9}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.26001799999999997, "inputTokens": 252188, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 1566, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 9}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 138358, "costStatus": "reported", "inputTokens": 13717, "outputTokens": 1990, "reportedCostUsd": 0.04091954999999999, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 154065}, "observedAndEstimatedCostUsd": 0.30093754999999994, "reportedCostUsd": 0.04091954999999999, "runtime": {"agentRunDurationMs": 23286, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T21-43-35-026Z", "caseId": "expanded-documents", "durationMs": 56477, "evidence": "tests/runner-e2e/results/local-2026-10-06T21-43-35-026Z/public-mcp/assistant-claude-haiku/local/expanded-documents/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-haiku.local.expanded-documents", "failedMatchers": [], "finishedAt": "2026-10-06T21:46:26.364Z", "kind": "case", "model": "claude-haiku-4-5-20251001", "profileId": "assistant-claude-haiku", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "b2196fae1a60077eb40dfe7f67e95f65f81ed8b2", "workflowRunUrl": null}, "startedAt": "2026-10-06T21:45:29.887Z", "status": "passed", "suiteDefinitionHash": "0d49f11ce0ebed41e74866616a1e5f03b1e52ead86131f3bbd239cbd36a81dba"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.20809499999999997, "inputTokens": 201785, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 1262, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 7}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.20809499999999997, "inputTokens": 201785, "model": "claude-haiku-4-5-20251001", "observedModels": ["claude-haiku-4-5-20251001"], "outputTokens": 1262, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 7}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 163706, "costStatus": "reported", "inputTokens": 13803, "outputTokens": 2155, "reportedCostUsd": 0.044384849999999997, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 179664}, "observedAndEstimatedCostUsd": 0.25247984999999995, "reportedCostUsd": 0.044384849999999997, "runtime": {"agentRunDurationMs": 24782, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T21-43-35-026Z", "caseId": "expanded-files", "durationMs": 56835, "evidence": "tests/runner-e2e/results/local-2026-10-06T21-43-35-026Z/public-mcp/assistant-claude-haiku/local/expanded-files/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-haiku.local.expanded-files", "failedMatchers": [], "finishedAt": "2026-10-06T21:47:32.164Z", "kind": "case", "model": "claude-haiku-4-5-20251001", "profileId": "assistant-claude-haiku", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "b2196fae1a60077eb40dfe7f67e95f65f81ed8b2", "workflowRunUrl": null}, "startedAt": "2026-10-06T21:46:35.329Z", "status": "passed", "suiteDefinitionHash": "0d49f11ce0ebed41e74866616a1e5f03b1e52ead86131f3bbd239cbd36a81dba"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.449166, "inputTokens": 143492, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 1246, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 5}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.449166, "inputTokens": 143492, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 1246, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 5}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 240258, "costStatus": "reported", "inputTokens": 30801, "outputTokens": 1936, "reportedCostUsd": 0.21661365, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 272995}, "observedAndEstimatedCostUsd": 0.66577965, "reportedCostUsd": 0.21661365, "runtime": {"agentRunDurationMs": 38570, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T21-43-35-026Z", "caseId": "expanded-agent-config", "durationMs": 76376, "evidence": "tests/runner-e2e/results/local-2026-10-06T21-43-35-026Z/public-mcp/assistant-claude-sonnet/local/expanded-agent-config/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-sonnet.local.expanded-agent-config", "failedMatchers": [], "finishedAt": "2026-10-06T21:50:38.635Z", "kind": "case", "model": "claude-sonnet-4-6", "profileId": "assistant-claude-sonnet", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "b2196fae1a60077eb40dfe7f67e95f65f81ed8b2", "workflowRunUrl": null}, "startedAt": "2026-10-06T21:49:22.259Z", "status": "passed", "suiteDefinitionHash": "0d49f11ce0ebed41e74866616a1e5f03b1e52ead86131f3bbd239cbd36a81dba"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.783279, "inputTokens": 252838, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 1651, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 9}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.783279, "inputTokens": 252838, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 1651, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 9}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 281622, "costStatus": "reported", "inputTokens": 28771, "outputTokens": 1644, "reportedCostUsd": 0.21702959999999993, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 312037}, "observedAndEstimatedCostUsd": 1.0003085999999999, "reportedCostUsd": 0.21702959999999993, "runtime": {"agentRunDurationMs": 31793, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T21-43-35-026Z", "caseId": "expanded-documents", "durationMs": 72422, "evidence": "tests/runner-e2e/results/local-2026-10-06T21-43-35-026Z/public-mcp/assistant-claude-sonnet/local/expanded-documents/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-sonnet.local.expanded-documents", "failedMatchers": [], "finishedAt": "2026-10-06T21:49:03.313Z", "kind": "case", "model": "claude-sonnet-4-6", "profileId": "assistant-claude-sonnet", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "b2196fae1a60077eb40dfe7f67e95f65f81ed8b2", "workflowRunUrl": null}, "startedAt": "2026-10-06T21:47:50.891Z", "status": "passed", "suiteDefinitionHash": "0d49f11ce0ebed41e74866616a1e5f03b1e52ead86131f3bbd239cbd36a81dba"}
{"assistantUsage": {"cachedInputTokens": 0, "estimatedCostUsd": 0.622386, "inputTokens": 201292, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 1234, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 7}, "billing": {"assistant": {"cachedInputTokens": 0, "estimatedCostUsd": 0.622386, "inputTokens": 201292, "model": "claude-sonnet-4-6", "observedModels": ["claude-sonnet-4-6"], "outputTokens": 1234, "pricingAsOf": "2026-09-30", "pricingUrl": "https://platform.claude.com/docs/en/about-claude/pricing", "provider": "anthropic", "requests": 7}, "complete": true, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 240369, "costStatus": "reported", "inputTokens": 30269, "outputTokens": 1376, "reportedCostUsd": 0.20625195000000002, "runCount": 1, "runsWithReportedCost": 1, "runsWithTokenUsage": 1, "totalTokens": 272014}, "observedAndEstimatedCostUsd": 0.8286379500000001, "reportedCostUsd": 0.20625195000000002, "runtime": {"agentRunDurationMs": 28102, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T21-43-35-026Z", "caseId": "expanded-files", "durationMs": 62264, "evidence": "tests/runner-e2e/results/local-2026-10-06T21-43-35-026Z/public-mcp/assistant-claude-sonnet/local/expanded-files/attempt-1/result.json", "executionId": "public-mcp.assistant-claude-sonnet.local.expanded-files", "failedMatchers": [], "finishedAt": "2026-10-06T21:49:14.082Z", "kind": "case", "model": "claude-sonnet-4-6", "profileId": "assistant-claude-sonnet", "provider": "claude", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "b2196fae1a60077eb40dfe7f67e95f65f81ed8b2", "workflowRunUrl": null}, "startedAt": "2026-10-06T21:48:11.818Z", "status": "passed", "suiteDefinitionHash": "0d49f11ce0ebed41e74866616a1e5f03b1e52ead86131f3bbd239cbd36a81dba"}
{"assistantUsage": {"cachedInputTokens": 94208, "estimatedCostUsd": 0.0147381, "inputTokens": 5154, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 846, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 5}, "billing": {"assistant": {"cachedInputTokens": 94208, "estimatedCostUsd": 0.0147381, "inputTokens": 5154, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 846, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 5}, "complete": false, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 92032, "costStatus": "unpriced", "inputTokens": 134331, "outputTokens": 1795, "reportedCostUsd": 0, "runCount": 1, "runsWithReportedCost": 0, "runsWithTokenUsage": 1, "totalTokens": 228158}, "observedAndEstimatedCostUsd": 0.0147381, "reportedCostUsd": 0, "runtime": {"agentRunDurationMs": 15145, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T21-43-35-026Z", "caseId": "expanded-agent-config", "durationMs": 42016, "evidence": "tests/runner-e2e/results/local-2026-10-06T21-43-35-026Z/public-mcp/assistant-codex-mini/local/expanded-agent-config/attempt-1/result.json", "executionId": "public-mcp.assistant-codex-mini.local.expanded-agent-config", "failedMatchers": [], "finishedAt": "2026-10-06T21:45:58.775Z", "kind": "case", "model": "gpt-5.4-mini", "profileId": "assistant-codex-mini", "provider": "codex", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "b2196fae1a60077eb40dfe7f67e95f65f81ed8b2", "workflowRunUrl": null}, "startedAt": "2026-10-06T21:45:16.759Z", "status": "passed", "suiteDefinitionHash": "0d49f11ce0ebed41e74866616a1e5f03b1e52ead86131f3bbd239cbd36a81dba"}
{"assistantUsage": {"cachedInputTokens": 149504, "estimatedCostUsd": 0.03271155, "inputTokens": 24435, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 705, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 9}, "billing": {"assistant": {"cachedInputTokens": 149504, "estimatedCostUsd": 0.03271155, "inputTokens": 24435, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 705, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 9}, "complete": false, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 53888, "costStatus": "unpriced", "inputTokens": 75296, "outputTokens": 1262, "reportedCostUsd": 0, "runCount": 1, "runsWithReportedCost": 0, "runsWithTokenUsage": 1, "totalTokens": 130446}, "observedAndEstimatedCostUsd": 0.03271155, "reportedCostUsd": 0, "runtime": {"agentRunDurationMs": 12408, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T21-43-35-026Z", "caseId": "expanded-documents", "durationMs": 47621, "evidence": "tests/runner-e2e/results/local-2026-10-06T21-43-35-026Z/public-mcp/assistant-codex-mini/local/expanded-documents/attempt-1/result.json", "executionId": "public-mcp.assistant-codex-mini.local.expanded-documents", "failedMatchers": [], "finishedAt": "2026-10-06T21:44:46.304Z", "kind": "case", "model": "gpt-5.4-mini", "profileId": "assistant-codex-mini", "provider": "codex", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "b2196fae1a60077eb40dfe7f67e95f65f81ed8b2", "workflowRunUrl": null}, "startedAt": "2026-10-06T21:43:58.683Z", "status": "passed", "suiteDefinitionHash": "0d49f11ce0ebed41e74866616a1e5f03b1e52ead86131f3bbd239cbd36a81dba"}
{"assistantUsage": {"cachedInputTokens": 155648, "estimatedCostUsd": 0.0210846, "inputTokens": 7724, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 804, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 8}, "billing": {"assistant": {"cachedInputTokens": 155648, "estimatedCostUsd": 0.0210846, "inputTokens": 7724, "model": "gpt-5.4-mini", "observedModels": ["gpt-5.4-mini-2026-03-17"], "outputTokens": 804, "pricingAsOf": "2026-09-30", "pricingUrl": "https://developers.openai.com/api/docs/models/gpt-5.4-mini", "provider": "openai", "requests": 8}, "complete": false, "estimatedRuntimeCostUsd": 0, "llm": {"cachedInputTokens": 97664, "costStatus": "unpriced", "inputTokens": 141747, "outputTokens": 2364, "reportedCostUsd": 0, "runCount": 1, "runsWithReportedCost": 0, "runsWithTokenUsage": 1, "totalTokens": 241775}, "observedAndEstimatedCostUsd": 0.0210846, "reportedCostUsd": 0, "runtime": {"agentRunDurationMs": 21257, "costSource": "local_not_metered", "costStatus": "not_metered", "leaseCount": 0, "leaseDurationMs": null, "provider": "local"}}, "campaignId": "local-2026-10-06T21-43-35-026Z", "caseId": "expanded-files", "durationMs": 56158, "evidence": "tests/runner-e2e/results/local-2026-10-06T21-43-35-026Z/public-mcp/assistant-codex-mini/local/expanded-files/attempt-1/result.json", "executionId": "public-mcp.assistant-codex-mini.local.expanded-files", "failedMatchers": [], "finishedAt": "2026-10-06T21:44:54.786Z", "kind": "case", "model": "gpt-5.4-mini", "profileId": "assistant-codex-mini", "provider": "codex", "source": {"ref": "codex/expanded-assistant-mcp", "sha": "b2196fae1a60077eb40dfe7f67e95f65f81ed8b2", "workflowRunUrl": null}, "startedAt": "2026-10-06T21:43:58.628Z", "status": "passed", "suiteDefinitionHash": "0d49f11ce0ebed41e74866616a1e5f03b1e52ead86131f3bbd239cbd36a81dba"}
doc/plans/2026-10-06-expanded-assistant-mcp-tools.md
+35
View File
@@ -0,0 +1,35 @@
# Expand Paperclip’s assistant MCP tools
## Outcome
Outside assistants manage work and configuration as the connected person through the existing experimental MCP endpoint. Human attribution, company isolation, revocation, approval gates, native execution ownership and scheduling remain authoritative.
## Implementation plan
- Add task update, finish and block tools; document writes and revisions; attachment upload/download links; deliverable registration/update; agent configuration and instructions; project creation/update/repositories; skill metadata/files; and explicitly registered API search/call.
- Reuse domain routes and validation. Reject task transitions that bypass execution review. Documents and instruction/skill files use revision checks. Binary replacement creates a new attachment and updates the deliverable reference.
- Keep paperclip:configure for agent/project/skill configuration internally. Per the subsequent user revision, one write-access checkbox approves both requested mutation scopes (checked by default for eligible roles). Existing grants do not gain authority. Exclude credentials, arbitrary commands, permission policies, agent creation, public skill sharing, approval decisions and runner lifecycle operations.
- Extend internal dispatch to PATCH/PUT, empty responses and sanitized errors. Share retry identity between named tools and generic API calls; report uncertain outcomes. API registry is default-deny, company-bound, with explicit fields and scopes; no arbitrary destination, credentials or actor overrides.
- Keep the central directory broker restricted to its existing ten tools on both discovery and invocation; expanded tools and generic API access are for direct connections.
- paperclip_get_upload_url creates a ten-minute file-specific transfer. Sending bytes saves the attachment and returns its ID automatically; there is no completion call. Retries recover the same attachment, conflicting bytes fail. paperclip_get_download_url returns a ten-minute URL for one authorized attachment.
- Hash transfer credentials, exclude them from logs and persistent mutation results, recheck live authority, enforce file ownership/content constraints and existing deployment size limits (10 MB default), and safely clean abandoned storage. No arbitrary remote source URLs or server-local paths.
- Narrow Cloud tenant transfer routing preserves host validation, header stripping and standby protection. Ordinary REST endpoints do not accept MCP tokens.
## Verification and delivery
- Fresh core and Cloud worktrees. Preserve the existing experimental setting.
- Cover company/role/scope restrictions, configure consent, feature off, revocation/expiry, human audit, idempotency and named/generic parity.
- Cover dependency cycles, reassignment, finish/block, native contention and approval bypass rejection; document/instruction/skill revision conflicts.
- Binary hash round trips, automatic creation, lost responses, duplicate/conflicting/partial/oversized uploads, expired/revoked/cross-company links.
- Preserve runner/OAuth/broker/Cloud regressions. Extend existing public-mcp paid Product E2E workflows for each new capability, failures and later-conversation retrieval. Start GPT-5.4 Mini and Claude Haiku, then Sonnet; retain failures, costs, fingerprints and independent durable assertions.
- Verify real OpenCode, Codex and Claude Code separately; Storybook configuration consent; local test drive and Butter staging walkthrough.
- Full repository checks, token gates, documentation, evidence and green reviewed PRs. Merging is a separate action.
## Execution record
- Core: codex/expanded-assistant-mcp in the managed expanded-assistant-mcp worktree.
- Cloud: codex/expanded-assistant-mcp in /private/tmp/paperclip-cloud-expanded-mcp.
- Core PR: https://github.com/paperclipai/paperclip/pull/15380; Cloud PR: https://github.com/paperclipai/paperclip-cloud/pull/678. Neither is merged by this task.
- Implementation includes 37 direct tools, configuration consent, durable transfer tickets, Cloud routing and the ten-tool directory boundary.
- Paid qualification: all eight expanded cases passed on Mini, Haiku and Sonnet (24/24). Source and suite fingerprints, failures, costs and limitations are preserved in the [verification record](2026-10-06-expanded-assistant-mcp-verification.md).
- Actual local Codex, Claude Code and OpenCode clients completed reads and mutations; OpenCode's file round trip matched by SHA-256. Configuration consent has interactive Storybook coverage.
- Review fixes cover pending native retries, model-specific reasoning configuration and legacy instruction revision safety. Butter staging and final CI are separate delivery gates tracked in the PR verification sections.
## Consent revision
- User requested one checkbox: **Write all of your Paperclip data**. It controls requested work and configuration scopes together in browser and device consent; there is no separate configuration checkbox. Viewer restrictions and already-issued grant scopes remain unchanged.
doc/plans/2026-10-06-expanded-assistant-mcp-verification.md
+116
View File
@@ -0,0 +1,116 @@
# Expanded assistant MCP qualification — 2026-10-06
This record separates model qualification from client interoperability and deployment. Core PR [15380](https://github.com/paperclipai/paperclip/pull/15380) and Cloud PR [678](https://github.com/paperclipai/paperclip-cloud/pull/678) contain implementation and final delivery status. Merging requires a separate action.
## Paid Product E2E
All eight added `public-mcp` cases passed on each of three models, with automatic retries disabled. Each case uses a real runner worker, a separate assistant, and independent durable-state assertions. The grader requires the expected report, one successful worker execution and no extra runs. Task editing additionally requires newly attributed human activity in edit → block → finish order. File tests independently compare SHA-256 hashes and automatic attachment creation. Calibration tests reject fabricated completion, missing transitions, wrong actors/order and secret leakage.
| Model | Initial qualified result | Source | Campaign |
| --- | --- | --- | --- |
| `gpt-5.4-mini` | 8/8 | `9089870a4da4b03b12f7eab0afd134311b840a02` | `local-2026-10-06T20-28-55-414Z` |
| `claude-haiku-4-5-20251001` | 8/8 | same | same |
| `claude-sonnet-4-6` | 8/8 | `b9c4be5f0f8a3f1143f83edf3e8a047d02888eea` | `local-2026-10-06T20-44-06-234Z` |
Cases: `expanded-task-edit`, `expanded-documents`, `expanded-files`, `expanded-agent-config`, `expanded-projects`, `expanded-skills`, `expanded-api`, `expanded-permissions`. Those initial runs use suite fingerprint `d86dc8451519467b8680e92b7d6c3bee975a4bff419e6beaace2d8eeb5ce43f5` and grader v15. Sonnet ran after the legacy-instruction guard; Mini/Haiku ran before that narrow rejection was added. The latter guard has named-tool/API route regressions at the final runtime revision.
The final cheap-model campaign reports $1.5474926 estimated assistant cost plus $0.4521314 provider-reported worker cost: **$1.999624 known, plus unpriced Mini worker usage**. Sonnet reports $4.175256 estimated assistant cost plus $1.6985064 worker cost: **$5.8737624**. These are qualification campaign costs, not total development spend. Cached usage, provider-reported costs and estimates remain separate in the evidence.
The selected new cases do not cover the entire existing 63-case/model `public-mcp` suite; campaign `complete: false` reflects that coverage distinction, not failed selected cases. Both final campaigns executed every selected case and passed cleanup. `billing.complete` separately represents missing pricing.
### Earlier attempts retained
| Campaign time (UTC) | Passed/executed | Result and response |
| --- | --- | --- |
| 19:42:13 | 0/1 | Worker used the wrong report key. Diagnostic run did not record source SHA; do not treat it as qualified evidence. |
| 19:46:03 | 0/2 | Wrong report key and invalid UUID. Added useful sanitized validation detail. |
| 19:53:49 | 2/13 | Canceled incomplete campaign; shared-memory exhaustion, deliberate transfer URL redaction, stale UI marker and incomplete worker execution. Preserved completed results; fixed environment and evidence sanitization. |
| 20:04:15 | 12/16 | Missing explicit worker completion/readback and wrong report marker. Made worker instructions explicit, preserving durable success assertions. |
| 20:13:15 | 8/8 | Earlier Sonnet run passed. Reran with stronger activity-order grading and final instruction guard. |
[Machine-readable evidence](2026-10-06-expanded-assistant-mcp-eval-evidence.jsonl) retains every available case result, source and suite fingerprints, failed matcher details, usage and costs. Raw transcripts/results remain in the implementation worktree under `tests/runner-e2e/results/<campaign>/`; paths in the evidence point to those retained files. The checked-in projection excludes credentials and transfer URLs. Missing campaign-level billing from the canceled run is not invented.
## Actual clients, independent of model evals
The local disposable instance was created with `paperclipai test-drive`, then configured for authenticated sign-in. No worktree database clone was used.
- **Codex CLI 0.153.4:** native OAuth/CIMD; read organization and agents, updated EXP-1, wrote the report document. A first attempt with a ChatGPT login could not use Mini, so the authorized API provider was used. A noninteractive approval-disabled attempt correctly refused mutation; explicitly approved only the two fixture mutation tools for that invocation.
- **Claude Code 2.1.245:** Paperclip device OAuth and the credential-protected stdio bridge. A later conversation read Codex's report, performed a revision-checked update, and read it back. Revision 2 preserved the original body and appended its own verification line.
- **OpenCode 1.18.17:** native remote OAuth; read organization/agents, created and updated a project, uploaded a file, downloaded it, compared hashes, then registered the existing attachment as a deliverable and listed it alongside the report. No completion tool was called.
The 79-byte file round trip matched SHA-256 `fdec086f7051ebf67a596661e7567d8a53c471d6b86e63303da7734b8afc046a`. OpenCode's shell tool UI can display temporary transfer URLs in history; walkthrough captures use its safe summary instead. Transfer URLs are credentials even though short-lived.
Storybook's configuration consent story renders the production page and exercises opt-in configuration. Protocol/domain tests separately assert unchecked configuration defaults and that old grants never gain this scope through refresh.
## Regression and deployment checks
At runtime revision `b9c4be5f0f8a3f1143f83edf3e8a047d02888eea`, 75 focused MCP tests pass. The combined catalog/grader checks pass (56 tests); full workspace typecheck/build and token gates passed at the immediately preceding merge revision, with server typecheck repeated after the final guard. Latest-head CI and the full local test run are tracked in the PR rather than inferred from these narrower results.
Cloud revision `7b7ad76b302282c9fe8007185fa80ef37ddded6d` passes 2,488 tests, with 73 explicit skips and no failures. The opt-in published migration replay also passes separately. The replay verifies that the exact earlier MCP preview lineage can rejoin mainline without losing OAuth clients/grants/tokens, pending PKCE/device requests, tasks or upload receipts. It uses immutable published package archives and tests an idempotent second migration pass. All other migration histories remain subject to the existing strict checks.
Butter deployment and the hosted screenshot walkthrough are separate acceptance gates. Consult the PR verification section for their final outcome; local results above do not substitute for hosted proof. Browser directory installation, store approval and hosted MCP Events are outside this direct-connection expansion and are not claimed by these tests.
### Full local suite and mainline integration
The local full-suite attempt completed its general-server phase with 15,614 tests
passing and two timeouts (heartbeat comment wake batching and the existing
40,000-file Git streaming fixture); later phases did not run after that failure.
The heartbeat case passed in isolation. The Git fixture also exceeded its
five-minute macOS limit in isolation; its Linux CI lane passed. This is not a
claim of a completely passing local full-suite run. At `8ac64ae0a`, all 52 Core CI
checks succeeded (two optional checks skipped).
Mainline subsequently added migration 0310. The merge preserves that migration,
regenerates the combined schema snapshot, and moves the byte-identical, replay-safe
MCP transfer migration to 0311. Both private-commentary and transfer log redaction
are retained. Post-merge checks and qualification are recorded in the PR.
### Merged-source repeat and review regressions
The full 24-cell repeat on `e81f114bc6c813e7784cde0d8199538880aebe1f`
(`local-2026-10-06T21-17-35-704Z`, grader v15) passed 23 cells. Haiku's
agent-configuration cell timed out during application startup before writing a
model result. The failed attempt is retained. Known campaign cost was
$7.56693965, including $5.5999777 estimated assistant cost and $1.96696195
reported worker cost, plus unpriced worker usage.
The final review found nested pool queries during upload and OAuth transactions.
Authorization now uses the held transaction, including live settings/membership
checks and the domain issue-read decision. A real one-connection PostgreSQL test
exercises PKCE, refresh, device approval/redemption and six concurrent/repeated
uploads yielding exactly three attachments. The MCP/privacy regressions pass
139 tests. Later-conversation grading now requires a successful read of the
expected company/task report containing both references and rejects all writes;
the grader advances to v16. Its combined calibration suite passes 76 tests.
Affected paid cases and the startup failure are rerun separately at the final
source; do not reinterpret the earlier v15 runs as v16 evidence.
Local revocation was also exercised: revoking only the Paperclip CLI device
grant makes the same stdio bridge reject subsequent MCP initialization. The
Codex and OpenCode grants stay independent. A Claude diagnostic attempt that
hallucinated a shell script is retained and is not counted as proof; the
independent protocol rejection supplies that proof.
### Final affected-case qualification
At `b2196fae1a60077eb40dfe7f67e95f65f81ed8b2`, campaign
`local-2026-10-06T21-43-35-026Z` passes **9/9**: documents, binary files
and agent configuration on Mini, Haiku and Sonnet, with no automatic retries.
The prior Haiku startup failure is now followed by a passing independent attempt.
All selected cases and cleanup passed. Grader v16 uses suite fingerprint
`0d49f11ce0ebed41e74866616a1e5f03b1e52ead86131f3bbd239cbd36a81dba`.
Known cost is **$3.3112918** ($2.54030125 estimated assistant cost and
$0.77099055 reported worker cost), plus unpriced Mini worker usage.
The other five expanded cases passed across all three models on the preceding
merged source; this targeted repeat covers the changed transfer/OAuth path and
stronger later-document grading.
Butter's merged-source deployment was verified on `e81f114bc` by workflow
[37535338093](https://github.com/paperclipai/paperclip-cloud/actions/runs/37535338093).
The public metadata advertises configuration scope; invalid upload and download
tickets both return 403 through the tenant gateway. In a fresh OpenCode workspace,
the copied invitation fetched setup instructions, added the remote server, started
OAuth and produced a real Butter consent page with configuration unchecked.
The hosted grant awaits human approval under the browser tool's persistent-access
confirmation rule. Successful hosted transfers/delegation are not yet claimed.
The follow-up deployment of `b2196fae1` is tracked in the PR.
doc/public-mcp.md
+76 -10
View File
@@ -104,7 +104,7 @@ without a store listing. For Codex CLI:
```sh
codex mcp add paperclip --url https://YOUR-PAPERCLIP-HOST/mcp/paperclip
codex mcp login paperclip --scopes paperclip:read,paperclip:write,offline_access
codex mcp login paperclip --scopes paperclip:read,paperclip:write,paperclip:configure,offline_access
```
For Claude Code:
@@ -206,7 +206,10 @@ expire one minute after consent and are single-use. Access tokens expire in
fifteen minutes. `offline_access` issues a thirty-day rotating refresh token;
replaying a consumed refresh token revokes its entire grant. Tokens and codes
are hashed at rest. `paperclip:read` is required; `paperclip:write` adds only task
creation and comments. Scope expansion requires a new consent flow.
work and attachment mutations on direct connections. The `paperclip:configure`
scope covers allowed configuration operations. One **Write all of your Paperclip
data** consent checkbox approves both requested mutation scopes. Scope expansion
requires a new consent flow; the directory broker retains its original operations.
Each grant records the person, client, company, resource and scopes. Membership
and company availability are rechecked at execution. Instance admin status does
@@ -225,7 +228,7 @@ undo in-flight mutations. Manage existing tasks and execution in Paperclip.
Audit records identify the human caller and connection/client for mutations.
OAuth credential bodies and redirect locations are redacted from HTTP logs.
## Tool surface
## Original directory tool surface
| Tool | Effect |
| --- | --- |
@@ -236,13 +239,13 @@ OAuth credential bodies and redirect locations are redacted from HTTP logs.
| `paperclip_read_task` | Current task plus recent comments/history |
| `paperclip_create_task` | Assigned task, submitted to existing scheduling |
| `paperclip_add_comment` | Human feedback; may wake or queue work |
| `paperclip_list_deliverables` | Documents and work-product references |
| `paperclip_read_document` | Durable document body |
| `paperclip_list_deliverables` | Documents, attachments and work-product references |
| `paperclip_read_document` | Durable document body and current revision |
| `paperclip_pending_approvals` | Pending approvals and existing decision links |
Every company-scoped call requires its explicit authorized company ID. The
server emits bounded projections rather than agent configurations or upstream
credentials. URLs include the company's prefix so unrelated browser company
server emits bounded projections without upstream credentials. Direct connections
can read the allowed operating configuration through the expanded tools below. URLs include the company's prefix so unrelated browser company
selection cannot redirect the user to a different team's approval interface.
Writes require a UUID `requestId`, unique per intended action. A durable receipt
@@ -260,8 +263,8 @@ task is not proof that an agent started or finished it.
## Hosted onboarding and release gates
When `PAPERCLIP_CLOUD_API_ORIGIN` is present, consent links to the existing Cloud
`/orgs/new` flow in a separate tab. Provisioning, mission/template selection,
Create a hosted organization through Cloud before approving a connection.
Provisioning, mission/template selection,
model credentials, execution capacity and spending remain owned by Cloud.
Installing a plugin never provisions a company or starts paid agents.
@@ -312,7 +315,8 @@ legacy MCP. It advertises `events` through `server/discover` and implements
`events/list`, `events/subscribe`, and `events/unsubscribe`. MCP 2.0 requests
include matching `MCP-Protocol-Version`/`Mcp-Method` headers, per-request version
and client-capability metadata, and `Mcp-Name` for tool calls. Existing
initialize-based clients keep the ten-tool connection.
initialize-based direct clients receive the same expanded tool catalog. The central
directory broker keeps its original ten-tool connection.
| Event | Required filters | Payload |
| --- | --- | --- |
@@ -437,3 +441,65 @@ and unavailable-organization variants remain separately inspectable.
Apply the additive nullable request-column migration before running this tenant
version, and deploy tenant support before the Cloud broker that sends the binding.
Existing direct requests and grants are unchanged.
## Expanded direct assistant tools
Direct instance connections expose 37 explicitly registered operations. The central
public directory broker retains its original ten tools and rejects expanded calls.
The existing experimental setting gates both surfaces; sharing a setup link still
conveys no authority. Consent has one **Write all of your Paperclip data** checkbox,
checked by default for eligible roles. It approves the requested work and
configuration scopes together; unchecking it grants read-only access. Clients must
request `paperclip:configure` to configure agents, projects and skills. Existing
write connections do not acquire this scope, including after refresh; reconnect
with the expanded scopes to approve it.
Normal Paperclip permissions are required in addition to the connection scope.
- Work (`paperclip:write`): update, finish or block tasks; write Markdown documents;
register or update deliverables; upload attachments. Assignment and status changes
can schedule agents. Pending execution reviews must be decided in Paperclip.
- Configuration (`paperclip:configure`): edit existing agents, model/budget and
existing connection bindings; edit instructions; create/update projects and choose
accessible repositories; create/update organization skills and their files.
Credentials, execution commands, policies and agent creation are excluded.
- Reads (`paperclip:read`): task/document history, agent instructions, skill files,
accessible repository choices and attachment downloads. Native domain read
permissions still apply. Revision IDs from reads are required for content edits.
`paperclip_search_api` describes the same allowlisted named operations and their
schemas. `paperclip_call_api` takes an operation identifier and validated arguments.
It cannot accept an arbitrary URL, REST path, credential or actor override. Both
interfaces share scope checks and mutation receipts. Reuse the same request UUID
and arguments after a timeout. An unknown result means execution is uncertain;
inspect current state or retry that identity instead of starting another action.
To attach a video, compute its byte size and SHA-256, then call
`paperclip_get_upload_url` with its task, filename, content type and request UUID.
PUT the exact bytes using the returned Content-Type. Success returns the saved
attachment immediately; **there is no completion call**. Recover a lost response
by repeating the upload or requesting its URL with the same identity. Conflicting
bytes are rejected. `paperclip_list_deliverables` includes attachments, documents
and work products; `paperclip_get_download_url` downloads one authorized attachment.
Replacing a binary deliverable uses a new attachment and updates its reference.
File URLs are ten-minute, transfer-specific credentials: keep them out of chat and
logs. Only their hashes are stored. Use host HTTP/file tools to send/receive bytes;
if unavailable, upload/download through the Paperclip task page manually. The server
never fetches a supplied remote URL or reads a supplied local path. Existing file
type and size limits apply (10 MB default). Grants, membership, organization,
experimental availability and write authority are checked again during transfer.
Revocation prevents future use, including of previously issued file URLs. Expired
pending storage is cleaned on subsequent transfer requests; completed attachments
remain durable. Cloud deployments also need the narrowly scoped transfer proxy.
Instruction edits require a managed instruction file with a current revision/hash.
The legacy `promptTemplate.legacy.md` entry has no version protection, so MCP
rejects edits to it with `MCP_LEGACY_INSTRUCTIONS_UNVERSIONED`. Migrate the legacy
prompt to a managed instruction file in Paperclip first; native runner behavior
is unchanged.
Client-side tool approval remains separate from OAuth consent. For example, a
noninteractive Codex invocation that forbids tool approvals may read successfully
but refuse a write. Approve the intended tool in the host rather than weakening
server permissions. The [qualification record](plans/2026-10-06-expanded-assistant-mcp-verification.md)
distinguishes paid model tests, actual-client proof and staging verification.
packages/adapter-utils/src/acpx-engine/execute.test.ts
+118 -12
View File
@@ -195,6 +195,7 @@ async function runExecutor(
const meta: Record<string, unknown>[] = [];
const logs: Array<{ stream: string; text: string }> = [];
const events: Array<{ eventType: string; payload?: Record<string, unknown> }> = [];
const receipts: unknown[] = [];
const execute = createAcpxEngineExecutor({
...(options.prepareRemoteManagedHome
? { prepareRemoteManagedHome: options.prepareRemoteManagedHome }
@@ -233,10 +234,11 @@ async function runExecutor(
onEvent: async (event: { eventType: string; payload?: Record<string, unknown> }) => {
events.push(event);
},
onUsage: async (receipt: unknown) => { receipts.push(receipt); },
} as never);
expect(result.exitCode).toBe(0);
return { logs, meta, events, runtimeOptions, configOptions, sessionInputs, turnInputs, result };
return { logs, meta, events, receipts, runtimeOptions, configOptions, sessionInputs, turnInputs, result };
}
// Under `vi.useFakeTimers()`, setup before `ensureSession` still performs real
@@ -540,7 +542,7 @@ describe("shared ACPX engine runtime behavior", () => {
});
it("sets Codex model, effort, and fast mode through CODEX_CONFIG without session config calls", async () => {
const { configOptions, meta } = await runExecutor({
const { configOptions, meta, receipts, result } = await runExecutor({
agent: "codex",
model: "gpt-5.6-sol",
modelReasoningEffort: "high",
@@ -554,6 +556,8 @@ describe("shared ACPX engine runtime behavior", () => {
features: { fast_mode: true },
});
expect(configOptions).toEqual([]);
expect(result.pricingContext).toEqual({ serviceTier: "fast" });
expect(receipts).toContainEqual(expect.objectContaining({ pricingContext: { serviceTier: "fast" }, complete: true }));
expect(meta[0]?.commandNotes).toContain(
"Requested ACPX model: gpt-5.6-sol (set via CODEX_CONFIG at startup).",
);
@@ -1478,6 +1482,45 @@ describe("shared ACPX engine runtime behavior", () => {
expect(statusLine?.text).toContain('"cost"');
});
it.each(["checkpoint", "terminal"])("returns received usage after the %s receipt sink rejects and still closes the runtime", async (failedSave) => {
const root = await makeTempRoot();
const close = vi.fn(async () => {});
const cancel = vi.fn(async () => {});
let reads = 0;
const execute = createAcpxEngineExecutor({
resolveBillingIdentity: () => ({ provider: "anthropic", biller: "anthropic", billingType: "api" }),
createRuntime: () => ({
...buildRuntime(),
getStatus: async () => ++reads === 1
? { usage: { cost: { amount: 0.4, currency: "USD" } } }
: { usage: { cumulative: { inputTokens: 120, outputTokens: 4500, cachedReadTokens: 900, cachedWriteTokens: 30 }, cost: { amount: 1.15, currency: "USD" } } },
startTurn: () => ({
events: (async function* () {
yield { type: "status", text: "usage", tag: "usage_update",
breakdown: { inputTokens: 100, outputTokens: 4000, cachedReadTokens: 800, cachedWriteTokens: 20 }, cost: { amount: 1.1, currency: "USD" } };
})(),
result: Promise.resolve({ status: "completed", stopReason: "end_turn" }), cancel,
}), close,
}) as never,
});
const onUsage = vi.fn(async (receipt: { complete: boolean }) => {
if (failedSave === "checkpoint" || receipt.complete) throw new Error("Receipt storage unavailable");
});
const result = await execute({ runId: `failed-${failedSave}-save`, agent: { id: "agent-1", companyId: "company-1" },
runtime: {}, config: { agent: "custom", agentCommand: "node ./fake-acp.js", stateDir: path.join(root, "state") },
context: {}, onMeta: async () => {}, onLog: async () => {}, onUsage } as never);
expect(result).toMatchObject({ exitCode: 1, usageBasis: "per_run", usageComplete: failedSave === "terminal",
provider: "anthropic", biller: "anthropic", billingType: "api" });
expect(result.errorMessage).toContain("Receipt storage unavailable");
expect(result.usage).toEqual(failedSave === "terminal"
? { inputTokens: 150, outputTokens: 4500, cachedInputTokens: 900 }
: { inputTokens: 120, outputTokens: 4000, cachedInputTokens: 800 });
expect(result.costUsd).toBeCloseTo(failedSave === "terminal" ? 0.75 : 0.7);
expect(onUsage).toHaveBeenCalledTimes(failedSave === "terminal" ? 2 : 1);
expect(cancel).toHaveBeenCalledTimes(1);
expect(close).toHaveBeenCalledTimes(1);
});
it("falls back to usage_update events when the runtime lacks getStatus", async () => {
const root = await makeTempRoot();
const stateDir = path.join(root, "state");
@@ -1527,6 +1570,59 @@ describe("shared ACPX engine runtime behavior", () => {
expect(result.billingType).toBe("unknown");
});
it.each(["failed", "cancelled"] as const)("keeps %s in-stream usage provisional without a final receipt", async (status) => {
const root = await makeTempRoot();
const stateDir = path.join(root, "state");
const execute = createAcpxEngineExecutor({
createRuntime: () => ({
ensureSession: async () => ({
backendSessionId: "backend-session",
agentSessionId: "agent-session",
runtimeSessionName: "runtime-session",
}),
startTurn: () => ({
events: (async function* () {
yield {
type: "status",
text: "usage",
tag: "usage_update",
cost: { amount: 0.31, currency: "USD" },
breakdown: { inputTokens: 40, outputTokens: 700, cachedReadTokens: 60 },
};
})(),
result: Promise.resolve({ status, error: { code: "INTERNAL", message: "fixture failure" } }),
cancel: async () => {},
}),
close: async () => {},
}) as never,
});
const onUsage = vi.fn();
const result = await execute({
runId: "run-usage-event-fallback",
agent: {
id: "agent-1",
companyId: "company-1",
},
runtime: {},
config: { agent: "custom", agentCommand: "node ./fake-acp.js", stateDir },
context: {},
onLog: async () => {},
onMeta: async () => {},
onUsage,
} as never);
expect(result.exitCode).not.toBe(0);
expect(result.usageComplete).toBe(false);
expect(onUsage).toHaveBeenLastCalledWith(expect.objectContaining({ complete: false, costUsd: 0.31 }));
expect(result.usage).toEqual({ inputTokens: 40, outputTokens: 700, cachedInputTokens: 60 });
expect(result.usageBasis).toBe("per_run");
expect(result.costUsd).toBeCloseTo(0.31);
expect(result.provider).toBe("acpx");
expect(result.billingType).toBe("unknown");
});
it.skipIf(process.platform === "win32")("materializes ACPX Claude skills without symlinked descendants", async () => {
const root = await makeTempRoot();
const skillRoot = path.join(root, "skills");
@@ -2814,7 +2910,9 @@ describe("shared ACPX engine runtime behavior", () => {
it.each(["claude", "codex", "grok", "gemini", "kimi"])("resumes %s with added and removed MCP servers and current credentials", async (agent) => {
const root = await makeTempRoot();
const config = { agent, cwd: root, stateDir: path.join(root, "state"), paperclipRuntimeSkills: [], paperclipSkillSync: { desiredSkills: [] } };
// This fixture injects its runtime. Keep session identity independent of
// host CLI discovery/version probes while testing MCP credential changes.
const config = { agent, agentCommand: "node ./fake-acp.js", cwd: root, stateDir: path.join(root, "state"), paperclipRuntimeSkills: [], paperclipSkillSync: { desiredSkills: [] } };
const first = await runExecutor(config);
const server = { name: "github", url: "https://example.test/github/mcp", connectionId: "github", token: "current-token" };
const second = await runExecutor(config, {
@@ -6317,9 +6415,10 @@ describe("ACPX engine run lifecycle corrections (F1: settle every failure after
expect(stagingLocks.size).toBe(0);
});
it("test_prompt_build_failure_returns_error_result_with_phase_prepare_turn", async () => {
it.each(["prompt", "metadata", "dispatch"])("records whether provider work could have started on %s failure", async (failure) => {
const root = await makeTempRoot();
const close = vi.fn(async () => {});
const startTurn = vi.fn(() => { throw new Error("dispatch failed"); });
const execute = createAcpxEngineExecutor({
createRuntime: () =>
({
@@ -6328,11 +6427,7 @@ describe("ACPX engine run lifecycle corrections (F1: settle every failure after
agentSessionId: "agent-session",
runtimeSessionName: "runtime-session",
}),
startTurn: () => ({
events: (async function* () {})(),
result: Promise.resolve({ status: "completed", stopReason: "end_turn" }),
cancel: async () => {},
}),
startTurn,
close,
}) as never,
});
@@ -6343,7 +6438,8 @@ describe("ACPX engine run lifecycle corrections (F1: settle every failure after
Object.defineProperty(context, "paperclipSessionHandoffMarkdown", {
enumerable: false,
get() {
throw new Error("prompt build boom");
if (failure === "prompt") throw new Error("prompt build boom");
return undefined;
},
});
@@ -6354,11 +6450,19 @@ describe("ACPX engine run lifecycle corrections (F1: settle every failure after
config: { agent: "custom", agentCommand: "node ./fake-acp.js", stateDir: path.join(root, "state") },
context,
onLog: async () => {},
onMeta: async () => {},
onMeta: async () => { if (failure === "metadata") throw new Error("metadata failed"); },
} as never);
expect(result.exitCode).toBe(1);
expect(result.resultJson?.phase).toBe("prepare_turn");
if (failure === "dispatch") {
expect(startTurn).toHaveBeenCalledTimes(1);
expect(result.executionRecovery).toBeUndefined();
expect(result.usageComplete).toBe(false);
} else {
expect(result.resultJson?.phase).toBe("prepare_turn");
expect(startTurn).not.toHaveBeenCalled();
expect(result.executionRecovery).toEqual({ kind: "bootstrap", providerWorkStarted: false });
}
// The established runtime is closed on the pre-turn failure, so no session
// leaks.
expect(close).toHaveBeenCalledTimes(1);
@@ -7316,6 +7420,7 @@ describe("ACPX engine sandbox bridge run-disposition seam (fail-closed)", () =>
// The lost channel overrides the nominally completed terminal to a failure.
expect(result.exitCode).not.toBe(0);
expect(result.errorCode).toBe("duplex_channel_lost");
expect(result.usageComplete).toBe(false);
// The message carries only the typed loss reason, not raw provider text.
expect(result.errorMessage).toContain("provider_exit");
expect(result.resultJson).toMatchObject({ status: "failed" });
@@ -7335,6 +7440,7 @@ describe("ACPX engine sandbox bridge run-disposition seam (fail-closed)", () =>
expect(result.exitCode).toBe(0);
expect(result.errorCode ?? null).toBeNull();
expect(result.usageComplete).toBe(true);
// The atomic settle step marked the orderly completion for the
// success-eligible terminal.
expect(fake.settleRunDisposition).toHaveBeenCalledTimes(1);
packages/adapter-utils/src/acpx-engine/execute.ts
+22
View File
@@ -3942,6 +3942,7 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) {
provider: billingIdentity?.provider ?? "acpx",
...(billingIdentity?.biller ? { biller: billingIdentity.biller } : {}),
billingType: billingIdentity?.billingType ?? ("unknown" as const),
pricingContext: undefined as AdapterExecutionResult["pricingContext"],
};
const warmIdleMs = asNumber(ctx.config.warmHandleIdleMs, DEFAULT_ACP_ENGINE_WARM_HANDLE_IDLE_MS);
// The host run site owns the warm-handle store on this run. It operates over
@@ -4246,6 +4247,7 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) {
}),
);
buildRuntimeSettled = true;
billingFields.pricingContext = prepared.fastMode ? { serviceTier: "fast" } : undefined;
// Capture acquired resources before the cancellation boundary so the
// normal settlement path also releases a just-completed build.
releaseStagingLease = prepared.sessionStagingLeaseRelease;
@@ -4751,8 +4753,12 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) {
};
let eventBreakdown: AcpRuntimeUsageBreakdown | null = null;
let eventCostUsd: number | null = null;
// Receipt persistence is fallible; retain the provider evidence before
// calling the sink so the failure result can still return it to the host.
let observedUsage: Pick<AdapterExecutionResult, "usage" | "usageBasis" | "costUsd" | "usageComplete"> = { usageComplete: false };
let lastRuntimeEventAt: number | null = null;
let observedRuntimeEvents = 0;
let turnDispatchAttempted = false;
// The turn-local state the sequence steps share. `promptBuild` sets the
// prompt, `preTurnUsage` sets the pre-turn status, `turnStart` sets the
// active turn, and `turnFinalize` reads all three. `activeTurn` is the run-
@@ -4837,6 +4843,9 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) {
};
const stepTurnStart = (signal: AbortSignal, startTimeoutMs: number | undefined): StartedTurn => {
ctx.signal?.throwIfAborted();
// After entering startTurn, a thrown error cannot prove the provider
// received no work. Only failures before this boundary are free to settle.
turnDispatchAttempted = true;
const turn = runtime.startTurn({
handle: sessionHandle,
text: runPrompt,
@@ -4953,6 +4962,10 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) {
if (event.type === "status" && event.tag === "usage_update") {
eventBreakdown = event.breakdown ?? eventBreakdown;
eventCostUsd = usdCostAmount(event.cost) ?? eventCostUsd;
const checkpoint = summarizeAcpxTurnUsage({ preStatus: preTurnStatus, postStatus: null, eventBreakdown, eventCostUsd });
observedUsage = { usage: checkpoint.usage ?? undefined, costUsd: checkpoint.costUsd, usageBasis: "per_run", usageComplete: false };
await ctx.onUsage?.({ ...billingFields, usage: checkpoint.usage ?? undefined, costUsd: checkpoint.costUsd,
model: prepared.requestedModel, usageBasis: "per_run", complete: false });
}
await emitRuntimeEvent(ctx, event, toolTitles, prepared.coalescePlaceholderToolUpdates);
}
@@ -5039,6 +5052,12 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) {
eventBreakdown,
eventCostUsd,
});
// An interrupted stream is provisional unless the runtime supplies its
// post-terminal receipt. An in-stream usage_update alone cannot close it.
const usageComplete = !channelLost && (turnSucceeded || postTurnStatus?.usage != null);
observedUsage = { usage: turnUsage.usage ?? undefined, costUsd: turnUsage.costUsd, usageBasis: "per_run", usageComplete };
await ctx.onUsage?.({ ...billingFields, usage: turnUsage.usage ?? undefined, costUsd: turnUsage.costUsd,
model: prepared.requestedModel, usageBasis: "per_run", complete: usageComplete });
const failedTurn = terminal.status === "failed" || terminal.status === "cancelled" || timedOut;
// ACPX can defer session/load until runTurn. Forget an unavailable
// session so the next bounded turn receives the full task conversation.
@@ -5124,6 +5143,7 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) {
model: prepared.requestedModel || null,
...(turnUsage.usage ? { usage: turnUsage.usage, usageBasis: "per_run" as const } : {}),
costUsd: turnUsage.costUsd,
usageComplete,
resultJson: {
status: channelLost ? "failed" : terminal.status,
...activityDiagnostics,
@@ -5257,6 +5277,8 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) {
errorMessage: message,
errorCode: timedOut ? "acpx_timeout" : (emitted?.classified.errorCode ?? null),
errorMeta: emitted?.classified.errorMeta,
...observedUsage,
...(!turnDispatchAttempted ? { executionRecovery: { kind: "bootstrap" as const, providerWorkStarted: false as const } } : {}),
...billingFields,
...referencedProjectStagingFailuresField,
model: prepared.requestedModel || null,
packages/adapter-utils/src/billing.test.ts
+20 -1
View File
@@ -1,5 +1,5 @@
import { describe, expect, it } from "vitest";
import { inferOpenAiCompatibleBiller } from "./billing.js";
import { inferOpenAiCompatibleBiller, resolveManagedOpenAiBilling } from "./billing.js";
describe("inferOpenAiCompatibleBiller", () => {
it("returns openrouter when OPENROUTER_API_KEY is present", () => {
@@ -25,4 +25,23 @@ describe("inferOpenAiCompatibleBiller", () => {
),
).toBe("openai");
});
it.each(["https://proxy.example/v1", "https://openrouter.ai.example/v1", "https://example.org/openrouter.ai", "http://api.openai.com/v1", "invalid"])("does not apply direct provider prices to %s", (baseUrl) => {
expect(inferOpenAiCompatibleBiller({ OPENAI_BASE_URL: baseUrl }, "openai")).toBe("unknown");
});
it("recognizes each supported base URL override", () => {
expect(inferOpenAiCompatibleBiller({ OPENAI_API_BASE: "https://proxy.example" })).toBe("unknown");
expect(inferOpenAiCompatibleBiller({ OPENAI_API_BASE_URL: "https://proxy.example" })).toBe("unknown");
expect(inferOpenAiCompatibleBiller({}, null)).toBeNull();
});
});
describe("managed Responses routing", () => {
it.each([["openrouter", "api_key", "openrouter"], ["custom", "api_key", "unknown"], ["local", "none", "unknown"]])("keeps %s/%s usage outside direct OpenAI pricing", (kind, auth, biller) => {
expect(resolveManagedOpenAiBilling({ kind, auth })).toEqual({ provider: "openai", biller, billingType: "api" });
});
it("leaves ordinary direct API and subscription detection unchanged", () => {
expect(resolveManagedOpenAiBilling(undefined)).toBeUndefined();
});
});
packages/adapter-utils/src/billing.ts
+22 -1
View File
@@ -14,7 +14,28 @@ export function inferOpenAiCompatibleBiller(
readEnv(env, "OPENAI_BASE_URL") ??
readEnv(env, "OPENAI_API_BASE") ??
readEnv(env, "OPENAI_API_BASE_URL");
if (baseUrl && /openrouter\.ai/i.test(baseUrl)) return "openrouter";
if (baseUrl) {
try {
const url = new URL(baseUrl);
if (url.protocol === "https:" && url.hostname === "openrouter.ai") return "openrouter";
// An OpenAI-compatible endpoint does not imply OpenAI prices. Do not
// assign direct-provider estimates to a proxy or an unknown endpoint.
if (url.protocol !== "https:" || url.hostname !== "api.openai.com") return "unknown";
} catch {
return "unknown";
}
}
return fallback;
}
/** Managed Responses routes use their own credentials and prices, including
* no-auth endpoints. Absence of OPENAI_API_KEY never makes them ChatGPT plans. */
export function resolveManagedOpenAiBilling(routing: unknown) {
if (!routing || typeof routing !== "object") return undefined;
return {
provider: "openai",
biller: (routing as { kind?: unknown }).kind === "openrouter" ? "openrouter" : "unknown",
billingType: "api" as const,
};
}
packages/adapter-utils/src/index.ts
+2 -1
View File
@@ -3,6 +3,7 @@ export type {
AgentRuntimeIdentity,
AdapterRuntime,
UsageSummary,
AdapterUsageCheckpoint,
AdapterBillingType,
AdapterRuntimeServiceReport,
AdapterExecutionResult,
@@ -88,7 +89,7 @@ export type {
RuntimeStatusSink,
RuntimeStatusUpdate,
} from "./runtime-progress.js";
export { inferOpenAiCompatibleBiller } from "./billing.js";
export { inferOpenAiCompatibleBiller, resolveManagedOpenAiBilling } from "./billing.js";
export {
ADAPTER_LOGIN_PANEL_MODES,
ADAPTER_LOGIN_TIMEOUT_POLICIES,
packages/adapter-utils/src/types.ts
+34 -1
View File
@@ -31,9 +31,32 @@ export interface AdapterRuntime {
// ---------------------------------------------------------------------------
export interface UsageSummary {
/** Uncached input plus cache writes; excludes cache reads. Total tokens are input + cachedInput + output. */
inputTokens: number;
outputTokens: number;
cachedInputTokens?: number;
/** Subset of inputTokens, not an additional token count. */
cacheWriteTokens?: number;
}
/** Accounting-only snapshot. Never include prompt, response, or credentials. */
export interface AdapterUsageCheckpoint {
attemptId?: string;
usage?: UsageSummary;
usageByModel?: Array<{ model: string; usage: UsageSummary; costUsd: number }>;
usageBasis?: "per_run" | "session_cumulative" | null;
provider?: string | null;
biller?: string | null;
model?: string | null;
billingType?: AdapterBillingType | null;
costUsd?: number | null;
costUsdExact?: string | null;
costStatus?: "reported" | "estimated" | "unpriced";
pricingContext?: { serviceTier?: string; contextTier?: "short" | "long" };
pricingProvenance?: { source: "provider_reported" | "provider_invoice" | "operator" | "rate_card" | "unknown"; version?: string; evidence?: string; inputCentsPerMillion?: string; cachedInputCentsPerMillion?: string; cacheWriteCentsPerMillion?: string; outputCentsPerMillion?: string; serviceTier?: string; contextTier?: "short" | "long" };
cacheAdjustedCostUsd?: number | null;
providerRequestId?: string | null;
complete: boolean;
}
export type AdapterBillingType =
@@ -93,11 +116,14 @@ export interface AdapterExecutionResult {
retryNotBefore?: string | null;
errorMeta?: Record<string, unknown>;
usage?: UsageSummary;
/** Complete per-model receipts, when supplied by the runtime. Their sums must match the run totals. */
usageByModel?: Array<{ model: string; usage: UsageSummary; costUsd: number }>;
/**
* How `usage` totals are scoped. "per_run" means the tokens cover only this
* execution; "session_cumulative" means they are running totals for the
* persisted session, and the server must delta consecutive runs. Absent
* means unknown — the server applies its legacy session-delta heuristic.
* defaults to per-run; the server must never infer cumulative usage from a
* reused session ID.
*/
usageBasis?: "per_run" | "session_cumulative" | null;
/**
@@ -111,6 +137,12 @@ export interface AdapterExecutionResult {
model?: string | null;
billingType?: AdapterBillingType | null;
costUsd?: number | null;
costUsdExact?: string | null;
costStatus?: "reported" | "estimated" | "unpriced";
pricingContext?: { serviceTier?: string; contextTier?: "short" | "long" };
pricingProvenance?: { source: "provider_reported" | "provider_invoice" | "operator" | "rate_card" | "unknown"; version?: string; evidence?: string; inputCentsPerMillion?: string; cachedInputCentsPerMillion?: string; cacheWriteCentsPerMillion?: string; outputCentsPerMillion?: string; serviceTier?: string; contextTier?: "short" | "long" };
providerRequestId?: string | null;
usageComplete?: boolean;
/**
* Provider-billed cost after prompt-cache discounts. Adapters should set
* this when they expose it separately; otherwise the server treats a
@@ -240,6 +272,7 @@ export interface AdapterExecutionContext {
onLog: (stream: "stdout" | "stderr", chunk: string) => Promise<void>;
onMeta?: (meta: AdapterInvocationMeta) => Promise<void>;
onEvent?: (event: AdapterRuntimeEvent) => Promise<void>;
onUsage?: (receipt: AdapterUsageCheckpoint) => Promise<void>;
onRuntimeProgress?: RuntimeStatusSink;
/**
* Reports that execution has crossed the adapter's dispatch boundary.
packages/adapter-utils/src/usage-checkpoint.test.ts
+132
View File
@@ -0,0 +1,132 @@
import { describe, expect, it, vi } from "vitest";
import { createUsageCheckpointLog } from "./usage-checkpoint.js";
import { runChildProcess } from "./server-utils.js";
import type { AdapterUsageCheckpoint } from "./types.js";
const createParser = () => {
let costUsd = 0;
return (stdout: string): AdapterUsageCheckpoint => {
const events = stdout.trim().split("\n").map(line => JSON.parse(line));
costUsd += events.reduce((sum, event) => sum + event.usage.costUsd, 0);
return { costUsd, complete: events.some(event => event.type === "result") };
};
};
const event = JSON.stringify({ type: "usage", usage: { inputTokens: 2, costUsd: 0.25 }, text: "private content" });
describe("usage checkpoint stream", () => {
it("redacts malformed counter text without making the counter absent", async () => {
const consume = vi.fn((_records: string) => null);
const log = createUsageCheckpointLog(vi.fn(), vi.fn(), consume);
await log("stdout", JSON.stringify({ type: "result", usage: {
input_tokens: 2, output_tokens: 1, cached_input_tokens: "private malformed counter",
} }) + "\n");
const retained = consume.mock.calls[0][0];
expect(retained).not.toContain("private malformed counter");
const usage = JSON.parse(retained).usage;
expect(Object.hasOwn(usage, "cached_input_tokens")).toBe(true);
expect(typeof usage.cached_input_tokens).not.toBe("number");
expect(usage.cached_input_tokens).not.toBeNull();
});
it.each([false, true])("persists reported zero usage on confirmed completion (newline=%s)", async newline => {
const saved = vi.fn();
const receipt = { usage: { inputTokens: 0, outputTokens: 0, cachedInputTokens: 0 }, costUsd: null, complete: false };
const log = createUsageCheckpointLog(vi.fn(), saved, () => receipt);
await log("stdout", JSON.stringify({ type: "step_finish", part: { tokens: { input: 0, output: 0 } } }) + (newline ? "\n" : ""));
expect(saved).not.toHaveBeenCalled();
await log.flush({ complete: true });
expect(saved).toHaveBeenCalledExactlyOnceWith(expect.objectContaining({ ...receipt, complete: true }));
await log.flush({ complete: true });
expect(saved).toHaveBeenCalledTimes(1);
});
it("propagates a zero-receipt persistence failure before cleanup and never invents absent usage", async () => {
const failure = new Error("Cannot persist zero receipt");
const saved = vi.fn().mockRejectedValue(failure);
const log = createUsageCheckpointLog(vi.fn(), saved, () => ({ usage: { inputTokens: 0, outputTokens: 0 }, costUsd: null, complete: false }));
await log.flush({ complete: true });
expect(saved).not.toHaveBeenCalled();
await log("stdout", '{"type":"step_finish","part":{"tokens":{"input":0,"output":0}}}');
await expect(log.flush({ complete: true })).rejects.toBe(failure);
});
it("flushes an unterminated Pi price and closes each attempt independently", async () => {
const saved = vi.fn(); const parser = vi.fn(createParser());
const first = createUsageCheckpointLog(vi.fn(), saved, parser);
await first("stdout", event);
expect(saved).not.toHaveBeenCalled();
await first.flush({ complete: true });
const second = createUsageCheckpointLog(vi.fn(), saved, createParser());
await second("stdout", event + "\n");
await second.flush({ complete: true });
expect(saved.mock.calls[0][0]).toMatchObject({ costUsd: 0.25, complete: true });
expect(saved.mock.calls.at(-1)![0]).toMatchObject({ costUsd: 0.25, complete: true });
expect(saved.mock.calls.at(-1)![0].attemptId).not.toBe(saved.mock.calls[0][0].attemptId);
expect(parser.mock.calls[0][0]).not.toContain("private content");
});
it("bounds large content strings before buffering and preserves usage on either side", async () => {
const saved = vi.fn(); const parser = vi.fn(createParser()); const output = vi.fn();
const log = createUsageCheckpointLog(output, saved, parser);
const line = JSON.stringify({ type: "result", usage: { inputTokens: 2, costUsd: 0.25 }, messages: [{ content: '"\\\n'.repeat(2 * 1024 * 1024) }], model: "after-large-content" });
for (let offset = 0; offset < line.length; offset += 4093) await log("stdout", line.slice(offset, offset + 4093));
await log.flush();
expect(saved).toHaveBeenLastCalledWith(expect.objectContaining({ costUsd: 0.25, complete: true }));
expect(parser.mock.calls[0][0]).toContain("after-large-content");
expect(parser.mock.calls[0][0].length).toBeLessThan(1000);
expect(output.mock.calls.map(call => call[1]).join("")).toBe(line);
});
it("preserves Unicode and escaped quotes across the string limit and chunk boundaries", async () => {
for (const suffix of ['\\u1234', '\\"', '\\\\']) {
const saved = vi.fn(); const parser = vi.fn(createParser());
const log = createUsageCheckpointLog(vi.fn(), saved, parser);
const line = '{"type":"result","text":"' + 'a'.repeat(249) + suffix + 'discarded","usage":{"costUsd":0.25}}\n';
for (const character of line) await log("stdout", character);
await log.flush();
expect(saved).toHaveBeenCalledWith(expect.objectContaining({ costUsd: 0.25 }));
}
});
it("surfaces a persistence failure outside the local process's best-effort log path", async () => {
const failure = new Error("Receipt storage unavailable");
const output = vi.fn(); const saved = vi.fn().mockRejectedValue(failure);
const log = createUsageCheckpointLog(output, saved, createParser());
const result = await runChildProcess("checkpoint-test", process.execPath, ["-e", `process.stdout.write(${JSON.stringify(event + "\n")})`], {
cwd: process.cwd(), env: {}, timeoutSec: 5, graceSec: 1, onLog: log,
});
expect(result.exitCode).toBe(0);
expect(output).toHaveBeenCalledWith("stdout", event + "\n");
await expect(log.flush()).rejects.toBe(failure);
});
it("does not promote partial usage to complete merely because the stream was flushed", async () => {
const saved = vi.fn(); const log = createUsageCheckpointLog(vi.fn(), saved, createParser());
await log("stdout", event); await log.flush();
expect(saved.mock.calls.at(-1)![0]).toMatchObject({ costUsd: 0.25, complete: false });
});
it("persists each changed total and the final receipt before publishing its chunk", async () => {
const order: string[] = [];
const saved = vi.fn(async (receipt: AdapterUsageCheckpoint) => { order.push(`saved:${receipt.costUsd}:${receipt.complete}`); });
const log = createUsageCheckpointLog(async () => { order.push("log"); }, saved, createParser());
await log("stdout", event + "\n");
await log("stdout", event + "\n");
await log("stdout", JSON.stringify({ type: "result", usage: { costUsd: 0 } }) + "\n");
expect(order).toEqual(["saved:0.25:false", "log", "saved:0.5:false", "log", "saved:0.5:true", "log"]);
});
it("saves every new receipt before logging with linear parsing work", async () => {
const parser = vi.fn(createParser()); const saved = vi.fn();
const log = createUsageCheckpointLog(vi.fn(), saved, parser);
for (let i = 0; i < 1024; i++) {
await log("stdout", event + "\n");
expect(saved.mock.calls.at(-1)![0].costUsd).toBe((i + 1) * 0.25);
}
await log.flush({ complete: true });
expect(saved.mock.calls.at(-1)![0]).toMatchObject({ costUsd: 256, complete: true });
expect(saved).toHaveBeenCalledTimes(1025); // all updates plus completion
expect(parser).toHaveBeenCalledTimes(1024);
expect(parser.mock.calls.reduce((sum, [input]) => sum + input.length, 0)).toBeLessThanOrEqual(1024 * (event.length + 1));
});
});
packages/adapter-utils/src/usage-checkpoint.ts
+126
View File
@@ -0,0 +1,126 @@
import type { AdapterUsageCheckpoint } from "./types.js";
/** Project accounting protocol fields before retaining a stream snapshot.
* Content-bearing strings are never retained in the accounting buffer. */
const fields = new Set([
"type", "subtype", "role", "id", "session_id", "sessionID", "thread_id", "model", "modelID", "providerID",
"usage", "usageMetadata", "modelUsage", "message", "part", "info", "stats", "tokens", "cost", "total",
"input", "output", "cache", "read", "write", "input_tokens", "output_tokens", "cached_input_tokens",
"cache_read_input_tokens", "cache_creation_input_tokens", "inputTokens", "outputTokens", "cacheReadInputTokens",
"cacheCreationInputTokens", "cachedInputTokens", "promptTokenCount", "candidatesTokenCount", "cachedContentTokenCount",
"reasoning", "cacheRead", "cacheWrite", "prompt", "candidates", "totalTokenCount", "total_tokens", "toolUsePromptTokenCount", "messages",
"thoughtsTokenCount", "inputTokens", "totalTokens", "cached", "thoughts", "total_cost_usd", "cost_usd", "costUSD", "costUsd",
]);
const stringFields = new Set(["type", "subtype", "role", "id", "session_id", "sessionID", "thread_id", "model", "modelID", "providerID"]);
function project(value: unknown, depth = 0, field = ""): unknown {
if (depth > 8) return undefined;
if (typeof value === "number" || typeof value === "boolean" || value === null) return value;
// Retain an invalid-type marker without retaining its contents. Omitting a
// malformed optional counter would let parsers default it to a valid zero.
if (typeof value === "string") return stringFields.has(field) ? value.slice(0, 250) : false;
if (Array.isArray(value)) return value.slice(0, 500).map(item => project(item, depth + 1));
if (!value || typeof value !== "object") return undefined;
return Object.fromEntries(Object.entries(value).filter(([key]) => fields.has(key))
.map(([key, entry]) => [key, key === "modelUsage" && entry && typeof entry === "object"
? Object.fromEntries(Object.entries(entry).map(([model, usage]) => [model.slice(0, 250), project(usage, depth + 1)]))
: project(entry, depth + 1, key)]));
}
/** Bound strings before buffering whole JSON lines. Provider records can carry
* many MiB of conversation text alongside a small usage object. Preserve JSON
* escapes across chunks so discarded text cannot corrupt the accounting fields. */
function createStringCompactor() {
let inString = false, length = 0, escapeRemaining = 0;
let escapeCode = false, keepEscape = false;
return (chunk: string) => {
let output = "";
for (const character of chunk) {
if (character === "\n") {
inString = false; escapeRemaining = 0; escapeCode = false;
output += character;
} else if (!inString) {
output += character;
if (character === '"') { inString = true; length = 0; }
} else if (escapeRemaining > 0) {
if (keepEscape) output += character;
escapeRemaining = escapeCode && character === "u" ? 4 : escapeRemaining - 1;
escapeCode = false;
length++;
} else if (character === '"') {
output += character; inString = false;
} else if (character === "\\") {
keepEscape = length < 250;
if (keepEscape) output += character;
escapeRemaining = 1; escapeCode = true; length++;
} else {
if (length < 250) output += character;
length++;
}
}
return output;
};
}
/** The consumer incrementally parses only new records (never the full history).
* A fresh instance belongs to one CLI attempt. Call flush() after its process
* exits, before retrying or final result handling. Checkpoint failures are
* retained outside runChildProcess's best-effort onLog error handler. */
export function createUsageCheckpointLog(
onLog: (stream: "stdout" | "stderr", chunk: string) => Promise<void>,
onUsage: ((receipt: AdapterUsageCheckpoint) => Promise<void>) | undefined,
consume: (records: string) => AdapterUsageCheckpoint | null,
) {
const attemptId = randomUUID();
const compactStrings = createStringCompactor();
let remainder = "", accounting = "", previous = "";
let snapshot: AdapterUsageCheckpoint | null = null;
let failure: unknown;
let failed = false;
function retain(line: string) {
let raw: unknown;
try { raw = JSON.parse(line); } catch { return; }
const compact = JSON.stringify(project(raw));
if (!compact || !/usage|tokens|cost|"result"|"turn.started"|"turn.completed"|"turn.failed"|"error"|"agent_end"|"step_finish"|"model"|"modelID"/.test(compact)) return;
accounting += compact + "\n";
if (accounting.length > 8 * 1024 * 1024) throw new Error("Accounting checkpoint chunk exceeds 8 MiB");
}
async function publish(complete = false) {
if (!onUsage) return;
if (accounting) {
snapshot = consume(accounting);
accounting = "";
}
const parsed = snapshot;
if (!parsed) return;
const receipt = { ...parsed, complete: parsed.complete || complete };
if (!receipt.complete && receipt.costStatus !== "unpriced" && receipt.costUsd == null && receipt.costUsdExact == null &&
!Object.values(receipt.usage ?? {}).some(value => typeof value === "number" && value > 0)) return;
const serialized = JSON.stringify(receipt);
if (serialized !== previous) { await onUsage({ ...receipt, attemptId }); previous = serialized; }
}
const log = async (stream: "stdout" | "stderr", chunk: string) => {
if (onUsage && !failed && stream === "stdout") {
try {
remainder += compactStrings(chunk);
const lines = remainder.split("\n"); remainder = lines.pop() ?? "";
if (remainder.length > 8 * 1024 * 1024) throw new Error("Accounting protocol line exceeds 8 MiB");
for (const line of lines) retain(line);
await publish();
} catch (error) {
failed = true; failure = error; remainder = ""; accounting = "";
}
}
// Receipt persistence and ordinary logs have separate failure contracts.
// A failed receipt must not hide the provider's diagnostic output.
await onLog(stream, chunk);
};
return Object.assign(log, {
async flush(options: { complete?: boolean } = {}) {
if (failed) throw failure;
if (!onUsage) return;
if (remainder) { retain(remainder); remainder = ""; }
await publish(options.complete ?? false);
},
});
}
import { randomUUID } from "node:crypto";
packages/adapters/claude-local/src/server/execute.ts
+34 -3
View File
@@ -1,4 +1,5 @@
import { createProviderStoppedBoundary } from "@paperclipai/adapter-utils/provider-stopped-boundary";
import { createUsageCheckpointLog } from "@paperclipai/adapter-utils/usage-checkpoint";
import fs from "node:fs/promises";
import path from "node:path";
import { fileURLToPath } from "node:url";
@@ -63,7 +64,8 @@ import {
} from "@paperclipai/adapter-utils/local-process-sandbox";
import {
claudeModelUsageTotals,
parseClaudeStreamJson,
claudeModelReceipts,
parseClaudeStreamJson, createClaudeStreamParser,
describeClaudeFailure,
detectClaudeLoginRequired,
extractClaudeRetryNotBefore,
@@ -964,6 +966,14 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
});
}
const consumeAccounting = createClaudeStreamParser();
const accountingLog = createUsageCheckpointLog(onLog, ctx.onUsage, stdout => {
const parsed = consumeAccounting(stdout);
return { usage: parsed.usage ?? undefined, usageBasis: "per_run", costUsd: parsed.costUsd,
provider: "anthropic", biller: isBedrockAuth(effectiveEnv) ? "aws_bedrock" : "anthropic",
billingType, model: Object.keys(parseObject(parsed.resultJson?.modelUsage)).length > 1 ? "mixed" : parsed.model || model,
usageByModel: claudeModelReceipts(parsed.resultJson?.modelUsage), complete: parsed.resultJson !== null };
});
const proc = await runAdapterExecutionTargetProcess(runId, runtimeExecutionTarget, command, args, {
onProcessStopped: providerStop.beginInvocation(),
cwd,
@@ -973,7 +983,7 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
graceSec,
onSpawn,
onRuntimeProgress: ctx.onRuntimeProgress,
onLog,
onLog: accountingLog,
runLogTail: paperclipBridge?.runLogTail,
settleRunDisposition: paperclipBridge?.settleRunDisposition,
terminalResultCleanup: {
@@ -982,6 +992,7 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
},
localProcessSandbox,
});
await accountingLog.flush();
const parsedStream = parseClaudeStreamJson(proc.stdout);
const parsed = parsedStream.resultJson ?? parseJson(proc.stdout);
@@ -1014,6 +1025,15 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
exitCode: proc.exitCode,
signal: proc.signal,
timedOut: true,
usageComplete: parsedStream.resultJson !== null,
usage: parsedStream.usage ?? undefined,
usageBasis: "per_run",
provider: "anthropic",
biller: isBedrockAuth(effectiveEnv) ? "aws_bedrock" : "anthropic",
model: Object.keys(parseObject(parsedStream.resultJson?.modelUsage)).length > 1 ? "mixed" : parsedStream.model || model,
usageByModel: claudeModelReceipts(parsedStream.resultJson?.modelUsage),
billingType,
costUsd: parsedStream.costUsd,
errorMessage: `Timed out after ${timeoutSec}s`,
errorCode: "timeout",
errorMeta,
@@ -1075,6 +1095,15 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
exitCode: proc.exitCode,
signal: proc.signal,
timedOut: false,
usageComplete: parsedStream.resultJson !== null,
usage: parsedStream.usage ?? undefined,
usageBasis: "per_run",
provider: "anthropic",
biller: isBedrockAuth(effectiveEnv) ? "aws_bedrock" : "anthropic",
model: Object.keys(parseObject(parsedStream.resultJson?.modelUsage)).length > 1 ? "mixed" : parsedStream.model || model,
usageByModel: claudeModelReceipts(parsedStream.resultJson?.modelUsage),
billingType,
costUsd: parsedStream.costUsd,
errorMessage: fallbackErrorMessage,
errorCode,
errorFamily,
@@ -1237,19 +1266,21 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
exitCode: proc.exitCode,
signal: proc.signal,
timedOut: false,
usageComplete: parsedStream.resultJson !== null || parsed.type === "result",
errorMessage,
errorCode: resolvedErrorCode,
errorFamily,
retryNotBefore: transientRetryNotBefore ? transientRetryNotBefore.toISOString() : null,
errorMeta,
usage,
usageByModel: claudeModelReceipts(parsed.modelUsage),
...(usageBasis ? { usageBasis } : {}),
sessionId: resolvedSessionId,
sessionParams: resolvedSessionParams,
sessionDisplayId: resolvedSessionId,
provider: "anthropic",
biller: isBedrockAuth(effectiveEnv) ? "aws_bedrock" : "anthropic",
model: parsedStream.model || asString(parsed.model, model),
model: Object.keys(parseObject(parsed.modelUsage)).length > 1 ? "mixed" : parsedStream.model || asString(parsed.model, model),
billingType,
costUsd: parsedStream.costUsd,
resultJson: mergedResultJson,
packages/adapters/claude-local/src/server/parse.test.ts
+14
View File
@@ -567,3 +567,17 @@ describe("parseClaudeStreamJson usage extraction", () => {
expect(parsed.usageBasis).toBe("per_run");
});
});
describe("interrupted Claude accounting", () => {
it("retains partial usage and replaces repeated message snapshots", () => {
const events = [
{ type: "assistant", message: { id: "m1", usage: { input_tokens: 10, cache_creation_input_tokens: 20, cache_read_input_tokens: 40, output_tokens: 3 } } },
{ type: "assistant", message: { id: "m1", usage: { input_tokens: 10, cache_creation_input_tokens: 20, cache_read_input_tokens: 40, output_tokens: 7 } } },
{ type: "assistant", message: { id: "m2", usage: { input_tokens: 5, output_tokens: 2 } } },
];
const parsed = parseClaudeStreamJson(events.map((event) => JSON.stringify(event)).join("\n"));
expect(parsed.usage).toEqual({ inputTokens: 35, cachedInputTokens: 40, outputTokens: 9 });
expect(parsed.usageBasis).toBe("per_run");
expect(parsed.costUsd).toBeNull();
});
});
packages/adapters/claude-local/src/server/parse.ts
+91 -55
View File
@@ -54,79 +54,115 @@ export function claudeModelUsageTotals(modelUsage: unknown): UsageSummary | null
return { inputTokens, outputTokens, cachedInputTokens };
}
export function claudeModelReceipts(modelUsage: unknown) {
const entries = Object.entries(parseObject(modelUsage));
if (!entries.length) return undefined;
const receipts = entries.map(([model, raw]) => {
const entry = parseObject(raw);
const usage = claudeModelUsageTotals({ [model]: entry });
const costUsd = entry.costUSD;
return usage && typeof costUsd === "number" && Number.isFinite(costUsd) && costUsd >= 0 ? { model, usage, costUsd } : null;
});
return receipts.every((part) => part !== null) ? receipts : undefined;
}
export function parseClaudeStreamJson(stdout: string) {
return createClaudeStreamParser()(stdout);
}
/** Consume complete JSONL records once, retaining protocol accounting state. */
export function createClaudeStreamParser() {
let sessionId: string | null = null;
let model = "";
let finalResult: Record<string, unknown> | null = null;
const assistantTexts: string[] = [];
const messageUsage = new Map<string, UsageSummary>();
let anonymousMessage = 0;
const observedTotals = { inputTokens: 0, outputTokens: 0, cachedInputTokens: 0 };
for (const rawLine of stdout.split(/\r?\n/)) {
const line = rawLine.trim();
if (!line) continue;
const event = parseJson(line);
if (!event) continue;
return (stdout: string) => {
for (const rawLine of stdout.split(/\r?\n/)) {
const line = rawLine.trim();
if (!line) continue;
const event = parseJson(line);
if (!event) continue;
const type = asString(event.type, "");
if (type === "system" && asString(event.subtype, "") === "init") {
sessionId = asString(event.session_id, sessionId ?? "") || sessionId;
model = asString(event.model, model);
continue;
}
const type = asString(event.type, "");
if (type === "system" && asString(event.subtype, "") === "init") {
sessionId = asString(event.session_id, sessionId ?? "") || sessionId;
model = asString(event.model, model);
continue;
}
if (type === "assistant") {
sessionId = asString(event.session_id, sessionId ?? "") || sessionId;
const message = parseObject(event.message);
const content = Array.isArray(message.content) ? message.content : [];
for (const entry of content) {
if (typeof entry !== "object" || entry === null || Array.isArray(entry)) continue;
const block = entry as Record<string, unknown>;
if (asString(block.type, "") === "text") {
const text = asString(block.text, "");
if (text) assistantTexts.push(text);
if (type === "assistant") {
sessionId = asString(event.session_id, sessionId ?? "") || sessionId;
const message = parseObject(event.message);
const observed = parseObject(message.usage);
if (Object.keys(observed).length > 0) {
const key = asString(message.id, "") || `anonymous:${anonymousMessage++}`;
const previous = messageUsage.get(key);
const next = {
inputTokens: Math.max(previous?.inputTokens ?? 0, asNumber(observed.input_tokens, 0) + asNumber(observed.cache_creation_input_tokens, 0)),
cachedInputTokens: Math.max(previous?.cachedInputTokens ?? 0, asNumber(observed.cache_read_input_tokens, 0)),
outputTokens: Math.max(previous?.outputTokens ?? 0, asNumber(observed.output_tokens, 0)),
};
observedTotals.inputTokens += next.inputTokens - (previous?.inputTokens ?? 0);
observedTotals.outputTokens += next.outputTokens - (previous?.outputTokens ?? 0);
observedTotals.cachedInputTokens += next.cachedInputTokens - (previous?.cachedInputTokens ?? 0);
messageUsage.set(key, next);
}
const content = Array.isArray(message.content) ? message.content : [];
for (const entry of content) {
if (typeof entry !== "object" || entry === null || Array.isArray(entry)) continue;
const block = entry as Record<string, unknown>;
if (asString(block.type, "") === "text") {
const text = asString(block.text, "");
if (text) assistantTexts.push(text);
}
}
continue;
}
if (type === "result") {
finalResult = event;
sessionId = asString(event.session_id, sessionId ?? "") || sessionId;
}
continue;
}
if (type === "result") {
finalResult = event;
sessionId = asString(event.session_id, sessionId ?? "") || sessionId;
if (!finalResult) {
return {
sessionId,
model,
costUsd: null as number | null,
usage: messageUsage.size ? { ...observedTotals } : null,
usageBasis: "per_run" as const,
summary: assistantTexts.join("\n\n").trim(),
resultJson: null as Record<string, unknown> | null,
};
}
}
if (!finalResult) {
const modelUsageTotals = claudeModelUsageTotals(finalResult.modelUsage);
const usageObj = parseObject(finalResult.usage);
const usage: UsageSummary = modelUsageTotals ?? {
inputTokens: asNumber(usageObj.input_tokens, 0) + asNumber(usageObj.cache_creation_input_tokens, 0),
cachedInputTokens: asNumber(usageObj.cache_read_input_tokens, 0),
outputTokens: asNumber(usageObj.output_tokens, 0),
};
const costRaw = finalResult.total_cost_usd;
const costUsd = typeof costRaw === "number" && Number.isFinite(costRaw) ? costRaw : null;
const summary = asString(finalResult.result, assistantTexts.join("\n\n")).trim();
return {
sessionId,
model,
costUsd: null as number | null,
usage: null as UsageSummary | null,
usageBasis: null as "per_run" | null,
summary: assistantTexts.join("\n\n").trim(),
resultJson: null as Record<string, unknown> | null,
costUsd,
usage,
// modelUsage covers exactly this CLI invocation, so mark it per-run to
// keep the server from applying its session-cumulative delta heuristic.
usageBasis: "per_run" as const,
summary,
resultJson: finalResult,
};
}
const modelUsageTotals = claudeModelUsageTotals(finalResult.modelUsage);
const usageObj = parseObject(finalResult.usage);
const usage: UsageSummary = modelUsageTotals ?? {
inputTokens: asNumber(usageObj.input_tokens, 0),
cachedInputTokens: asNumber(usageObj.cache_read_input_tokens, 0),
outputTokens: asNumber(usageObj.output_tokens, 0),
};
const costRaw = finalResult.total_cost_usd;
const costUsd = typeof costRaw === "number" && Number.isFinite(costRaw) ? costRaw : null;
const summary = asString(finalResult.result, assistantTexts.join("\n\n")).trim();
return {
sessionId,
model,
costUsd,
usage,
// modelUsage covers exactly this CLI invocation, so mark it per-run to
// keep the server from applying its session-cumulative delta heuristic.
usageBasis: "per_run" as const,
summary,
resultJson: finalResult,
};
}
packages/adapters/claude-local/src/server/quota.ts
+3 -2
View File
@@ -265,14 +265,15 @@ export async function fetchWithTimeout(url: string, init: RequestInit, ms = 8000
const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), ms);
try {
return await fetch(url, { ...init, signal: controller.signal });
return await fetch(url, { ...init, signal: init.signal ? AbortSignal.any([init.signal, controller.signal]) : controller.signal });
} finally {
clearTimeout(timer);
}
}
export async function fetchClaudeQuota(token: string): Promise<QuotaWindow[]> {
export async function fetchClaudeQuota(token: string, signal?: AbortSignal): Promise<QuotaWindow[]> {
const resp = await fetchWithTimeout("https://api.anthropic.com/api/oauth/usage", {
signal,
headers: {
Authorization: `Bearer ${token}`,
"anthropic-beta": "oauth-2025-04-20",
packages/adapters/codex-local/src/server/acp.test.ts
+6
View File
@@ -1426,6 +1426,12 @@ describe("codex_local ACP lane", () => {
});
describe("resolveCodexAcpBillingIdentity", () => {
it.each([["openrouter", "api_key", "openrouter"], ["custom", "api_key", "unknown"], ["local", "none", "unknown"]])("classifies managed %s/%s independently of host authentication", (kind, auth, biller) => {
expect(resolveCodexAcpBillingIdentity({ config: {
managedAiRouting: { kind, auth },
env: { OPENAI_API_KEY: "", PAPERCLIP_AI_PROVIDER_KEY: auth === "none" ? "" : "fixture" },
} })).toEqual({ provider: "openai", biller, billingType: "api" });
});
const originalOpenAiKey = process.env.OPENAI_API_KEY;
const originalOpenRouterKey = process.env.OPENROUTER_API_KEY;
packages/adapters/codex-local/src/server/acp.ts
+3 -1
View File
@@ -13,7 +13,7 @@ import {
parseLocalProcessFilesystemScope,
parseLocalProcessNetworkScope,
} from "@paperclipai/adapter-utils/local-process-sandbox";
import { inferOpenAiCompatibleBiller } from "@paperclipai/adapter-utils";
import { inferOpenAiCompatibleBiller, resolveManagedOpenAiBilling } from "@paperclipai/adapter-utils";
import {
ensureAdapterExecutionTargetCommandResolvable,
readAdapterExecutionTarget,
@@ -330,6 +330,8 @@ export function resolveCodexAcpBillingIdentity(
ctx: Pick<AdapterExecutionContext, "config"> &
Partial<Pick<AdapterExecutionContext, "executionTarget" | "executionTransport">>,
): { provider: string; biller: string; billingType: AdapterBillingType } {
const managedBilling = resolveManagedOpenAiBilling(ctx.config.managedAiRouting);
if (managedBilling) return managedBilling;
const envConfig = parseObject(parseObject(ctx.config).env);
const target = readAdapterExecutionTarget({
executionTarget: ctx.executionTarget,
packages/adapters/codex-local/src/server/codex-auth-cache.test.ts
+34 -2
View File
@@ -463,13 +463,15 @@ describe("codex auth cache store", () => {
releaseFirstCaller = resolve;
});
const entered = deferredSignal();
const firstCall = withAccountHomeSecretMutationLock(env, "company-shared", async () => {
events.push("first-enter");
entered.resolve();
await firstCallerGate;
events.push("first-exit");
return "first";
});
await new Promise((resolve) => setTimeout(resolve, 20));
await entered.promise;
const secondCall = withAccountHomeSecretMutationLock(env, "company-shared", async () => {
events.push("second-enter");
events.push("second-exit");
@@ -497,12 +499,14 @@ describe("codex auth cache store", () => {
releaseCompanyA = resolve;
});
const entered = deferredSignal();
const companyACall = withAccountHomeSecretMutationLock(env, "company-a", async () => {
events.push("a-enter");
entered.resolve();
await companyAGate;
events.push("a-exit");
});
await new Promise((resolve) => setTimeout(resolve, 20));
await entered.promise;
const companyBCall = withAccountHomeSecretMutationLock(env, "company-b", async () => {
events.push("b-enter");
events.push("b-exit");
@@ -536,6 +540,28 @@ describe("codex auth cache store", () => {
expect(events).toEqual(["promotion-enter", "mutation-enter", "mutation-exit", "promotion-exit"]);
});
it("allows awaited nested writes but expires inherited ownership after release", async () => {
const env = envFor(await makeInstanceRoot());
const detachedGate = deferredSignal();
let detached!: Promise<void>;
let enteredDetached = false;
await withAccountHomeSecretMutationLock(env, "nested", async () => {
await expect(withAccountHomeSecretMutationLock(env, "nested", async () => "nested write")).resolves.toBe("nested write");
detached = detachedGate.promise.then(() => withAccountHomeSecretMutationLock(env, "nested", async () => { enteredDetached = true; }));
});
const holding = deferredSignal();
const release = deferredSignal();
const other = withAccountHomeSecretMutationLock(env, "nested", async () => { holding.resolve(); await release.promise; });
await holding.promise;
detachedGate.resolve();
try {
await new Promise(resolve => setTimeout(resolve, 30));
expect(enteredDetached).toBe(false);
} finally { release.resolve(); }
await Promise.all([other, detached]);
expect(enteredDetached).toBe(true);
});
});
describe("Phase 8: account-home directory validity before a secret write commits", () => {
@@ -581,3 +607,9 @@ describe("codex auth cache store", () => {
});
});
});
function deferredSignal() {
let resolve!: () => void;
const promise = new Promise<void>(done => { resolve = done; });
return { promise, resolve };
}
packages/adapters/codex-local/src/server/codex-auth-cache.ts
+17 -2
View File
@@ -1,4 +1,5 @@
import { lstat, mkdir, open, readFile, rename, rm } from "node:fs/promises";
import { AsyncLocalStorage } from "node:async_hooks";
import { lstat, mkdir, open, readFile, realpath, rename, rm } from "node:fs/promises";
import path from "node:path";
import { randomUUID } from "node:crypto";
import { resolvePaperclipInstanceRootForAdapter } from "@paperclipai/adapter-utils/server-utils";
@@ -334,6 +335,11 @@ export async function withCodexAccountHomePromotionLock<T>(
* create) can still call into a secrets-service write that takes this lock
* without deadlocking on itself.
*/
// An outer credential transaction acquires this lock before database row locks.
// Nested secret-service writes must share that ownership, not acquire it again.
// Expire ownership on return so detached work cannot inherit a released lock.
const secretMutationOwnership = new AsyncLocalStorage<Map<string, { active: boolean }>>();
export async function withAccountHomeSecretMutationLock<T>(
env: NodeJS.ProcessEnv = process.env,
companyId: string,
@@ -341,7 +347,16 @@ export async function withAccountHomeSecretMutationLock<T>(
): Promise<T> {
const lockDir = resolveCodexAuthCacheNamedLockDir(env, companyId, "codex-account-home-secret-mutation-lock");
await ensurePrivateDir(lockDir);
return withDirectoryMergeLock(lockDir, fn, env);
const key = await realpath(lockDir);
const inherited = secretMutationOwnership.getStore();
if (inherited?.get(key)?.active) return fn();
return withDirectoryMergeLock(lockDir, async () => {
const ownership = { active: true };
const scope = new Map(inherited);
scope.set(key, ownership);
try { return await secretMutationOwnership.run(scope, fn); }
finally { ownership.active = false; }
}, env);
}
/**
packages/adapters/codex-local/src/server/execute.ts
+36 -8
View File
@@ -1,8 +1,9 @@
import { createProviderStoppedBoundary } from "@paperclipai/adapter-utils/provider-stopped-boundary";
import { createUsageCheckpointLog } from "@paperclipai/adapter-utils/usage-checkpoint";
import fs from "node:fs/promises";
import path from "node:path";
import { fileURLToPath } from "node:url";
import { inferOpenAiCompatibleBiller, type AdapterExecutionContext, type AdapterExecutionResult } from "@paperclipai/adapter-utils";
import { inferOpenAiCompatibleBiller, resolveManagedOpenAiBilling, type AdapterExecutionContext, type AdapterExecutionResult } from "@paperclipai/adapter-utils";
import { buildCodexAuthInboundProvision } from "./codex-auth-merge-scripts.js";
import { copyBackCodexAuth } from "./codex-auth-copyback.js";
import {
@@ -61,7 +62,7 @@ import {
type LocalProcessSandboxOptions,
} from "@paperclipai/adapter-utils/local-process-sandbox";
import {
parseCodexJsonl,
parseCodexJsonl, createCodexJsonlParser,
classifyCodexAuthRefreshFailure,
extractCodexRetryNotBefore,
isCodexHarnessCrash,
@@ -993,7 +994,9 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
(entry): entry is [string, string] => typeof entry[1] === "string",
),
);
const billingType = resolveCodexBillingType(effectiveEnv);
const managedBilling = resolveManagedOpenAiBilling(config.managedAiRouting);
const billingType = managedBilling?.billingType ?? resolveCodexBillingType(effectiveEnv);
const biller = managedBilling?.biller ?? resolveCodexBiller(effectiveEnv, billingType);
const networkScope = parseLocalProcessNetworkScope(config.networkScope);
const filesystemScope = parseLocalProcessFilesystemScope(config.filesystemScope);
const localProcessSandbox: LocalProcessSandboxOptions | null =
@@ -1228,6 +1231,7 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
},
);
const args = execArgs.args;
const pricingContext = execArgs.fastModeApplied ? { serviceTier: "fast" } : undefined;
if (ctx.agentIdentity) {
const identityNames = ["PAPERCLIP_AGENT_KEY_ID", "PAPERCLIP_AGENT_PUBLIC_KEY", "PAPERCLIP_AGENT_PRIVATE_KEY"];
const shellKeys = [...new Set([
@@ -1330,6 +1334,11 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
}
};
const consumeAccounting = createCodexJsonlParser();
const accountingLog = createUsageCheckpointLog(onLog, ctx.onUsage, stdout => {
const parsed = consumeAccounting(stdout);
return { usage: parsed.usageReported ? parsed.usage : undefined, costStatus: parsed.usageComplete ? undefined : "unpriced", usageBasis: "per_run", provider: "openai", biller, billingType, model, pricingContext, costUsd: null, complete: parsed.usageComplete };
});
try {
const proc = await runAdapterExecutionTargetProcess(runId, runtimeExecutionTarget, command, args, {
onProcessStopped: providerStop.beginInvocation(),
@@ -1343,7 +1352,7 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
onLog: async (stream, chunk) => {
monitor?.noteOutputChunk(stream, chunk);
if (stream === "stdout") {
await onLog(stream, chunk);
await accountingLog(stream, chunk);
return;
}
const cleaned = stripCodexRolloutNoise(chunk);
@@ -1354,6 +1363,7 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
settleRunDisposition: paperclipBridge?.settleRunDisposition,
localProcessSandbox,
});
await accountingLog.flush();
const cleanedStderr = stripCodexRolloutNoise(proc.stderr);
return {
proc: {
@@ -1362,6 +1372,7 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
},
rawStderr: proc.stderr,
parsed: parseCodexJsonl(proc.stdout),
pricingContext,
monitor: monitorFired
? {
fired: true as const,
@@ -1390,6 +1401,7 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
proc: { exitCode: number | null; signal: string | null; timedOut: boolean; stdout: string; stderr: string; errorCode?: string | null };
rawStderr: string;
parsed: ReturnType<typeof parseCodexJsonl>;
pricingContext?: AdapterExecutionResult["pricingContext"];
monitor?:
| { fired: false }
| { fired: true; terminationSignal: NodeJS.Signals | null; elapsedMsSinceLastEvent: number; timeoutMs: number };
@@ -1403,19 +1415,22 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
exitCode: null,
signal: attempt.monitor.terminationSignal ?? attempt.proc.signal,
timedOut: false,
usageComplete: attempt.parsed.usageComplete,
costStatus: attempt.parsed.usageComplete ? undefined : "unpriced",
errorMessage,
errorCode: "codex_output_inactivity_monitor",
errorFamily: null,
usage: attempt.parsed.usage,
usage: attempt.parsed.usageReported ? attempt.parsed.usage : undefined,
usageBasis: attempt.parsed.usageBasis,
sessionId: null,
sessionParams: null,
sessionDisplayId: null,
provider: "openai",
biller: resolveCodexBiller(effectiveEnv, billingType),
biller,
model,
billingType,
costUsd: null,
pricingContext: attempt.pricingContext,
resultJson: {
stdout: attempt.proc.stdout,
stderr: attempt.proc.stderr,
@@ -1435,6 +1450,16 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
exitCode: attempt.proc.exitCode,
signal: attempt.proc.signal,
timedOut: true,
usageComplete: attempt.parsed.usageComplete,
costStatus: attempt.parsed.usageComplete ? undefined : "unpriced",
usage: attempt.parsed.usageReported ? attempt.parsed.usage : undefined,
usageBasis: "per_run",
provider: "openai",
biller,
model,
billingType,
costUsd: null,
pricingContext: attempt.pricingContext,
errorMessage: `Timed out after ${timeoutSec}s`,
clearSession: clearSessionOnMissingSession,
};
@@ -1514,6 +1539,8 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
exitCode: attempt.proc.exitCode,
signal: attempt.proc.signal,
timedOut: false,
usageComplete: attempt.parsed.usageComplete,
costStatus: attempt.parsed.usageComplete ? undefined : "unpriced",
errorMessage:
(attempt.proc.exitCode ?? 0) === 0
? null
@@ -1535,16 +1562,17 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
: null,
errorFamily,
retryNotBefore: transientRetryNotBefore ? transientRetryNotBefore.toISOString() : null,
usage: attempt.parsed.usage,
usage: attempt.parsed.usageReported ? attempt.parsed.usage : undefined,
usageBasis: attempt.parsed.usageBasis,
sessionId: resolvedSessionId,
sessionParams: resolvedSessionParams,
sessionDisplayId: resolvedSessionId,
provider: "openai",
biller: resolveCodexBiller(effectiveEnv, billingType),
biller,
model,
billingType,
costUsd: null,
pricingContext: attempt.pricingContext,
resultJson: {
stdout: attempt.proc.stdout,
stderr: attempt.proc.stderr,
packages/adapters/codex-local/src/server/parse.test.ts
+19 -2
View File
@@ -10,6 +10,19 @@ import {
} from "./parse.js";
describe("parseCodexJsonl", () => {
it.each(["turn.failed", "error", "turn.completed"])("does not invent zero usage for %s without counters", (type) => {
expect(parseCodexJsonl(JSON.stringify({ type }))).toMatchObject({
sawProtocolTerminalEvent: true, usageReported: false, usageComplete: false,
});
});
it.each([{}, { input_tokens: 0 }, { input_tokens: -1, output_tokens: 0 }, { input_tokens: 1, output_tokens: 0, cached_input_tokens: 2 }])("rejects missing or invalid completion counters: %j", (usage) => {
expect(parseCodexJsonl(JSON.stringify({ type: "turn.completed", usage })).usageComplete).toBe(false);
});
it("distinguishes explicit zero usage from missing usage", () => {
expect(parseCodexJsonl(JSON.stringify({ type: "turn.completed", usage: { input_tokens: 0, output_tokens: 0 } }))).toMatchObject({
usageReported: true, usageComplete: true, usage: { inputTokens: 0, outputTokens: 0, cachedInputTokens: 0 },
});
});
it("captures session id, assistant summary, usage, and error message", () => {
const stdout = [
JSON.stringify({ type: "thread.started", thread_id: "thread_123" }),
@@ -28,7 +41,7 @@ describe("parseCodexJsonl", () => {
sessionId: "thread_123",
summary: "Recovered response",
usage: {
inputTokens: 10,
inputTokens: 8,
cachedInputTokens: 2,
outputTokens: 4,
},
@@ -36,6 +49,8 @@ describe("parseCodexJsonl", () => {
errorMessage: "resume failed",
sawProtocolEvent: true,
sawProtocolTerminalEvent: true,
usageReported: true,
usageComplete: false,
});
});
@@ -64,7 +79,7 @@ describe("parseCodexJsonl", () => {
sessionId: "thread_123",
summary: "Fixed the issue and verified the targeted tests pass.",
usage: {
inputTokens: 10,
inputTokens: 8,
cachedInputTokens: 2,
outputTokens: 4,
},
@@ -72,6 +87,8 @@ describe("parseCodexJsonl", () => {
errorMessage: null,
sawProtocolEvent: true,
sawProtocolTerminalEvent: true,
usageReported: true,
usageComplete: true,
});
});
});
packages/adapters/codex-local/src/server/parse.ts
+64 -49
View File
@@ -1,6 +1,5 @@
import {
asString,
asNumber,
parseObject,
parseJson,
} from "@paperclipai/adapter-utils/server-utils";
@@ -28,72 +27,88 @@ export type CodexAuthRefreshFailureClass =
| "refresh_token_invalidated";
export function parseCodexJsonl(stdout: string) {
return createCodexJsonlParser()(stdout);
}
/** Consume complete JSONL records once, retaining protocol accounting state. */
export function createCodexJsonlParser() {
let sessionId: string | null = null;
let finalMessage: string | null = null;
let errorMessage: string | null = null;
let sawProtocolEvent = false;
let sawProtocolTerminalEvent = false;
let usageReported = false;
let usageComplete = false;
const usage = {
inputTokens: 0,
cachedInputTokens: 0,
outputTokens: 0,
};
for (const rawLine of stdout.split(/\r?\n/)) {
const line = rawLine.trim();
if (!line) continue;
return (stdout: string) => {
for (const rawLine of stdout.split(/\r?\n/)) {
const line = rawLine.trim();
if (!line) continue;
const event = parseJson(line);
if (!event) continue;
const event = parseJson(line);
if (!event) continue;
const type = asString(event.type, "");
if (type) sawProtocolEvent = true;
if (type === "error" || type === "turn.completed" || type === "turn.failed") {
sawProtocolTerminalEvent = true;
}
if (type === "thread.started") {
sessionId = asString(event.thread_id, sessionId ?? "") || sessionId;
continue;
}
const type = asString(event.type, "");
if (type) sawProtocolEvent = true;
if (type === "error" || type === "turn.completed" || type === "turn.failed") {
sawProtocolTerminalEvent = true;
const reported = parseObject(event.usage);
const validCount = (value: unknown): value is number => typeof value === "number" && Number.isSafeInteger(value) && value >= 0;
const cached = reported.cached_input_tokens ?? 0;
usageComplete = validCount(reported.input_tokens) && validCount(reported.output_tokens)
&& validCount(cached) && cached <= reported.input_tokens;
if (usageComplete) {
usage.inputTokens = reported.input_tokens as number;
usage.cachedInputTokens = cached as number;
usage.outputTokens = reported.output_tokens as number;
usageReported = true;
}
}
if (type === "turn.started") usageComplete = false;
if (type === "thread.started") {
sessionId = asString(event.thread_id, sessionId ?? "") || sessionId;
continue;
}
if (type === "error") {
const msg = asString(event.message, "").trim();
if (msg) errorMessage = msg;
continue;
}
if (type === "error") {
const msg = asString(event.message, "").trim();
if (msg) errorMessage = msg;
continue;
}
if (type === "item.completed") {
const item = parseObject(event.item);
if (asString(item.type, "") === "agent_message") {
const text = asString(item.text, "");
if (text) finalMessage = text;
if (type === "item.completed") {
const item = parseObject(event.item);
if (asString(item.type, "") === "agent_message") {
const text = asString(item.text, "");
if (text) finalMessage = text;
}
continue;
}
if (type === "turn.failed") {
const err = parseObject(event.error);
const msg = asString(err.message, "").trim();
if (msg) errorMessage = msg;
}
continue;
}
if (type === "turn.completed") {
const usageObj = parseObject(event.usage);
usage.inputTokens = asNumber(usageObj.input_tokens, usage.inputTokens);
usage.cachedInputTokens = asNumber(usageObj.cached_input_tokens, usage.cachedInputTokens);
usage.outputTokens = asNumber(usageObj.output_tokens, usage.outputTokens);
continue;
}
if (type === "turn.failed") {
const err = parseObject(event.error);
const msg = asString(err.message, "").trim();
if (msg) errorMessage = msg;
}
}
return {
sessionId,
summary: finalMessage?.trim() ?? "",
usage,
usageBasis: "per_run" as const,
errorMessage,
sawProtocolEvent,
sawProtocolTerminalEvent,
return {
sessionId,
summary: finalMessage?.trim() ?? "",
// Codex includes cache hits in input_tokens; Paperclip stores them separately.
usage: { ...usage, inputTokens: Math.max(0, usage.inputTokens - usage.cachedInputTokens) },
usageBasis: "per_run" as const,
errorMessage,
sawProtocolEvent,
sawProtocolTerminalEvent,
usageReported,
usageComplete,
};
};
}
packages/adapters/codex-local/src/server/quota-spawn-error.test.ts
+182 -3
View File
@@ -17,7 +17,7 @@ vi.mock("node:child_process", async (importOriginal) => {
};
});
import { fetchCodexQuota, getQuotaWindows } from "./quota.js";
import { fetchCodexQuota, fetchCodexRpcQuota, getQuotaWindows } from "./quota.js";
function createChildThatErrorsOnMicrotask(err: Error): ChildProcess {
const child = new EventEmitter() as ChildProcess;
@@ -27,7 +27,7 @@ function createChildThatErrorsOnMicrotask(err: Error): ChildProcess {
Object.assign(child, {
stdout: stream,
stderr: Object.assign(new EventEmitter(), { setEncoding: () => {} }),
stdin: { write: vi.fn(), end: vi.fn() },
stdin: Object.assign(new EventEmitter(), { write: vi.fn(), end: vi.fn() }),
kill: vi.fn(),
});
queueMicrotask(() => {
@@ -36,6 +36,31 @@ function createChildThatErrorsOnMicrotask(err: Error): ChildProcess {
return child;
}
type RpcRequest = { id?: number; method: string };
function createRpcChild(response?: (request: RpcRequest) => unknown) {
const child = new EventEmitter() as ChildProcess;
const stdout = Object.assign(new EventEmitter(), { setEncoding: () => {} });
const stderr = Object.assign(new EventEmitter(), { setEncoding: () => {} });
const stdin = Object.assign(new EventEmitter(), {
write: vi.fn((line: string) => {
const request = JSON.parse(line) as RpcRequest;
if (request.id == null) return;
const message = response ? response(request) : { id: request.id, result: {} };
if (message !== undefined) {
queueMicrotask(() => stdout.emit("data", JSON.stringify(message) + "\n"));
}
}),
end: vi.fn(),
});
const kill = vi.fn((_signal?: string | number) => {
queueMicrotask(() => child.emit("close", 0));
return true;
});
Object.assign(child, { pid: 12345, stdout, stderr, stdin, kill });
return { child, stdout, stderr, stdin, kill };
}
describe("CodexRpcClient spawn failures", () => {
let previousCodexHome: string | undefined;
let isolatedCodexHome: string | undefined;
@@ -51,6 +76,7 @@ describe("CodexRpcClient spawn failures", () => {
});
afterEach(() => {
vi.useRealTimers();
vi.unstubAllGlobals();
if (isolatedCodexHome) {
try {
@@ -67,6 +93,159 @@ describe("CodexRpcClient spawn failures", () => {
}
});
it("reads account quota with a supported approval policy and never starts a model turn", async () => {
const { child, stdin, kill } = createRpcChild((request) => {
const result = request.method === "account/rateLimits/read"
? { rateLimits: { primary: { usedPercent: 37, windowDurationMins: 300 } } }
: {};
return { id: request.id, result };
});
mockSpawn.mockImplementation((_command, args: string[]) => {
if (args.includes("untrusted")) return createChildThatErrorsOnMicrotask(new Error("invalid approval policy"));
return child;
});
const result = await fetchCodexRpcQuota();
expect(result.windows).toEqual([expect.objectContaining({ usedPercent: 37 })]);
expect(mockSpawn).toHaveBeenCalledWith("codex", ["-s", "read-only", "-a", "on-request", "app-server"], expect.any(Object));
expect(stdin.write.mock.calls.map(([line]) => JSON.parse(line).method)).toEqual([
"initialize", "initialized", "account/rateLimits/read", "account/read",
]);
expect(stdin.end).toHaveBeenCalledOnce();
expect(kill).toHaveBeenCalledWith("SIGTERM");
expect(kill).not.toHaveBeenCalledWith("SIGKILL");
});
it("rejects RPC error envelopes and preserves their authentication classification", async () => {
const { child, kill } = createRpcChild((request) => request.method === "account/rateLimits/read"
? { id: request.id, error: { code: -32000, message: "OAuth failed: refresh token has expired" } }
: { id: request.id, result: {} });
mockSpawn.mockReturnValue(child);
const result = await getQuotaWindows();
expect(result).toMatchObject({ ok: false, source: "codex-rpc", errorFamily: "refresh_token_expired", windows: [] });
expect(kill).toHaveBeenCalledWith("SIGTERM");
});
it.each([null, [], "invalid", 17, undefined])("rejects an invalid result envelope (%s)", async (result) => {
const { child } = createRpcChild((request) => ({ id: request.id, result }));
mockSpawn.mockReturnValue(child);
await expect(fetchCodexRpcQuota()).rejects.toThrow("invalid quota response");
});
it("tolerates non-object lines and parses fragmented, out-of-order replies", async () => {
const probe = createRpcChild();
mockSpawn.mockReturnValue(probe.child);
probe.stdin.write.mockImplementation((line) => {
const request = JSON.parse(line) as RpcRequest;
if (request.method === "initialize") {
queueMicrotask(() => probe.stdout.emit("data", 'null\n[]\nnot-json\n{"id":1,"result":{}}\n'));
} else if (request.method === "account/read") {
queueMicrotask(() => {
probe.stdout.emit("data", '{"id":3,"result":{"account":{"email":"test@example.com"}}}\n{"id":2,');
probe.stdout.emit("data", '"result":{"rateLimits":{"primary":{"usedPercent":37}}}}\n');
});
}
});
const result = await fetchCodexRpcQuota();
expect(result.email).toBe("test@example.com");
expect(result.windows[0].usedPercent).toBe(37);
});
it.each([false, true])("bounds stdout buffering with and without a completed frame (newline: %s)", async (newline) => {
const probe = createRpcChild(() => undefined);
mockSpawn.mockReturnValue(probe.child);
const result = fetchCodexRpcQuota();
probe.stdout.emit("data", "x".repeat(64 * 1024));
probe.stdout.emit("data", "x" + (newline ? "\n" : ""));
await expect(result).rejects.toThrow("response exceeded the quota probe limit");
expect(probe.kill).toHaveBeenCalledWith("SIGTERM");
});
it("bounds stderr diagnostics when the process exits", async () => {
const probe = createRpcChild(() => undefined);
mockSpawn.mockReturnValue(probe.child);
const result = fetchCodexRpcQuota().catch((error: Error) => error);
probe.stderr.emit("data", "x".repeat(128 * 1024));
probe.stderr.emit("data", " end of diagnostic");
probe.child.emit("exit", 1);
const error = await result as Error;
expect(error.message.length).toBeLessThanOrEqual(4_000);
expect(error.message).toMatch(/end of diagnostic$/);
});
it("handles a broken stdin without an uncaught stream error", async () => {
const probe = createRpcChild(() => undefined);
mockSpawn.mockReturnValue(probe.child);
const result = fetchCodexRpcQuota();
probe.stdin.emit("error", new Error("write EPIPE"));
await expect(result).rejects.toThrow("write EPIPE");
expect(probe.kill).toHaveBeenCalledWith("SIGTERM");
});
it("clears other pending requests when a probe times out and kills a child that ignores SIGTERM", async () => {
vi.useFakeTimers();
const probe = createRpcChild((request) => request.method === "initialize"
? { id: request.id, result: {} }
: undefined);
probe.kill.mockImplementation((signal) => {
if (signal === "SIGKILL") queueMicrotask(() => probe.child.emit("close", null));
return true;
});
mockSpawn.mockReturnValue(probe.child);
const result = fetchCodexRpcQuota();
const rejected = expect(result).rejects.toThrow("timed out on account/rateLimits/read");
await vi.advanceTimersByTimeAsync(6_000);
expect(probe.stdin.end).toHaveBeenCalledOnce();
expect(probe.kill.mock.calls).toEqual([["SIGTERM"]]);
await vi.advanceTimersByTimeAsync(250);
await rejected;
expect(probe.kill.mock.calls).toEqual([["SIGTERM"], ["SIGKILL"]]);
expect(vi.getTimerCount()).toBe(0);
expect(probe.child.listenerCount("close")).toBe(1);
});
it("bounds shutdown even if a killed child never emits close", async () => {
vi.useFakeTimers();
const probe = createRpcChild();
probe.kill.mockImplementation(() => true);
mockSpawn.mockReturnValue(probe.child);
const result = fetchCodexRpcQuota();
await vi.advanceTimersByTimeAsync(500);
await expect(result).resolves.toMatchObject({ windows: [] });
expect(probe.kill.mock.calls).toEqual([["SIGTERM"], ["SIGKILL"]]);
expect(vi.getTimerCount()).toBe(0);
expect(probe.child.listenerCount("close")).toBe(1);
});
it("reaps a real subprocess that ignores SIGTERM", async () => {
const { spawn: spawnReal } = await vi.importActual<typeof import("node:child_process")>("node:child_process");
const child = spawnReal(process.execPath, ["--input-type=commonjs", "-e", `
process.on("SIGTERM", () => {});
setInterval(() => {}, 60_000);
require("node:readline").createInterface({ input: process.stdin }).on("line", (line) => {
const request = JSON.parse(line);
if (request.id == null) return;
const result = request.method === "account/rateLimits/read"
? { rateLimits: { primary: { usedPercent: 37 } } }
: {};
process.stdout.write(JSON.stringify({ id: request.id, result }) + "\\n");
});
`], { stdio: ["pipe", "pipe", "pipe"] });
mockSpawn.mockReturnValue(child);
try {
const result = await fetchCodexRpcQuota();
expect(result.windows[0].usedPercent).toBe(37);
expect(child.signalCode).toBe("SIGKILL");
expect(child.stdin.destroyed).toBe(true);
expect(child.stdout.destroyed).toBe(true);
expect(child.stderr.destroyed).toBe(true);
} finally {
if (child.exitCode === null && child.signalCode === null) child.kill("SIGKILL");
}
});
it("classifies app-server refresh-token failures as quota probe auth errors", async () => {
mockSpawn.mockImplementation(() => createChildThatErrorsOnMicrotask(new Error("OAuth failed: refresh token has expired")));
@@ -110,7 +289,7 @@ describe("CodexRpcClient spawn failures", () => {
expect(result.windows).toEqual([
expect.objectContaining({
label: "5h limit",
usedPercent: 50,
usedPercent: 0.5,
resetsAt: "2024-03-22T12:38:31.000Z",
}),
]);
packages/adapters/codex-local/src/server/quota.ts
+85 -24
View File
@@ -11,6 +11,8 @@ import {
const CODEX_USAGE_SOURCE_RPC = "codex-rpc";
const CODEX_USAGE_SOURCE_WHAM = "codex-wham";
const MAX_QUOTA_ERROR_BODY_BYTES = 4_000;
const MAX_RPC_FRAME_BYTES = 64 * 1024;
const RPC_SHUTDOWN_GRACE_MS = 250;
export function codexHomeDir(): string {
const fromEnv = process.env.CODEX_HOME;
@@ -227,7 +229,7 @@ export async function fetchWithTimeout(
const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), ms);
try {
return await fetch(url, { ...init, signal: controller.signal });
return await fetch(url, { ...init, signal: init.signal ? AbortSignal.any([init.signal, controller.signal]) : controller.signal });
} finally {
clearTimeout(timer);
}
@@ -272,20 +274,22 @@ async function readResponseTextPrefix(
}
function normalizeCodexUsedPercent(rawPct: number | null | undefined): number | null {
if (rawPct == null) return null;
return Math.min(100, Math.round(rawPct < 1 ? rawPct * 100 : rawPct));
if (typeof rawPct !== "number" || !Number.isFinite(rawPct)) return null;
// Both RPC and WHAM report percentage points, including values below 1%.
return Math.max(0, Math.min(100, rawPct));
}
export async function fetchCodexQuota(
token: string,
accountId: string | null,
signal?: AbortSignal,
): Promise<QuotaWindow[]> {
const headers: Record<string, string> = {
Authorization: `Bearer ${token}`,
};
if (accountId) headers["ChatGPT-Account-Id"] = accountId;
const resp = await fetchWithTimeout("https://chatgpt.com/backend-api/wham/usage", { headers });
const resp = await fetchWithTimeout("https://chatgpt.com/backend-api/wham/usage", { headers, signal });
if (!resp.ok) {
const message = `chatgpt wham api returned ${resp.status}`;
const responseText = await readResponseTextPrefix(resp);
@@ -470,7 +474,9 @@ type PendingRequest = {
class CodexRpcClient {
private proc = spawn(
"codex",
["-s", "read-only", "-a", "untrusted", "app-server"],
// Only account-read RPCs are sent; no agent turn is started. Keep the
// read-only sandbox and use the approval policy supported by current CLIs.
["-s", "read-only", "-a", "on-request", "app-server"],
{ stdio: ["pipe", "pipe", "pipe"], env: process.env },
);
@@ -478,69 +484,101 @@ class CodexRpcClient {
private buffer = "";
private pending = new Map<number, PendingRequest>();
private stderr = "";
private failure: Error | null = null;
private closed = false;
constructor() {
this.proc.stdout.setEncoding("utf8");
this.proc.stderr.setEncoding("utf8");
this.proc.stdout.on("data", (chunk: string) => this.onStdout(chunk));
this.proc.stderr.on("data", (chunk: string) => {
this.stderr += chunk;
this.stderr = (this.stderr + chunk).slice(-MAX_QUOTA_ERROR_BODY_BYTES);
});
this.proc.on("exit", () => {
for (const request of this.pending.values()) {
clearTimeout(request.timer);
request.reject(new Error(this.stderr.trim() || "codex app-server closed unexpectedly"));
}
this.pending.clear();
this.fail(new Error(this.stderr.trim() || "codex app-server closed unexpectedly"));
});
this.proc.on("error", (err: Error) => {
for (const request of this.pending.values()) {
clearTimeout(request.timer);
request.reject(err);
}
this.pending.clear();
this.proc.on("close", () => {
this.closed = true;
});
this.proc.on("error", (err: Error) => this.fail(err));
// A child may close its input between writing a request and exiting.
this.proc.stdin.on("error", (err: Error) => this.fail(err));
}
private fail(error: Error) {
this.failure ??= error;
this.buffer = "";
for (const request of this.pending.values()) {
clearTimeout(request.timer);
request.reject(this.failure);
}
this.pending.clear();
}
private onStdout(chunk: string) {
if (this.failure) return;
this.buffer += chunk;
while (true) {
const newlineIndex = this.buffer.indexOf("\n");
const frame = newlineIndex < 0 ? this.buffer : this.buffer.slice(0, newlineIndex);
if (Buffer.byteLength(frame, "utf8") > MAX_RPC_FRAME_BYTES) {
this.fail(new Error("codex app-server response exceeded the quota probe limit"));
return;
}
if (newlineIndex < 0) break;
const line = this.buffer.slice(0, newlineIndex).trim();
this.buffer = this.buffer.slice(newlineIndex + 1);
if (!line) continue;
let parsed: Record<string, unknown>;
let value: unknown;
try {
parsed = JSON.parse(line) as Record<string, unknown>;
value = JSON.parse(line);
} catch {
continue;
}
if (!value || typeof value !== "object" || Array.isArray(value)) continue;
const parsed = value as Record<string, unknown>;
const id = typeof parsed.id === "number" ? parsed.id : null;
if (id == null) continue;
const pending = this.pending.get(id);
if (!pending) continue;
this.pending.delete(id);
clearTimeout(pending.timer);
pending.resolve(parsed);
if (parsed.error != null) {
const error = parsed.error as { message?: unknown };
pending.reject(new Error(typeof error.message === "string"
? error.message.slice(0, MAX_QUOTA_ERROR_BODY_BYTES)
: "codex app-server rejected the quota request"));
} else if (!parsed.result || typeof parsed.result !== "object" || Array.isArray(parsed.result)) {
pending.reject(new Error("codex app-server returned an invalid quota response"));
} else {
pending.resolve(parsed);
}
}
}
private request(method: string, params: Record<string, unknown> = {}, timeoutMs = 6_000): Promise<Record<string, unknown>> {
if (this.failure) return Promise.reject(this.failure);
const id = this.nextId++;
const payload = JSON.stringify({ id, method, params }) + "\n";
return new Promise<Record<string, unknown>>((resolve, reject) => {
const timer = setTimeout(() => {
this.pending.delete(id);
reject(new Error(`codex app-server timed out on ${method}`));
this.fail(new Error(`codex app-server timed out on ${method}`));
}, timeoutMs);
this.pending.set(id, { resolve, reject, timer });
this.proc.stdin.write(payload);
this.write(payload);
});
}
private write(payload: string) {
try {
this.proc.stdin.write(payload);
} catch (error) {
this.fail(error instanceof Error ? error : new Error("codex app-server input failed"));
}
}
private notify(method: string, params: Record<string, unknown> = {}) {
this.proc.stdin.write(JSON.stringify({ method, params }) + "\n");
this.write(JSON.stringify({ method, params }) + "\n");
}
async initialize() {
@@ -568,7 +606,30 @@ class CodexRpcClient {
}
async shutdown() {
this.fail(new Error("codex app-server quota probe closed"));
this.proc.stdin.end();
// A failed spawn has no process to signal. Never leave a live probe behind
// when it ignores SIGTERM, including after a request or transport failure.
if (this.closed || this.proc.pid == null) return;
this.proc.kill("SIGTERM");
await this.waitForClose();
if (!this.closed) {
this.proc.kill("SIGKILL");
await this.waitForClose();
}
}
private async waitForClose(): Promise<void> {
if (this.closed) return;
await new Promise<void>((resolve) => {
const done = () => {
clearTimeout(timer);
this.proc.removeListener("close", done);
resolve();
};
const timer = setTimeout(done, RPC_SHUTDOWN_GRACE_MS);
this.proc.once("close", done);
});
}
}
packages/adapters/cursor-local/src/server/execute.test.ts
+34 -2
View File
@@ -138,6 +138,36 @@ function createFreshLeaseSandboxRunner(options: {
}
describe("cursor execute", () => {
it.each([0, 7])("settles legacy step usage only after a clean process exit (%s)", async (exitCode) => {
setPrepareCursorSandboxCommand.mockReset();
setPrepareCursorSandboxCommand.mockImplementation(async (input) => ({
command: input.command, env: input.env, remoteSystemHomeDir: null,
addedPathEntry: null, preferredCommandPath: null,
}));
const root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-cursor-legacy-"));
const command = path.join(root, "agent.sh");
await fs.writeFile(command, `#!/bin/sh
cat >/dev/null
printf '%s\\n' '{"type":"step_finish","part":{"tokens":{"input":20,"output":5},"cost":0.01}}'
exit ${exitCode}
`, { mode: 0o755 });
const onUsage = vi.fn();
try {
const result = await execute({
runId: "run-legacy", agent: { id: "agent-1", companyId: "company-1", name: "Cursor", adapterType: "cursor", adapterConfig: {} },
runtime: { sessionId: null, sessionParams: null, sessionDisplayId: null, taskKey: null },
config: { command, cwd: root }, context: createPromptContextFixture(),
authToken: "fixture-run-token", onLog: async () => {}, onUsage,
});
expect(result.usage).toMatchObject({ inputTokens: 20, outputTokens: 5 });
expect(result.costUsd).toBe(0.01);
expect(result.usageComplete).toBe(exitCode === 0);
expect(onUsage).toHaveBeenCalledWith(expect.objectContaining({ complete: false, costUsd: 0.01 }));
} finally {
await fs.rm(root, { recursive: true, force: true });
}
});
it.each([
{ detail: "Authentication failed", structured: "", expected: "Authentication failed" },
{ detail: "", structured: "", expected: "Cursor exited with code 7" },
@@ -265,6 +295,7 @@ exit 7
await fs.mkdir(workspaceDir, { recursive: true });
await fs.mkdir(remoteWorkspace, { recursive: true });
const preferredAgentScript = `#!/bin/sh
cat >/dev/null
printf '%s\\n' '{"type":"system","subtype":"init","session_id":"cursor-session-fresh-1","model":"auto"}'
printf '%s\\n' '{"type":"assistant","message":{"content":[{"type":"output_text","text":"hello"}]}}'
printf '%s\\n' '{"type":"result","subtype":"success","session_id":"cursor-session-fresh-1","result":"ok"}'
@@ -288,7 +319,7 @@ printf '%s\\n' '{"type":"result","subtype":"success","session_id":"cursor-sessio
finalPreparedCommand = preferredCommandPath;
const runtimeEnv = {
...input.env,
PATH: `${path.join(systemHomeDir, ".local", "bin")}${path.delimiter}${input.env.PATH}`,
PATH: `${path.join(systemHomeDir, ".local", "bin")}${path.delimiter}${input.env.PATH ?? process.env.PATH ?? "/usr/bin:/bin"}`,
};
await fs.mkdir(path.dirname(preferredCommandPath), { recursive: true });
await fs.writeFile(preferredCommandPath, preferredAgentScript);
@@ -312,13 +343,14 @@ printf '%s\\n' '{"type":"result","subtype":"success","session_id":"cursor-sessio
// shell seam faithful to that protocol instead of returning empty stdout
// for every shell command.
const runner = {
execute: async (input: { command: string; args?: string[]; env?: Record<string, string> }) => {
execute: async (input: { command: string; args?: string[]; env?: Record<string, string>; stdin?: string }) => {
runnerState.commands.push(input.command);
// Exercise actual bounded file reads during managed-home restoration;
// reporting empty success for every shell command hides missing bytes.
return runChildProcess(`cursor-fresh-lease-${runnerState.commands.length}`, input.command, input.args ?? [], {
cwd: remoteWorkspace,
env: { ...input.env, PATH: `${input.env?.PATH ?? ""}:/usr/bin:/bin` },
stdin: input.stdin,
timeoutSec: 30,
graceSec: 5,
onLog: async () => {},
packages/adapters/cursor-local/src/server/execute.ts
+35 -3
View File
@@ -1,4 +1,5 @@
import { createProviderStoppedBoundary } from "@paperclipai/adapter-utils/provider-stopped-boundary";
import { createUsageCheckpointLog } from "@paperclipai/adapter-utils/usage-checkpoint";
import fs from "node:fs/promises";
import os from "node:os";
import path from "node:path";
@@ -53,7 +54,7 @@ import {
joinPromptSections,
} from "@paperclipai/adapter-utils/server-utils";
import { DEFAULT_CURSOR_LOCAL_MODEL, SANDBOX_INSTALL_COMMAND } from "../index.js";
import { firstCursorDiagnosticLine, parseCursorJsonl, isCursorUnknownSessionError } from "./parse.js";
import { firstCursorDiagnosticLine, parseCursorJsonl, createCursorJsonlParser, isCursorUnknownSessionError } from "./parse.js";
import { prepareCursorSandboxCommand } from "./remote-command.js";
import { normalizeCursorStreamLine } from "../shared/stream.js";
import { hasCursorTrustBypassArg } from "../shared/trust.js";
@@ -626,11 +627,25 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
});
}
const consumeAccounting = createCursorJsonlParser();
const accountingLog = createUsageCheckpointLog(onLog, ctx.onUsage, stdout => {
const parsed = consumeAccounting(stdout);
const provider = resolveProviderFromModel(model);
const hasCompleteAccounting = parsed.usageComplete || parsed.costUsd != null;
return {
usage: parsed.usageReported ? parsed.usage : undefined,
costUsd: parsed.costUsd,
costStatus: hasCompleteAccounting ? undefined : "unpriced",
usageBasis: "per_run", provider,
biller: resolveCursorBiller(effectiveEnv, billingType, provider), billingType, model,
complete: parsed.sawResult && hasCompleteAccounting,
};
});
let stdoutLineBuffer = "";
const emitNormalizedStdoutLine = async (rawLine: string) => {
const normalized = normalizeCursorStreamLine(rawLine);
if (!normalized.line) return;
await onLog(normalized.stream ?? "stdout", `${normalized.line}\n`);
await accountingLog(normalized.stream ?? "stdout", `${normalized.line}\n`);
};
const flushStdoutChunk = async (chunk: string, finalize = false) => {
const combined = `${stdoutLineBuffer}${chunk}`;
@@ -670,6 +685,7 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
settleRunDisposition: paperclipBridge?.settleRunDisposition,
});
await flushStdoutChunk("", true);
await accountingLog.flush();
return {
proc,
@@ -693,11 +709,24 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
},
clearSessionOnMissingSession = false,
): AdapterExecutionResult => {
const hasCompleteAccounting = attempt.parsed.usageComplete || attempt.parsed.costUsd != null;
const usageComplete = hasCompleteAccounting && (attempt.parsed.sawResult || (attempt.parsed.sawLegacyStep
&& attempt.proc.exitCode === 0 && !attempt.proc.signal && !attempt.proc.timedOut
&& !attempt.parsed.errorMessage));
if (attempt.proc.timedOut) {
return {
exitCode: attempt.proc.exitCode,
signal: attempt.proc.signal,
timedOut: true,
usageComplete,
usage: attempt.parsed.usageReported ? attempt.parsed.usage : undefined,
usageBasis: "per_run",
provider: providerFromModel,
biller: resolveCursorBiller(effectiveEnv, billingType, providerFromModel),
model,
billingType,
costUsd: attempt.parsed.costUsd,
costStatus: hasCompleteAccounting ? undefined : "unpriced",
errorMessage: `Timed out after ${timeoutSec}s`,
clearSession: clearSessionOnMissingSession,
};
@@ -729,6 +758,8 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
exitCode: attempt.proc.exitCode,
signal: attempt.proc.signal,
timedOut: false,
usageComplete,
usageBasis: "per_run",
errorMessage:
(attempt.proc.exitCode ?? 0) === 0
? null
@@ -737,7 +768,7 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
// lost duplex control channel surfaces the typed `duplex_channel_lost`
// code; every other result carries no code here.
errorCode: attempt.proc.errorCode ?? null,
usage: attempt.parsed.usage,
usage: attempt.parsed.usageReported ? attempt.parsed.usage : undefined,
sessionId: resolvedSessionId,
sessionParams: resolvedSessionParams,
sessionDisplayId: resolvedSessionId,
@@ -746,6 +777,7 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
model,
billingType,
costUsd: attempt.parsed.costUsd,
costStatus: hasCompleteAccounting ? undefined : "unpriced",
resultJson: {
stdout: attempt.proc.stdout,
stderr: attempt.proc.stderr,
packages/adapters/cursor-local/src/server/parse.ts
+114 -77
View File
@@ -1,5 +1,5 @@
import { stripVTControlCharacters } from "node:util";
import { asString, asNumber, parseObject, parseJson } from "@paperclipai/adapter-utils/server-utils";
import { asString, parseObject, parseJson } from "@paperclipai/adapter-utils/server-utils";
import { normalizeCursorStreamLine } from "../shared/stream.js";
/** Select diagnostics without mistaking Cursor's trace-file notice for an error. */
@@ -61,102 +61,139 @@ function readSessionId(event: Record<string, unknown>): string | null {
}
export function parseCursorJsonl(stdout: string) {
return createCursorJsonlParser()(stdout);
}
/** Consume complete JSONL records once, retaining protocol accounting state. */
export function createCursorJsonlParser() {
let sessionId: string | null = null;
let sawResult = false;
let sawLegacyStep = false;
let usageReported = false;
let missingUsage = false;
const messages: string[] = [];
let errorMessage: string | null = null;
let totalCostUsd = 0;
let reportedCost = false;
let missingCost = false;
const addCost = (value: unknown) => {
if (typeof value === "number" && Number.isFinite(value) && value >= 0) {
totalCostUsd += value;
reportedCost = true;
} else missingCost = true;
};
const usage = {
inputTokens: 0,
cachedInputTokens: 0,
outputTokens: 0,
};
for (const rawLine of stdout.split(/\r?\n/)) {
const line = normalizeCursorStreamLine(rawLine).line;
if (!line) continue;
const event = parseJson(line);
if (!event) continue;
const foundSession = readSessionId(event);
if (foundSession) sessionId = foundSession;
const type = asString(event.type, "").trim();
if (type === "assistant") {
messages.push(...collectAssistantText(event.message));
continue;
// Missing counters are not reported zeroes. Preserve earlier valid counts,
// but never present their partial sum as the complete invocation total.
const addUsage = (input: unknown, output: unknown, cached: unknown = 0, written: unknown = 0) => {
const counts = [input, output, cached, written];
if (!counts.every(value => typeof value === "number" && Number.isSafeInteger(value) && value >= 0)) {
missingUsage = true;
return;
}
const [inputTokens, outputTokens, cachedInputTokens, cacheWriteTokens] = counts as number[];
const nextInput = usage.inputTokens + inputTokens + cacheWriteTokens;
const nextOutput = usage.outputTokens + outputTokens;
const nextCached = usage.cachedInputTokens + cachedInputTokens;
if (![nextInput, nextOutput, nextCached].every(Number.isSafeInteger)) {
missingUsage = true;
return;
}
usage.inputTokens = nextInput;
usage.outputTokens = nextOutput;
usage.cachedInputTokens = nextCached;
usageReported = true;
};
if (type === "result") {
const usageObj = parseObject(event.usage);
usage.inputTokens += asNumber(
usageObj.input_tokens,
asNumber(usageObj.inputTokens, 0),
);
usage.cachedInputTokens += asNumber(
usageObj.cached_input_tokens,
asNumber(usageObj.cachedInputTokens, asNumber(usageObj.cache_read_input_tokens, 0)),
);
usage.outputTokens += asNumber(
usageObj.output_tokens,
asNumber(usageObj.outputTokens, 0),
);
totalCostUsd += asNumber(event.total_cost_usd, asNumber(event.cost_usd, asNumber(event.cost, 0)));
return (stdout: string) => {
for (const rawLine of stdout.split(/\r?\n/)) {
const line = normalizeCursorStreamLine(rawLine).line;
if (!line) continue;
const isError = event.is_error === true || asString(event.subtype, "").toLowerCase() === "error";
const resultText = asString(event.result, "").trim();
if (resultText && messages.length === 0) {
messages.push(resultText);
const event = parseJson(line);
if (!event) continue;
const foundSession = readSessionId(event);
if (foundSession) sessionId = foundSession;
const type = asString(event.type, "").trim();
if (type === "assistant") {
messages.push(...collectAssistantText(event.message));
continue;
}
if (isError) {
const resultError = asErrorText(event.error ?? event.message ?? event.result).trim();
if (resultError) errorMessage = resultError;
if (type === "result") {
sawResult = true;
const usageObj = parseObject(event.usage);
addUsage(
usageObj.input_tokens ?? usageObj.inputTokens,
usageObj.output_tokens ?? usageObj.outputTokens,
usageObj.cached_input_tokens ?? usageObj.cachedInputTokens ?? usageObj.cache_read_input_tokens,
);
addCost(event.total_cost_usd ?? event.cost_usd ?? event.cost);
const isError = event.is_error === true || asString(event.subtype, "").toLowerCase() === "error";
const resultText = asString(event.result, "").trim();
if (resultText && messages.length === 0) {
messages.push(resultText);
}
if (isError) {
const resultError = asErrorText(event.error ?? event.message ?? event.result).trim();
if (resultError) errorMessage = resultError;
}
continue;
}
continue;
}
if (type === "error") {
const message = asErrorText(event.message ?? event.error ?? event.detail).trim();
if (message) errorMessage = message;
continue;
}
if (type === "system") {
const subtype = asString(event.subtype, "").trim().toLowerCase();
if (subtype === "error") {
if (type === "error") {
const message = asErrorText(event.message ?? event.error ?? event.detail).trim();
if (message) errorMessage = message;
continue;
}
if (type === "system") {
const subtype = asString(event.subtype, "").trim().toLowerCase();
if (subtype === "error") {
const message = asErrorText(event.message ?? event.error ?? event.detail).trim();
if (message) errorMessage = message;
}
continue;
}
// Compatibility with older stream-json shapes.
if (type === "text") {
const part = parseObject(event.part);
const text = asString(part.text, "").trim();
if (text) messages.push(text);
continue;
}
if (type === "step_finish") {
sawLegacyStep = true;
const part = parseObject(event.part);
const tokens = parseObject(part.tokens);
const cache = parseObject(tokens.cache);
addUsage(tokens.input, tokens.output, cache.read, cache.write);
addCost(part.cost);
continue;
}
continue;
}
// Compatibility with older stream-json shapes.
if (type === "text") {
const part = parseObject(event.part);
const text = asString(part.text, "").trim();
if (text) messages.push(text);
continue;
}
if (type === "step_finish") {
const part = parseObject(event.part);
const tokens = parseObject(part.tokens);
const cache = parseObject(tokens.cache);
usage.inputTokens += asNumber(tokens.input, 0);
usage.cachedInputTokens += asNumber(cache.read, 0);
usage.outputTokens += asNumber(tokens.output, 0);
totalCostUsd += asNumber(part.cost, 0);
continue;
}
}
return {
sessionId,
summary: messages.join("\n\n").trim(),
usage,
costUsd: totalCostUsd > 0 ? totalCostUsd : null,
errorMessage,
return {
sawResult,
sawLegacyStep,
usageReported,
usageComplete: usageReported && !missingUsage,
sessionId,
summary: messages.join("\n\n").trim(),
usage: { ...usage },
costUsd: reportedCost && !missingCost ? totalCostUsd : null,
errorMessage,
};
};
}
packages/adapters/gemini-local/src/server/execute.ts
+19 -2
View File
@@ -1,4 +1,5 @@
import { createProviderStoppedBoundary } from "@paperclipai/adapter-utils/provider-stopped-boundary";
import { createUsageCheckpointLog } from "@paperclipai/adapter-utils/usage-checkpoint";
import fs from "node:fs/promises";
import type { Dirent } from "node:fs";
import os from "node:os";
@@ -62,7 +63,7 @@ import {
isGeminiTransientNetworkError,
isGeminiTurnLimitResult,
isGeminiSessionUnrecoverableError,
parseGeminiJsonl,
parseGeminiJsonl, createGeminiJsonlParser,
} from "./parse.js";
import { firstNonEmptyLine } from "./utils.js";
import {
@@ -642,6 +643,11 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
});
}
const consumeAccounting = createGeminiJsonlParser();
const accountingLog = createUsageCheckpointLog(onLog, ctx.onUsage, stdout => {
const parsed = consumeAccounting(stdout);
return { usage: parsed.usage, costUsd: parsed.costUsd, usageBasis: "per_run", provider: "google", biller: "google", billingType, model, complete: parsed.resultEvent !== null };
});
const proc = await runAdapterExecutionTargetProcess(runId, runtimeExecutionTarget, command, args, {
onProcessStopped: providerStop.beginInvocation(),
cwd,
@@ -650,10 +656,11 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
graceSec,
onSpawn,
onRuntimeProgress: ctx.onRuntimeProgress,
onLog,
onLog: accountingLog,
runLogTail: paperclipBridge?.runLogTail,
settleRunDisposition: paperclipBridge?.settleRunDisposition,
});
await accountingLog.flush();
return {
proc,
parsed: parseGeminiJsonl(proc.stdout),
@@ -687,6 +694,14 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
exitCode: attempt.proc.exitCode,
signal: attempt.proc.signal,
timedOut: true,
usageComplete: attempt.parsed.resultEvent !== null,
usage: attempt.parsed.usage,
usageBasis: "per_run",
provider: "google",
biller: "google",
model,
billingType,
costUsd: attempt.parsed.costUsd,
errorMessage: `Timed out after ${timeoutSec}s`,
errorCode: authMeta.requiresAuth
? "gemini_auth_required"
@@ -743,6 +758,8 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
exitCode: attempt.proc.exitCode,
signal: attempt.proc.signal,
timedOut: false,
usageComplete: attempt.parsed.resultEvent !== null,
usageBasis: "per_run",
errorMessage: failed ? fallbackErrorMessage : null,
// Forward the transport-level error code from the run-disposition seam
// first. A lost duplex control channel surfaces the typed
packages/adapters/gemini-local/src/server/parse.test.ts
+38 -2
View File
@@ -1,5 +1,6 @@
import { describe, expect, it } from "vitest";
import {
createGeminiJsonlParser,
detectGeminiAuthRequired,
isGeminiTransientNetworkError,
isGeminiSessionUnrecoverableError,
@@ -88,8 +89,8 @@ describe("parseGeminiJsonl", () => {
expect(result.summary).toBe("hello.");
expect(result.sessionId).toBe("session-abc");
expect(result.errorMessage).toBeNull();
expect(result.usage.inputTokens).toBe(9095);
expect(result.usage.outputTokens).toBe(29);
expect(result.usage.inputTokens).toBe(963);
expect(result.usage.outputTokens).toBe(373);
expect(result.usage.cachedInputTokens).toBe(8132);
});
@@ -213,3 +214,38 @@ describe("isGeminiTransientNetworkError", () => {
).toBe(false);
});
});
describe("Gemini accounting semantics", () => {
it.each([
[0, "result"], [1.25, "result"], [0, "step_finish"], [1.25, "step_finish"], [0, "usage"], [1.25, "usage"],
] as const)("invalidates an earlier $%s price when a later %s adds unpriced usage", (cost, type) => {
const consume = createGeminiJsonlParser();
const priced = JSON.stringify({ type: "step_finish", usage: { input_tokens: 10, output_tokens: 2 }, cost });
const unpriced = JSON.stringify({ type, usage: { input_tokens: 20, output_tokens: 3 } });
expect(consume(priced).costUsd).toBe(cost);
const parsed = consume(unpriced);
expect(parsed.usage).toEqual({ inputTokens: 30, cachedInputTokens: 0, outputTokens: 5 });
expect(parsed.costUsd).toBeNull();
expect(parseGeminiJsonl(`${priced}\n${unpriced}`)).toEqual(parsed);
});
it.each([0, 2.5])("accepts a final reported total of $%s after earlier unpriced usage", (cost) => {
const consume = createGeminiJsonlParser();
expect(consume(JSON.stringify({ type: "step_finish", usage: { input_tokens: 10 } })).costUsd).toBeNull();
expect(consume(JSON.stringify({ type: "result", total_cost_usd: cost })).costUsd).toBe(cost);
});
it("retains a reported price when later events add no usage", () => {
const consume = createGeminiJsonlParser();
consume(JSON.stringify({ type: "step_finish", usage: { input_tokens: 10 }, cost: 1.25 }));
expect(consume(JSON.stringify({ type: "message", role: "assistant", content: "Done" })).costUsd).toBe(1.25);
expect(consume(JSON.stringify({ type: "result", status: "success" })).costUsd).toBe(1.25);
});
it("keeps cache reads disjoint and includes thinking tokens in API metadata", () => {
const parsed = parseGeminiJsonl(JSON.stringify({ type: "result", usageMetadata: { promptTokenCount: 100, cachedContentTokenCount: 80, candidatesTokenCount: 10, thoughtsTokenCount: 5, totalTokenCount: 115 } }));
expect(parsed.usage).toEqual({ inputTokens: 20, cachedInputTokens: 80, outputTokens: 15 });
});
it("preserves explicit zero and distinguishes absent prices", () => {
expect(parseGeminiJsonl(JSON.stringify({ type: "result", total_cost_usd: 0 })).costUsd).toBe(0);
expect(parseGeminiJsonl(JSON.stringify({ type: "result" })).costUsd).toBeNull();
});
});
packages/adapters/gemini-local/src/server/parse.ts
+122 -107
View File
@@ -58,24 +58,27 @@ function accumulateUsage(
const usageMetadata = parseObject(usage.usageMetadata);
const source = Object.keys(usageMetadata).length > 0 ? usageMetadata : usage;
target.inputTokens += asNumber(
source.input_tokens,
asNumber(source.inputTokens, asNumber(source.promptTokenCount, 0)),
);
target.cachedInputTokens += asNumber(
source.cached_input_tokens,
asNumber(
source.cachedInputTokens,
asNumber(source.cachedContentTokenCount, asNumber(source.cached, 0)),
),
);
target.outputTokens += asNumber(
source.output_tokens,
asNumber(source.outputTokens, asNumber(source.candidatesTokenCount, 0)),
);
// Gemini CLI's input_tokens is the full prompt; input is uncached input.
// https://github.com/google-gemini/gemini-cli/blob/main/packages/core/src/output/stream-json-formatter.ts
const cached = asNumber(source.cached_input_tokens,
asNumber(source.cachedInputTokens, asNumber(source.cachedContentTokenCount, asNumber(source.cached, 0))));
const prompt = asNumber(source.input_tokens, asNumber(source.inputTokens, asNumber(source.promptTokenCount, 0)));
const toolInput = asNumber(source.toolUsePromptTokenCount, 0);
target.inputTokens += Math.max(0, asNumber(source.input, prompt - cached)) + toolInput;
target.cachedInputTokens += cached;
// The streaming CLI omits a separate thought count, but includes it in total_tokens.
const response = asNumber(source.output_tokens, asNumber(source.outputTokens, asNumber(source.candidatesTokenCount, 0)));
const total = asNumber(source.total_tokens, asNumber(source.totalTokenCount, 0));
target.outputTokens += Math.max(response + asNumber(source.thoughtsTokenCount, 0), total - prompt - toolInput);
}
export function parseGeminiJsonl(stdout: string) {
return createGeminiJsonlParser()(stdout);
}
/** Consume complete JSONL records once, retaining protocol accounting state. */
export function createGeminiJsonlParser() {
let sessionId: string | null = null;
const messages: string[] = [];
let errorMessage: string | null = null;
@@ -87,115 +90,127 @@ export function parseGeminiJsonl(stdout: string) {
cachedInputTokens: 0,
outputTokens: 0,
};
const recordAccounting = (event: Record<string, unknown>, usageRaw: unknown) => {
const before = { ...usage };
accumulateUsage(usage, usageRaw);
const reportedCost = [event.total_cost_usd, event.cost_usd, event.cost]
.find((value) => typeof value === "number" && Number.isFinite(value) && value >= 0);
if (typeof reportedCost === "number") costUsd = reportedCost;
else if (usage.inputTokens !== before.inputTokens || usage.cachedInputTokens !== before.cachedInputTokens || usage.outputTokens !== before.outputTokens) {
// An earlier price, including explicit zero, cannot cover new unpriced
// usage. Text-only or terminal events with no new usage leave it intact.
costUsd = null;
}
};
for (const rawLine of stdout.split(/\r?\n/)) {
const line = rawLine.trim();
if (!line) continue;
return (stdout: string) => {
for (const rawLine of stdout.split(/\r?\n/)) {
const line = rawLine.trim();
if (!line) continue;
const event = parseJson(line);
if (!event) continue;
const event = parseJson(line);
if (!event) continue;
const foundSessionId = readSessionId(event);
if (foundSessionId) sessionId = foundSessionId;
const foundSessionId = readSessionId(event);
if (foundSessionId) sessionId = foundSessionId;
const type = asString(event.type, "").trim();
const type = asString(event.type, "").trim();
if (type === "assistant") {
messages.push(...collectMessageText(event.message));
const messageObj = parseObject(event.message);
const content = Array.isArray(messageObj.content) ? messageObj.content : [];
for (const partRaw of content) {
const part = parseObject(partRaw);
if (asString(part.type, "").trim() === "question") {
question = {
prompt: asString(part.prompt, "").trim(),
choices: (Array.isArray(part.choices) ? part.choices : []).map((choiceRaw) => {
const choice = parseObject(choiceRaw);
return {
key: asString(choice.key, "").trim(),
label: asString(choice.label, "").trim(),
description: asString(choice.description, "").trim() || undefined,
};
}),
};
break; // only one question per message
if (type === "assistant") {
messages.push(...collectMessageText(event.message));
const messageObj = parseObject(event.message);
const content = Array.isArray(messageObj.content) ? messageObj.content : [];
for (const partRaw of content) {
const part = parseObject(partRaw);
if (asString(part.type, "").trim() === "question") {
question = {
prompt: asString(part.prompt, "").trim(),
choices: (Array.isArray(part.choices) ? part.choices : []).map((choiceRaw) => {
const choice = parseObject(choiceRaw);
return {
key: asString(choice.key, "").trim(),
label: asString(choice.label, "").trim(),
description: asString(choice.description, "").trim() || undefined,
};
}),
};
break; // only one question per message
}
}
continue;
}
continue;
}
// Gemini CLI v0.38+ stream-json schema emits assistant turns as:
// {"type":"message","role":"assistant","content":"...","delta":true}
// These are discrete final messages (one per assistant turn), not
// cumulative streaming tokens, so collecting all of them produces the
// expected concatenated turn-by-turn summary rather than duplicated text.
if (type === "message") {
const role = asString(event.role, "").trim().toLowerCase();
if (role === "assistant") {
messages.push(...collectMessageText(event.content));
// Gemini CLI v0.38+ stream-json schema emits assistant turns as:
// {"type":"message","role":"assistant","content":"...","delta":true}
// These are discrete final messages (one per assistant turn), not
// cumulative streaming tokens, so collecting all of them produces the
// expected concatenated turn-by-turn summary rather than duplicated text.
if (type === "message") {
const role = asString(event.role, "").trim().toLowerCase();
if (role === "assistant") {
messages.push(...collectMessageText(event.content));
}
continue;
}
continue;
}
if (type === "result") {
resultEvent = event;
accumulateUsage(usage, event.usage ?? event.usageMetadata ?? event.stats);
const resultText =
asString(event.result, "").trim() ||
asString(event.text, "").trim() ||
asString(event.response, "").trim();
if (resultText && messages.length === 0) messages.push(resultText);
costUsd = asNumber(event.total_cost_usd, asNumber(event.cost_usd, asNumber(event.cost, costUsd ?? 0))) || costUsd;
const status = asString(event.status, "").toLowerCase();
const isError =
event.is_error === true ||
asString(event.subtype, "").toLowerCase() === "error" ||
status === "error" ||
status === "failed";
if (isError) {
const text = asErrorText(event.error ?? event.message ?? event.result).trim();
if (text) errorMessage = text;
if (type === "result") {
resultEvent = event;
recordAccounting(event, event.usage ?? event.usageMetadata ?? event.stats);
const resultText =
asString(event.result, "").trim() ||
asString(event.text, "").trim() ||
asString(event.response, "").trim();
if (resultText && messages.length === 0) messages.push(resultText);
const status = asString(event.status, "").toLowerCase();
const isError =
event.is_error === true ||
asString(event.subtype, "").toLowerCase() === "error" ||
status === "error" ||
status === "failed";
if (isError) {
const text = asErrorText(event.error ?? event.message ?? event.result).trim();
if (text) errorMessage = text;
}
continue;
}
continue;
}
if (type === "error") {
const text = asErrorText(event.error ?? event.message ?? event.detail).trim();
if (text) errorMessage = text;
continue;
}
if (type === "system") {
const subtype = asString(event.subtype, "").trim().toLowerCase();
if (subtype === "error") {
if (type === "error") {
const text = asErrorText(event.error ?? event.message ?? event.detail).trim();
if (text) errorMessage = text;
continue;
}
if (type === "system") {
const subtype = asString(event.subtype, "").trim().toLowerCase();
if (subtype === "error") {
const text = asErrorText(event.error ?? event.message ?? event.detail).trim();
if (text) errorMessage = text;
}
continue;
}
if (type === "text") {
const part = parseObject(event.part);
const text = asString(part.text, "").trim();
if (text) messages.push(text);
continue;
}
if (type === "step_finish" || event.usage || event.usageMetadata) {
recordAccounting(event, event.usage ?? event.usageMetadata);
continue;
}
continue;
}
if (type === "text") {
const part = parseObject(event.part);
const text = asString(part.text, "").trim();
if (text) messages.push(text);
continue;
}
if (type === "step_finish" || event.usage || event.usageMetadata) {
accumulateUsage(usage, event.usage ?? event.usageMetadata);
costUsd = asNumber(event.total_cost_usd, asNumber(event.cost_usd, asNumber(event.cost, costUsd ?? 0))) || costUsd;
continue;
}
}
return {
sessionId,
summary: messages.join("\n\n").trim(),
usage,
costUsd,
errorMessage,
resultEvent,
question,
return {
sessionId,
summary: messages.join("\n\n").trim(),
usage: { ...usage },
costUsd,
errorMessage,
resultEvent,
question,
};
};
}
packages/adapters/kimi-local/src/server/execute.test.ts
+3
View File
@@ -326,6 +326,9 @@ describe("kimi_local execute", () => {
const result = await execute(makeContext(root, { config: { cwd: root, timeoutSec: 5 } }));
expect(result.timedOut).toBe(true);
expect(result.usageComplete).toBe(true);
expect(result.costStatus).toBe("unpriced");
expect(result.costUsd).toBeNull();
expect(result.errorMessage).toContain("5s");
});
packages/adapters/kimi-local/src/server/execute.ts
+13
View File
@@ -648,6 +648,16 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
exitCode: attempt.proc.exitCode,
signal: attempt.proc.signal,
timedOut: true,
// This stream protocol has no usage receipt to recover after exit.
// Close collection with an explicitly unknown price, never a free run.
usageComplete: true,
costStatus: "unpriced",
usageBasis: "per_run",
provider: "moonshot",
biller: "moonshot",
model: model || null,
billingType,
costUsd: null,
errorMessage: `Timed out after ${timeoutSec}s`,
errorCode: authMeta.requiresAuth
? "kimi_auth_required"
@@ -704,6 +714,9 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
exitCode: attempt.proc.exitCode,
signal: attempt.proc.signal,
timedOut: false,
usageComplete: true,
costStatus: "unpriced",
usageBasis: "per_run",
errorMessage: failed ? fallbackErrorMessage : null,
// Forward the transport-level error code from the run-disposition seam
// first. A lost duplex control channel surfaces the typed
packages/adapters/opencode-local/src/server/execute.ts
+45 -13
View File
@@ -1,4 +1,5 @@
import { createProviderStoppedBoundary } from "@paperclipai/adapter-utils/provider-stopped-boundary";
import { createUsageCheckpointLog } from "@paperclipai/adapter-utils/usage-checkpoint";
import fs from "node:fs/promises";
import os from "node:os";
import path from "node:path";
@@ -52,7 +53,7 @@ import {
readPaperclipIssueWorkModeFromContext,
resolveLegacyPaperclipDesiredSkillNames,
} from "@paperclipai/adapter-utils/server-utils";
import { isOpenCodeUnknownSessionError, parseOpenCodeJsonl } from "./parse.js";
import { isOpenCodeUnknownSessionError, parseOpenCodeJsonl, createOpenCodeJsonlParser } from "./parse.js";
import {
ensureOpenCodeModelConfiguredAndAvailable,
isTruthyEnvFlag,
@@ -83,7 +84,7 @@ function parseModelProvider(model: string | null): string | null {
}
function resolveOpenCodeBiller(env: Record<string, string>, provider: string | null): string {
return inferOpenAiCompatibleBiller(env, null) ?? provider ?? "unknown";
return provider === "openai" ? inferOpenAiCompatibleBiller(env, "openai") ?? "unknown" : provider ?? "unknown";
}
const REMOTE_OPENCODE_MODELS_PROBE_DEFAULT_TIMEOUT_SEC = 20;
@@ -645,6 +646,16 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
});
}
const consumeAccounting = createOpenCodeJsonlParser();
let hasAccounting = false;
const accountingLog = createUsageCheckpointLog(onLog, ctx.onUsage ?? (async () => {}), stdout => {
hasAccounting = true;
const parsed = consumeAccounting(stdout);
const provider = parseModelProvider(model || null);
return { usage: parsed.usageReported ? parsed.usage : undefined, costUsd: parsed.costUsd,
costStatus: parsed.usageComplete || parsed.costUsd != null ? undefined : "unpriced",
usageBasis: "per_run", provider, biller: resolveOpenCodeBiller(runtimeEnv, provider), billingType: "unknown", model, complete: false };
});
const proc = await runAdapterExecutionTargetProcess(runId, runtimeExecutionTarget, command, args, {
onProcessStopped: providerStop.beginInvocation(),
cwd,
@@ -654,15 +665,27 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
graceSec,
onSpawn,
onRuntimeProgress: ctx.onRuntimeProgress,
onLog,
onLog: accountingLog,
runLogTail: paperclipBridge?.runLogTail,
settleRunDisposition: paperclipBridge?.settleRunDisposition,
});
return {
proc,
rawStderr: proc.stderr,
parsed: parseOpenCodeJsonl(proc.stdout),
};
// Parse any unterminated final record before deciding whether its usage
// is complete. A clean exit alone cannot turn absent counters into zero.
await accountingLog.flush();
const retainedAccounting = consumeAccounting("");
await accountingLog.flush({ complete: proc.exitCode === 0 && !proc.timedOut && !proc.signal
&& (retainedAccounting.usageComplete || retainedAccounting.costUsd != null) });
// Display output is capped by the process transport. Keep accounting
// from the full stream, including when no checkpoint callback is installed.
const parsed = parseOpenCodeJsonl(proc.stdout);
if (hasAccounting) {
const retained = consumeAccounting("");
parsed.usage = retained.usage;
parsed.usageReported = retained.usageReported;
parsed.usageComplete = retained.usageComplete;
parsed.costUsd = retained.costUsd;
}
return { proc, rawStderr: proc.stderr, parsed };
};
const toResult = (
@@ -678,6 +701,15 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
exitCode: attempt.proc.exitCode,
signal: attempt.proc.signal,
timedOut: true,
usageComplete: false,
usage: attempt.parsed.usageReported ? attempt.parsed.usage : undefined,
usageBasis: "per_run",
provider: parseModelProvider(model || null),
biller: resolveOpenCodeBiller(runtimeEnv, parseModelProvider(model || null)),
model,
billingType: "unknown",
costUsd: attempt.parsed.costUsd,
costStatus: attempt.parsed.usageComplete || attempt.parsed.costUsd != null ? undefined : "unpriced",
errorMessage: `Timed out after ${timeoutSec}s`,
clearSession: clearSessionOnMissingSession,
};
@@ -715,16 +747,15 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
exitCode: synthesizedExitCode,
signal: attempt.proc.signal,
timedOut: false,
usageComplete: attempt.proc.exitCode === 0 && !attempt.proc.signal
&& (attempt.parsed.usageComplete || attempt.parsed.costUsd != null),
usageBasis: "per_run",
errorMessage: (synthesizedExitCode ?? 0) === 0 ? null : fallbackErrorMessage,
// Forward the transport-level error code from the run-disposition seam.
// A lost duplex control channel surfaces the typed `duplex_channel_lost`
// code; every other result carries no code here.
errorCode: attempt.proc.errorCode ?? null,
usage: {
inputTokens: attempt.parsed.usage.inputTokens,
outputTokens: attempt.parsed.usage.outputTokens,
cachedInputTokens: attempt.parsed.usage.cachedInputTokens,
},
usage: attempt.parsed.usageReported ? attempt.parsed.usage : undefined,
sessionId: resolvedSessionId,
sessionParams: resolvedSessionParams,
sessionDisplayId: resolvedSessionId,
@@ -733,6 +764,7 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
model: modelId,
billingType: "unknown",
costUsd: attempt.parsed.costUsd,
costStatus: attempt.parsed.usageComplete || attempt.parsed.costUsd != null ? undefined : "unpriced",
resultJson: {
stdout: attempt.proc.stdout,
stderr: attempt.proc.stderr,
packages/adapters/opencode-local/src/server/parse.test.ts
+20
View File
@@ -75,3 +75,23 @@ describe("parseOpenCodeJsonl", () => {
expect(isOpenCodeUnknownSessionError("all good", "")).toBe(false);
});
});
describe("OpenCode price availability", () => {
it("distinguishes missing cost from reported zero and counts cache writes", () => {
const event = { type: "step_finish", part: { tokens: { input: 10, output: 3, cache: { read: 100, write: 20 } } } };
expect(parseOpenCodeJsonl(JSON.stringify(event))).toMatchObject({ costUsd: null, usage: { inputTokens: 30, cachedInputTokens: 100, outputTokens: 3 } });
expect(parseOpenCodeJsonl(JSON.stringify({ ...event, part: { ...event.part, cost: 0 } })).costUsd).toBe(0);
expect(parseOpenCodeJsonl([event, { ...event, part: { ...event.part, cost: 1 } }].map((row) => JSON.stringify(row)).join("\n")).costUsd).toBeNull();
});
});
describe("OpenCode reported usage completeness", () => {
it.each([undefined, {}, { input: 1 }, { input: -1, output: 2 }, { input: 1, output: 0.5 }, { input: 1, output: 2, cache: { read: -1 } }, { input: 1, output: 2, cache: { read: null } }, { input: 1, output: 2, reasoning: null }])("preserves unknown usage for incomplete counters %j", tokens => {
expect(parseOpenCodeJsonl(JSON.stringify({ type: "step_finish", part: { tokens } }))).toMatchObject({ usageReported: false, usageComplete: false, costUsd: null });
});
it("accepts explicit zeros but keeps a later missing step incomplete", () => {
const line = JSON.stringify({ type: "step_finish", part: { tokens: { input: 0, output: 0 } } });
expect(parseOpenCodeJsonl(line)).toMatchObject({ usageReported: true, usageComplete: true, usage: { inputTokens: 0, outputTokens: 0 }, costUsd: null });
expect(parseOpenCodeJsonl(line + '\n' + JSON.stringify({ type: "step_finish" }))).toMatchObject({ usageReported: true, usageComplete: false });
});
});
packages/adapters/opencode-local/src/server/parse.ts
+65 -47
View File
@@ -1,4 +1,4 @@
import { asNumber, asString, parseJson, parseObject } from "@paperclipai/adapter-utils/server-utils";
import { asString, parseJson, parseObject } from "@paperclipai/adapter-utils/server-utils";
function errorText(value: unknown): string {
if (typeof value === "string") return value;
@@ -20,6 +20,11 @@ function errorText(value: unknown): string {
}
export function parseOpenCodeJsonl(stdout: string) {
return createOpenCodeJsonlParser()(stdout);
}
/** Consume complete JSONL records once, retaining protocol accounting state. */
export function createOpenCodeJsonlParser() {
let sessionId: string | null = null;
const messages: string[] = [];
const errors: string[] = [];
@@ -29,62 +34,75 @@ export function parseOpenCodeJsonl(stdout: string) {
cachedInputTokens: 0,
outputTokens: 0,
};
let costUsd = 0;
let costUsd: number | null = null;
let missingCost = false;
let usageReported = false;
let missingUsage = false;
for (const rawLine of stdout.split(/\r?\n/)) {
const line = rawLine.trim();
if (!line) continue;
return (stdout: string) => {
for (const rawLine of stdout.split(/\r?\n/)) {
const line = rawLine.trim();
if (!line) continue;
const event = parseJson(line);
if (!event) continue;
const event = parseJson(line);
if (!event) continue;
const currentSessionId = asString(event.sessionID, "").trim();
if (currentSessionId) sessionId = currentSessionId;
const currentSessionId = asString(event.sessionID, "").trim();
if (currentSessionId) sessionId = currentSessionId;
const type = asString(event.type, "");
const type = asString(event.type, "");
if (type === "text") {
const part = parseObject(event.part);
const text = asString(part.text, "").trim();
if (text) messages.push(text);
continue;
}
if (type === "text") {
const part = parseObject(event.part);
const text = asString(part.text, "").trim();
if (text) messages.push(text);
continue;
}
if (type === "step_finish") {
const part = parseObject(event.part);
const tokens = parseObject(part.tokens);
const cache = parseObject(tokens.cache);
usage.inputTokens += asNumber(tokens.input, 0);
usage.cachedInputTokens += asNumber(cache.read, 0);
usage.outputTokens += asNumber(tokens.output, 0) + asNumber(tokens.reasoning, 0);
costUsd += asNumber(part.cost, 0);
continue;
}
if (type === "step_finish") {
const part = parseObject(event.part);
const tokens = parseObject(part.tokens);
const cache = parseObject(tokens.cache);
const counts = [tokens.input, tokens.output, cache.read === undefined ? 0 : cache.read, cache.write === undefined ? 0 : cache.write, tokens.reasoning === undefined ? 0 : tokens.reasoning];
if (counts.every(value => typeof value === "number" && Number.isSafeInteger(value) && value >= 0)) {
const [input, output, cached, written, reasoning] = counts as number[];
const next = { inputTokens: usage.inputTokens + input + written,
outputTokens: usage.outputTokens + output + reasoning, cachedInputTokens: usage.cachedInputTokens + cached };
if (Object.values(next).every(Number.isSafeInteger)) { Object.assign(usage, next); usageReported = true; }
else missingUsage = true;
} else missingUsage = true;
if (typeof part.cost === "number" && Number.isFinite(part.cost) && part.cost >= 0) costUsd = (costUsd ?? 0) + part.cost;
else missingCost = true;
continue;
}
if (type === "tool_use") {
const part = parseObject(event.part);
const state = parseObject(part.state);
if (asString(state.status, "") === "error") {
const text = asString(state.error, "").trim();
if (text) toolErrors.push(text);
if (type === "tool_use") {
const part = parseObject(event.part);
const state = parseObject(part.state);
if (asString(state.status, "") === "error") {
const text = asString(state.error, "").trim();
if (text) toolErrors.push(text);
}
continue;
}
if (type === "error") {
const text = errorText(event.error ?? event.message).trim();
if (text) errors.push(text);
continue;
}
continue;
}
if (type === "error") {
const text = errorText(event.error ?? event.message).trim();
if (text) errors.push(text);
continue;
}
}
return {
sessionId,
summary: messages.join("\n\n").trim(),
usage,
costUsd,
errorMessage: errors.length > 0 ? errors.join("\n") : null,
toolErrors,
return {
sessionId,
summary: messages.join("\n\n").trim(),
usage: { ...usage },
usageReported,
usageComplete: usageReported && !missingUsage,
costUsd: missingCost ? null : costUsd,
errorMessage: errors.length > 0 ? errors.join("\n") : null,
toolErrors,
};
};
}
packages/adapters/pi-local/src/server/execute.ts
+25 -8
View File
@@ -1,4 +1,5 @@
import { createProviderStoppedBoundary } from "@paperclipai/adapter-utils/provider-stopped-boundary";
import { createUsageCheckpointLog } from "@paperclipai/adapter-utils/usage-checkpoint";
import fs from "node:fs/promises";
import os from "node:os";
import path from "node:path";
@@ -54,7 +55,7 @@ import {
runChildProcess,
} from "@paperclipai/adapter-utils/server-utils";
import { shellQuote } from "@paperclipai/adapter-utils/ssh";
import { isPiUnknownSessionError, parsePiJsonl } from "./parse.js";
import { isPiUnknownSessionError, parsePiJsonl, createPiJsonlParser } from "./parse.js";
import { ensurePiModelConfiguredAndAvailable } from "./models.js";
import { preparePiRuntimeConfig } from "./runtime-config.js";
import { SANDBOX_INSTALL_COMMAND } from "../index.js";
@@ -141,7 +142,7 @@ async function buildPiSkillsDir(config: Record<string, unknown>): Promise<string
}
function resolvePiBiller(env: Record<string, string>, provider: string | null): string {
return inferOpenAiCompatibleBiller(env, null) ?? provider ?? "unknown";
return provider === "openai" ? inferOpenAiCompatibleBiller(env, "openai") ?? "unknown" : provider ?? "unknown";
}
async function ensureSessionsDir(): Promise<string> {
@@ -736,6 +737,13 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
}
};
const consumeAccounting = createPiJsonlParser();
let hasAccounting = false;
const accountingLog = createUsageCheckpointLog(bufferedOnLog, ctx.onUsage ?? (async () => {}), stdout => {
hasAccounting = true;
const parsed = consumeAccounting(stdout);
return { usage: parsed.usage, costUsd: parsed.usage.costUsd, usageBasis: "per_run", provider, biller: resolvePiBiller(runtimeEnv, provider), billingType: "unknown", model, complete: parsed.sawAgentEnd };
});
const proc = await runAdapterExecutionTargetProcess(runId, runtimeExecutionTarget, command, args, {
onProcessStopped: providerStop.beginInvocation(),
cwd,
@@ -744,21 +752,26 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
graceSec,
onSpawn,
onRuntimeProgress: ctx.onRuntimeProgress,
onLog: bufferedOnLog,
onLog: accountingLog,
runLogTail: paperclipBridge?.runLogTail,
settleRunDisposition: paperclipBridge?.settleRunDisposition,
});
await accountingLog.flush();
// Flush any remaining buffer content
if (stdoutBuffer) {
await onLog("stdout", stdoutBuffer);
}
return {
proc,
rawStderr: proc.stderr,
parsed: parsePiJsonl(proc.stdout),
};
// Display output is capped by the process transport. Keep accounting
// from the full stream, including when no checkpoint callback is installed.
const parsed = parsePiJsonl(proc.stdout);
if (hasAccounting) {
const retained = consumeAccounting("");
parsed.usage = retained.usage;
parsed.sawAgentEnd = retained.sawAgentEnd;
}
return { proc, rawStderr: proc.stderr, parsed };
};
const toResult = (
@@ -774,6 +787,8 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
exitCode: attempt.proc.exitCode,
signal: attempt.proc.signal,
timedOut: true,
usageComplete: attempt.parsed.sawAgentEnd,
usage: attempt.parsed.usage, usageBasis: "per_run", provider, biller: resolvePiBiller(runtimeEnv, provider), model, billingType: "unknown", costUsd: attempt.parsed.usage.costUsd,
errorMessage: `Timed out after ${timeoutSec}s`,
clearSession: clearSessionOnMissingSession,
};
@@ -805,6 +820,8 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
exitCode: effectiveExitCode,
signal: attempt.proc.signal,
timedOut: false,
usageComplete: attempt.parsed.sawAgentEnd,
usageBasis: "per_run",
errorMessage: (effectiveExitCode ?? 0) === 0 ? null : fallbackErrorMessage,
// Forward the transport-level error code from the run-disposition seam.
// A lost duplex control channel surfaces the typed `duplex_channel_lost`
packages/adapters/pi-local/src/server/parse.test.ts
+9
View File
@@ -288,3 +288,12 @@ describe("terminal provider failures", () => {
.toEqual(["Pi provider request failed."]);
});
});
describe("Pi price availability", () => {
it("retains unpriced usage and counts cache writes without treating a missing price as free", () => {
const event = { type: "turn_end", message: { role: "assistant", content: [], usage: { input: 10, output: 3, cacheRead: 100, cacheWrite: 20 } } };
expect(parsePiJsonl(JSON.stringify(event)).usage).toEqual({ inputTokens: 30, cachedInputTokens: 100, outputTokens: 3, costUsd: null });
event.message.usage = { ...event.message.usage, cost: { total: 0 } } as typeof event.message.usage;
expect(parsePiJsonl(JSON.stringify(event)).usage.costUsd).toBe(0);
});
});
packages/adapters/pi-local/src/server/parse.ts
+170 -160
View File
@@ -1,6 +1,7 @@
import { asNumber, asString, parseJson, parseObject } from "@paperclipai/adapter-utils/server-utils";
interface ParsedPiOutput {
sawAgentEnd: boolean;
sessionId: string | null;
messages: string[];
errors: string[];
@@ -8,7 +9,7 @@ interface ParsedPiOutput {
inputTokens: number;
outputTokens: number;
cachedInputTokens: number;
costUsd: number;
costUsd: number | null;
};
finalMessage: string | null;
toolCalls: Array<{ toolCallId: string; toolName: string; args: unknown; result: string | null; isError: boolean }>;
@@ -28,8 +29,14 @@ function extractTextContent(content: string | Array<{ type: string; text?: strin
.join("");
}
export function parsePiJsonl(stdout: string): ParsedPiOutput {
export function parsePiJsonl(stdout: string) {
return createPiJsonlParser()(stdout);
}
/** Consume complete JSONL records once, retaining protocol accounting state. */
export function createPiJsonlParser() {
const result: ParsedPiOutput = {
sawAgentEnd: false,
sessionId: null,
messages: [],
errors: [],
@@ -37,198 +44,201 @@ export function parsePiJsonl(stdout: string): ParsedPiOutput {
inputTokens: 0,
outputTokens: 0,
cachedInputTokens: 0,
costUsd: 0,
costUsd: null,
},
finalMessage: null,
toolCalls: [],
};
let missingCost = false;
function addCost(value: unknown) {
if (typeof value === "number" && Number.isFinite(value) && value >= 0) result.usage.costUsd = (result.usage.costUsd ?? 0) + value;
else missingCost = true;
}
let currentToolCall: { toolCallId: string; toolName: string; args: unknown } | null = null;
for (const rawLine of stdout.split(/\r?\n/)) {
const line = rawLine.trim();
if (!line) continue;
return (stdout: string) => {
for (const rawLine of stdout.split(/\r?\n/)) {
const line = rawLine.trim();
if (!line) continue;
const event = parseJson(line);
if (!event) continue;
const event = parseJson(line);
if (!event) continue;
const eventType = asString(event.type, "");
const eventType = asString(event.type, "");
// Pi can exit successfully after a provider failure. The terminal assistant
// message carries that failure in both message_end and turn_end envelopes.
const terminalMessages = eventType === "agent_end"
? (Array.isArray(event.messages) ? event.messages : [])
: eventType === "message_end" || eventType === "turn_end"
? [event.message]
: [];
for (const rawMessage of terminalMessages) {
const message = asRecord(rawMessage);
if (message?.role !== "assistant" || message.stopReason !== "error") continue;
const error = asString(message.errorMessage, "").trim() || "Pi provider request failed.";
if (!result.errors.includes(error)) result.errors.push(error);
}
// RPC protocol messages - skip these (internal implementation detail)
if (eventType === "response" || eventType === "extension_ui_request" || eventType === "extension_ui_response" || eventType === "extension_error") {
continue;
}
// Agent lifecycle
if (eventType === "agent_start") {
continue;
}
if (eventType === "agent_end") {
const messages = event.messages as Array<Record<string, unknown>> | undefined;
if (messages && messages.length > 0) {
const lastMessage = messages[messages.length - 1];
if (lastMessage?.role === "assistant") {
const content = lastMessage.content as string | Array<{ type: string; text?: string }>;
result.finalMessage = extractTextContent(content);
}
// Pi can exit successfully after a provider failure. The terminal assistant
// message carries that failure in both message_end and turn_end envelopes.
const terminalMessages = eventType === "agent_end"
? (Array.isArray(event.messages) ? event.messages : [])
: eventType === "message_end" || eventType === "turn_end"
? [event.message]
: [];
for (const rawMessage of terminalMessages) {
const message = asRecord(rawMessage);
if (message?.role !== "assistant" || message.stopReason !== "error") continue;
const error = asString(message.errorMessage, "").trim() || "Pi provider request failed.";
if (!result.errors.includes(error)) result.errors.push(error);
}
continue;
}
if (eventType === "auto_retry_end") {
const succeeded = event.success === true;
if (!succeeded) {
const finalError = asString(event.finalError, "").trim();
result.errors.push(finalError || "Pi exhausted automatic retries without producing a response.");
// RPC protocol messages - skip these (internal implementation detail)
if (eventType === "response" || eventType === "extension_ui_request" || eventType === "extension_ui_response" || eventType === "extension_error") {
continue;
}
continue;
}
// Turn lifecycle
if (eventType === "turn_start") {
continue;
}
// Agent lifecycle
if (eventType === "agent_start") {
continue;
}
if (eventType === "turn_end") {
const message = asRecord(event.message);
if (message) {
const content = message.content as string | Array<{ type: string; text?: string }>;
const text = extractTextContent(content);
if (text) {
result.finalMessage = text;
result.messages.push(text);
}
// Extract usage and cost from assistant message
const usage = asRecord(message.usage);
if (usage) {
result.usage.inputTokens += asNumber(usage.input, 0);
result.usage.outputTokens += asNumber(usage.output, 0);
result.usage.cachedInputTokens += asNumber(usage.cacheRead, 0);
// Pi stores cost in usage.cost.total (and broken down in usage.cost.input, etc.)
const cost = asRecord(usage.cost);
if (cost) {
result.usage.costUsd += asNumber(cost.total, 0);
if (eventType === "agent_end") {
result.sawAgentEnd = true;
const messages = event.messages as Array<Record<string, unknown>> | undefined;
if (messages && messages.length > 0) {
const lastMessage = messages[messages.length - 1];
if (lastMessage?.role === "assistant") {
const content = lastMessage.content as string | Array<{ type: string; text?: string }>;
result.finalMessage = extractTextContent(content);
}
}
continue;
}
// Tool results are in toolResults array
const toolResults = event.toolResults as Array<Record<string, unknown>> | undefined;
if (toolResults) {
for (const tr of toolResults) {
const toolCallId = asString(tr.toolCallId, "");
const content = tr.content;
const isError = tr.isError === true;
// Find matching tool call by toolCallId
const existingCall = result.toolCalls.find((tc) => tc.toolCallId === toolCallId);
if (existingCall) {
existingCall.result = typeof content === "string" ? content : JSON.stringify(content);
existingCall.isError = isError;
if (eventType === "auto_retry_end") {
const succeeded = event.success === true;
if (!succeeded) {
const finalError = asString(event.finalError, "").trim();
result.errors.push(finalError || "Pi exhausted automatic retries without producing a response.");
}
continue;
}
// Turn lifecycle
if (eventType === "turn_start") {
continue;
}
if (eventType === "turn_end") {
const message = asRecord(event.message);
if (message) {
const content = message.content as string | Array<{ type: string; text?: string }>;
const text = extractTextContent(content);
if (text) {
result.finalMessage = text;
result.messages.push(text);
}
// Extract usage and cost from assistant message
const usage = asRecord(message.usage);
if (usage) {
result.usage.inputTokens += asNumber(usage.input, 0) + asNumber(usage.cacheWrite, 0);
result.usage.outputTokens += asNumber(usage.output, 0);
result.usage.cachedInputTokens += asNumber(usage.cacheRead, 0);
// Pi stores cost in usage.cost.total (and broken down in usage.cost.input, etc.)
const cost = asRecord(usage.cost);
addCost(cost?.total);
}
}
}
continue;
}
// Message updates (streaming)
if (eventType === "message_update") {
const assistantEvent = asRecord(event.assistantMessageEvent);
if (assistantEvent) {
const msgType = asString(assistantEvent.type, "");
if (msgType === "text_delta") {
const delta = asString(assistantEvent.delta, "");
if (delta) {
// Append to last message or create new
if (result.messages.length === 0) {
result.messages.push(delta);
} else {
result.messages[result.messages.length - 1] += delta;
// Tool results are in toolResults array
const toolResults = event.toolResults as Array<Record<string, unknown>> | undefined;
if (toolResults) {
for (const tr of toolResults) {
const toolCallId = asString(tr.toolCallId, "");
const content = tr.content;
const isError = tr.isError === true;
// Find matching tool call by toolCallId
const existingCall = result.toolCalls.find((tc) => tc.toolCallId === toolCallId);
if (existingCall) {
existingCall.result = typeof content === "string" ? content : JSON.stringify(content);
existingCall.isError = isError;
}
}
}
continue;
}
continue;
}
if (eventType === "error") {
const message = asString(event.message, "").trim();
if (message) {
result.errors.push(message);
// Message updates (streaming)
if (eventType === "message_update") {
const assistantEvent = asRecord(event.assistantMessageEvent);
if (assistantEvent) {
const msgType = asString(assistantEvent.type, "");
if (msgType === "text_delta") {
const delta = asString(assistantEvent.delta, "");
if (delta) {
// Append to last message or create new
if (result.messages.length === 0) {
result.messages.push(delta);
} else {
result.messages[result.messages.length - 1] += delta;
}
}
}
}
continue;
}
continue;
}
// Tool execution
if (eventType === "tool_execution_start") {
const toolCallId = asString(event.toolCallId, "");
const toolName = asString(event.toolName, "");
const args = event.args;
currentToolCall = { toolCallId, toolName, args };
result.toolCalls.push({
toolCallId,
toolName,
args,
result: null,
isError: false,
});
continue;
}
if (eventType === "tool_execution_end") {
const toolCallId = asString(event.toolCallId, "");
const toolName = asString(event.toolName, "");
const toolResult = event.result;
const isError = event.isError === true;
// Find the tool call by toolCallId (not toolName, to handle multiple calls to same tool)
const existingCall = result.toolCalls.find((tc) => tc.toolCallId === toolCallId);
if (existingCall) {
existingCall.result = typeof toolResult === "string" ? toolResult : JSON.stringify(toolResult);
existingCall.isError = isError;
if (eventType === "error") {
const message = asString(event.message, "").trim();
if (message) {
result.errors.push(message);
}
continue;
}
currentToolCall = null;
continue;
}
// Usage tracking if available in the event (fallback for standalone usage events)
if (eventType === "usage" || event.usage) {
const usage = asRecord(event.usage);
if (usage) {
// Support both Pi format (input/output/cacheRead) and generic format (inputTokens/outputTokens/cachedInputTokens)
result.usage.inputTokens += asNumber(usage.inputTokens ?? usage.input, 0);
result.usage.outputTokens += asNumber(usage.outputTokens ?? usage.output, 0);
result.usage.cachedInputTokens += asNumber(usage.cachedInputTokens ?? usage.cacheRead, 0);
// Cost may be in usage.costUsd (direct) or usage.cost.total (Pi format)
const cost = asRecord(usage.cost);
if (cost) {
result.usage.costUsd += asNumber(cost.total ?? usage.costUsd, 0);
} else {
result.usage.costUsd += asNumber(usage.costUsd, 0);
// Tool execution
if (eventType === "tool_execution_start") {
const toolCallId = asString(event.toolCallId, "");
const toolName = asString(event.toolName, "");
const args = event.args;
currentToolCall = { toolCallId, toolName, args };
result.toolCalls.push({
toolCallId,
toolName,
args,
result: null,
isError: false,
});
continue;
}
if (eventType === "tool_execution_end") {
const toolCallId = asString(event.toolCallId, "");
const toolName = asString(event.toolName, "");
const toolResult = event.result;
const isError = event.isError === true;
// Find the tool call by toolCallId (not toolName, to handle multiple calls to same tool)
const existingCall = result.toolCalls.find((tc) => tc.toolCallId === toolCallId);
if (existingCall) {
existingCall.result = typeof toolResult === "string" ? toolResult : JSON.stringify(toolResult);
existingCall.isError = isError;
}
currentToolCall = null;
continue;
}
// Usage tracking if available in the event (fallback for standalone usage events)
if (eventType === "usage" || event.usage) {
const usage = asRecord(event.usage);
if (usage) {
// Support both Pi format (input/output/cacheRead) and generic format (inputTokens/outputTokens/cachedInputTokens)
result.usage.inputTokens += asNumber(usage.inputTokens ?? usage.input, 0) + asNumber(usage.cacheWrite, 0);
result.usage.outputTokens += asNumber(usage.outputTokens ?? usage.output, 0);
result.usage.cachedInputTokens += asNumber(usage.cachedInputTokens ?? usage.cacheRead, 0);
// Cost may be in usage.costUsd (direct) or usage.cost.total (Pi format)
const cost = asRecord(usage.cost);
addCost(cost?.total ?? usage.costUsd);
}
}
}
}
return result;
if (missingCost) result.usage.costUsd = null;
return { ...result, usage: { ...result.usage }, messages: [...result.messages], errors: [...result.errors], toolCalls: [...result.toolCalls] };
};
}
export function isPiUnknownSessionError(stdout: string, stderr: string): boolean {
packages/db/src/client.ts
+2 -2
View File
@@ -466,8 +466,8 @@ async function recordMigrationHistoryEntry(
// blocks. Execute each statement at top level so DDL locks and completed batches
// are released before the next batch. History is recorded only after completion.
const MIGRATIONS_WITH_BATCH_COMMITS = new Set([
"0311_private_task_access.sql",
"0312_private_task_draft_assets.sql",
"0312_private_task_access.sql",
"0313_private_task_draft_assets.sql",
]);
async function applyPendingMigrationsManually(
packages/db/src/heartbeat-run-issue-privacy-migration.test.ts
+1 -1
View File
@@ -8,7 +8,7 @@ import {
startEmbeddedPostgresTestDatabase,
} from "./test-embedded-postgres.js";
const MIGRATION_FILE = "0311_private_task_access.sql";
const MIGRATION_FILE = "0312_private_task_access.sql";
const cleanups: Array<() => Promise<void>> = [];
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
packages/db/src/migrations/0311_mcp_file_transfers.sql
+48
View File
@@ -0,0 +1,48 @@
CREATE TABLE IF NOT EXISTS "mcp_attachment_uploads" (
"id" uuid PRIMARY KEY DEFAULT gen_random_uuid() NOT NULL,
"company_id" uuid NOT NULL,
"user_id" text NOT NULL,
"request_id" uuid NOT NULL,
"task_id" uuid NOT NULL,
"arguments_hash" text NOT NULL,
"original_filename" text NOT NULL,
"content_type" text NOT NULL,
"byte_size" integer NOT NULL,
"sha256" text NOT NULL,
"attachment_id" uuid,
"object_key" text NOT NULL,
"storage_provider" text NOT NULL,
"expires_at" timestamp with time zone NOT NULL,
"cleaned_at" timestamp with time zone,
"created_at" timestamp with time zone DEFAULT now() NOT NULL
);
--> statement-breakpoint
CREATE TABLE IF NOT EXISTS "mcp_file_tickets" (
"token_hash" text PRIMARY KEY NOT NULL,
"company_id" uuid NOT NULL,
"grant_id" uuid NOT NULL,
"upload_id" uuid,
"attachment_id" uuid,
"expires_at" timestamp with time zone NOT NULL,
"created_at" timestamp with time zone DEFAULT now() NOT NULL
);
--> statement-breakpoint
DO $$ BEGIN
ALTER TABLE "mcp_attachment_uploads" ADD CONSTRAINT "mcp_attachment_uploads_company_id_companies_id_fk" FOREIGN KEY ("company_id") REFERENCES "public"."companies"("id") ON DELETE cascade ON UPDATE no action;
EXCEPTION WHEN duplicate_object THEN NULL; END $$;--> statement-breakpoint
DO $$ BEGIN
ALTER TABLE "mcp_attachment_uploads" ADD CONSTRAINT "mcp_attachment_uploads_user_id_user_id_fk" FOREIGN KEY ("user_id") REFERENCES "public"."user"("id") ON DELETE cascade ON UPDATE no action;
EXCEPTION WHEN duplicate_object THEN NULL; END $$;--> statement-breakpoint
DO $$ BEGIN
ALTER TABLE "mcp_file_tickets" ADD CONSTRAINT "mcp_file_tickets_company_id_companies_id_fk" FOREIGN KEY ("company_id") REFERENCES "public"."companies"("id") ON DELETE cascade ON UPDATE no action;
EXCEPTION WHEN duplicate_object THEN NULL; END $$;--> statement-breakpoint
DO $$ BEGIN
ALTER TABLE "mcp_file_tickets" ADD CONSTRAINT "mcp_file_tickets_grant_id_mcp_oauth_grants_id_fk" FOREIGN KEY ("grant_id") REFERENCES "public"."mcp_oauth_grants"("id") ON DELETE cascade ON UPDATE no action;
EXCEPTION WHEN duplicate_object THEN NULL; END $$;--> statement-breakpoint
DO $$ BEGIN
ALTER TABLE "mcp_file_tickets" ADD CONSTRAINT "mcp_file_tickets_upload_id_mcp_attachment_uploads_id_fk" FOREIGN KEY ("upload_id") REFERENCES "public"."mcp_attachment_uploads"("id") ON DELETE cascade ON UPDATE no action;
EXCEPTION WHEN duplicate_object THEN NULL; END $$;--> statement-breakpoint
CREATE UNIQUE INDEX IF NOT EXISTS "mcp_attachment_upload_request_uq" ON "mcp_attachment_uploads" USING btree ("company_id","user_id","request_id");--> statement-breakpoint
CREATE INDEX IF NOT EXISTS "mcp_attachment_upload_expiry_idx" ON "mcp_attachment_uploads" USING btree ("expires_at");--> statement-breakpoint
CREATE INDEX IF NOT EXISTS "mcp_file_ticket_expiry_idx" ON "mcp_file_tickets" USING btree ("expires_at");--> statement-breakpoint
CREATE INDEX IF NOT EXISTS "mcp_file_ticket_grant_idx" ON "mcp_file_tickets" USING btree ("grant_id");
packages/db/src/migrations/0311_private_task_access.sql → packages/db/src/migrations/0312_private_task_access.sql
View File
File renamed without changes.
packages/db/src/migrations/0312_private_task_draft_assets.sql → packages/db/src/migrations/0313_private_task_draft_assets.sql
View File
File renamed without changes.
packages/db/src/migrations/meta/0311_snapshot.json
Generated
+312 -503
View File
@@ -1,5 +1,5 @@
{
"id": "9e1af935-0e07-42f1-ace0-6b6f81211c97",
"id": "53940222-21d7-4ccc-8b24-d8db300f4ec3",
"prevId": "8a60ce2b-6ab6-435d-90b9-7dc7fc67d53f",
"version": "7",
"dialect": "postgresql",
@@ -24987,19 +24987,6 @@
"primaryKey": false,
"notNull": true
},
"scope_kind": {
"name": "scope_kind",
"type": "text",
"primaryKey": false,
"notNull": true,
"default": "'company'"
},
"issue_id": {
"name": "issue_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"invocation_source": {
"name": "invocation_source",
"type": "text",
@@ -25497,33 +25484,6 @@
"method": "btree",
"with": {}
},
"heartbeat_runs_company_issue_created_idx": {
"name": "heartbeat_runs_company_issue_created_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "issue_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "created_at",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
},
"heartbeat_runs_company_responsible_user_idx": {
"name": "heartbeat_runs_company_responsible_user_idx",
"columns": [
@@ -25762,19 +25722,6 @@
"onDelete": "no action",
"onUpdate": "no action"
},
"heartbeat_runs_issue_id_issues_id_fk": {
"name": "heartbeat_runs_issue_id_issues_id_fk",
"tableFrom": "heartbeat_runs",
"tableTo": "issues",
"columnsFrom": [
"issue_id"
],
"columnsTo": [
"id"
],
"onDelete": "set null",
"onUpdate": "no action"
},
"heartbeat_runs_wakeup_request_id_agent_wakeup_requests_id_fk": {
"name": "heartbeat_runs_wakeup_request_id_agent_wakeup_requests_id_fk",
"tableFrom": "heartbeat_runs",
@@ -25825,12 +25772,7 @@
}
},
"policies": {},
"checkConstraints": {
"heartbeat_runs_scope_binding_check": {
"name": "heartbeat_runs_scope_binding_check",
"value": "(\"heartbeat_runs\".\"scope_kind\" = 'company' AND \"heartbeat_runs\".\"issue_id\" IS NULL)\n OR \"heartbeat_runs\".\"scope_kind\" = 'issue'"
}
},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.inbox_dismissals": {
@@ -27273,166 +27215,6 @@
"checkConstraints": {},
"isRLSEnabled": false
},
"public.issue_access_grants": {
"name": "issue_access_grants",
"schema": "",
"columns": {
"id": {
"name": "id",
"type": "uuid",
"primaryKey": true,
"notNull": true,
"default": "gen_random_uuid()"
},
"issue_id": {
"name": "issue_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"subject_type": {
"name": "subject_type",
"type": "text",
"primaryKey": false,
"notNull": true
},
"subject_id": {
"name": "subject_id",
"type": "text",
"primaryKey": false,
"notNull": true
},
"source": {
"name": "source",
"type": "text",
"primaryKey": false,
"notNull": true
},
"granted_by_user_id": {
"name": "granted_by_user_id",
"type": "text",
"primaryKey": false,
"notNull": false
},
"granted_by_agent_id": {
"name": "granted_by_agent_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
},
"revoked_at": {
"name": "revoked_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
}
},
"indexes": {
"issue_access_grants_subject_active_issue_idx": {
"name": "issue_access_grants_subject_active_issue_idx",
"columns": [
{
"expression": "subject_type",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "subject_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "revoked_at",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "issue_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
},
"issue_access_grants_issue_idx": {
"name": "issue_access_grants_issue_idx",
"columns": [
{
"expression": "issue_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "created_at",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {
"issue_access_grants_issue_id_issues_id_fk": {
"name": "issue_access_grants_issue_id_issues_id_fk",
"tableFrom": "issue_access_grants",
"tableTo": "issues",
"columnsFrom": [
"issue_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"issue_access_grants_granted_by_agent_id_agents_id_fk": {
"name": "issue_access_grants_granted_by_agent_id_agents_id_fk",
"tableFrom": "issue_access_grants",
"tableTo": "agents",
"columnsFrom": [
"granted_by_agent_id"
],
"columnsTo": [
"id"
],
"onDelete": "set null",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {
"issue_access_grants_subject_type_check": {
"name": "issue_access_grants_subject_type_check",
"value": "\"issue_access_grants\".\"subject_type\" in ('user', 'agent')"
},
"issue_access_grants_source_check": {
"name": "issue_access_grants_source_check",
"value": "\"issue_access_grants\".\"source\" in ('explicit', 'assignment', 'project')"
}
},
"isRLSEnabled": false
},
"public.issue_approvals": {
"name": "issue_approvals",
"schema": "",
@@ -32305,25 +32087,6 @@
"primaryKey": false,
"notNull": false
},
"visibility": {
"name": "visibility",
"type": "text",
"primaryKey": false,
"notNull": true,
"default": "'open'"
},
"privacy_root_issue_id": {
"name": "privacy_root_issue_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"privacy_parent_issue_id": {
"name": "privacy_parent_issue_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"title": {
"name": "title",
"type": "text",
@@ -32808,48 +32571,6 @@
"method": "btree",
"with": {}
},
"issues_company_privacy_parent_idx": {
"name": "issues_company_privacy_parent_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "privacy_parent_issue_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
},
"issues_company_privacy_root_idx": {
"name": "issues_company_privacy_root_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "privacy_root_issue_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
},
"issues_company_project_idx": {
"name": "issues_company_project_idx",
"columns": [
@@ -33419,32 +33140,6 @@
"onDelete": "no action",
"onUpdate": "no action"
},
"issues_privacy_root_issue_id_issues_id_fk": {
"name": "issues_privacy_root_issue_id_issues_id_fk",
"tableFrom": "issues",
"tableTo": "issues",
"columnsFrom": [
"privacy_root_issue_id"
],
"columnsTo": [
"id"
],
"onDelete": "no action",
"onUpdate": "no action"
},
"issues_privacy_parent_issue_id_issues_id_fk": {
"name": "issues_privacy_parent_issue_id_issues_id_fk",
"tableFrom": "issues",
"tableTo": "issues",
"columnsFrom": [
"privacy_parent_issue_id"
],
"columnsTo": [
"id"
],
"onDelete": "set null",
"onUpdate": "no action"
},
"issues_assignee_agent_id_agents_id_fk": {
"name": "issues_assignee_agent_id_agents_id_fk",
"tableFrom": "issues",
@@ -33524,10 +33219,6 @@
},
"policies": {},
"checkConstraints": {
"issues_visibility_check": {
"name": "issues_visibility_check",
"value": "\"issues\".\"visibility\" in ('open', 'private')"
},
"issues_conversation_identity_check": {
"name": "issues_conversation_identity_check",
"value": "(\n \"issues\".\"conversation_agent_id\" is null and \"issues\".\"conversation_user_id\" is null and \"issues\".\"conversation_state\" is null\n ) or (\n \"issues\".\"conversation_agent_id\" is not null and \"issues\".\"conversation_user_id\" is not null\n and \"issues\".\"assignee_agent_id\" = \"issues\".\"conversation_agent_id\" and \"issues\".\"assignee_agent_id\" is not null\n and \"issues\".\"assignee_user_id\" is null and \"issues\".\"conversation_state\" is not null\n and \"issues\".\"conversation_state\" in ('active', 'waiting')\n and \"issues\".\"status\" not in ('done', 'cancelled')\n )"
@@ -38817,150 +38508,6 @@
"checkConstraints": {},
"isRLSEnabled": false
},
"public.project_access_members": {
"name": "project_access_members",
"schema": "",
"columns": {
"id": {
"name": "id",
"type": "uuid",
"primaryKey": true,
"notNull": true,
"default": "gen_random_uuid()"
},
"company_id": {
"name": "company_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"project_id": {
"name": "project_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"subject_type": {
"name": "subject_type",
"type": "text",
"primaryKey": false,
"notNull": true
},
"subject_id": {
"name": "subject_id",
"type": "text",
"primaryKey": false,
"notNull": true
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {
"project_access_members_project_subject_uq": {
"name": "project_access_members_project_subject_uq",
"columns": [
{
"expression": "project_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "subject_type",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "subject_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": true,
"concurrently": false,
"method": "btree",
"with": {}
},
"project_access_members_subject_lookup_idx": {
"name": "project_access_members_subject_lookup_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "subject_type",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "subject_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "project_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {
"project_access_members_company_id_companies_id_fk": {
"name": "project_access_members_company_id_companies_id_fk",
"tableFrom": "project_access_members",
"tableTo": "companies",
"columnsFrom": [
"company_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"project_access_members_project_id_projects_id_fk": {
"name": "project_access_members_project_id_projects_id_fk",
"tableFrom": "project_access_members",
"tableTo": "projects",
"columnsFrom": [
"project_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {
"project_access_members_subject_type_check": {
"name": "project_access_members_subject_type_check",
"value": "\"project_access_members\".\"subject_type\" in ('user', 'agent')"
}
},
"isRLSEnabled": false
},
"public.project_goals": {
"name": "project_goals",
"schema": "",
@@ -39590,25 +39137,6 @@
"primaryKey": false,
"notNull": false
},
"visibility": {
"name": "visibility",
"type": "text",
"primaryKey": false,
"notNull": true,
"default": "'open'"
},
"privacy_owner_user_id": {
"name": "privacy_owner_user_id",
"type": "text",
"primaryKey": false,
"notNull": false
},
"personal_owner_user_id": {
"name": "personal_owner_user_id",
"type": "text",
"primaryKey": false,
"notNull": false
},
"status": {
"name": "status",
"type": "text",
@@ -39700,28 +39228,6 @@
"concurrently": false,
"method": "btree",
"with": {}
},
"projects_company_personal_owner_uq": {
"name": "projects_company_personal_owner_uq",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "personal_owner_user_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": true,
"where": "\"projects\".\"personal_owner_user_id\" is not null",
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {
@@ -39768,12 +39274,7 @@
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {
"projects_visibility_check": {
"name": "projects_visibility_check",
"value": "\"projects\".\"visibility\" in ('open', 'private')"
}
},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.provider_trace_records": {
@@ -51992,6 +51493,187 @@
"checkConstraints": {},
"isRLSEnabled": false
},
"public.mcp_attachment_uploads": {
"name": "mcp_attachment_uploads",
"schema": "",
"columns": {
"id": {
"name": "id",
"type": "uuid",
"primaryKey": true,
"notNull": true,
"default": "gen_random_uuid()"
},
"company_id": {
"name": "company_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"user_id": {
"name": "user_id",
"type": "text",
"primaryKey": false,
"notNull": true
},
"request_id": {
"name": "request_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"task_id": {
"name": "task_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"arguments_hash": {
"name": "arguments_hash",
"type": "text",
"primaryKey": false,
"notNull": true
},
"original_filename": {
"name": "original_filename",
"type": "text",
"primaryKey": false,
"notNull": true
},
"content_type": {
"name": "content_type",
"type": "text",
"primaryKey": false,
"notNull": true
},
"byte_size": {
"name": "byte_size",
"type": "integer",
"primaryKey": false,
"notNull": true
},
"sha256": {
"name": "sha256",
"type": "text",
"primaryKey": false,
"notNull": true
},
"attachment_id": {
"name": "attachment_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"object_key": {
"name": "object_key",
"type": "text",
"primaryKey": false,
"notNull": true
},
"storage_provider": {
"name": "storage_provider",
"type": "text",
"primaryKey": false,
"notNull": true
},
"expires_at": {
"name": "expires_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true
},
"cleaned_at": {
"name": "cleaned_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {
"mcp_attachment_upload_request_uq": {
"name": "mcp_attachment_upload_request_uq",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "user_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "request_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": true,
"concurrently": false,
"method": "btree",
"with": {}
},
"mcp_attachment_upload_expiry_idx": {
"name": "mcp_attachment_upload_expiry_idx",
"columns": [
{
"expression": "expires_at",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {
"mcp_attachment_uploads_company_id_companies_id_fk": {
"name": "mcp_attachment_uploads_company_id_companies_id_fk",
"tableFrom": "mcp_attachment_uploads",
"tableTo": "companies",
"columnsFrom": [
"company_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"mcp_attachment_uploads_user_id_user_id_fk": {
"name": "mcp_attachment_uploads_user_id_user_id_fk",
"tableFrom": "mcp_attachment_uploads",
"tableTo": "user",
"columnsFrom": [
"user_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.mcp_event_admissions": {
"name": "mcp_event_admissions",
"schema": "",
@@ -52388,6 +52070,133 @@
"checkConstraints": {},
"isRLSEnabled": false
},
"public.mcp_file_tickets": {
"name": "mcp_file_tickets",
"schema": "",
"columns": {
"token_hash": {
"name": "token_hash",
"type": "text",
"primaryKey": true,
"notNull": true
},
"company_id": {
"name": "company_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"grant_id": {
"name": "grant_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"upload_id": {
"name": "upload_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"attachment_id": {
"name": "attachment_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"expires_at": {
"name": "expires_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {
"mcp_file_ticket_expiry_idx": {
"name": "mcp_file_ticket_expiry_idx",
"columns": [
{
"expression": "expires_at",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
},
"mcp_file_ticket_grant_idx": {
"name": "mcp_file_ticket_grant_idx",
"columns": [
{
"expression": "grant_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {
"mcp_file_tickets_company_id_companies_id_fk": {
"name": "mcp_file_tickets_company_id_companies_id_fk",
"tableFrom": "mcp_file_tickets",
"tableTo": "companies",
"columnsFrom": [
"company_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"mcp_file_tickets_grant_id_mcp_oauth_grants_id_fk": {
"name": "mcp_file_tickets_grant_id_mcp_oauth_grants_id_fk",
"tableFrom": "mcp_file_tickets",
"tableTo": "mcp_oauth_grants",
"columnsFrom": [
"grant_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"mcp_file_tickets_upload_id_mcp_attachment_uploads_id_fk": {
"name": "mcp_file_tickets_upload_id_mcp_attachment_uploads_id_fk",
"tableFrom": "mcp_file_tickets",
"tableTo": "mcp_attachment_uploads",
"columnsFrom": [
"upload_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.mcp_mutation_receipts": {
"name": "mcp_mutation_receipts",
"schema": "",
@@ -53248,4 +53057,4 @@
"schemas": {},
"tables": {}
}
}
}
packages/db/src/migrations/meta/0312_snapshot.json
Generated
+311 -3
View File
@@ -1,6 +1,6 @@
{
"id": "5c3530f7-ecaa-40ba-93c4-03e44507df86",
"prevId": "9e1af935-0e07-42f1-ace0-6b6f81211c97",
"id": "e4c03d22-6ac5-4109-99b7-8c51604072a3",
"prevId": "53940222-21d7-4ccc-8b24-d8db300f4ec3",
"version": "7",
"dialect": "postgresql",
"tables": {
@@ -51992,6 +51992,187 @@
"checkConstraints": {},
"isRLSEnabled": false
},
"public.mcp_attachment_uploads": {
"name": "mcp_attachment_uploads",
"schema": "",
"columns": {
"id": {
"name": "id",
"type": "uuid",
"primaryKey": true,
"notNull": true,
"default": "gen_random_uuid()"
},
"company_id": {
"name": "company_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"user_id": {
"name": "user_id",
"type": "text",
"primaryKey": false,
"notNull": true
},
"request_id": {
"name": "request_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"task_id": {
"name": "task_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"arguments_hash": {
"name": "arguments_hash",
"type": "text",
"primaryKey": false,
"notNull": true
},
"original_filename": {
"name": "original_filename",
"type": "text",
"primaryKey": false,
"notNull": true
},
"content_type": {
"name": "content_type",
"type": "text",
"primaryKey": false,
"notNull": true
},
"byte_size": {
"name": "byte_size",
"type": "integer",
"primaryKey": false,
"notNull": true
},
"sha256": {
"name": "sha256",
"type": "text",
"primaryKey": false,
"notNull": true
},
"attachment_id": {
"name": "attachment_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"object_key": {
"name": "object_key",
"type": "text",
"primaryKey": false,
"notNull": true
},
"storage_provider": {
"name": "storage_provider",
"type": "text",
"primaryKey": false,
"notNull": true
},
"expires_at": {
"name": "expires_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true
},
"cleaned_at": {
"name": "cleaned_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {
"mcp_attachment_upload_request_uq": {
"name": "mcp_attachment_upload_request_uq",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "user_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "request_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": true,
"concurrently": false,
"method": "btree",
"with": {}
},
"mcp_attachment_upload_expiry_idx": {
"name": "mcp_attachment_upload_expiry_idx",
"columns": [
{
"expression": "expires_at",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {
"mcp_attachment_uploads_company_id_companies_id_fk": {
"name": "mcp_attachment_uploads_company_id_companies_id_fk",
"tableFrom": "mcp_attachment_uploads",
"tableTo": "companies",
"columnsFrom": [
"company_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"mcp_attachment_uploads_user_id_user_id_fk": {
"name": "mcp_attachment_uploads_user_id_user_id_fk",
"tableFrom": "mcp_attachment_uploads",
"tableTo": "user",
"columnsFrom": [
"user_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.mcp_event_admissions": {
"name": "mcp_event_admissions",
"schema": "",
@@ -52388,6 +52569,133 @@
"checkConstraints": {},
"isRLSEnabled": false
},
"public.mcp_file_tickets": {
"name": "mcp_file_tickets",
"schema": "",
"columns": {
"token_hash": {
"name": "token_hash",
"type": "text",
"primaryKey": true,
"notNull": true
},
"company_id": {
"name": "company_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"grant_id": {
"name": "grant_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"upload_id": {
"name": "upload_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"attachment_id": {
"name": "attachment_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"expires_at": {
"name": "expires_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {
"mcp_file_ticket_expiry_idx": {
"name": "mcp_file_ticket_expiry_idx",
"columns": [
{
"expression": "expires_at",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
},
"mcp_file_ticket_grant_idx": {
"name": "mcp_file_ticket_grant_idx",
"columns": [
{
"expression": "grant_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {
"mcp_file_tickets_company_id_companies_id_fk": {
"name": "mcp_file_tickets_company_id_companies_id_fk",
"tableFrom": "mcp_file_tickets",
"tableTo": "companies",
"columnsFrom": [
"company_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"mcp_file_tickets_grant_id_mcp_oauth_grants_id_fk": {
"name": "mcp_file_tickets_grant_id_mcp_oauth_grants_id_fk",
"tableFrom": "mcp_file_tickets",
"tableTo": "mcp_oauth_grants",
"columnsFrom": [
"grant_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"mcp_file_tickets_upload_id_mcp_attachment_uploads_id_fk": {
"name": "mcp_file_tickets_upload_id_mcp_attachment_uploads_id_fk",
"tableFrom": "mcp_file_tickets",
"tableTo": "mcp_attachment_uploads",
"columnsFrom": [
"upload_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.mcp_mutation_receipts": {
"name": "mcp_mutation_receipts",
"schema": "",
@@ -53248,4 +53556,4 @@
"schemas": {},
"tables": {}
}
}
}
packages/db/src/migrations/meta/0306_snapshot.json → packages/db/src/migrations/meta/0313_snapshot.json
Generated
+5161 -3973
View File
File diff suppressed because it is too large. Load diff
packages/db/src/migrations/meta/_journal.json
Generated
+12 -5
View File
@@ -2168,16 +2168,23 @@
{
"idx": 311,
"version": "7",
"when": 1791314887097,
"tag": "0311_private_task_access",
"when": 1791321362665,
"tag": "0311_mcp_file_transfers",
"breakpoints": true
},
{
"idx": 312,
"version": "7",
"when": 1791314887098,
"tag": "0312_private_task_draft_assets",
"when": 1791336853102,
"tag": "0312_private_task_access",
"breakpoints": true
},
{
"idx": 313,
"version": "7",
"when": 1791336854116,
"tag": "0313_private_task_draft_assets",
"breakpoints": true
}
]
}
}
packages/db/src/private-task-draft-assets-migration.test.ts
+1 -1
View File
@@ -12,7 +12,7 @@ itEmbeddedPostgres("backfills owned draft images without attaching another uploa
const database = await startEmbeddedPostgresTestDatabase("paperclip-draft-privacy-");
const sql = postgres(database.connectionString, { max: 1, onnotice: () => {} });
try {
const migration = await fs.readFile(new URL("./migrations/0312_private_task_draft_assets.sql", import.meta.url), "utf8");
const migration = await fs.readFile(new URL("./migrations/0313_private_task_draft_assets.sql", import.meta.url), "utf8");
const hash = createHash("sha256").update(migration).digest("hex");
const company = randomUUID(), task = randomUUID(), owned = randomUUID(), foreign = randomUUID();
await sql`INSERT INTO companies (id, name, issue_prefix) VALUES (${company}, 'Draft privacy', 'DRAFT')`;
packages/db/src/schema/public_mcp.ts
+34
View File
@@ -154,3 +154,37 @@ export const mcpEventDeliveries = pgTable("mcp_event_deliveries", {
uniqueIndex("mcp_event_deliveries_activity_uq").on(t.subscriptionId, t.activityId),
index("mcp_event_deliveries_due_idx").on(t.nextAttemptAt),
]);
// Upload identities outlive their short-lived tickets so retries cannot duplicate files.
export const mcpAttachmentUploads = pgTable("mcp_attachment_uploads", {
id: uuid("id").primaryKey().defaultRandom(),
companyId: uuid("company_id").notNull().references(() => companies.id, { onDelete: "cascade" }),
userId: text("user_id").notNull().references(() => authUsers.id, { onDelete: "cascade" }),
requestId: uuid("request_id").notNull(),
taskId: uuid("task_id").notNull(),
argumentsHash: text("arguments_hash").notNull(),
originalFilename: text("original_filename").notNull(),
contentType: text("content_type").notNull(),
byteSize: integer("byte_size").notNull(),
sha256: text("sha256").notNull(),
attachmentId: uuid("attachment_id"),
// Allocated before storage writes, so a crash never loses cleanup provenance.
objectKey: text("object_key").notNull(),
storageProvider: text("storage_provider").notNull(),
expiresAt: timestamp("expires_at", { withTimezone: true }).notNull(),
cleanedAt: timestamp("cleaned_at", { withTimezone: true }),
createdAt: timestamp("created_at", { withTimezone: true }).notNull().defaultNow(),
}, (t) => [
uniqueIndex("mcp_attachment_upload_request_uq").on(t.companyId, t.userId, t.requestId),
index("mcp_attachment_upload_expiry_idx").on(t.expiresAt),
]);
export const mcpFileTickets = pgTable("mcp_file_tickets", {
tokenHash: text("token_hash").primaryKey(),
companyId: uuid("company_id").notNull().references(() => companies.id, { onDelete: "cascade" }),
grantId: uuid("grant_id").notNull().references(() => mcpOauthGrants.id, { onDelete: "cascade" }),
uploadId: uuid("upload_id").references(() => mcpAttachmentUploads.id, { onDelete: "cascade" }),
attachmentId: uuid("attachment_id"),
expiresAt: timestamp("expires_at", { withTimezone: true }).notNull(),
createdAt: timestamp("created_at", { withTimezone: true }).notNull().defaultNow(),
}, (t) => [index("mcp_file_ticket_expiry_idx").on(t.expiresAt), index("mcp_file_ticket_grant_idx").on(t.grantId)]);
packages/paperclip-runner/src/drivers/codex/codex-app-server-driver.events.test.ts
+22
View File
@@ -294,6 +294,28 @@ describe("Codex app-server Codex driver", () => {
expect(events.some((event) => event.eventType === "turn.interrupted")).toBe(true);
});
it.each(["output", "cache"])("does not certify partial provider usage using retained %s counters", async omitted => {
const transport = new FakeCodexTransport();
const session = await makeDriver([transport]).openSession({
runId: "run-partial-usage", normalizedSessionId: "normalized-partial-usage", workingDirectory: WORKSPACE,
});
const turn = await session.startTurn({ message: { role: "user", text: "Work." } });
transport.push("turn/started", { threadId: "thread-1", turn: { id: turn.turnId, status: "inProgress" } });
const first = { inputTokens: 10, outputTokens: 4, ...(omitted === "cache" ? { cachedInputTokens: 2 } : {}) };
const second = { inputTokens: 20, ...(omitted === "cache" ? { outputTokens: 8 } : {}) };
for (const total of [first, second]) {
transport.push("thread/tokenUsage/updated", { threadId: "thread-1", turnId: turn.turnId, tokenUsage: { total } });
}
transport.push("turn/completed", { threadId: "thread-1", turn: { id: turn.turnId, status: "completed", items: [] } });
const events = await collectUntilTerminal(session.events());
const usage = events.filter(event => event.payload.kind === "usage");
expect(usage.map(event => event.payload.usage)).toMatchObject([
{ runDelta: first, runDeltaComplete: true },
{ runDelta: { ...first, ...second }, runDeltaComplete: false },
]);
expect(await session.usage()).toMatchObject({ runDeltaComplete: false });
});
it("accepts a thread usage snapshot replayed before a resumed turn starts", async () => {
const transport = new FakeCodexTransport();
const session = await makeDriver([transport]).openSession({
packages/paperclip-runner/src/drivers/codex/codex-session-notifications.ts
+13 -3
View File
@@ -533,9 +533,19 @@ async function mapNotificationBody(state: CodexSessionState, notification: Codex
}
if (notification.method === "thread/tokenUsage/updated") {
state.usageSnapshot = boundedPayload(record(params.tokenUsage));
if (state.driverKind === "codex_app_server" && Object.keys(record(record(params.tokenUsage).total)).length > 0) {
state.codexUsageBaseline = observeCodexUsage(state.codexUsageBaseline, record(params.tokenUsage).total, false);
state.usageSnapshot = { ...state.usageSnapshot, ...codexRunUsage(state.codexUsageBaseline) };
if (state.driverKind === "codex_app_server") {
const reported = record(record(params.tokenUsage).total);
const validCount = (value: unknown) => typeof value === "number" && Number.isSafeInteger(value) && value >= 0;
// The monotonic snapshot retains absent counters from older reports.
// Preserve whether this provider report can actually close accounting.
const runDeltaComplete = validCount(reported.inputTokens) && validCount(reported.outputTokens)
&& Object.entries(reported).every(([key, value]) => !key.endsWith("Tokens") || validCount(value))
&& Object.keys(state.codexUsageBaseline?.latest ?? {}).every(key => !key.endsWith("Tokens") || validCount(reported[key]));
if (Object.keys(reported).length > 0) {
state.codexUsageBaseline = observeCodexUsage(state.codexUsageBaseline, reported, false);
state.usageSnapshot = { ...state.usageSnapshot, ...codexRunUsage(state.codexUsageBaseline) };
}
state.usageSnapshot = { ...state.usageSnapshot, runDeltaComplete };
}
// Codex can replay a thread-scoped usage snapshot while a resumed thread
// is being attached, before the next turn has started. Keep the snapshot,
packages/shared/src/mcp-setup.ts
+1 -1
View File
@@ -18,7 +18,7 @@ export function mcpSetupSteps(serverUrl: string, assistant: AssistantClient) {
switch (assistant) {
case "codex": return [
{ text: "Check for an existing Paperclip server first. Reuse it only if its URL matches this instance. Add the remote server with the Codex CLI.", code: `codex mcp add paperclip --url ${quoted}` },
{ text: "Start browser authorization and give the user the approval link.", code: "codex mcp login paperclip --scopes paperclip:read,paperclip:write,offline_access" },
{ text: "Start browser authorization and give the user the approval link.", code: "codex mcp login paperclip --scopes paperclip:read,paperclip:write,paperclip:configure,offline_access" },
{ text: "If the current conversation does not reload its MCP tools, start a new conversation or restart the client. Verify the connection with paperclip_connection before claiming success." },
];
case "claude": return [
packages/shared/src/public-mcp.ts
+3 -1
View File
@@ -1,11 +1,12 @@
import { z } from "zod";
export const PUBLIC_MCP_PATH = "/mcp/paperclip";
export const PUBLIC_MCP_SCOPES = ["paperclip:read", "paperclip:write", "offline_access"] as const;
export const PUBLIC_MCP_SCOPES = ["paperclip:read", "paperclip:write", "paperclip:configure", "offline_access"] as const;
export const mcpConsentSchema = z.object({
decision: z.enum(["approve", "deny"]),
companyId: z.string().uuid().optional(),
allowWrites: z.boolean().default(false),
allowConfiguration: z.boolean().default(false),
}).strict();
export interface McpConnectionRequest {
@@ -14,6 +15,7 @@ export interface McpConnectionRequest {
redirectOrigin: string;
clientOrigin?: string | null;
requestedWrite: boolean;
requestedConfigure?: boolean;
offlineAccess: boolean;
requiresSignIn: boolean;
/** Fixed by the authorization request; null permits direct-instance selection. */
packages/shared/src/types/quota.ts
+4
View File
@@ -16,6 +16,10 @@ export interface QuotaWindow {
export interface ProviderQuotaResult {
/** provider slug, e.g. "anthropic", "openai" */
provider: string;
/** Connection, grant and credential revision; never a credential value. */
accountKey?: string;
accountLabel?: string;
capturedAt?: string;
/** source label when the provider reports where the quota data came from */
source?: string | null;
/** true when the fetch succeeded and windows is populated */
scripts/__tests__/run-vitest-stable-shard.test.mjs
+21 -1
View File
@@ -92,17 +92,37 @@ test("a route/authz suite never leaks into the general-server shards", () => {
const shard = dryRunJson(["--mode", "general", "--group", "general-server", "--shard-index", "0", "--shard-count", SHARD_COUNT.toString()]);
for (const file of shard.selectedGeneralServerSuites) {
assert.ok(
!/[^/]*(?:route|routes|authz)[^/]*\.test\.ts$/.test(file),
!(file.startsWith("server/src/__tests__/") && /[^/]*(?:route|routes|authz)[^/]*\.test\.ts$/.test(file)),
`route/authz suite must stay in the serialized lane, not general-server: ${file}`,
);
}
});
test("general shards retain scripts and route-named suites outside the serialized directory", () => {
const general = dryRunJson(["--mode", "general", "--group", "general-server", "--shard-index", "0", "--shard-count", "1"]);
for (const file of [
"server/scripts/verify-runner-vendor-dependencies.test.mjs",
"server/src/services/openrouter-models.test.ts",
"server/src/routes/setup-token-route.test.ts",
]) assert.ok(general.selectedGeneralServerSuites.includes(file), `missing configured server suite: ${file}`);
});
test("shard flags are rejected for the workspaces-b group", () => {
const result = dryRun(["--mode", "general", "--group", "general-workspaces-b", "--shard-index", "0", "--shard-count", "3"]);
assert.notEqual(result.status, 0, "workspaces-b must not accept shard flags");
});
test("workspace lanes cover every non-server project in the root Vitest configuration", () => {
const config = readFileSync(path.join(repoRoot, "vitest.config.ts"), "utf8");
const roots = [...config.matchAll(/^\s+"([^"]+)",?\s*$/gm)].map(match => match[1]);
assert.ok(roots.includes("server"), "expected the explicit root Vitest project list");
const expected = roots.filter(root => root !== "server")
.map(root => JSON.parse(readFileSync(path.join(repoRoot, root, "package.json"), "utf8")).name).sort();
const actual = ["general-workspaces-a", "general-workspaces-b"]
.flatMap(group => dryRunJson(["--mode", "general", "--group", group]).workspaceProjects).sort();
assert.deepEqual(actual, expected, "no configured project may be silently omitted or run twice");
});
test("workspaces-a shards map to Vitest native --shard slices over a stable project list", () => {
const shards = [0, 1].map((index) =>
dryRunJson([
scripts/run-vitest-stable.mjs
+16 -3
View File
@@ -19,6 +19,7 @@ const serializedShardDurations = loadShardDurations(
const serverRoot = path.join(repoRoot, "server");
const serverSrcDir = path.join(repoRoot, "server", "src");
const serverTestsDir = path.join(repoRoot, "server", "src", "__tests__");
const serverScriptsDir = path.join(repoRoot, "server", "scripts");
const nonServerProjects = [
"@paperclipai/shared",
"@paperclipai/skills-catalog",
@@ -26,9 +27,14 @@ const nonServerProjects = [
"@paperclipai/adapter-utils",
"@paperclipai/adapter-claude-local",
"@paperclipai/adapter-codex-local",
"@paperclipai/adapter-cursor-cloud",
"@paperclipai/adapter-cursor-local",
"@paperclipai/adapter-gemini-local",
"@paperclipai/adapter-grok-local",
"@paperclipai/adapter-kimi-local",
"@paperclipai/adapter-openclaw-gateway",
"@paperclipai/adapter-opencode-local",
"@paperclipai/adapter-pi-local",
"@paperclipai/plugin-daytona",
"@paperclipai/plugin-sdk",
"@paperclipai/create-paperclip-plugin",
@@ -345,6 +351,8 @@ function runVitest(args, label, testShard = null) {
const env = {
...process.env,
NODE_ENV: "test",
PAPERCLIP_TEST_HOST_HOME: process.env.PAPERCLIP_TEST_HOST_HOME
?? (process.env.PAPERCLIP_HOME?.trim() || path.join(os.homedir(), ".paperclip")),
PAPERCLIP_HOME: path.join(testRoot, "h"),
// Config discovery otherwise prefers the checkout's .paperclip/config.json
// over PAPERCLIP_HOME, importing preview scheduling policy into unit tests.
@@ -539,10 +547,15 @@ const routeTests = walk(serverTestsDir)
// config pins maxWorkers to 1, so the only way to parallelize is across jobs.
// Suites are partitioned by recorded duration (scripts/general-server-shard.mjs)
// rather than round-robin, so one slow suite cluster can't stretch a single shard.
const generalServerTestFiles = walk(serverSrcDir)
const serializedRepoPaths = new Set(routeTests.map(test => test.repoPath));
const generalServerTestFiles = [
...walk(serverSrcDir).filter(file => file.endsWith(".test.ts")),
...walk(serverScriptsDir).filter(file => file.endsWith(".test.mjs")),
]
.map((file) => toRepoPath(file))
.filter((repoPath) => repoPath.endsWith(".test.ts"))
.filter((repoPath) => !isRouteOrAuthzTest(repoPath))
// Only exclude suites actually assigned to the serialized lane. A name
// such as services/openrouter-models.test.ts is not a serialized route.
.filter((repoPath) => !serializedRepoPaths.has(repoPath))
.sort((a, b) => a.localeCompare(b));
const options = parseCliOptions(process.argv.slice(2));
server/src/__tests__/adapter-cost-failure-paths.test.ts
+327
View File
@@ -0,0 +1,327 @@
import { mkdtemp, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { afterEach, describe, expect, it, vi } from "vitest";
import type { AdapterExecutionContext, AdapterExecutionResult } from "@paperclipai/adapter-utils";
const processResult = vi.hoisted(() => vi.fn());
vi.mock("@paperclipai/adapter-utils/execution-target", async (original) => ({
...await original<typeof import("@paperclipai/adapter-utils/execution-target")>(),
runAdapterExecutionTargetProcess: processResult,
}));
type Execute = (context: AdapterExecutionContext) => Promise<AdapterExecutionResult>;
const cases: Array<{ name: string; execute: () => Promise<Execute>; event: unknown; tokens?: { inputTokens: number; outputTokens: number; cachedInputTokens: number }; price: number | null }> = [
{ name: "codex", execute: async () => (await import("../../../packages/adapters/codex-local/src/server/execute.js")).execute,
event: { type: "turn.completed", usage: { input_tokens: 120, cached_input_tokens: 100, output_tokens: 10 } }, tokens: { inputTokens: 20, cachedInputTokens: 100, outputTokens: 10 }, price: null },
{ name: "claude", execute: async () => (await import("../../../packages/adapters/claude-local/src/server/execute.js")).execute,
event: { type: "assistant", message: { id: "m1", usage: { input_tokens: 20, cache_read_input_tokens: 100, output_tokens: 10 }, content: [] } }, tokens: { inputTokens: 20, cachedInputTokens: 100, outputTokens: 10 }, price: null },
{ name: "opencode", execute: async () => (await import("../../../packages/adapters/opencode-local/src/server/execute.js")).execute,
event: { type: "step_finish", part: { tokens: { input: 20, output: 10, cache: { read: 100 } }, cost: 0.004 } }, tokens: { inputTokens: 20, cachedInputTokens: 100, outputTokens: 10 }, price: 0.004 },
{ name: "pi", execute: async () => (await import("../../../packages/adapters/pi-local/src/server/execute.js")).execute,
event: { type: "turn_end", message: { role: "assistant", content: [], usage: { input: 20, output: 10, cacheRead: 100, cost: { total: 0.004 } } } }, tokens: { inputTokens: 20, cachedInputTokens: 100, outputTokens: 10 }, price: 0.004 },
{ name: "cursor", execute: async () => (await import("../../../packages/adapters/cursor-local/src/server/execute.js")).execute,
event: { type: "result", usage: { input_tokens: 20, cache_read_input_tokens: 100, output_tokens: 10 }, total_cost_usd: 0.004 }, tokens: { inputTokens: 20, cachedInputTokens: 100, outputTokens: 10 }, price: 0.004 },
{ name: "gemini", execute: async () => (await import("../../../packages/adapters/gemini-local/src/server/execute.js")).execute,
event: { type: "result", usage: { input_tokens: 120, cached_input_tokens: 100, output_tokens: 10 }, total_cost_usd: 0.004 }, tokens: { inputTokens: 20, cachedInputTokens: 100, outputTokens: 10 }, price: 0.004 },
{ name: "kimi", execute: async () => (await import("../../../packages/adapters/kimi-local/src/server/execute.js")).execute,
event: { type: "assistant", content: "partial work" }, price: null },
];
describe("CLI adapter accounting on timeout", () => {
const directories: string[] = [];
afterEach(async () => { vi.clearAllMocks(); for (const dir of directories.splice(0)) await rm(dir, { recursive: true, force: true }); });
it.each(cases.filter(f => ["codex", "cursor", "opencode"].includes(f.name)))("$name preserves malformed optional counters through the checkpoint wrapper", async fixture => {
const dir = await mkdtemp(join(tmpdir(), "paperclip-invalid-counter-")); directories.push(dir);
const command = join(dir, "runtime"); await writeFile(command, "#!/bin/sh\nprintf 'openai test\\n'\n", { mode: 0o755 });
const execute = await fixture.execute();
for (const cached of ["private malformed counter", false, {}, 0, undefined]) for (const newline of [false, true]) {
const event = fixture.name === "opencode"
? { type: "step_finish", part: { tokens: { input: 20, output: 10, cache: { read: cached } } } }
: { type: fixture.name === "codex" ? "turn.completed" : "result", usage: { input_tokens: 20, output_tokens: 10, cached_input_tokens: cached } };
const stdout = JSON.stringify(event) + (newline ? "\n" : "");
processResult.mockImplementation(async (_run, _target, _command, _args, options) => {
for (let offset = 0; offset < stdout.length; offset += 7) await options.onLog("stdout", stdout.slice(offset, offset + 7));
return { exitCode: 0, signal: null, timedOut: false, stdout, stderr: "", pid: 123, startedAt: new Date().toISOString() };
});
const onUsage = vi.fn();
const result = await execute({
runId: "invalid-counter-run", agent: { id: "agent", companyId: "company", name: "Worker", adapterType: `${fixture.name}_local`, adapterConfig: {} },
runtime: { sessionId: null, sessionParams: null, sessionDisplayId: null, taskKey: null },
config: { engine: "cli", command, cwd: dir, model: fixture.name === "opencode" ? "openai/test" : "gpt-6-astra", paperclipRuntimeSkills: [],
env: { OPENAI_API_KEY: "fixture", OPENCODE_ALLOW_ALL_MODELS: "1" } },
context: {}, onLog: async () => {}, onUsage,
});
const valid = cached === 0 || cached === undefined;
expect(result.usageComplete, `${fixture.name}: ${JSON.stringify(cached)}`).toBe(valid);
expect(result.usage).toEqual(valid ? { inputTokens: 20, outputTokens: 10, cachedInputTokens: 0 } : undefined);
expect(onUsage.mock.calls.at(-1)![0].complete).toBe(valid);
if (!valid) {
expect(result.costStatus).toBe("unpriced");
expect(onUsage.mock.calls.every(([receipt]) => !receipt.complete && receipt.costStatus === "unpriced")).toBe(true);
}
}
});
it.each(cases.filter(fixture => fixture.name !== "kimi"))("$name flushes its attempt and surfaces checkpoint persistence failures", async (fixture) => {
const dir = await mkdtemp(join(tmpdir(), "paperclip-accounting-adapter-")); directories.push(dir);
const command = join(dir, "runtime"); await writeFile(command, "#!/bin/sh\nprintf 'openai test\\n'\n", { mode: 0o755 });
const stdout = JSON.stringify(fixture.event);
processResult.mockImplementation(async (_run, _target, _command, _args, options) => {
// Local process logging catches callback failures. Finalization must
// independently enforce receipt durability despite that contract.
await options.onLog("stdout", stdout).catch(() => {});
return { exitCode: 0, signal: null, timedOut: false, stdout, stderr: "", pid: 123, startedAt: new Date().toISOString() };
});
const onUsage = vi.fn();
const context: AdapterExecutionContext = {
runId: "test-run", agent: { id: "test-agent", companyId: "test-company", name: "Accounting", adapterType: `${fixture.name}_local`, adapterConfig: {} },
runtime: { sessionId: null, sessionParams: null, sessionDisplayId: null, taskKey: null },
config: { engine: "cli", command, cwd: dir, model: ["opencode", "pi"].includes(fixture.name) ? "openai/test" : fixture.name === "cursor" ? "gpt-5" : "test", paperclipRuntimeSkills: [],
env: { OPENAI_API_KEY: "test-placeholder", ANTHROPIC_API_KEY: "test-placeholder", GEMINI_API_KEY: "test-placeholder", OPENCODE_ALLOW_ALL_MODELS: "1", CLAUDE_CONFIG_DIR: dir } },
context: {}, onLog: async () => {}, onUsage,
};
const execute = await fixture.execute();
const initial = await execute(context);
const complete = !["claude", "pi"].includes(fixture.name);
expect(initial.usageComplete).toBe(complete);
expect(onUsage.mock.calls.at(-1)![0]).toMatchObject({ usage: fixture.tokens, costUsd: fixture.price, complete });
if (fixture.name === "codex") {
for (const type of ["turn.failed", "error", "turn.completed"]) {
processResult.mockImplementationOnce(async (_run, _target, _command, _args, options) => {
const failedOutput = JSON.stringify({ type }) + "\n";
await options.onLog("stdout", failedOutput);
return { exitCode: 1, signal: null, timedOut: false, stdout: failedOutput, stderr: "", pid: 123, startedAt: new Date().toISOString() };
});
const failed = await execute({ ...context, config: { ...context.config, model: "gpt-6-astra" } });
expect(failed).toMatchObject({ usage: undefined, usageComplete: false, costStatus: "unpriced", costUsd: null });
expect(onUsage.mock.calls.at(-1)![0]).toMatchObject({ usage: undefined, complete: false, costStatus: "unpriced", costUsd: null });
}
for (const [kind, auth, biller] of [["openrouter", "api_key", "openrouter"], ["custom", "api_key", "unknown"], ["local", "none", "unknown"]]) {
const routed = await execute({ ...context, config: { ...context.config,
managedAiRouting: { kind, auth }, env: { ...(context.config.env as object), OPENAI_API_KEY: "", PAPERCLIP_AI_PROVIDER_KEY: auth === "none" ? "" : "fixture" },
} });
expect(routed).toMatchObject({ billingType: "api", biller, costUsd: null });
expect(onUsage.mock.calls.at(-1)![0]).toMatchObject({ billingType: "api", biller, costUsd: null });
}
for (const safer of [false, true]) {
const priced = await execute({ ...context,
config: { ...context.config, model: "gpt-5.4", fastMode: true },
context: safer ? { codexTransientFallbackMode: "fresh_session_safer_invocation" } : {},
});
const expected = safer ? undefined : { serviceTier: "fast" };
expect(priced.pricingContext).toEqual(expected);
expect(onUsage.mock.calls.at(-1)![0].pricingContext).toEqual(expected);
}
}
if (fixture.name === "opencode") {
for (const kind of ["zero_usage", "zero_price", "missing", "partial"]) for (const timedOut of [false, true]) {
const part = kind === "zero_usage" ? { tokens: { input: 0, output: 0 } } : kind === "zero_price" ? { cost: 0 } : kind === "partial" ? { tokens: { input: 2 } } : {};
processResult.mockImplementationOnce(async (_run, _target, _command, _args, options) => {
const output = JSON.stringify({ type: "step_finish", part });
await options.onLog("stdout", output);
return { exitCode: timedOut ? null : 0, signal: timedOut ? "SIGTERM" : null, timedOut, stdout: output, stderr: "", pid: 123, startedAt: new Date().toISOString() };
});
onUsage.mockClear();
const reported = await execute(context);
const complete = !timedOut && kind.startsWith("zero_");
expect(reported.usageComplete).toBe(complete);
expect(reported.usage).toEqual(kind === "zero_usage" ? { inputTokens: 0, outputTokens: 0, cachedInputTokens: 0 } : undefined);
expect(reported.costUsd).toBe(kind === "zero_price" ? 0 : null);
if (complete || !kind.startsWith("zero_")) expect(onUsage.mock.calls.at(-1)![0]).toMatchObject({ complete });
if (!kind.startsWith("zero_")) expect(reported.costStatus).toBe("unpriced");
}
}
if (fixture.name === "cursor") {
for (const type of ["result", "step_finish"]) for (const timedOut of [false, true]) {
processResult.mockImplementationOnce(async (_run, _target, _command, _args, options) => {
const output = JSON.stringify({ type }) + "\n";
await options.onLog("stdout", output);
return { exitCode: timedOut ? null : 0, signal: timedOut ? "SIGTERM" : null, timedOut, stdout: output, stderr: "", pid: 123, startedAt: new Date().toISOString() };
});
const missing = await execute({ ...context, config: { ...context.config, model: "gpt-5.6-sol" } });
expect(missing).toMatchObject({ usage: undefined, usageComplete: false, costStatus: "unpriced", costUsd: null });
expect(onUsage.mock.calls.at(-1)![0]).toMatchObject({ usage: undefined, complete: false, costStatus: "unpriced", costUsd: null });
}
for (const explicitPrice of [false, true]) {
processResult.mockImplementationOnce(async (_run, _target, _command, _args, options) => {
const output = JSON.stringify({ type: "result", ...(explicitPrice ? { total_cost_usd: 0 } : { usage: { input_tokens: 0, output_tokens: 0 } }) }) + "\n";
await options.onLog("stdout", output);
return { exitCode: 0, signal: null, timedOut: false, stdout: output, stderr: "", pid: 123, startedAt: new Date().toISOString() };
});
const reported = await execute(context);
expect(reported.usageComplete).toBe(true);
expect(reported.usage).toEqual(explicitPrice ? undefined : { inputTokens: 0, outputTokens: 0, cachedInputTokens: 0 });
expect(reported.costUsd).toBe(explicitPrice ? 0 : null);
expect(onUsage.mock.calls.at(-1)![0].complete).toBe(true);
}
}
if (["claude", "pi", "opencode"].includes(fixture.name)) {
processResult.mockImplementationOnce(async (_run, _target, _command, _args, options) => {
await options.onLog("stdout", stdout).catch(() => {});
return { exitCode: 1, signal: null, timedOut: false, stdout, stderr: "process failed", pid: 123, startedAt: new Date().toISOString() };
});
expect((await execute(context)).usageComplete).toBe(false);
expect(onUsage.mock.calls.at(-1)![0].complete).toBe(false);
}
if (fixture.name === "claude") {
const mixed = [
{ type: "system", subtype: "init", model: "actual-model" },
{ type: "result", total_cost_usd: 0.003, usage: { input_tokens: 3, output_tokens: 3 }, modelUsage: {
"model-a": { inputTokens: 1, outputTokens: 1, costUSD: 0.001 },
"model-b": { inputTokens: 2, outputTokens: 2, costUSD: 0.002 },
} },
];
processResult.mockImplementation(async (_run, _target, _command, _args, options) => {
const output = mixed.map(item => JSON.stringify(item)).join("\n");
await options.onLog("stdout", output).catch(() => {});
return { exitCode: 0, signal: null, timedOut: false, stdout: output, stderr: "", pid: 123, startedAt: new Date().toISOString() };
});
await execute(context);
expect(onUsage.mock.calls.at(-1)![0]).toMatchObject({ model: "mixed", costUsd: 0.003, complete: true, usageByModel: [
{ model: "model-a", costUsd: 0.001 }, { model: "model-b", costUsd: 0.002 },
] });
delete (mixed[1] as { modelUsage?: unknown }).modelUsage;
await execute(context);
expect(onUsage.mock.calls.at(-1)![0]).toMatchObject({ model: "actual-model", complete: true });
}
onUsage.mockRejectedValue(new Error("Receipt persistence failed"));
await expect(execute(context)).rejects.toThrow("Receipt persistence failed");
});
it.each(cases.filter(f => ["pi", "opencode"].includes(f.name)))("$name preserves full-stream accounting after stdout truncation", async (fixture) => {
const dir = await mkdtemp(join(tmpdir(), "paperclip-accounting-tail-")); directories.push(dir);
const command = join(dir, "runtime"); await writeFile(command, "#!/bin/sh\nprintf 'anthropic test\\n'\n", { mode: 0o755 });
const execute = await fixture.execute();
for (const checkpoint of [true, false]) for (const timedOut of [false, true]) for (const missingPrice of [false, true]) {
const early = structuredClone(fixture.event) as any;
if (missingPrice) {
if (fixture.name === "pi") delete early.message.usage.cost;
else delete early.part.cost;
}
const end = fixture.name === "pi" && !timedOut ? '\n{"type":"agent_end","messages":[]}' : "";
const first = JSON.stringify(early) + "\n";
const noise = JSON.stringify({ type: "text", part: { text: "x".repeat(9 * 1024 * 1024) } }) + "\n";
const last = JSON.stringify(fixture.event) + end;
processResult.mockImplementation(async (_run, _target, _command, _args, options) => {
for (const record of [first, noise, last]) {
for (let offset = 0; offset < record.length; offset += 65521) await options.onLog("stdout", record.slice(offset, offset + 65521));
}
return { exitCode: timedOut ? null : 0, signal: timedOut ? "SIGTERM" : null, timedOut, stdout: (first + noise + last).slice(-4 * 1024 * 1024), stderr: "", pid: 123, startedAt: new Date().toISOString() };
});
const onUsage = checkpoint ? vi.fn() : undefined;
const result = await execute({
runId: "tail-run", agent: { id: "test-agent", companyId: "test-company", name: "Accounting", adapterType: `${fixture.name}_local`, adapterConfig: {} },
runtime: { sessionId: null, sessionParams: null, sessionDisplayId: null, taskKey: null },
config: { engine: "cli", command, cwd: dir, model: "anthropic/test", paperclipRuntimeSkills: [], env: { ANTHROPIC_API_KEY: "fixture", OPENAI_BASE_URL: "https://unrelated-proxy.example/v1", OPENROUTER_API_KEY: "unrelated-key", OPENCODE_ALLOW_ALL_MODELS: "1" } },
context: {}, onLog: async () => {}, onUsage,
});
expect(result.usage).toMatchObject({ inputTokens: 40, outputTokens: 20, cachedInputTokens: 200 });
expect(result.costUsd).toBe(missingPrice ? null : 0.008);
expect(result.biller).toBe("anthropic");
expect(result.usageComplete).toBe(!timedOut);
if (onUsage) expect(onUsage.mock.calls.at(-1)![0]).toMatchObject({ usage: result.usage, costUsd: result.costUsd });
}
});
it.each(cases)("$name retains observed accounting when the process times out", async (fixture) => {
const dir = await mkdtemp(join(tmpdir(), "paperclip-accounting-adapter-")); directories.push(dir);
const command = join(dir, "runtime"); await writeFile(command, "#!/bin/sh\nprintf 'openai test\\n'\n", { mode: 0o755 });
processResult.mockResolvedValue({ exitCode: null, signal: "SIGTERM", timedOut: true, stdout: JSON.stringify(fixture.event), stderr: "", pid: 123, startedAt: new Date().toISOString() });
const execute = await fixture.execute();
const result = await execute({
runId: "test-run", agent: { id: "test-agent", companyId: "test-company", name: "Accounting", adapterType: `${fixture.name}_local`, adapterConfig: {} },
runtime: { sessionId: null, sessionParams: null, sessionDisplayId: null, taskKey: null },
config: { engine: "cli", command, cwd: dir, model: ["opencode", "pi"].includes(fixture.name) ? "openai/test" : fixture.name === "cursor" ? "gpt-5" : "test", paperclipRuntimeSkills: [],
env: { OPENAI_API_KEY: "test-placeholder", ANTHROPIC_API_KEY: "test-placeholder", GEMINI_API_KEY: "test-placeholder", MOONSHOT_API_KEY: "test-placeholder", OPENCODE_ALLOW_ALL_MODELS: "1", CLAUDE_CONFIG_DIR: dir } },
context: {}, onLog: async () => {},
});
expect(result.timedOut).toBe(true);
expect(result.usageComplete).toBe(["codex", "cursor", "gemini", "kimi"].includes(fixture.name));
if (fixture.name === "kimi") expect(result.costStatus).toBe("unpriced");
expect(result.usageBasis).toBe("per_run");
expect(result.provider).toBeTruthy();
expect(result.billingType).toBeTruthy();
if (fixture.tokens) expect(result.usage).toMatchObject(fixture.tokens);
expect(result.costUsd).toBe(fixture.price);
});
});
describe("incremental protocol accounting", () => {
it.each(["turn.failed", "error", "turn.started"])("saves an unpriced Codex %s checkpoint before exposing its log", async (type) => {
const { createCodexJsonlParser } = await import("../../../packages/adapters/codex-local/src/server/parse.js");
const { createUsageCheckpointLog } = await import("@paperclipai/adapter-utils/usage-checkpoint");
const consume = createCodexJsonlParser();
const order: string[] = [];
const saved = vi.fn(async () => { order.push("receipt"); });
const parsed = vi.fn((records: string) => {
const result = consume(records);
return { usage: result.usageReported ? result.usage : undefined, complete: result.usageComplete, costStatus: "unpriced" as const, costUsd: null };
});
const log = createUsageCheckpointLog(async () => { order.push("log"); }, saved, parsed);
await log("stdout", JSON.stringify({ type, message: "private provider failure" }) + "\n");
// No flush or adapter return has happened: a host crash after this point
// must preserve the unknown usage and keep the reservation pending.
expect(saved).toHaveBeenCalledWith(expect.objectContaining({ complete: false, costUsd: null, costStatus: "unpriced", usage: undefined }));
expect(order).toEqual(["receipt", "log"]);
expect(parsed.mock.calls[0][0]).not.toContain("private provider failure");
});
it("keeps Claude message updates idempotent and preserves immutable snapshots", async () => {
const { createClaudeStreamParser } = await import("../../../packages/adapters/claude-local/src/server/parse.js");
const consume = createClaudeStreamParser();
consume(JSON.stringify({ type: "system", subtype: "init", model: "actual-model" }));
const message = (id: string, output: number) => JSON.stringify({ type: "assistant", message: { id, usage: { input_tokens: 2, output_tokens: output } } });
const first = consume(message("one", 1));
expect(first.usage).toMatchObject({ inputTokens: 2, outputTokens: 1 });
expect(consume(message("one", 3)).usage).toMatchObject({ inputTokens: 2, outputTokens: 3 });
expect(consume(message("two", 2)).usage).toMatchObject({ inputTokens: 4, outputTokens: 5 });
expect(first.usage).toMatchObject({ inputTokens: 2, outputTokens: 1 });
const final = consume(JSON.stringify({ type: "result", total_cost_usd: 1, usage: { input_tokens: 9, output_tokens: 8 } }));
expect(final).toMatchObject({ model: "actual-model", costUsd: 1, usage: { inputTokens: 9, outputTokens: 8 } });
});
it.each(cases.filter(item => !["claude", "kimi"].includes(item.name)))("updates $name totals once per newly received record", async fixture => {
const factories = {
codex: async () => (await import("../../../packages/adapters/codex-local/src/server/parse.js")).createCodexJsonlParser(),
cursor: async () => (await import("../../../packages/adapters/cursor-local/src/server/parse.js")).createCursorJsonlParser(),
gemini: async () => (await import("../../../packages/adapters/gemini-local/src/server/parse.js")).createGeminiJsonlParser(),
pi: async () => (await import("../../../packages/adapters/pi-local/src/server/parse.js")).createPiJsonlParser(),
opencode: async () => (await import("../../../packages/adapters/opencode-local/src/server/parse.js")).createOpenCodeJsonlParser(),
};
const consume = await factories[fixture.name as keyof typeof factories]();
const line = JSON.stringify(fixture.event);
const first = consume(line);
const second = consume(line);
expect(first.usage).toMatchObject(fixture.tokens!);
const multiplier = fixture.name === "codex" ? 1 : 2;
expect(second.usage).toMatchObject(Object.fromEntries(Object.entries(fixture.tokens!).map(([key, count]) => [key, count * multiplier])));
});
});
describe("process adapter bootstrap accounting proof", () => {
const context = (config: Record<string, unknown>): AdapterExecutionContext => ({
runId: `bootstrap-${crypto.randomUUID()}`,
agent: { id: "agent", companyId: "company", name: "Process", adapterType: "process", adapterConfig: {} },
runtime: { sessionId: null, sessionParams: null, sessionDisplayId: null, taskKey: null },
config, context: {}, onLog: async () => {},
});
it.each([{}, { command: "/nonexistent-paperclip-accounting-command" },
{ command: process.execPath, cwd: "/nonexistent-paperclip-accounting-directory" }])(
"proves no provider work for a failure before spawn: %j", async (config) => {
const { execute } = await import("../adapters/process/execute.js");
const onSpawn = vi.fn();
const result = await execute({ ...context(config), onSpawn });
expect(result).toMatchObject({ exitCode: 1, executionRecovery: { kind: "bootstrap", providerWorkStarted: false } });
expect(result.errorMessage).toBeTruthy();
expect(onSpawn).not.toHaveBeenCalled();
},
);
it("does not invent bootstrap proof when a spawned process fails", async () => {
const { execute } = await import("../adapters/process/execute.js");
const onSpawn = vi.fn(async () => { throw new Error("Metadata persistence failed"); });
const result = await execute({ ...context({ command: process.execPath, args: ["-e", "process.exit(7)"] }), onSpawn });
expect(onSpawn).toHaveBeenCalledOnce();
expect(result.exitCode).toBe(7);
expect(result.executionRecovery).toBeUndefined();
});
});
server/src/__tests__/agent-hire-ai-connections.test.ts
+34
View File
@@ -1,3 +1,5 @@
import * as codexAdapter from "@paperclipai/adapter-codex-local/server";
import { WORKSPACE_RESTORE_LOCK_TIMEOUT_CODE } from "@paperclipai/adapter-utils/workspace-restore-merge";
import { randomUUID } from "node:crypto";
import { mkdtemp, rm } from "node:fs/promises";
import os from "node:os";
@@ -470,6 +472,38 @@ describe("agent-created hires use managed AI connections", () => {
});
describe("hired agents sharing a subscription", () => {
it("keeps a credential-lock timeout on automatic retry without blocking the task or starting a provider", async () => {
const f = await fixture("openai", "subscription");
const agent = hired(await request(f.app).post(`/api/companies/${f.companyId}/agent-hires`).send({ name: "Waiting teammate", role: "engineer", adapterType: f.adapterType, reportsTo: f.agentId, adapterConfig: { engine: "cli" }, runtimeConfig: { heartbeat: { enabled: false } } }));
// Only an operator may set host execution paths after the agent is hired.
await db.update(agents).set({ adapterConfig: { ...agent.adapterConfig, cwd: home } }).where(eq(agents.id, agent.id));
const [issue] = await db.insert(issues).values({ companyId: f.companyId, title: "Wait for credential rotation", status: "todo", assigneeAgentId: agent.id, responsibleUserId: f.userId, createdByUserId: f.userId }).returning();
const execute = vi.fn(async () => ({ exitCode: 0, signal: null, timedOut: false, resultJson: {} }));
registerServerAdapter({ ...getServerAdapter(f.adapterType), execute });
const lock = vi.spyOn(codexAdapter, "withAccountHomeSecretMutationLock")
.mockRejectedValueOnce(Object.assign(new Error("quota refresh holds company lock"), { code: WORKSPACE_RESTORE_LOCK_TIMEOUT_CODE }));
const heartbeat = heartbeatService(db);
try {
const run = await heartbeat.invoke(agent.id, "assignment", { issueId: issue.id, wakeReason: "issue_assigned", responsibleUserId: f.userId }, "system");
expect(run).not.toBeNull();
await expect.poll(async () => (await heartbeat.getRun(run!.id))?.status, { timeout: 20_000 }).toBe("cancelled");
await heartbeat.drainActiveRunExecutions();
expect(await heartbeat.getRun(run!.id)).toMatchObject({ errorCode: "ai_connection_busy", resultJson: { executionRecovery: { providerWorkStarted: false } } });
const retries = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, run!.id));
expect(retries).toHaveLength(1);
expect(retries[0]).toMatchObject({ status: "scheduled_retry", scheduledRetryReason: "ai_connection_busy" });
const [savedIssue] = await db.select().from(issues).where(eq(issues.id, issue.id));
expect(savedIssue.status).not.toBe("blocked");
expect(savedIssue.executionRunId).toBe(retries[0].id);
expect(execute).not.toHaveBeenCalled();
expect(await aiConnectionService(db).list(f.companyId, f.userId)).toEqual([expect.objectContaining({ status: "connected" })]);
} finally {
lock.mockRestore();
await heartbeat.drainActiveRunExecutions();
unregisterServerAdapter(f.adapterType);
}
});
it.each(["openai", "anthropic"] as const)("runs the %s child alongside a live parent and inherits its connection", async (provider) => {
const f = await fixture(provider, "subscription");
const agent = hired(await request(f.app).post(`/api/companies/${f.companyId}/agent-hires`).send({ name: "Concurrent teammate", role: "engineer", adapterType: f.adapterType, reportsTo: f.agentId, adapterConfig: { engine: "cli" }, runtimeConfig: { heartbeat: { enabled: false } } }));
server/src/__tests__/agent-instructions-routes.test.ts
+8 -2
View File
@@ -211,7 +211,7 @@ function makeReflectionCoachAgent(overrides: Record<string, unknown> = {}) {
}
describe("agent instructions bundle routes", () => {
beforeEach(() => {
beforeEach(async () => {
vi.resetModules();
vi.doUnmock("../routes/agents.js");
vi.doUnmock("../routes/authz.js");
@@ -288,7 +288,13 @@ describe("agent instructions bundle routes", () => {
instructionsFilePath: "/tmp/agent-1/AGENTS.md",
},
});
});
// Cold route compilation belongs to fixture setup. Timing it as a request
// let a timed-out import resume under the next case's reset mocks.
await Promise.all([
vi.importActual("../routes/agents.js"),
vi.importActual("../middleware/index.js"),
]);
}, 60_000);
it("returns bundle metadata", async () => {
const res = await requestApp(
server/src/__tests__/agent-test-environment-routes.test.ts
+4
View File
@@ -110,6 +110,10 @@ const mockPrepareManagedAiRuntime = vi.hoisted(() => vi.fn());
vi.mock("../services/ai-connection-runtime.js", async (importOriginal) => ({
...(await importOriginal<typeof import("../services/ai-connection-runtime.js")>()),
prepareManagedAiRuntime: mockPrepareManagedAiRuntime,
withManagedAiProbe: async (_db: unknown, input: unknown, probe: (runtime: unknown) => Promise<unknown>) => {
const runtime = await mockPrepareManagedAiRuntime(_db, input);
try { return await probe(runtime); } finally { await runtime.cleanup(); }
},
}));
const mockValidateAiApiKey = vi.hoisted(() => vi.fn(async () => undefined));
vi.mock("../routes/ai-connections.js", async (importOriginal) => ({
server/src/__tests__/ai-connections.test.ts
+644 -2
View File
@@ -7,15 +7,21 @@ import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
import { createHash, randomUUID } from "node:crypto";
import { spawnSync } from "node:child_process";
import { localEncryptedProvider } from "../secrets/local-encrypted-provider.js";
import { promises as fs } from "node:fs";
import { mkdtemp, realpath, rm, access, readFile, writeFile, stat } from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { and, eq, sql } from "drizzle-orm";
import { createDb, companies, agents, agentTaskSessions, heartbeatRuns, companyMemberships, connectionGrants, toolApplications, connectionGrantDelegations, connectionGrantMembers, toolConnections, toolConnectionInstalls, aiConnectionDefaults, aiProviderDefaults, adapterAuthSessions, environments, issues, issueThreadInteractions, issueRecoveryActions, connectionIntentDeliveries, agentWakeupRequests, companySecrets, principalPermissionGrants } from "@paperclipai/db";
import { createDb, companies, agents, agentTaskSessions, heartbeatRuns, companyMemberships, connectionGrants, toolApplications, connectionGrantDelegations, connectionGrantMembers, toolConnections, toolConnectionInstalls, aiConnectionDefaults, aiProviderDefaults, adapterAuthSessions, environments, issues, issueThreadInteractions, issueRecoveryActions, connectionIntentDeliveries, agentWakeupRequests, companySecrets, companySecretVersions, userSecretDefinitions, principalPermissionGrants } from "@paperclipai/db";
import { startEmbeddedPostgresTestDatabase } from "@paperclipai/db/test-embedded-postgres";
import { fetchCompanyQuotaWindows } from "../services/quota-windows.js";
import { quotaCredentialRecoveryPath, quotaCredentialRecovery, quotaCredentialHash } from "../services/quota-credential-recovery.js";
import { aiConnectionService } from "../services/ai-connections.js";
import { syncConnectionCredentialBindings } from "../services/connection-credential-bindings.js";
import * as codexAdapter from "@paperclipai/adapter-codex-local/server";
import { WORKSPACE_RESTORE_LOCK_TIMEOUT_CODE } from "@paperclipai/adapter-utils/workspace-restore-merge";
import * as executionTarget from "@paperclipai/adapter-utils/execution-target";
import { prepareManagedAiRuntime, assertManagedAiProjectAuth } from "../services/ai-connection-runtime.js";
import { prepareManagedAiRuntime, withManagedAiProbe, assertManagedAiProjectAuth, isAiConnectionBusy } from "../services/ai-connection-runtime.js";
import { execute as executeGemini, testEnvironment as testGeminiEnvironment } from "@paperclipai/adapter-gemini-local/server";
import { toolAccessService } from "../services/tool-access.js";
import { secretService } from "../services/secrets.js";
@@ -54,6 +60,625 @@ beforeAll(async () => {
afterAll(async () => { await database?.cleanup(); vi.unstubAllEnvs(); if (home) await rm(home, { recursive: true, force: true }); });
describe("managed AI connections", () => {
it("persists first-time recovery directories before consuming a single-use token", async () => {
const owner = "quota-durable-path";
await db.insert(companyMemberships).values({ companyId, principalId: owner, principalType: "user", status: "active", membershipRole: "member" });
const original = JSON.stringify({ tokens: { account_id: owner, access_token: "expired", refresh_token: "single-use" } });
await service.save(companyId, owner, { provider: "openai", method: "subscription", ownership: "personal", name: owner, loginSessionId: "fixture", agentIds: [], allAgents: true }, original);
const [row] = await service.quotaAccounts(companyId, owner);
const file = quotaCredentialRecoveryPath(companyId, row.grant.id);
const recoveryRoot = path.dirname(path.dirname(file));
await expect(access(recoveryRoot)).rejects.toMatchObject({ code: "ENOENT" });
const synced: string[] = [];
let rejectRoot = true;
const realOpen = fs.open.bind(fs);
const open = vi.spyOn(fs, "open").mockImplementation(async (...args: Parameters<typeof fs.open>) => {
const handle = await realOpen(...args);
const sync = handle.sync.bind(handle);
vi.spyOn(handle, "sync").mockImplementation(async () => {
if (rejectRoot && args[0] === recoveryRoot) throw new Error("Injected recovery directory flush failure");
await sync(); synced.push(String(args[0]));
});
return handle;
});
const request = vi.spyOn(globalThis, "fetch").mockImplementation(async () => {
expect(synced).toContain(recoveryRoot);
expect(synced).toContain(path.dirname(recoveryRoot));
return Response.json({ access_token: "fresh-durable", refresh_token: "replacement-durable" });
});
try {
await expect(service.refreshQuotaCredential(row, original, new AbortController().signal)).rejects.toThrow("Injected recovery directory flush failure");
expect(request).not.toHaveBeenCalled();
expect(await service.credential(row)).toBe(original);
rejectRoot = false;
await service.refreshQuotaCredential(row, original, new AbortController().signal);
expect(request).toHaveBeenCalledTimes(1);
expect(JSON.parse(await service.credential(row))).toMatchObject({ tokens: { access_token: "fresh-durable", refresh_token: "replacement-durable" } });
} finally { open.mockRestore(); request.mockRestore(); }
});
it.each(["timeout", "unexpected"] as const)("classifies a manual probe's %s lock failure before starting the provider", async (kind) => {
const owner = `manual-probe-lock-${kind}`;
await db.insert(companyMemberships).values({ companyId, principalId: owner, principalType: "user", status: "active", membershipRole: "member" });
const auth = JSON.stringify({ tokens: { account_id: owner, access_token: "valid-access", refresh_token: "valid-refresh" } });
await service.save(companyId, owner, { provider: "openai", method: "subscription", ownership: "personal", name: owner, loginSessionId: "fixture", agentIds: [], allAgents: true }, auth);
const [row] = await service.quotaAccounts(companyId, owner);
const probe = vi.fn().mockResolvedValue("ok");
const input = { companyId, agentId, responsibleUserId: owner, adapterType: "codex_local",
binding: { provider: "openai", method: "subscription", mode: "responsible_user" } as const, config: {} };
const failure = Object.assign(new Error("injected private lock diagnostic"), { code: kind === "timeout" ? WORKSPACE_RESTORE_LOCK_TIMEOUT_CODE : "EACCES" });
const lock = vi.spyOn(codexAdapter, "withAccountHomeSecretMutationLock").mockRejectedValueOnce(failure);
try {
const error = await withManagedAiProbe(db, input, probe).then(() => { throw new Error("Probe must not start"); }, error => error);
if (kind === "timeout") {
expect(isAiConnectionBusy(error)).toBe(true);
expect(error).toMatchObject({ status: 422, message: "AI credentials are being updated. Retry shortly.", details: { code: "ai_connection_busy" } });
} else expect(error).toBe(failure);
expect(probe).not.toHaveBeenCalled();
expect((await db.select().from(connectionGrants).where(eq(connectionGrants.id, row.grant.id)))[0].status).toBe("active");
} finally { lock.mockRestore(); }
expect(await withManagedAiProbe(db, input, probe)).toBe("ok");
expect(probe).toHaveBeenCalledTimes(1);
expect(await service.credential(row)).toBe(auth);
});
it("keeps quota refresh outside a manual probe until its rotated credential is saved", async () => {
const owner = "manual-probe-race";
await db.insert(companyMemberships).values({ companyId, principalId: owner, principalType: "user", status: "active", membershipRole: "member" });
const auth = (hour: string) => JSON.stringify({ tokens: { account_id: owner, id_token: "identity", access_token: `access-${hour}`, refresh_token: `refresh-${hour}` }, last_refresh: `2026-09-10T${hour}:00:00Z` });
await service.save(companyId, owner, { provider: "openai", method: "subscription", ownership: "personal", name: owner, loginSessionId: "fixture", agentIds: [], allAgents: true }, auth("10"));
const [row] = await service.quotaAccounts(companyId, owner);
const entered = deferredSignal(), release = deferredSignal();
const request = vi.spyOn(globalThis, "fetch").mockRejectedValue(new Error("Quota must not exchange the probe's token"));
const probing = withManagedAiProbe(db, { companyId, agentId, responsibleUserId: owner, adapterType: "codex_local",
binding: { provider: "openai", method: "subscription", mode: "responsible_user" }, config: {} }, async runtime => {
entered.resolve(); await release.promise;
await writeFile(path.join(String(runtime.config.env.CODEX_HOME), "auth.json"), auth("11"));
});
let refreshing: Promise<string> | undefined;
let lock: ReturnType<typeof vi.spyOn> | undefined;
try {
await entered.promise;
lock = vi.spyOn(codexAdapter, "withAccountHomeSecretMutationLock");
// Start outside the probe's async scope; it is a separate HTTP request.
refreshing = service.refreshQuotaCredential(row, auth("10"), new AbortController().signal);
await vi.waitFor(() => expect(lock).toHaveBeenCalled());
expect(request).not.toHaveBeenCalled();
release.resolve(); await probing;
expect(await refreshing).toBe(auth("11"));
expect(await service.credential(row)).toBe(auth("11"));
expect(request).not.toHaveBeenCalled();
} finally { release.resolve(); await Promise.allSettled([probing, refreshing]); lock?.mockRestore(); request.mockRestore(); }
});
it("rejects preparation when reconnect replaces the selected credential reference", async () => {
const owner = "replaced-ref-owner";
await db.insert(companyMemberships).values({ companyId, principalId: owner, principalType: "user", status: "active", membershipRole: "member" });
const auth = (id: string) => JSON.stringify({ tokens: { account_id: id, access_token: id, refresh_token: id } });
await service.save(companyId, owner, { provider: "openai", method: "subscription", ownership: "personal", name: owner, loginSessionId: "fixture", agentIds: [], allAgents: true }, auth("old-account"));
const [row] = await service.quotaAccounts(companyId, owner);
const definition = await secretService(db).createUserSecretDefinition(companyId, { key: "replacement_credential", name: "Replacement", provider: "local_encrypted" }, { userId: owner });
const replacement = await secretService(db).createCurrentUserSecretValue(companyId, owner, { definitionId: definition.id, value: auth("new-account") }, { userId: owner });
const prepare = () => prepareManagedAiRuntime(db, { companyId, agentId, responsibleUserId: owner, adapterType: "codex_local",
binding: { provider: "openai", method: "subscription", mode: "responsible_user" }, config: {} });
const original = codexAdapter.withAccountHomeSecretMutationLock;
const lock = vi.spyOn(codexAdapter, "withAccountHomeSecretMutationLock").mockImplementationOnce(async (env, company, callback) => {
await db.update(connectionGrants).set({ credentialSecretRefs: row.grant.credentialSecretRefs.map(ref => ref.configPath === "ai.credential" ? { ...ref, secretId: replacement.id } : ref) }).where(eq(connectionGrants.id, row.grant.id));
await syncConnectionCredentialBindings(db, row.connection);
return original(env, company, callback);
});
try { await expect(prepare()).rejects.toThrow("AI credential changed during preparation"); }
finally { lock.mockRestore(); }
const runtime = await prepare();
try {
expect(runtime.sessionIdentity).toContain(replacement.id);
expect(await readFile(path.join(String(runtime.config.env.CODEX_HOME), "auth.json"), "utf8")).toBe(auth("new-account"));
} finally { await runtime.cleanup(); }
});
it("caches quota after a real secret read but invalidates credential rotation and revocation", async () => {
const owner = "quota-cache-owner";
await db.insert(companyMemberships).values({ companyId, principalId: owner, principalType: "user", status: "active", membershipRole: "member" });
const account = await service.save(companyId, owner, { provider: "anthropic", method: "subscription", ownership: "personal", name: "Quota cache account", loginSessionId: "fixture", agentIds: [], allAgents: true }, "quota-cache-original");
const [grant] = await db.select().from(connectionGrants).where(eq(connectionGrants.id, account.grantId));
const secretId = grant.credentialSecretRefs.find(ref => ref.configPath === "ai.credential")!.secretId;
const [before] = await db.select().from(companySecrets).where(eq(companySecrets.id, secretId));
const fetchSpy = vi.spyOn(globalThis, "fetch").mockImplementation(async () => new Response(JSON.stringify({ five_hour: { utilization: 42 } })));
try {
const first = await fetchCompanyQuotaWindows(db, companyId, owner);
const [after] = await db.select().from(companySecrets).where(eq(companySecrets.id, secretId));
expect(after.lastResolvedAt).not.toBeNull();
expect(after.updatedAt.getTime()).toBeGreaterThan(before.updatedAt.getTime());
expect(first[0].ok).toBe(true);
expect(await fetchCompanyQuotaWindows(db, companyId, owner)).toEqual(first);
expect(fetchSpy).toHaveBeenCalledTimes(1);
await secretService(db).rotate(secretId, { value: "quota-cache-rotated" });
const rotated = await fetchCompanyQuotaWindows(db, companyId, owner);
expect(rotated[0].ok).toBe(true);
expect(rotated[0].accountKey).not.toBe(first[0].accountKey);
expect(fetchSpy).toHaveBeenCalledTimes(2);
expect(fetchSpy).toHaveBeenLastCalledWith(expect.any(String), expect.objectContaining({ headers: expect.objectContaining({ Authorization: "Bearer quota-cache-rotated" }) }));
await db.update(userSecretDefinitions).set({ status: "disabled" }).where(eq(userSecretDefinitions.id, before.userSecretDefinitionId!));
const disabled = await fetchCompanyQuotaWindows(db, companyId, owner);
expect(disabled[0]).toMatchObject({ ok: false, errorFamily: "credentials_unavailable", windows: [] });
expect(disabled[0].accountKey).not.toBe(rotated[0].accountKey);
expect(fetchSpy).toHaveBeenCalledTimes(2);
await db.update(userSecretDefinitions).set({ status: "active" }).where(eq(userSecretDefinitions.id, before.userSecretDefinitionId!));
expect((await fetchCompanyQuotaWindows(db, companyId, owner))[0].ok).toBe(true);
await db.update(companySecretVersions).set({ status: "disabled", revokedAt: new Date() }).where(and(eq(companySecretVersions.secretId, secretId), eq(companySecretVersions.version, 2)));
const revoked = await fetchCompanyQuotaWindows(db, companyId, owner);
expect(revoked[0].ok).toBe(false);
expect(revoked[0].windows).toEqual([]);
expect(fetchSpy).toHaveBeenCalledTimes(2);
} finally { fetchSpy.mockRestore(); }
});
it("caches refreshed quota under the saved grant and secret revision", async () => {
const owner = "quota-refreshed-cache-owner";
await db.insert(companyMemberships).values({ companyId, principalId: owner, principalType: "user", status: "active", membershipRole: "member" });
const original = JSON.stringify({ tokens: { access_token: "expired-cache", refresh_token: "single-use", account_id: "cache-account" } });
await service.save(companyId, owner, { provider: "openai", method: "subscription", ownership: "personal", name: "Refreshed cache", loginSessionId: "fixture", agentIds: [], allAgents: true }, original);
const request = vi.spyOn(globalThis, "fetch").mockImplementation(async (url, options) => {
if (String(url).endsWith("/oauth/token")) return Response.json({ access_token: "fresh-cache", refresh_token: "replacement" });
if (new Headers(options?.headers).get("authorization") === "Bearer expired-cache") return new Response("Expired", { status: 401 });
return Response.json({ rate_limit: { primary_window: { used_percent: 42 } } });
});
try {
const first = await fetchCompanyQuotaWindows(db, companyId, owner);
expect(first[0]).toMatchObject({ ok: true, windows: [{ usedPercent: 42 }] });
expect(await fetchCompanyQuotaWindows(db, companyId, owner)).toEqual(first);
expect(request).toHaveBeenCalledTimes(3); // rejected quota, OAuth, refreshed quota
} finally { request.mockRestore(); }
});
it.each(["rotation", "secret_metadata", "activity", "commit"] as const)("recovers exchanged OAuth tokens after %s persistence failure without another exchange", async (failure) => {
const owner = `quota-rollback-${failure}`;
await db.insert(companyMemberships).values({ companyId, principalId: owner, principalType: "user", status: "active", membershipRole: "member" });
const original = JSON.stringify({ tokens: { access_token: `expired-${failure}`, refresh_token: "single-use", account_id: owner } });
await service.save(companyId, owner, { provider: "openai", method: "subscription", ownership: "personal", name: owner, loginSessionId: "fixture", agentIds: [], allAgents: true }, original);
const [row] = await service.quotaAccounts(companyId, owner);
const ref = row.grant.credentialSecretRefs.find(ref => ref.configPath === "ai.credential")!;
const table = failure === "secret_metadata" ? "company_secrets" : failure === "activity" ? "activity_log" : "company_secret_versions";
const condition = failure === "secret_metadata" ? `new.id = '${ref.secretId}'::uuid and new.latest_version > old.latest_version` : failure === "activity" ? `new.action = 'ai_connection.credential_refreshed'` : `new.secret_id = '${ref.secretId}'::uuid`;
await db.execute(sql.raw(`create function quota_save_failure() returns trigger language plpgsql as $$ begin
if ${condition} then raise exception 'injected quota save failure'; end if; return new; end $$;
create ${failure === "commit" ? "constraint " : ""}trigger quota_save_failure ${failure === "commit" ? "after" : "before"} ${failure === "secret_metadata" ? "update" : "insert"} on ${table}
${failure === "commit" ? "deferrable initially deferred" : ""} for each row execute function quota_save_failure();`));
const request = vi.spyOn(globalThis, "fetch").mockResolvedValue(Response.json({ access_token: `fresh-${failure}`, refresh_token: `replacement-${failure}` }));
const file = quotaCredentialRecoveryPath(companyId, row.grant.id);
let triggerDropped = false;
try {
await expect(service.refreshQuotaCredential(row, original, new AbortController().signal)).rejects.toThrow();
expect(request).toHaveBeenCalledTimes(1);
expect(await service.credential(row)).toBe(original);
const encrypted = await readFile(file, "utf8");
expect(encrypted).not.toContain(`fresh-${failure}`);
expect(encrypted).not.toContain(`replacement-${failure}`);
await db.execute(sql.raw(`drop trigger quota_save_failure on ${table}; drop function quota_save_failure();`));
triggerDropped = true;
// A new service instance recovers from disk, without an in-memory cache.
const recovered = aiConnectionService(db);
await recovered.refreshQuotaCredential(row, original, new AbortController().signal);
expect(JSON.parse(await recovered.credential(row))).toMatchObject({ tokens: { access_token: `fresh-${failure}`, refresh_token: `replacement-${failure}` } });
expect(request).toHaveBeenCalledTimes(1);
await expect(access(file)).rejects.toThrow();
expect((await db.select().from(companySecretVersions).where(eq(companySecretVersions.secretId, ref.secretId))).map(version => version.version).sort()).toEqual([1, 2]);
} finally {
request.mockRestore();
if (!triggerDropped) await db.execute(sql.raw(`drop trigger quota_save_failure on ${table}; drop function quota_save_failure();`));
}
});
it.each(["company_file", "grant_row", "secret_row"] as const)("defers runtime preparation on a contended %s lock and succeeds once it clears", async (kind) => {
const owner = `runtime-lock-${kind}`;
await db.insert(companyMemberships).values({ companyId, principalId: owner, principalType: "user", status: "active", membershipRole: "member" });
const auth = JSON.stringify({ tokens: { access_token: "valid-access", refresh_token: "valid-refresh", account_id: owner } });
await service.save(companyId, owner, { provider: "openai", method: "subscription", ownership: "personal", name: owner, loginSessionId: "fixture", agentIds: [], allAgents: true }, auth);
const [row] = await service.quotaAccounts(companyId, owner);
const ref = row.grant.credentialSecretRefs.find(ref => ref.configPath === "ai.credential")!;
const prepare = () => prepareManagedAiRuntime(db, { companyId, agentId, responsibleUserId: owner, adapterType: "codex_local",
binding: { provider: "openai", method: "subscription", mode: "responsible_user" }, config: {} });
const held = deferredSignal(), release = deferredSignal();
const lock = kind === "company_file" ? vi.spyOn(codexAdapter, "withAccountHomeSecretMutationLock")
.mockRejectedValueOnce(Object.assign(new Error("company lock held by another account"), { code: WORKSPACE_RESTORE_LOCK_TIMEOUT_CODE })) : undefined;
const holding = kind === "company_file" ? Promise.resolve() : db.transaction(async tx => {
if (kind === "grant_row") await tx.select().from(connectionGrants).where(eq(connectionGrants.id, row.grant.id)).for("update");
else await tx.select().from(companySecrets).where(eq(companySecrets.id, ref.secretId)).for("update");
held.resolve(); await release.promise;
});
try {
if (kind !== "company_file") await held.promise;
const error = await prepare().then(() => { throw new Error("Runtime must defer"); }, error => error);
expect(isAiConnectionBusy(error)).toBe(true);
expect(error).toMatchObject({ status: 422, details: { code: "ai_connection_busy" } });
expect((await db.select().from(connectionGrants).where(eq(connectionGrants.id, row.grant.id)))[0].status).toBe("active");
} finally { release.resolve(); await holding; lock?.mockRestore(); }
const run = await prepare();
try { expect(await readFile(path.join(String(run.config.env.CODEX_HOME), "auth.json"), "utf8")).toBe(auth); }
finally { await run.cleanup(); }
});
it.each([
["runtime", "rotation"], ["runtime", "commit"], ["failed_run", "rotation"], ["failed_run", "commit"],
] as const)("recovers pending quota credentials before %s can reuse or invalidate the consumed token after %s failure", async (consumer, failure) => {
const owner = `quota-consumer-${consumer}-${failure}`;
await db.insert(companyMemberships).values({ companyId, principalId: owner, principalType: "user", status: "active", membershipRole: "member" });
const auth = (marker: string) => JSON.stringify({ tokens: { access_token: `${marker}-access`, refresh_token: `${marker}-refresh`, account_id: owner } });
await service.save(companyId, owner, { provider: "openai", method: "subscription", ownership: "personal", name: owner, loginSessionId: "fixture", agentIds: [], allAgents: true }, auth("old"));
const [row] = await service.quotaAccounts(companyId, owner);
const prepare = () => prepareManagedAiRuntime(db, { companyId, agentId, responsibleUserId: owner, adapterType: "codex_local",
binding: { provider: "openai", method: "subscription", mode: "responsible_user" }, config: {} });
const oldRun = await prepare();
const ref = row.grant.credentialSecretRefs.find(ref => ref.configPath === "ai.credential")!;
const journal = quotaCredentialRecovery(companyId, row.grant.id);
await journal.write({ companyId, grantId: row.grant.id, connectionId: row.connection.id, secretId: ref.secretId,
baseHash: quotaCredentialHash(auth("old")), value: auth("new") });
const fail = () => service.markAuthenticationFailed({ companyId, agentId, runStartedAt: new Date(),
attribution: { ...oldRun.attribution, identity: oldRun.identity } });
await db.execute(sql.raw(`create function reject_consumer_recovery() returns trigger language plpgsql as $$ begin
if new.secret_id = '${ref.secretId}'::uuid then raise exception 'recovery unavailable'; end if; return new; end $$;
create ${failure === "commit" ? "constraint " : ""}trigger reject_consumer_recovery ${failure === "commit" ? "after" : "before"} insert on company_secret_versions
${failure === "commit" ? "deferrable initially deferred" : ""} for each row execute function reject_consumer_recovery();`));
try {
if (consumer === "runtime") {
const error = await prepare().then(() => { throw new Error("Runtime must defer"); }, error => error);
expect(isAiConnectionBusy(error)).toBe(true);
expect(error).toMatchObject({ status: 422, details: { code: "ai_connection_busy" } });
} else await expect(fail()).rejects.toThrow();
expect((await db.select().from(connectionGrants).where(eq(connectionGrants.id, row.grant.id)))[0].status).toBe("active");
expect(await service.credential(row)).toBe(auth("old"));
expect(await journal.read()).not.toBeNull();
} finally {
await db.execute(sql`drop trigger reject_consumer_recovery on company_secret_versions; drop function reject_consumer_recovery()`);
}
const request = vi.spyOn(globalThis, "fetch").mockRejectedValue(new Error("Recovery must not exchange again"));
let nextRun: Awaited<ReturnType<typeof prepare>> | undefined;
try {
if (consumer === "failed_run") await fail();
nextRun = await prepare();
expect(await readFile(path.join(String(nextRun.config.env.CODEX_HOME), "auth.json"), "utf8")).toBe(auth("new"));
expect(nextRun.sessionIdentity).toBe(oldRun.sessionIdentity);
await fail(); // A late failure of the old run cannot invalidate the recovered identity.
expect((await db.select().from(connectionGrants).where(eq(connectionGrants.id, row.grant.id)))[0].status).toBe("active");
expect(await service.credential(row)).toBe(auth("new"));
expect(await journal.read()).toBeNull();
expect(request).not.toHaveBeenCalled();
expect(await db.select().from(companySecretVersions).where(eq(companySecretVersions.secretId, ref.secretId))).toHaveLength(2);
} finally { request.mockRestore(); await oldRun.cleanup(); await nextRun?.cleanup(); await journal.clear(); }
});
it.each(["rollback", "lost_commit_reply"] as const)("automatically saves one exchanged token across %s", async (failure) => {
const owner = `quota-once-${failure}`;
await db.insert(companyMemberships).values({ companyId, principalId: owner, principalType: "user", status: "active", membershipRole: "member" });
const original = JSON.stringify({ tokens: { access_token: "expired", refresh_token: "single-use", account_id: owner } });
await service.save(companyId, owner, { provider: "openai", method: "subscription", ownership: "personal", name: owner, loginSessionId: "fixture", agentIds: [], allAgents: true }, original);
const [row] = await service.quotaAccounts(companyId, owner);
const transaction = db.transaction.bind(db);
const interrupted = vi.spyOn(db, "transaction").mockImplementationOnce(async (work, config) => {
await transaction(async (tx) => {
await work(tx);
if (failure === "rollback") throw new Error("Injected rollback after writes");
}, config);
throw new Error("Injected lost commit response");
});
const request = vi.spyOn(globalThis, "fetch").mockResolvedValue(Response.json({ access_token: "fresh-once", refresh_token: "replacement-once" }));
try {
await service.refreshQuotaCredential(row, original, new AbortController().signal);
expect(JSON.parse(await service.credential(row))).toMatchObject({ tokens: { access_token: "fresh-once", refresh_token: "replacement-once" } });
expect(request).toHaveBeenCalledTimes(1);
await expect(access(quotaCredentialRecoveryPath(companyId, row.grant.id))).rejects.toThrow();
} finally { interrupted.mockRestore(); request.mockRestore(); }
});
it("recovers the current credential's pending replacement before answering a delayed older poll", async () => {
const owner = "delayed-quota-poll";
await db.insert(companyMemberships).values({ companyId, principalId: owner, principalType: "user", status: "active", membershipRole: "member" });
const original = JSON.stringify({ tokens: { access_token: "A", refresh_token: "refresh-A", account_id: owner } });
await service.save(companyId, owner, { provider: "openai", method: "subscription", ownership: "personal", name: owner, loginSessionId: "fixture", agentIds: [], allAgents: true }, original);
const [row] = await service.quotaAccounts(companyId, owner);
const ref = row.grant.credentialSecretRefs.find(ref => ref.configPath === "ai.credential")!;
const journal = quotaCredentialRecovery(companyId, row.grant.id);
const request = vi.spyOn(globalThis, "fetch")
.mockResolvedValueOnce(Response.json({ access_token: "B", refresh_token: "refresh-B" }))
.mockResolvedValueOnce(Response.json({ access_token: "C", refresh_token: "refresh-C" }));
let triggerInstalled = false;
try {
const current = await service.refreshQuotaCredential(row, original, new AbortController().signal);
await db.execute(sql.raw(`create function delayed_quota_commit_failure() returns trigger language plpgsql as $$ begin
if new.secret_id = '${ref.secretId}'::uuid then raise exception 'injected delayed quota commit failure'; end if; return new; end $$;
create constraint trigger delayed_quota_commit_failure after insert on company_secret_versions deferrable initially deferred
for each row execute function delayed_quota_commit_failure();`));
triggerInstalled = true;
await expect(service.refreshQuotaCredential(row, current, new AbortController().signal)).rejects.toThrow();
expect(await service.credential(row)).toBe(current);
expect(JSON.parse((await journal.read())!.value).tokens.access_token).toBe("C");
await db.execute(sql`drop trigger delayed_quota_commit_failure on company_secret_versions; drop function delayed_quota_commit_failure()`);
triggerInstalled = false;
const recovered = await service.refreshQuotaCredential(row, original, new AbortController().signal);
expect(JSON.parse(recovered).tokens).toMatchObject({ access_token: "C", refresh_token: "refresh-C" });
expect(await service.credential(row)).toBe(recovered);
expect(await journal.read()).toBeNull();
expect(request).toHaveBeenCalledTimes(2);
} finally {
if (triggerInstalled) await db.execute(sql`drop trigger delayed_quota_commit_failure on company_secret_versions; drop function delayed_quota_commit_failure()`);
request.mockRestore(); await journal.clear();
}
});
it.each(["reconnected", "expired_reconnect", "revoked"] as const)("does not apply a pending refresh over a %s identity", async (change) => {
const owner = `quota-replay-${change}`;
await db.insert(companyMemberships).values({ companyId, principalId: owner, principalType: "user", status: "active", membershipRole: "member" });
const original = JSON.stringify({ tokens: { access_token: "expired", refresh_token: "single-use", account_id: owner } });
await service.save(companyId, owner, { provider: "openai", method: "subscription", ownership: "personal", name: owner, loginSessionId: "fixture", agentIds: [], allAgents: true }, original);
const [row] = await service.quotaAccounts(companyId, owner);
const ref = row.grant.credentialSecretRefs.find(ref => ref.configPath === "ai.credential")!;
const journal = quotaCredentialRecovery(companyId, row.grant.id);
await journal.write({ companyId, grantId: row.grant.id, connectionId: row.connection.id, secretId: ref.secretId, baseHash: quotaCredentialHash(original), value: "old-pending-replacement" });
const request = vi.spyOn(globalThis, "fetch");
try {
if (change === "reconnected") {
await secretService(db).rotate(ref.secretId, { value: "new-authorized-identity" });
expect(await service.refreshQuotaCredential(row, original, new AbortController().signal)).toBe("new-authorized-identity");
expect(await service.credential(row)).toBe("new-authorized-identity");
expect(await journal.read()).toBeNull();
} else if (change === "expired_reconnect") {
const reconnected = JSON.stringify({ tokens: { access_token: "expired-new-account", refresh_token: "new-account-refresh", account_id: owner } });
await secretService(db).rotate(ref.secretId, { value: reconnected });
request.mockResolvedValueOnce(Response.json({ access_token: "new-account-fresh", refresh_token: "new-account-replacement" }));
await service.refreshQuotaCredential(row, reconnected, new AbortController().signal);
expect(request).toHaveBeenCalledOnce();
expect(JSON.parse(String(request.mock.calls[0][1]?.body))).toMatchObject({ refresh_token: "new-account-refresh" });
expect(JSON.parse(await service.credential(row))).toMatchObject({ tokens: { access_token: "new-account-fresh", refresh_token: "new-account-replacement" } });
expect(await journal.read()).toBeNull();
} else {
await db.update(connectionGrants).set({ status: "revoked" }).where(eq(connectionGrants.id, row.grant.id));
await expect(service.refreshQuotaCredential(row, original, new AbortController().signal)).rejects.toThrow("credentials_unavailable");
expect(await journal.read()).not.toBeNull();
}
if (change !== "expired_reconnect") expect(request).not.toHaveBeenCalled();
} finally { request.mockRestore(); await journal.clear(); }
});
it("saves exchanged tokens when the quota deadline expires during response-body reading", async () => {
const owner = "quota-body-deadline-owner";
await db.insert(companyMemberships).values({ companyId, principalId: owner, principalType: "user", status: "active", membershipRole: "member" });
const original = JSON.stringify({ tokens: { access_token: "expired", refresh_token: "single-use", account_id: "deadline-account" } });
await service.save(companyId, owner, { provider: "openai", method: "subscription", ownership: "personal", name: "Slow token body", loginSessionId: "fixture", agentIds: [], allAgents: true }, original);
const [row] = await service.quotaAccounts(companyId, owner);
const reading = deferredSignal(), releaseBody = deferredSignal(), quotaDeadline = new AbortController();
let refreshSignal: AbortSignal | null | undefined;
const request = vi.spyOn(globalThis, "fetch").mockImplementation(async (_url, options) => {
refreshSignal = options?.signal;
return { ok: true, json: async () => {
reading.resolve(); await releaseBody.promise;
refreshSignal?.throwIfAborted();
return { access_token: "fresh-after-deadline", refresh_token: "replacement" };
} } as Response;
});
const saving = service.refreshQuotaCredential(row, original, quotaDeadline.signal);
try {
await reading.promise;
quotaDeadline.abort();
expect(refreshSignal?.aborted).toBe(false);
releaseBody.resolve();
await saving;
expect(JSON.parse(await service.credential(row))).toMatchObject({ tokens: { access_token: "fresh-after-deadline", refresh_token: "replacement" } });
await expect(service.refreshQuotaCredential(row, original, quotaDeadline.signal)).rejects.toThrow();
expect(request).toHaveBeenCalledTimes(1);
} finally { releaseBody.resolve(); await saving.catch(() => {}); request.mockRestore(); }
});
it("retries another account's token write-back while a slow quota exchange holds the company lock", async () => {
const auth = (account: string, marker: string) => JSON.stringify({ tokens: { account_id: account, id_token: `id-${marker}`, access_token: `access-${marker}`, refresh_token: `refresh-${marker}` }, last_refresh: marker === "old" ? "2026-09-10T10:00:00Z" : "2026-09-10T11:00:00Z" });
const quotaOwner = "slow-quota-owner", runOwner = "slow-quota-other-run";
await db.insert(companyMemberships).values([quotaOwner, runOwner].map(principalId => ({ companyId, principalId, principalType: "user" as const, status: "active" as const, membershipRole: "member" as const })));
for (const owner of [quotaOwner, runOwner]) await service.save(companyId, owner, { provider: "openai", method: "subscription", ownership: "personal", name: owner, loginSessionId: "fixture", agentIds: [], allAgents: true }, auth(owner, "old"));
const [quotaRow] = await service.quotaAccounts(companyId, quotaOwner);
// Runtime credential reads now serialize with quota exchanges. Prepare this
// already-running, different account before testing its contended write-back.
const preparedRun = await prepareManagedAiRuntime(db, { companyId, agentId, responsibleUserId: runOwner, adapterType: "codex_local", binding: { provider: "openai", method: "subscription", mode: "responsible_user" }, config: {} });
const started = deferredSignal(), release = deferredSignal();
const request = vi.spyOn(globalThis, "fetch").mockImplementation(async () => {
started.resolve(); await release.promise;
return Response.json({ access_token: "quota-fresh", refresh_token: "quota-replacement" });
});
const refreshing = service.refreshQuotaCredential(quotaRow, auth(quotaOwner, "old"), new AbortController().signal);
let run: Awaited<ReturnType<typeof prepareManagedAiRuntime>> | undefined;
let cleaning: Promise<void> | undefined;
let lock: ReturnType<typeof vi.spyOn> | undefined;
try {
await started.promise;
run = preparedRun;
const authFile = path.join(String(run.config.env.CODEX_HOME), "auth.json");
await writeFile(authFile, auth(runOwner, "replacement"));
// Inject the two 30-second acquisition expirations, then use the real
// still-held lock and real Postgres write-back for the final attempt.
lock = vi.spyOn(codexAdapter, "withAccountHomeSecretMutationLock")
.mockRejectedValueOnce(Object.assign(new Error("lock timeout"), { code: WORKSPACE_RESTORE_LOCK_TIMEOUT_CODE }))
.mockRejectedValueOnce(Object.assign(new Error("lock timeout"), { code: WORKSPACE_RESTORE_LOCK_TIMEOUT_CODE }));
cleaning = run.cleanup();
await vi.waitFor(() => expect(lock).toHaveBeenCalledTimes(3));
expect(await readFile(authFile, "utf8")).toBe(auth(runOwner, "replacement"));
release.resolve();
await Promise.all([refreshing, cleaning]);
const [runRow] = await service.quotaAccounts(companyId, runOwner);
expect(await service.credential(runRow)).toBe(auth(runOwner, "replacement"));
expect(JSON.parse(await service.credential(quotaRow))).toMatchObject({ tokens: { access_token: "quota-fresh", refresh_token: "quota-replacement" } });
await expect(access(run.home!)).rejects.toMatchObject({ code: "ENOENT" });
} finally { release.resolve(); await Promise.allSettled([refreshing, cleaning]); lock?.mockRestore(); request.mockRestore(); }
});
it.each(["new", "reconnect"])("retains a completed %s sign-in across two lock waits behind a quota refresh", async kind => {
const owner = `slow-quota-login-${kind}`, loginOwner = kind === "reconnect" ? owner : "contended-login-owner";
await db.insert(companyMemberships).values([...new Set([owner, loginOwner])].map(principalId => ({ companyId, principalId, principalType: "user", status: "active", membershipRole: "member" })));
const original = JSON.stringify({ tokens: { account_id: owner, access_token: "old", refresh_token: "old-refresh" } });
await service.save(companyId, owner, { provider: "openai", method: "subscription", ownership: "personal", name: owner, loginSessionId: "fixture", agentIds: [], allAgents: true }, original);
const [quotaRow] = await service.quotaAccounts(companyId, owner);
const [environment] = await db.insert(environments).values({ name: "Contended AI login", driver: "sandbox" }).returning();
const intent = { provider: "openai", method: "subscription", ownership: "personal", name: "Contended sign-in", agentIds: [], allAgents: true, ...(kind === "reconnect" ? { connectionId: quotaRow.connection.id } : {}) } as const;
const sessionId = randomUUID();
await db.insert(adapterAuthSessions).values({ id: sessionId, publicSessionId: randomUUID(), companyId, environmentId: environment.id, adapterType: "codex_local", startedByUserId: loginOwner, status: "promoting", aiConnection: { ...intent, agentIds: [] }, expiresAt: new Date(Date.now() + 600_000) });
const started = deferredSignal(), release = deferredSignal();
const request = vi.spyOn(globalThis, "fetch").mockImplementation(async () => {
started.resolve(); await release.promise;
return Response.json({ access_token: "quota-fresh", refresh_token: "quota-replacement" });
});
const refreshing = service.refreshQuotaCredential(quotaRow, original, new AbortController().signal);
let saving: ReturnType<typeof service.save> | undefined;
let lock: ReturnType<typeof vi.spyOn> | undefined;
const credential = JSON.stringify({ tokens: { account_id: "signed-in", access_token: "login-token", refresh_token: "login-refresh" } });
try {
await started.promise;
// Represent two expired 30-second acquisitions, then wait on the real
// held lock. The credential must survive until that final save commits.
lock = vi.spyOn(codexAdapter, "withAccountHomeSecretMutationLock")
.mockRejectedValueOnce(Object.assign(new Error("lock timeout"), { code: WORKSPACE_RESTORE_LOCK_TIMEOUT_CODE }))
.mockRejectedValueOnce(Object.assign(new Error("lock timeout"), { code: WORKSPACE_RESTORE_LOCK_TIMEOUT_CODE }));
saving = service.save(companyId, loginOwner, { ...intent, agentIds: [] }, credential, sessionId);
void saving.catch(() => {});
await vi.waitFor(() => expect(lock).toHaveBeenCalledTimes(3));
expect((await db.select().from(adapterAuthSessions).where(eq(adapterAuthSessions.id, sessionId)))[0].connectionId).toBeNull();
release.resolve();
const [, saved] = await Promise.all([refreshing, saving]);
const [row] = (await service.quotaAccounts(companyId, loginOwner)).filter(row => row.connection.id === saved.connectionId);
expect(await service.credential(row)).toBe(credential);
expect(await service.save(companyId, loginOwner, { ...intent, agentIds: [] }, credential, sessionId)).toEqual(saved);
if (kind === "new") expect(JSON.parse(await service.credential(quotaRow))).toMatchObject({ tokens: { access_token: "quota-fresh", refresh_token: "quota-replacement" } });
} finally {
release.resolve(); await Promise.allSettled([refreshing, saving]); lock?.mockRestore(); request.mockRestore();
await db.delete(adapterAuthSessions).where(eq(adapterAuthSessions.id, sessionId));
await db.delete(environments).where(eq(environments.id, environment.id));
}
});
it.each(["quota", "recovery", "runtime"])("allows an in-flight reconnect after automatic %s credential rotation", async kind => {
const owner = `reconnect-refresh-${kind}`;
await db.insert(companyMemberships).values({ companyId, principalId: owner, principalType: "user", status: "active", membershipRole: "member" });
const auth = (label: string, hour: number) => JSON.stringify({ tokens: { account_id: owner, id_token: "identity", access_token: `access-${label}`, refresh_token: `refresh-${label}` }, last_refresh: `2026-09-10T${hour}:00:00Z` });
const intent = { provider: "openai", method: "subscription", ownership: "personal", name: owner, loginSessionId: "fixture", agentIds: [], allAgents: true } as const;
const account = await service.save(companyId, owner, { ...intent, agentIds: [] }, auth("old", 10));
const [row] = await service.quotaAccounts(companyId, owner);
const ref = row.grant.credentialSecretRefs.find(ref => ref.configPath === "ai.credential")!;
const before = (await db.select().from(companySecrets).where(eq(companySecrets.id, ref.secretId)))[0];
const started = new Date();
if (kind === "quota") {
const request = vi.spyOn(globalThis, "fetch").mockResolvedValue(Response.json({ access_token: "access-fresh", refresh_token: "refresh-fresh" }));
try { await service.refreshQuotaCredential(row, auth("old", 10), new AbortController().signal); }
finally { request.mockRestore(); }
} else if (kind === "recovery") {
const recovery = quotaCredentialRecovery(companyId, row.grant.id);
await recovery.write({ companyId, grantId: row.grant.id, connectionId: row.connection.id, secretId: ref.secretId, baseHash: quotaCredentialHash(auth("old", 10)), value: auth("fresh", 11) });
expect(await service.runtimeCredential(row)).toBe(auth("fresh", 11));
} else {
const runtime = await prepareManagedAiRuntime(db, { companyId, agentId, responsibleUserId: owner, adapterType: "codex_local", binding: { provider: "openai", method: "subscription", mode: "responsible_user" }, config: {} });
await writeFile(path.join(String(runtime.config.env.CODEX_HOME), "auth.json"), auth("fresh", 11));
await runtime.cleanup();
}
const after = (await db.select().from(companySecrets).where(eq(companySecrets.id, ref.secretId)))[0];
expect(after.latestVersion).toBeGreaterThan(before.latestVersion);
expect(after.aiSessionEpoch).toBe(before.aiSessionEpoch);
const saved = await service.save(companyId, owner, { ...intent, agentIds: [], connectionId: account.connectionId }, auth("reconnected", 12), undefined, started);
expect(saved).toEqual(account);
expect(await service.credential(row)).toBe(auth("reconnected", 12));
await expect(service.save(companyId, owner, { ...intent, agentIds: [], connectionId: account.connectionId }, auth("older-login", 13), undefined, started)).rejects.toThrow("changed");
// A real later revocation still fences this same old reconnect attempt.
await db.update(connectionGrants).set({ status: "revoked", updatedAt: new Date() }).where(eq(connectionGrants.id, account.grantId));
await expect(service.save(companyId, owner, { ...intent, agentIds: [], connectionId: account.connectionId }, auth("stale", 13), undefined, started)).rejects.toThrow("changed");
});
it.each(["lock", "database"])("preserves replacement tokens after exhausted %s write-back and permits a later retry", async failure => {
const owner = `retained-writeback-${failure}`;
await db.insert(companyMemberships).values({ companyId, principalId: owner, principalType: "user", status: "active", membershipRole: "member" });
const auth = (hour: string) => JSON.stringify({ tokens: { account_id: owner, id_token: "identity", access_token: `access-${hour}`, refresh_token: `refresh-${hour}` }, last_refresh: `2026-09-10T${hour}:00:00Z` });
await service.save(companyId, owner, { provider: "openai", method: "subscription", ownership: "personal", name: owner, loginSessionId: "fixture", agentIds: [], allAgents: true }, auth("10"));
const run = await prepareManagedAiRuntime(db, { companyId, agentId, responsibleUserId: owner, adapterType: "codex_local", binding: { provider: "openai", method: "subscription", mode: "responsible_user" }, config: {} });
const authFile = path.join(String(run.config.env.CODEX_HOME), "auth.json");
await writeFile(authFile, auth("11"));
const injected = failure === "lock"
? vi.spyOn(codexAdapter, "withAccountHomeSecretMutationLock").mockRejectedValue(Object.assign(new Error("lock timeout"), { code: WORKSPACE_RESTORE_LOCK_TIMEOUT_CODE }))
: vi.spyOn(db, "transaction").mockRejectedValue(new Error("database unavailable"));
try {
await expect(run.cleanup()).rejects.toThrow(failure === "lock" ? "lock timeout" : "database unavailable");
expect(injected).toHaveBeenCalledTimes(failure === "lock" ? 3 : 1);
expect(await readFile(authFile, "utf8")).toBe(auth("11"));
} finally { injected.mockRestore(); }
await run.cleanup();
const [row] = await service.quotaAccounts(companyId, owner);
expect(await service.credential(row)).toBe(auth("11"));
await expect(access(run.home!)).rejects.toMatchObject({ code: "ENOENT" });
});
it("serializes quota exchange with a concurrent secret edit and runtime credential read", async () => {
const owner = "quota-race-owner";
await db.insert(companyMemberships).values({ companyId, principalId: owner, principalType: "user", status: "active", membershipRole: "member" });
const original = JSON.stringify({ tokens: { access_token: "old-access", refresh_token: "single-use", id_token: "identity", account_id: "race-account" } });
const account = await service.save(companyId, owner, { provider: "openai", method: "subscription", ownership: "personal", name: "Racing refresh", loginSessionId: "fixture", agentIds: [], allAgents: true }, original);
const [row] = await service.quotaAccounts(companyId, owner);
const secretId = row.grant.credentialSecretRefs.find(ref => ref.configPath === "ai.credential")!.secretId;
const exchangeStarted = deferredSignal();
const releaseExchange = deferredSignal();
const request = vi.spyOn(globalThis, "fetch").mockImplementation(async () => {
exchangeStarted.resolve(); await releaseExchange.promise;
return Response.json({ access_token: "new-access", refresh_token: "new-refresh" });
});
let runtime: Awaited<ReturnType<typeof prepareManagedAiRuntime>> | undefined;
let lock: ReturnType<typeof vi.spyOn> | undefined;
const refreshing = service.refreshQuotaCredential(row, original, new AbortController().signal);
try {
await exchangeStarted.promise;
const edit = secretService(db).update(secretId, { description: "Changed during refresh" });
lock = vi.spyOn(codexAdapter, "withAccountHomeSecretMutationLock");
const preparing = prepareManagedAiRuntime(db, { companyId, agentId, responsibleUserId: owner, adapterType: "codex_local", binding: { provider: "openai", method: "subscription", mode: "responsible_user" }, config: {} });
// Runtime reads now wait at the company file lock before touching the
// old credential. The returned home must contain the committed rotation.
let prepared = false;
void preparing.then(() => { prepared = true; });
await vi.waitFor(() => expect(lock).toHaveBeenCalled());
expect(prepared).toBe(false);
releaseExchange.resolve();
const results = await Promise.all([refreshing, edit, preparing]);
runtime = results[2];
expect(JSON.parse(await readFile(path.join(String(runtime.config.env.CODEX_HOME), "auth.json"), "utf8"))).toMatchObject({ tokens: { access_token: "new-access", refresh_token: "new-refresh" } });
expect(JSON.parse(await service.credential(row))).toMatchObject({ tokens: { access_token: "new-access", refresh_token: "new-refresh" } });
expect((await secretService(db).getById(secretId))?.description).toBe("Changed during refresh");
} finally { releaseExchange.resolve(); await runtime?.cleanup(); request.mockRestore(); lock?.mockRestore(); }
}, 10_000);
it("serializes quota credential refresh, persists rotation, and defers while the provider is active", async () => {
const owner = "quota-refresh-owner";
await db.insert(companyMemberships).values({ companyId, principalId: owner, principalType: "user", status: "active", membershipRole: "member" });
const original = JSON.stringify({ tokens: { access_token: "expired-access", refresh_token: "single-use", id_token: "identity", account_id: null } });
const account = await service.save(companyId, owner, { provider: "openai", method: "subscription", ownership: "personal", name: "Refresh account", loginSessionId: "fixture", agentIds: [], allAgents: true }, original);
const [row] = await service.quotaAccounts(companyId, owner);
const request = vi.spyOn(globalThis, "fetch").mockImplementation(async () => new Response(JSON.stringify({ access_token: "fresh-access", refresh_token: "rotated-once" })));
const signal = new AbortController().signal;
try {
// A pre-existing transaction may own a secret row before asking for the
// company file lock. Refresh must defer before exchanging its token.
await db.transaction(async tx => {
const secretId = row.grant.credentialSecretRefs.find(ref => ref.configPath === "ai.credential")!.secretId;
await tx.select().from(companySecrets).where(eq(companySecrets.id, secretId)).for("update");
await expect(service.refreshQuotaCredential(row, original, signal)).rejects.toThrow();
expect(request).not.toHaveBeenCalled();
});
const activeId = randomUUID();
await db.insert(heartbeatRuns).values({ id: activeId, companyId, agentId, invocationSource: "on_demand", status: "running", contextSnapshot: { aiConnection: { grantId: account.grantId } } });
await expect(service.refreshQuotaCredential(row, original, signal)).rejects.toThrow("provider_unavailable");
expect(request).not.toHaveBeenCalled();
await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, activeId));
const [first, second] = await Promise.all([
service.refreshQuotaCredential(row, original, signal),
service.refreshQuotaCredential(row, original, signal),
]);
expect(first).toBe(second);
expect(request).toHaveBeenCalledTimes(1);
expect(JSON.parse(await service.credential(row))).toMatchObject({ tokens: { access_token: "fresh-access", refresh_token: "rotated-once", account_id: null } });
await db.update(connectionGrants).set({ status: "revoked" }).where(eq(connectionGrants.id, account.grantId));
await expect(service.refreshQuotaCredential(row, first, signal)).rejects.toThrow("credentials_unavailable");
expect(request).toHaveBeenCalledTimes(1);
} finally { request.mockRestore(); }
});
it.each([
["openai", "codex_local", "responses"],
["anthropic", "claude_local", "messages"],
@@ -380,6 +1005,17 @@ console.log(JSON.stringify({ type: "result", subtype: "success", result: "hello"
expect(await intents.requestForRunAuthFailure(runId)).toBeNull();
});
it("lists quotas only for the current member's accessible subscription accounts", async () => {
const user = `quota-${randomUUID()}`;
await db.insert(companyMemberships).values({ companyId, principalId: user, principalType: "user", status: "active", membershipRole: "member" });
const subscription = await service.save(companyId, user, { provider: "openai", method: "subscription", ownership: "personal", name: "Private quota", loginSessionId: "fixture", allAgents: true, agentIds: [] }, JSON.stringify({ tokens: { access_token: "fixture", account_id: "fixture", refresh_token: "fixture" } }));
expect((await service.quotaAccounts(companyId, user)).some(row => row.connection.id === subscription.connectionId)).toBe(true);
expect((await service.quotaAccounts(companyId, "bob")).some(row => row.connection.id === subscription.connectionId)).toBe(false);
expect(await service.quotaAccounts(otherCompanyId, user)).toEqual([]);
await db.update(companyMemberships).set({ status: "inactive" }).where(and(eq(companyMemberships.companyId, companyId), eq(companyMemberships.principalId, user)));
expect(await service.quotaAccounts(companyId, user)).toEqual([]);
});
it("repairs a teammate's missing onboarding key with their own inline AI connection", async () => {
const userId = `new-teammate-${randomUUID()}`;
const id = randomUUID();
@@ -1273,3 +1909,9 @@ describe("AI connection recovery delivery", () => {
}, 30000,
);
});
function deferredSignal() {
let resolve!: () => void;
const promise = new Promise<void>(done => { resolve = done; });
return { promise, resolve };
}
server/src/__tests__/chat-channels.integration.test.ts
+12 -6
View File
@@ -7679,6 +7679,8 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
).toBe(1);
expect(deferred).toHaveLength(1);
// The production reorder window is 750 ms; allow the subsequent database
// drain to finish on loaded CI instead of sharing the default one-second budget.
deferred.shift()?.();
await vi.waitFor(async () => {
const rows = await db
@@ -7686,7 +7688,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
.from(chatConversations)
.where(eq(chatConversations.endpointId, endpoint.id));
expect(rows).toHaveLength(1);
});
}, { timeout: 10_000 });
const [conversation] = await db
.select()
.from(chatConversations)
@@ -7702,7 +7704,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
"unmentioned follow-up delivered first",
]);
expect(wakeup).toHaveBeenCalledTimes(2);
});
}, { timeout: 10_000 });
await service.shutdown();
});
@@ -60633,19 +60635,23 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
db
.select({ id: issueComments.id })
.from(issueComments)
.where(eq(issueComments.companyId, fixture.companyId)),
.where(eq(issueComments.companyId, fixture.companyId))
.orderBy(issueComments.id),
db
.select({ id: issues.id })
.from(issues)
.where(eq(issues.companyId, fixture.companyId)),
.where(eq(issues.companyId, fixture.companyId))
.orderBy(issues.id),
db
.select({ id: heartbeatRuns.id })
.from(heartbeatRuns)
.where(eq(heartbeatRuns.companyId, fixture.companyId)),
.where(eq(heartbeatRuns.companyId, fixture.companyId))
.orderBy(heartbeatRuns.id),
db
.select({ id: chatPublications.id })
.from(chatPublications)
.where(eq(chatPublications.endpointId, endpoint.id)),
.where(eq(chatPublications.endpointId, endpoint.id))
.orderBy(chatPublications.id),
]);
const baseline = await unchangedRows();
const wakeupCount = wakeup.mock.calls.length;
server/src/__tests__/cli-invocation-safety.test.ts
+31 -1
View File
@@ -1,3 +1,4 @@
import { execFileSync } from "node:child_process";
import { mkdirSync, mkdtempSync, readdirSync, readFileSync, rmSync, writeFileSync } from "node:fs";
import os from "node:os";
import path from "node:path";
@@ -367,6 +368,13 @@ function isTestFile(relPath: string): boolean {
function listGuidanceFiles(rootDir = repoRoot): string[] {
const found: string[] = [];
// Include tracked files and new authored files, while respecting gitignored
// local settings. A tracked file remains scanned even if an ignore matches.
const authored = new Set(execFileSync(
"git",
["ls-files", "--cached", "--others", "--exclude-standard", "-z"],
{ cwd: rootDir, encoding: "utf8" },
).split("\0"));
function walk(absDir: string, relDir: string): void {
for (const entry of readdirSync(absDir, { withFileTypes: true })) {
@@ -382,7 +390,7 @@ function listGuidanceFiles(rootDir = repoRoot): string[] {
if (!SCAN_EXTENSIONS.has(path.extname(entry.name))) continue;
if (isTestFile(relPath)) continue;
if (SKIP_PATH_PREFIXES.some((prefix) => relPath.startsWith(prefix))) continue;
found.push(relPath);
if (authored.has(relPath)) found.push(relPath);
}
}
@@ -486,6 +494,7 @@ describe("paperclipai CLI invocation safety", () => {
"tests/runner-e2e/catalog.ts",
];
try {
execFileSync("git", ["init", "--quiet"], { cwd: root });
for (const relPath of [
...sourcePaths,
"tests/runner-e2e/results/campaign/attempt-1/snapshots/api-state.json",
@@ -503,6 +512,7 @@ describe("paperclipai CLI invocation safety", () => {
it("excludes root runtime recordings but still scans unsafe docs and source guidance", () => {
const fixtureRoot = mkdtempSync(path.join(os.tmpdir(), "paperclip-cli-guidance-"));
try {
execFileSync("git", ["init", "--quiet"], { cwd: fixtureRoot });
const fixtures = [
"doc/CLI.md",
"src/guidance.ts",
@@ -528,6 +538,26 @@ describe("paperclipai CLI invocation safety", () => {
}
});
it("scans tracked and ordinary untracked guidance but excludes ignored untracked files", () => {
const root = mkdtempSync(path.join(os.tmpdir(), "paperclip-cli-guidance-"));
try {
execFileSync("git", ["init", "--quiet"], { cwd: root });
writeFileSync(path.join(root, ".gitignore"), "ignored.md\ntracked.md\n");
for (const name of ["ignored.md", "untracked.md", "tracked.md"]) {
writeFileSync(path.join(root, name), "pnpm paperclipai issue get issue-1\n");
}
execFileSync("git", ["add", "--force", "tracked.md"], { cwd: root });
const files = listGuidanceFiles(root).sort();
expect(files).toEqual(["tracked.md", "untracked.md"]);
for (const name of files) {
expect(scanText(name, readFileSync(path.join(root, name), "utf8"))).toHaveLength(1);
}
} finally {
rmSync(root, { recursive: true, force: true });
}
});
it("allows only exact-allowlist pnpm paperclipai commands on every guidance surface", () => {
const offenders = scanForOffenders();
expect(
server/src/__tests__/codex-local-adapter.test.ts
+1 -1
View File
@@ -16,7 +16,7 @@ describe("codex_local parser", () => {
expect(parsed.sessionId).toBe("thread-123");
expect(parsed.summary).toBe("hello");
expect(parsed.usage).toEqual({
inputTokens: 10,
inputTokens: 8,
cachedInputTokens: 2,
outputTokens: 4,
});
server/src/__tests__/company-quota-windows.test.ts
+256
View File
@@ -0,0 +1,256 @@
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import type { Db } from "@paperclipai/db";
const mocks = vi.hoisted(() => ({
accounts: vi.fn(),
credential: vi.fn(),
refresh: vi.fn(),
codex: vi.fn(),
claude: vi.fn(),
adapters: vi.fn(),
}));
vi.mock("../services/ai-connections.js", () => ({
aiConnectionService: () => ({
quotaAccounts: mocks.accounts,
credential: mocks.credential,
refreshQuotaCredential: mocks.refresh,
}),
}));
vi.mock("../adapters/registry.js", () => ({
listServerAdapters: mocks.adapters,
}));
vi.mock("@paperclipai/adapter-codex-local/server", () => ({
fetchCodexQuota: mocks.codex,
}));
vi.mock("@paperclipai/adapter-claude-local/server", () => ({
fetchClaudeQuota: mocks.claude,
}));
import { fetchCompanyQuotaWindows } from "../services/quota-windows.js";
let revision = 1;
const db = {
select: () => ({
from: () => ({
leftJoin: () => ({
leftJoin: () => ({
where: async () => [
{
id: "secret",
updatedAt: new Date(0),
latestVersion: revision,
status: "active",
},
],
}),
}),
}),
}),
} as unknown as Db;
const account = (id: string, provider = "openai", status = "connected") => ({
connection: { id, updatedAt: new Date(0) },
grant: {
id,
updatedAt: new Date(0),
credentialSecretRefs: [{ secretId: "secret" }],
},
summary: { provider, name: id, status },
});
describe("connected account quotas", () => {
beforeEach(() => {
vi.clearAllMocks();
revision = 1;
mocks.refresh.mockRejectedValue(new Error("authentication_required"));
mocks.credential.mockResolvedValue(
JSON.stringify({
tokens: { access_token: "private-token", account_id: "actual-account" },
}),
);
mocks.codex.mockResolvedValue([
{ label: "5h", usedPercent: 0.5, resetsAt: null, valueLabel: null },
]);
});
afterEach(() => vi.useRealTimers());
it("uses the selected account without invoking a local CLI or fallback login", async () => {
mocks.accounts.mockResolvedValue([account("remote-codex")]);
const result = await fetchCompanyQuotaWindows(db, "company-1", "user-1");
expect(mocks.accounts).toHaveBeenCalledWith("company-1", "user-1");
expect(mocks.codex).toHaveBeenCalledWith("private-token", "actual-account", expect.any(AbortSignal));
expect(mocks.adapters).not.toHaveBeenCalled();
expect(result[0]).toMatchObject({
ok: true,
accountLabel: "remote-codex",
source: "managed-connection",
});
expect(result[0].capturedAt).toBeDefined();
expect(JSON.stringify(result)).not.toContain("private-token");
});
it("deduplicates account requests and invalidates on credential revision", async () => {
mocks.accounts.mockResolvedValue([account("cached-codex")]);
const [a, b] = await Promise.all([
fetchCompanyQuotaWindows(db, "company-2", "user-2"),
fetchCompanyQuotaWindows(db, "company-2", "user-2"),
]);
expect(mocks.codex).toHaveBeenCalledTimes(1);
expect(a[0].capturedAt).toBe(b[0].capturedAt);
revision++;
const next = await fetchCompanyQuotaWindows(db, "company-2", "user-2");
expect(mocks.codex).toHaveBeenCalledTimes(2);
expect(next[0].accountKey).not.toBe(a[0].accountKey);
});
it("keeps two accounts for the same provider separate", async () => {
mocks.accounts.mockResolvedValue([account("first"), account("second")]);
const result = await fetchCompanyQuotaWindows(db, "company-3", "user-3");
expect(result).toHaveLength(2);
expect(result[0].accountKey).not.toBe(result[1].accountKey);
});
it("does not expose failures and marks authentication failures for stale-window removal", async () => {
mocks.accounts.mockResolvedValue([account("expired")]);
mocks.codex.mockRejectedValue(
new Error("401 private-token raw provider diagnostics"),
);
const [result] = await fetchCompanyQuotaWindows(db, "company-4", "user-4");
expect(result.errorFamily).toBe("authentication_required");
expect(result.windows).toEqual([]);
expect(JSON.stringify(result)).not.toContain("private-token");
});
it("does not read revoked credentials or use the server login for an unconnected remote account", async () => {
mocks.accounts.mockResolvedValue([account("revoked", "openai", "revoked")]);
const [result] = await fetchCompanyQuotaWindows(db, "company-5", "user-5");
expect(result.errorFamily).toBe("credentials_unavailable");
expect(mocks.credential).not.toHaveBeenCalled();
mocks.accounts.mockResolvedValue([]);
const empty = await fetchCompanyQuotaWindows(db, "company-5", "user-5");
expect(
empty.every((row) => row.errorFamily === "credentials_unavailable"),
).toBe(true);
expect(mocks.adapters).not.toHaveBeenCalled();
});
it("uses a managed Claude OAuth credential and keeps failed reads private", async () => {
mocks.accounts.mockResolvedValue([account("claude-account", "anthropic")]);
mocks.credential.mockResolvedValue("claude-private-oauth");
mocks.claude.mockResolvedValue([
{ label: "Weekly", usedPercent: 12, resetsAt: null, valueLabel: null },
]);
const [result] = await fetchCompanyQuotaWindows(db, "company-6", "user-6");
expect(result.ok).toBe(true);
expect(mocks.claude).toHaveBeenCalledWith("claude-private-oauth", expect.any(AbortSignal));
mocks.accounts.mockResolvedValue([account("secret-failure")]);
mocks.credential.mockRejectedValue(
new Error("private credential store failure"),
);
const [failed] = await fetchCompanyQuotaWindows(db, "company-6", "user-6");
expect(failed.errorFamily).toBe("provider_unavailable");
expect(JSON.stringify(failed)).not.toContain("private credential");
});
it("reads quota for a verified login without an optional account ID", async () => {
mocks.accounts.mockResolvedValue([account("missing-identity")]);
mocks.credential.mockResolvedValue(
JSON.stringify({ accessToken: "private" }),
);
const [result] = await fetchCompanyQuotaWindows(db, "company-7", "user-7");
expect(result.ok).toBe(true);
expect(mocks.codex).toHaveBeenCalledWith("private", null, expect.any(AbortSignal));
});
it("distinguishes permission errors from revoked credentials", async () => {
mocks.accounts.mockResolvedValue([account("permission-error")]);
mocks.codex.mockRejectedValueOnce(new Error("chatgpt wham api returned 403"));
const [result] = await fetchCompanyQuotaWindows(db, "permission-company", "user");
expect(result.errorFamily).toBe("permission_denied");
expect(mocks.refresh).not.toHaveBeenCalled();
});
it("refreshes an expired selected credential once before retrying quota", async () => {
mocks.accounts.mockResolvedValue([account("refresh-account")]);
mocks.codex.mockRejectedValueOnce(new Error("chatgpt wham api returned 401"));
mocks.refresh.mockImplementationOnce(async () => {
const value = JSON.stringify({ tokens: { access_token: "new-token", account_id: "same-account" } });
revision++;
mocks.credential.mockResolvedValue(value);
return value;
});
const [result] = await fetchCompanyQuotaWindows(db, "refresh-company", "user");
expect(result.ok).toBe(true);
expect(mocks.refresh).toHaveBeenCalledTimes(1);
expect(mocks.codex).toHaveBeenLastCalledWith("new-token", "same-account", expect.any(AbortSignal));
expect(JSON.stringify(result)).not.toContain("new-token");
const again = await fetchCompanyQuotaWindows(db, "refresh-company", "user");
expect(again[0]).toEqual(result);
expect(mocks.codex).toHaveBeenCalledTimes(2); // initial 401 and one refreshed read
});
it("does not cache a quota read under a credential revision that changed during resolution", async () => {
mocks.accounts.mockResolvedValue([account("rotating-during-refresh")]);
mocks.codex.mockRejectedValueOnce(new Error("401"));
mocks.refresh.mockResolvedValue(JSON.stringify({ accessToken: "refreshed" }));
mocks.credential.mockResolvedValueOnce(JSON.stringify({ accessToken: "expired" })).mockImplementationOnce(async () => {
revision++;
return JSON.stringify({ accessToken: "changed-again" });
});
const [result] = await fetchCompanyQuotaWindows(db, "changing-revision", "user");
expect(result).toMatchObject({ ok: false, errorFamily: "provider_unavailable" });
expect(mocks.codex).toHaveBeenCalledTimes(1);
});
it("bounds a stalled credential lookup and preserves its account identity", async () => {
vi.useFakeTimers();
mocks.accounts.mockResolvedValue([account("stalled")]);
mocks.credential.mockReturnValue(new Promise(() => {}));
const pending = fetchCompanyQuotaWindows(db, "company-8", "user-8");
await vi.advanceTimersByTimeAsync(20_001);
const [result] = await pending;
expect(result).toMatchObject({
ok: false,
accountLabel: "stalled",
windows: [],
});
expect(result.accountKey).toBeTruthy();
expect(result.error).not.toContain("timed out");
});
it("does not start a provider request when a credential resolves after its deadline", async () => {
vi.useFakeTimers();
mocks.accounts.mockResolvedValue([account("late-credential")]);
let resolveCredential!: (value: string) => void;
mocks.credential.mockReturnValue(new Promise<string>((resolve) => { resolveCredential = resolve; }));
const pending = fetchCompanyQuotaWindows(db, "late-company", "late-user");
await vi.advanceTimersByTimeAsync(20_001);
expect((await pending)[0].ok).toBe(false);
resolveCredential(JSON.stringify({ accessToken: "private", accountId: "actual" }));
await vi.advanceTimersByTimeAsync(0);
expect(mocks.codex).not.toHaveBeenCalled();
});
it("cancels an active provider read when the account deadline expires", async () => {
vi.useFakeTimers();
mocks.accounts.mockResolvedValue([account("slow-provider")]);
let signal!: AbortSignal;
mocks.codex.mockImplementation((_token, _id, abortSignal) => {
signal = abortSignal;
return new Promise((_resolve, reject) => signal.addEventListener("abort", () => reject(signal.reason), { once: true }));
});
const pending = fetchCompanyQuotaWindows(db, "slow-company", "slow-user");
await vi.advanceTimersByTimeAsync(20_001);
expect((await pending)[0].ok).toBe(false);
expect(signal.aborted).toBe(true);
});
it("limits account probes to four concurrent requests and preserves account order", async () => {
mocks.accounts.mockResolvedValue(
Array.from({ length: 9 }, (_, i) => account(`parallel-${i}`)),
);
mocks.credential.mockResolvedValue(
JSON.stringify({ accessToken: "private", accountId: "actual" }),
);
let active = 0,
maximum = 0;
mocks.codex.mockImplementation(async () => {
active++;
maximum = Math.max(maximum, active);
await new Promise((resolve) => setTimeout(resolve, 5));
active--;
return [];
});
const results = await fetchCompanyQuotaWindows(db, "company-9", "user-9");
expect(maximum).toBe(4);
expect(results.map((row) => row.accountLabel)).toEqual(
Array.from({ length: 9 }, (_, i) => `parallel-${i}`),
);
});
});
server/src/__tests__/cursor-local-adapter.test.ts
+42
View File
@@ -4,6 +4,39 @@ import { parseCursorStdoutLine } from "@paperclipai/adapter-cursor-local/ui";
import { printCursorStreamEvent } from "@paperclipai/adapter-cursor-local/cli";
describe("cursor parser", () => {
it.each([
{ type: "result" },
{ type: "result", usage: {} },
{ type: "result", usage: { input_tokens: 0 } },
{ type: "result", usage: { input_tokens: -1, output_tokens: 0 } },
{ type: "step_finish", part: {} },
])("does not treat absent or malformed counters as reported zeroes (%j)", event => {
const parsed = parseCursorJsonl(JSON.stringify(event));
expect(parsed.usageReported).toBe(false);
expect(parsed.usageComplete).toBe(false);
expect(parsed.costUsd).toBeNull();
});
it.each([
{ type: "result", usage: { input_tokens: 0, output_tokens: 0 } },
{ type: "result", usage: { inputTokens: 0, outputTokens: 0 } },
{ type: "step_finish", part: { tokens: { input: 0, output: 0 } } },
])("preserves explicitly reported zero counters (%j)", event => {
expect(parseCursorJsonl(JSON.stringify(event))).toMatchObject({
usageReported: true, usageComplete: true,
usage: { inputTokens: 0, cachedInputTokens: 0, outputTokens: 0 },
});
});
it("keeps known partial usage without claiming a complete total", () => {
const parsed = parseCursorJsonl([
{ type: "step_finish", part: { tokens: { input: 20, output: 5 } } },
{ type: "result" },
].map(event => JSON.stringify(event)).join("\n"));
expect(parsed).toMatchObject({ usageReported: true, usageComplete: false,
usage: { inputTokens: 20, outputTokens: 5 }, costUsd: null });
});
it("extracts session, summary, usage, cost, and terminal error message", () => {
const stdout = [
JSON.stringify({ type: "system", subtype: "init", session_id: "chat_123", model: "gpt-5" }),
@@ -403,3 +436,12 @@ describe("cursor cli formatter", () => {
}
});
});
describe("Cursor price completeness", () => {
it("preserves reported zero and keeps a partially priced stream unpriced", () => {
expect(parseCursorJsonl(JSON.stringify({ type: "result", cost_usd: 0 })).costUsd).toBe(0);
expect(parseCursorJsonl(JSON.stringify({ type: "result" })).costUsd).toBeNull();
expect(parseCursorJsonl([JSON.stringify({ type: "step_finish", part: { cost: 1 } }), JSON.stringify({ type: "step_finish", part: {} })].join("\n")).costUsd).toBeNull();
});
});
server/src/__tests__/cursor-local-execute.test.ts
+19 -1
View File
@@ -1,10 +1,23 @@
import { describe, expect, it } from "vitest";
import { describe, expect, it, vi } from "vitest";
import fs from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { runChildProcess } from "@paperclipai/adapter-utils/server-utils";
import { execute } from "@paperclipai/adapter-cursor-local/server";
// These fixtures verify command discovery, environment and real CLI execution.
// Callback bridge startup/teardown has its own execution-target coverage; starting
// that server here adds unrelated process and polling work to command assertions.
vi.mock("@paperclipai/adapter-utils/execution-target", async () => {
const actual = await vi.importActual<typeof import("@paperclipai/adapter-utils/execution-target")>(
"@paperclipai/adapter-utils/execution-target",
);
return {
...actual,
startAdapterExecutionTargetPaperclipBridge: async () => null,
};
});
async function writeFakeCursorCommand(commandPath: string): Promise<void> {
const script = `#!/usr/bin/env node
const fs = require("node:fs");
@@ -88,6 +101,11 @@ function createLocalSandboxRunner() {
onSpawn?: (meta: { pid: number; startedAt: string }) => Promise<void>;
}) => {
counter += 1;
// Both sandbox fixtures install their fake CLI up front. Never fall back
// to a network install if discovery regresses on another host or shell.
if (input.args?.some(arg => arg.includes("curl https://cursor.com/install"))) {
throw new Error("Sandbox fixture did not discover its preinstalled Cursor CLI");
}
return await runChildProcess(`cursor-sandbox-execute-${counter}`, input.command, input.args ?? [], {
cwd: input.cwd ?? process.cwd(),
env: input.env ?? {},
server/src/__tests__/document-annotation-routes.test.ts
+6 -1
View File
@@ -187,7 +187,7 @@ async function createApp(actor: "board" | "agent" = "board", actorCompanyId = co
}
describe("document annotation routes", () => {
beforeEach(() => {
beforeEach(async () => {
vi.resetModules();
vi.doUnmock("../routes/issues.js");
vi.doUnmock("../middleware/index.js");
@@ -226,6 +226,11 @@ describe("document annotation routes", () => {
mockAnnotationService.addComment.mockResolvedValue(annotationComment);
mockAnnotationService.updateThread.mockResolvedValue({ ...annotationThread, status: "resolved" });
mockAnnotationService.remapOpenThreadsForDocument.mockResolvedValue([]);
// Cold route transforms belong to fixture setup, not the request timeout.
await Promise.all([
vi.importActual("../routes/issues.js"),
vi.importActual("../middleware/index.js"),
]);
});
it("includes compact open annotations without comment bodies by default for agent document reads", async () => {
server/src/__tests__/durable-directory.test.ts
+243
View File
@@ -0,0 +1,243 @@
import { createRequire } from "node:module";
import { spawnSync } from "node:child_process";
import { promises as fs } from "node:fs";
import os from "node:os";
import path from "node:path";
import { afterEach, describe, expect, it, vi } from "vitest";
import { ensureDurableDirectory } from "../lib/durable-directory.js";
const roots: string[] = [];
afterEach(async () => {
vi.restoreAllMocks();
await Promise.all(roots.splice(0).map(root => fs.rm(root, { recursive: true, force: true })));
});
describe.skipIf(process.platform === "win32")("durable directory creation", () => {
async function fixture() {
const root = await fs.realpath(await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-durable-directory-")));
roots.push(root);
return { root, directory: path.join(root, "instance", "recovery", "company") };
}
function observeSyncs(failOn?: string) {
const synced: string[] = [];
const open = fs.open.bind(fs);
vi.spyOn(fs, "open").mockImplementation(async (...args: Parameters<typeof fs.open>) => {
const handle = await open(...args);
const sync = handle.sync.bind(handle);
vi.spyOn(handle, "sync").mockImplementation(async () => {
if (args[0] === failOn) throw new Error("Injected directory fsync failure");
await sync();
synced.push(String(args[0]));
});
return handle;
});
return synced;
}
it.each([false, true])("flushes symlink targets and the directories containing each link (chained: %s)", async (chained) => {
const { root } = await fixture();
const targetParent = path.join(root, "storage");
const target = path.join(targetParent, "new-home");
const writtenParent = path.join(root, "configured");
const links = path.join(root, "links");
await fs.mkdir(target, { recursive: true });
await fs.mkdir(writtenParent);
await fs.mkdir(links);
await fs.symlink("../storage/new-home", path.join(links, "middle"));
const alias = path.join(writtenParent, "home");
await fs.symlink(chained ? "../links/middle" : target, alias);
const directory = path.join(alias, "recovery");
const synced = observeSyncs();
await ensureDurableDirectory(directory);
expect(synced).toContain(targetParent);
expect(synced).toContain(writtenParent);
if (chained) expect(synced).toContain(links);
expect(synced).toContain(path.join(target, "recovery"));
synced.length = 0;
await ensureDurableDirectory(directory);
expect(synced).toEqual([]);
});
it("resolves parent components after following a symlink target", async () => {
const { root } = await fixture();
const storage = path.join(root, "storage");
await fs.mkdir(path.join(storage, "nested"), { recursive: true });
await fs.mkdir(path.join(storage, "home"));
await fs.symlink(path.join(storage, "nested"), path.join(root, "through"));
await fs.symlink("through/../home", path.join(root, "alias"));
const synced = observeSyncs();
await ensureDurableDirectory(path.join(root, "alias", "recovery"));
expect(synced).toContain(storage);
expect(synced).toContain(path.join(storage, "home", "recovery"));
});
it("invalidates the proof when a symlink moves the same leaf under a new parent", async () => {
const { root } = await fixture();
const original = path.join(root, "original");
const parent = path.join(root, "new-parent");
const alias = path.join(root, "alias");
await fs.mkdir(original);
await fs.symlink(original, alias);
const directory = path.join(alias, "recovery");
await ensureDurableDirectory(directory);
const inode = (await fs.stat(directory)).ino;
await fs.mkdir(parent);
await fs.rename(original, path.join(parent, "moved"));
await fs.unlink(alias);
await fs.symlink(path.join(parent, "moved"), alias);
expect((await fs.stat(directory)).ino).toBe(inode);
const synced = observeSyncs();
await ensureDurableDirectory(directory);
expect(synced).toContain(parent);
});
it("retries failed real-parent flushes through a symlink", async () => {
const { root } = await fixture();
const parent = path.join(root, "storage");
await fs.mkdir(path.join(parent, "home"), { recursive: true });
const alias = path.join(root, "alias");
await fs.symlink(path.join(parent, "home"), alias);
const directory = path.join(alias, "recovery");
observeSyncs(parent);
await expect(ensureDurableDirectory(directory)).rejects.toThrow("Injected directory fsync failure");
await expect(ensureDurableDirectory(directory)).rejects.toThrow("Injected directory fsync failure");
vi.restoreAllMocks();
const synced = observeSyncs();
await ensureDurableDirectory(directory);
expect(synced).toContain(parent);
});
it("flushes every created path entry and avoids ancestor fsyncs on subsequent writes", async () => {
const { root, directory } = await fixture();
const synced = observeSyncs();
await ensureDurableDirectory(directory);
expect(synced.slice(0, 4)).toEqual([directory, path.dirname(directory), path.join(root, "instance"), root]);
synced.length = 0;
await ensureDurableDirectory(directory);
expect(synced).toEqual([]);
await fs.rm(directory, { recursive: true });
await ensureDurableDirectory(directory);
expect(synced.slice(0, 4)).toEqual([directory, path.dirname(directory), path.join(root, "instance"), root]);
});
it("fails closed for existing storage beneath an unflushable ancestor", async () => {
const { root, directory } = await fixture();
const storage = path.join(root, "instance");
await fs.mkdir(storage);
const open = fs.open.bind(fs);
vi.spyOn(fs, "open").mockImplementation(async (...args: Parameters<typeof fs.open>) => {
if (args[0] === root) throw Object.assign(new Error("execute-only ancestor"), { code: "EACCES" });
return open(...args);
});
await expect(ensureDurableDirectory(directory)).rejects.toMatchObject({ code: "EACCES" });
expect((await fs.stat(directory)).isDirectory()).toBe(true);
});
it("requires permission to flush a newly created entry's parent, including on retry", async () => {
const { root, directory } = await fixture();
const open = fs.open.bind(fs);
vi.spyOn(fs, "open").mockImplementation(async (...args: Parameters<typeof fs.open>) => {
if (args[0] === root) throw Object.assign(new Error("write-only creation parent"), { code: "EACCES" });
return open(...args);
});
await expect(ensureDurableDirectory(directory)).rejects.toMatchObject({ code: "EACCES" });
// mkdir now succeeds without creating anything. Its first attempt still
// owes a parent flush, so a retry cannot silently acknowledge that path.
await expect(ensureDurableDirectory(directory)).rejects.toMatchObject({ code: "EACCES" });
vi.restoreAllMocks();
const synced = observeSyncs();
await ensureDurableDirectory(directory);
expect(synced).toContain(root);
});
it.each([
{ sibling: false, symlink: false }, { sibling: true, symlink: false },
{ sibling: false, symlink: true }, { sibling: true, symlink: true },
])("retains a failed parent flush across fresh processes ($sibling, symlink: $symlink)", async ({ sibling, symlink }) => {
const { root, directory: original } = await fixture();
let directory = original;
let parent = root;
if (symlink) {
parent = path.join(root, "storage");
await fs.mkdir(path.join(parent, "home"), { recursive: true });
await fs.symlink(path.join(parent, "home"), path.join(root, "alias"));
directory = path.join(root, "alias", "recovery", "company");
}
const source = new URL("../lib/durable-directory.ts", import.meta.url).href;
const child = (target: string, fail: boolean) => spawnSync(process.execPath, ["--import", createRequire(import.meta.url).resolve("tsx"), "--input-type=module", "-e", `
import { promises as fs } from "node:fs";
import { ensureDurableDirectory } from ${JSON.stringify(source)};
const [directory, parent, fail] = process.argv.slice(1);
const open = fs.open.bind(fs);
fs.open = async (...args) => {
if (fail === "true" && args[0] === parent) throw Object.assign(new Error("unflushed parent"), { code: "EACCES" });
return open(...args);
};
try { await ensureDurableDirectory(directory); }
catch (error) { console.error(error.code); process.exitCode = error.code === "EACCES" ? 23 : 1; }
`, target, parent, String(fail)], { encoding: "utf8", timeout: 10000 });
const first = child(directory, true);
expect(first.stderr).toContain("EACCES");
expect(first.status).toBe(23);
expect((await fs.stat(directory)).isDirectory()).toBe(true);
const retryTarget = sibling ? path.join(path.dirname(directory), "other-company") : directory;
const retry = child(retryTarget, true);
expect(retry.status, retry.stderr).toBe(23);
expect(retry.stderr).toContain("EACCES");
const recovered = child(retryTarget, false);
expect(recovered.status, recovered.stderr).toBe(0);
});
it("retries all ancestors after a failed flush even though mkdir already succeeded", async () => {
const { root, directory } = await fixture();
observeSyncs(root);
await expect(ensureDurableDirectory(directory)).rejects.toThrow("Injected directory fsync failure");
expect((await fs.stat(directory)).isDirectory()).toBe(true);
vi.restoreAllMocks();
const synced = observeSyncs();
await ensureDurableDirectory(directory);
expect(synced).toContain(root);
expect(synced).toContain(path.dirname(root));
});
it.each([false, true])("keeps a shared parent flush required for sibling writes (first call pending: %s)", async (concurrent) => {
const { root, directory } = await fixture();
const sibling = path.join(path.dirname(directory), "other-company");
const open = fs.open.bind(fs);
let releaseFirst!: () => void;
let reachedParent!: () => void;
const held = new Promise<void>(resolve => { releaseFirst = resolve; });
const reached = new Promise<void>(resolve => { reachedParent = resolve; });
let firstParent = true;
vi.spyOn(fs, "open").mockImplementation(async (...args: Parameters<typeof fs.open>) => {
if (args[0] === root) {
if (firstParent) {
firstParent = false;
reachedParent();
if (concurrent) await held;
}
throw Object.assign(new Error("unflushed shared parent"), { code: "EACCES" });
}
return open(...args);
});
const first = ensureDurableDirectory(directory).then(() => null, error => error);
try {
await reached;
if (!concurrent) expect(await first).toMatchObject({ code: "EACCES" });
await expect(ensureDurableDirectory(sibling)).rejects.toMatchObject({ code: "EACCES" });
} finally {
releaseFirst();
await first;
}
vi.restoreAllMocks();
const synced = observeSyncs();
await ensureDurableDirectory(sibling);
expect(synced).toContain(root);
await ensureDurableDirectory(directory);
synced.length = 0;
await ensureDurableDirectory(sibling);
await ensureDurableDirectory(directory);
expect(synced).toEqual([]);
});
});
server/src/__tests__/environment-execution-target.test.ts
+17
View File
@@ -713,6 +713,23 @@ describe("resolveEnvironmentExecutionTarget", () => {
expect(delivered).toEqual([["stdout", "full"]]);
});
it("does not replay a capped provider tail after streaming more than 4 MiB", async () => {
const receipt = JSON.stringify({ type: "turn_end", message: { usage: { input: 20, output: 10, cost: { total: 0.004 } } } }) + "\n";
const output = "x".repeat(5 * 1024 * 1024) + "\n" + receipt;
const { tracer } = createRecordingExecTracer();
const runner = await runnerWithExecute({ provider: "createos", tracer,
execute: async (input: unknown) => {
const typed = input as { onLog?: (s: "stdout" | "stderr", c: string) => Promise<void> };
for (let offset = 0; offset < output.length; offset += 65536) await typed.onLog?.("stdout", output.slice(offset, offset + 65536));
await typed.onLog?.("stderr", "diagnostic prefix\nlast warning\n");
return { exitCode: 0, signal: null, timedOut: false, stdout: output.slice(-4 * 1024 * 1024), stderr: "last warning\n" };
},
});
const delivered = await runExecuteCollectingLogs(runner as { execute(input: unknown): Promise<unknown> });
expect(delivered.filter(([stream]) => stream === "stdout").map(([, chunk]) => chunk).join("")).toBe(output);
expect(delivered.filter(([stream]) => stream === "stderr")).toEqual([["stderr", "diagnostic prefix\nlast warning\n"]]);
});
it("delivers the full captured output when the provider streams nothing incrementally", async () => {
// The provider streams no incremental chunk, so the whole final result is
// the suffix and reaches the sink once.
server/src/__tests__/environment-runtime.test.ts
+2
View File
@@ -11,6 +11,7 @@ import {
stopSshEnvLabFixture,
} from "@paperclipai/adapter-utils/ssh";
import {
costEvents,
agents,
builtInManagedResources,
companies,
@@ -219,6 +220,7 @@ describeEmbeddedPostgres("environmentRuntimeService", () => {
}
await db.delete(environmentLeases);
await db.delete(issues);
await db.delete(costEvents);
await db.delete(heartbeatRuns);
await db.delete(agents);
await db.delete(environments);
server/src/__tests__/execution-lock-orphan-cleanup.test.ts
+2
View File
@@ -2,6 +2,7 @@ import { randomUUID } from "node:crypto";
import { eq } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest";
import {
costEvents,
activityLog,
agents,
agentWakeupRequests,
@@ -40,6 +41,7 @@ describeEmbeddedPostgres("execution lock orphan cleanup", () => {
await db.delete(activityLog);
await db.delete(heartbeatRunEvents);
await db.delete(issues);
await db.delete(costEvents);
await db.delete(heartbeatRuns);
await db.delete(agents);
await db.delete(companies);
server/src/__tests__/gemini-local-adapter.test.ts
+1 -1
View File
@@ -34,7 +34,7 @@ describe("gemini_local parser", () => {
expect(parsed.sessionId).toBe("gemini-session-1");
expect(parsed.summary).toBe("hello");
expect(parsed.usage).toEqual({
inputTokens: 12,
inputTokens: 9,
cachedInputTokens: 3,
outputTokens: 7,
});
server/src/__tests__/heartbeat-accepted-plan-workspace-refresh.test.ts
+2
View File
@@ -7,6 +7,7 @@ import { promisify } from "node:util";
import { eq, ne } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
import {
costEvents,
activityLog,
agentRuntimeState,
agentTaskSessions,
@@ -136,6 +137,7 @@ describeEmbeddedPostgres("accepted plan workspace refresh", () => {
await db.delete(activityLog);
await db.delete(heartbeatRunEvents);
try {
await db.delete(costEvents);
await db.delete(heartbeatRuns);
break;
} catch (error) {
server/src/__tests__/heartbeat-archived-company-guard.test.ts
+2
View File
@@ -1,6 +1,7 @@
import { randomUUID } from "node:crypto";
import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest";
import {
costEvents,
agents,
agentWakeupRequests,
companies,
@@ -35,6 +36,7 @@ describeEmbeddedPostgres("heartbeat archived-company guard", () => {
afterEach(async () => {
await db.delete(heartbeatRunEvents);
await db.delete(costEvents);
await db.delete(heartbeatRuns);
await db.delete(agentWakeupRequests);
await db.delete(issues);
server/src/__tests__/heartbeat-dependency-scheduling.test.ts
+2
View File
@@ -2,6 +2,7 @@ import { randomUUID } from "node:crypto";
import { and, eq, inArray, sql } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
import {
costEvents,
activityLog,
agents,
agentRuntimeState,
@@ -143,6 +144,7 @@ describeEmbeddedPostgres("heartbeat dependency-aware queued run selection", () =
await db.delete(issues);
await db.delete(heartbeatRunEvents);
await db.delete(activityLog);
await db.delete(costEvents);
await db.delete(heartbeatRuns);
await db.delete(agentWakeupRequests);
await db.delete(agentRuntimeState);
server/src/__tests__/heartbeat-issue-rewake-throttle.test.ts
+2
View File
@@ -2,6 +2,7 @@ import { randomUUID } from "node:crypto";
import { and, desc, eq, sql } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
import {
costEvents,
activityLog,
agentRuntimeState,
agentWakeupRequests,
@@ -90,6 +91,7 @@ describeEmbeddedPostgres("heartbeat issue rewake throttle", () => {
await db.delete(issues);
await db.delete(heartbeatRunEvents);
await db.delete(activityLog);
await db.delete(costEvents);
await db.delete(heartbeatRuns);
await db.delete(agentWakeupRequests);
await db.delete(agentRuntimeState);
server/src/__tests__/heartbeat-lock-release-on-reassignment.test.ts
+2
View File
@@ -2,6 +2,7 @@ import { randomUUID } from "node:crypto";
import { and, eq } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
import {
costEvents,
agents,
agentRuntimeState,
agentWakeupRequests,
@@ -59,6 +60,7 @@ describeEmbeddedPostgres("heartbeat lock release on cross-agent reassignment", (
afterEach(async () => {
await drainHeartbeatRunsToQuiescence(db, heartbeat);
await db.delete(heartbeatRunEvents);
await db.delete(costEvents);
await db.delete(heartbeatRuns);
await db.delete(agentWakeupRequests);
await db.delete(issues);
server/src/__tests__/heartbeat-orphaned-active-lease-sweep.test.ts
+2
View File
@@ -2,6 +2,7 @@ import { randomUUID } from "node:crypto";
import { eq } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
import {
costEvents,
agents,
companies,
createDb,
@@ -62,6 +63,7 @@ describeEmbeddedPostgres("heartbeat sweepOrphanedActiveLeases", () => {
afterEach(async () => {
await db.delete(environmentLeases);
await db.delete(costEvents);
await db.delete(heartbeatRuns);
await db.delete(environments);
await db.delete(agents);
server/src/__tests__/heartbeat-pending-cleanup-sweep.test.ts
+2
View File
@@ -2,6 +2,7 @@ import { randomUUID } from "node:crypto";
import { eq, sql } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
import {
costEvents,
activityLog,
agents,
heartbeatRuns,
@@ -71,6 +72,7 @@ describeEmbeddedPostgres("heartbeat sweepPendingCleanupLeases", () => {
afterEach(async () => {
await db.delete(activityLog);
await db.delete(environmentLeases);
await db.delete(costEvents);
await db.delete(heartbeatRuns);
await db.delete(environments);
await db.delete(agents);
server/src/__tests__/heartbeat-process-recovery.test.ts
+153 -130
View File
@@ -357,6 +357,22 @@ async function waitForHeartbeatIdle(
}
}
async function stopRegisteredProcessesForCleanup() {
await Promise.all([...runningProcesses.values()].map(async ({ child, processGroupId }) => {
if (child.exitCode !== null || child.signalCode !== null || typeof child.kill !== "function") return;
await new Promise<void>((resolve) => {
child.once("close", () => resolve());
try {
if (processGroupId) process.kill(-processGroupId, "SIGKILL");
else child.kill("SIGKILL");
} catch {
child.kill("SIGKILL");
}
});
}));
runningProcesses.clear();
}
async function cancelActiveRunsForCleanup(
db: ReturnType<typeof createDb>,
timeoutMs = 3_000,
@@ -511,7 +527,7 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
provider: "test",
model: "test-model",
}));
runningProcesses.clear();
await stopRegisteredProcessesForCleanup();
for (const child of childProcesses) {
child.kill("SIGKILL");
}
@@ -550,7 +566,9 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
}
await new Promise((resolve) => setTimeout(resolve, 50));
await waitForHeartbeatIdle(db, 5_000);
await new Promise((resolve) => setTimeout(resolve, 100));
// Terminal rows do not mean their background finalizers have returned.
// Drain tracked wakeups too: one can still create a run after the row poll.
await heartbeatService(db).drainActiveRunExecutions();
await db.delete(activityLog);
await db.delete(agentRuntimeState);
await db.delete(companySkills);
@@ -669,7 +687,7 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
}
}
cleanupPids.clear();
runningProcesses.clear();
await stopRegisteredProcessesForCleanup();
if (externalTestDatabaseUrl) {
await closeRegisteredClients(externalTestDatabaseUrl);
}
@@ -821,13 +839,12 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
}
it("does not reap active adapter executions started by another heartbeat service instance", async () => {
let releaseAdapter: (() => void) | null = null;
let releaseAdapter!: () => void;
const adapterRelease = new Promise<void>(resolve => { releaseAdapter = resolve; });
const adapterStarted = new Promise<void>((resolve) => {
mockAdapterExecute.mockImplementationOnce(async () => {
resolve();
await new Promise<void>((release) => {
releaseAdapter = release;
});
await adapterRelease;
return {
exitCode: 0,
signal: null,
@@ -851,52 +868,55 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
const executorHeartbeat = heartbeatService(db);
const reaperHeartbeat = heartbeatService(db);
await executorHeartbeat.resumeQueuedRuns();
await Promise.race([
adapterStarted,
new Promise<never>((_, reject) => {
setTimeout(
() =>
reject(
new Error("Timed out waiting for adapter execution to start"),
),
3_000,
);
}),
]);
try {
await executorHeartbeat.resumeQueuedRuns();
await Promise.race([
adapterStarted,
new Promise<never>((_, reject) => {
setTimeout(
() =>
reject(
new Error("Timed out waiting for adapter execution to start"),
),
3_000,
);
}),
]);
await db
.update(heartbeatRuns)
.set({
updatedAt: new Date("2026-03-19T00:00:00.000Z"),
})
.where(eq(heartbeatRuns.id, runId));
await db
.update(heartbeatRuns)
.set({
updatedAt: new Date("2026-03-19T00:00:00.000Z"),
})
.where(eq(heartbeatRuns.id, runId));
const result = await reaperHeartbeat.reapOrphanedRuns({
staleThresholdMs: 1,
});
expect(result).toEqual({ reaped: 0, runIds: [] });
const result = await reaperHeartbeat.reapOrphanedRuns({
staleThresholdMs: 1,
});
expect(result).toEqual({ reaped: 0, runIds: [] });
const activeRun = await reaperHeartbeat.getRun(runId);
expect(activeRun?.status).toBe("running");
expect(activeRun?.errorCode).toBeNull();
const activeRun = await reaperHeartbeat.getRun(runId);
expect(activeRun?.status).toBe("running");
expect(activeRun?.errorCode).toBeNull();
const wakeup = await db
.select()
.from(agentWakeupRequests)
.where(eq(agentWakeupRequests.id, wakeupRequestId))
.then((rows) => rows[0] ?? null);
expect(wakeup?.status).toBe("claimed");
const wakeup = await db
.select()
.from(agentWakeupRequests)
.where(eq(agentWakeupRequests.id, wakeupRequestId))
.then((rows) => rows[0] ?? null);
expect(wakeup?.status).toBe("claimed");
if (!releaseAdapter)
throw new Error("Adapter release handle was not captured");
releaseAdapter();
const settledRun = await waitForRunToSettle(
executorHeartbeat,
runId,
5_000,
);
expect(settledRun?.status).toBe("succeeded");
releaseAdapter();
const settledRun = await waitForRunToSettle(
executorHeartbeat,
runId,
5_000,
);
expect(settledRun?.status).toBe("succeeded");
} finally {
releaseAdapter();
await executorHeartbeat.drainActiveRunExecutions();
}
});
async function seedStrandedIssueFixture(input: {
@@ -3615,7 +3635,8 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
});
it("persists codex_local spawn identity before hot restart and never loses the live run for missing metadata", async () => {
let releaseAdapter: (() => void) | null = null;
let releaseAdapter!: () => void;
const adapterRelease = new Promise<void>(resolve => { releaseAdapter = resolve; });
let spawnedPid: number | null = null;
const adapterStarted = new Promise<void>((resolve) => {
mockAdapterExecute.mockImplementationOnce(async (rawInput?: unknown) => {
@@ -3637,9 +3658,7 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
startedAt: new Date("2026-07-30T07:00:00.000Z").toISOString(),
});
resolve();
await new Promise<void>((release) => {
releaseAdapter = release;
});
await adapterRelease;
return {
exitCode: 0,
signal: null,
@@ -3664,87 +3683,90 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
includeIssue: false,
});
const heartbeat = heartbeatService(db);
await heartbeat.resumeQueuedRuns();
await Promise.race([
adapterStarted,
new Promise<never>((_, reject) => {
setTimeout(
() =>
reject(
new Error("Timed out waiting for codex_local spawn identity"),
),
3_000,
);
}),
]);
const running = await waitForValue(async () =>
db
.select()
.from(heartbeatRuns)
.where(eq(heartbeatRuns.id, runId))
.then((rows) => {
const row = rows[0] ?? null;
return row?.status === "running" && row.processPid ? row : null;
try {
await heartbeat.resumeQueuedRuns();
await Promise.race([
adapterStarted,
new Promise<never>((_, reject) => {
setTimeout(
() =>
reject(
new Error("Timed out waiting for codex_local spawn identity"),
),
3_000,
);
}),
);
const observedProcessStartedAt = await readProcessStartedAt(spawnedPid!);
expect(observedProcessStartedAt).not.toBeNull();
expect(running).toMatchObject({
id: runId,
status: "running",
processPid: spawnedPid,
processGroupId: null,
processStartedAt: new Date(observedProcessStartedAt!),
});
]);
await withTempPaperclipHome(async (home) => {
await writeHotRestartIntent({
previousServerPid: process.pid,
previousServerVersion: "old-version",
requestedAt: new Date("2026-07-30T07:01:00.000Z"),
});
await heartbeat.prepareHotRestartShutdown(
"SIGTERM",
new Date("2026-07-30T07:02:00.000Z"),
);
const adoption = await heartbeat.reconcileHotRestartAdoption(
new Date("2026-07-30T07:03:00.000Z"),
);
expect(adoption).toMatchObject({
mode: "reported",
adoptedRunIds: [runId],
finalizedWhileDownRunIds: [],
lostRunIds: [],
});
const report = JSON.parse(
await fs.readFile(resolveHotRestartReportPath(home), "utf8"),
) as { runs?: Array<Record<string, unknown>> };
expect(report.runs).toEqual(
expect.arrayContaining([
expect.objectContaining({
runId,
classification: "adopted",
reason: "process_pid_alive",
const running = await waitForValue(async () =>
db
.select()
.from(heartbeatRuns)
.where(eq(heartbeatRuns.id, runId))
.then((rows) => {
const row = rows[0] ?? null;
return row?.status === "running" && row.processPid ? row : null;
}),
]),
);
expect(report.runs).not.toEqual(
expect.arrayContaining([
expect.objectContaining({
runId,
reason: "missing_process_metadata",
}),
]),
);
});
const observedProcessStartedAt = await readProcessStartedAt(spawnedPid!);
expect(observedProcessStartedAt).not.toBeNull();
expect(running).toMatchObject({
id: runId,
status: "running",
processPid: spawnedPid,
processGroupId: null,
processStartedAt: new Date(observedProcessStartedAt!),
});
if (!releaseAdapter)
throw new Error("Adapter release handle was not captured");
releaseAdapter();
const settled = await waitForRunToSettle(heartbeat, runId, 5_000);
expect(settled?.status).toBe("succeeded");
await withTempPaperclipHome(async (home) => {
await writeHotRestartIntent({
previousServerPid: process.pid,
previousServerVersion: "old-version",
requestedAt: new Date("2026-07-30T07:01:00.000Z"),
});
await heartbeat.prepareHotRestartShutdown(
"SIGTERM",
new Date("2026-07-30T07:02:00.000Z"),
);
const adoption = await heartbeat.reconcileHotRestartAdoption(
new Date("2026-07-30T07:03:00.000Z"),
);
expect(adoption).toMatchObject({
mode: "reported",
adoptedRunIds: [runId],
finalizedWhileDownRunIds: [],
lostRunIds: [],
});
const report = JSON.parse(
await fs.readFile(resolveHotRestartReportPath(home), "utf8"),
) as { runs?: Array<Record<string, unknown>> };
expect(report.runs).toEqual(
expect.arrayContaining([
expect.objectContaining({
runId,
classification: "adopted",
reason: "process_pid_alive",
}),
]),
);
expect(report.runs).not.toEqual(
expect.arrayContaining([
expect.objectContaining({
runId,
reason: "missing_process_metadata",
}),
]),
);
});
releaseAdapter();
const settled = await waitForRunToSettle(heartbeat, runId, 5_000);
expect(settled?.status).toBe("succeeded");
} finally {
releaseAdapter();
await heartbeat.drainActiveRunExecutions();
}
});
it("reports adopted hot-restart runs before startup reap can mark them process_lost", async () => {
@@ -7486,7 +7508,7 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
}
});
it("dispatches interrupted CLI input after the executor releases its lease", async () => {
it.each([false, true])("dispatches interrupted CLI input and cleans up early fixture exits (%s)", async (earlyExit) => {
const actualProcess = await vi.importActual<typeof import("../adapters/process/execute.js")>("../adapters/process/execute.js");
const { companyId, agentId, issueId, runId } = await seedRunFixture({
runtimeMode: "legacy", adapterType: "codex_local", agentStatus: "idle", runStatus: "queued",
@@ -7499,6 +7521,7 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
const heartbeat = heartbeatService(db);
await heartbeat.resumeQueuedRuns();
expect(await waitForValue(async () => runningProcesses.get(runId))).toBeTruthy();
if (earlyExit) return; // Exercise afterEach while the unbounded CLI child is still live.
const [comment] = await db.insert(issueComments).values({ companyId, issueId, authorUserId: "responsible-user", body: "continue" }).returning();
const [wake] = await db.insert(agentWakeupRequests).values({
companyId, agentId, source: "automation", reason: "issue_commented", status: "deferred_issue_execution",
@@ -10739,7 +10762,7 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
expect(beforePromotion?.status).toBe("scheduled_retry");
// Simulate a server restart: no in-memory process/timer state carries over.
runningProcesses.clear();
await stopRegisteredProcessesForCleanup();
const restarted = heartbeatService(db);
const promotion = await restarted.promoteDueScheduledRetries(
scheduled.dueAt,
server/src/__tests__/heartbeat-responsible-user-invariant.test.ts
+2
View File
@@ -2,6 +2,7 @@ import { randomUUID } from "node:crypto";
import { and, eq } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
import {
costEvents,
activityLog,
agents,
agentRuntimeState,
@@ -62,6 +63,7 @@ async function deleteHeartbeatRunsAfterEvents(db: ReturnType<typeof createDb>) {
for (let attempt = 0; attempt < 5; attempt += 1) {
await db.delete(heartbeatRunEvents);
try {
await db.delete(costEvents);
await db.delete(heartbeatRuns);
return;
} catch (error) {
server/src/__tests__/heartbeat-retry-scheduling.test.ts
+2
View File
@@ -2,6 +2,7 @@ import { randomUUID } from "node:crypto";
import { and, eq, inArray, sql } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
import {
costEvents,
agents,
approvals,
issueApprovals,
@@ -164,6 +165,7 @@ describeEmbeddedPostgres("heartbeat bounded retry scheduling", () => {
}
async function cleanupRetryFixtureOnce() {
await db.delete(costEvents);
await db.delete(activityLog);
await db.delete(environmentLeases);
await db.delete(issueRelations);
server/src/__tests__/heartbeat-run-lease-release-terminalization.test.ts
+2
View File
@@ -2,6 +2,7 @@ import { randomUUID } from "node:crypto";
import { eq } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
import {
costEvents,
agents,
companies,
createDb,
@@ -42,6 +43,7 @@ describeEmbeddedPostgres("heartbeat terminalizeRunOnLeaseRelease", () => {
afterEach(async () => {
await db.delete(heartbeatRunEvents);
await db.delete(issues);
await db.delete(costEvents);
await db.delete(heartbeatRuns);
await db.delete(agents);
await db.delete(companies);
server/src/__tests__/heartbeat-run-terminalize-before-release.test.ts
+2
View File
@@ -2,6 +2,7 @@ import { randomUUID } from "node:crypto";
import { eq } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
import {
costEvents,
agents,
companies,
createDb,
@@ -71,6 +72,7 @@ describeEmbeddedPostgres("heartbeat teardown terminalizes the run before releasi
afterEach(async () => {
await db.delete(heartbeatRunEvents);
await db.delete(issues);
await db.delete(costEvents);
await db.delete(heartbeatRuns);
await db.delete(agents);
await db.delete(companies);
server/src/__tests__/heartbeat-runtime-state.test.ts
+2
View File
@@ -2,6 +2,7 @@ import { randomUUID } from "node:crypto";
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
import { eq } from "drizzle-orm";
import {
costEvents,
agents,
agentRuntimeState,
agentWakeupRequests,
@@ -52,6 +53,7 @@ describeEmbeddedPostgres("heartbeat runtime state deduplication", () => {
afterEach(async () => {
clearAllHeartbeatRunRuntimeStatuses();
await db.delete(heartbeatRunEvents);
await db.delete(costEvents);
await db.delete(heartbeatRuns);
await db.delete(agentWakeupRequests);
await db.delete(agentRuntimeState);
server/src/__tests__/heartbeat-stale-queue-invalidation.test.ts
+36
View File
@@ -4,6 +4,7 @@ import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest
import {
agents,
agentWakeupRequests,
agentRuntimeState,
companies,
costEvents,
createDb,
@@ -1227,6 +1228,41 @@ describeEmbeddedPostgres("heartbeat stale queued-run invalidation", () => {
);
});
it.each(["codex_local", "gemini_local"].flatMap(adapterType => [
{ adapterType, inputTokens: 50_000, cachedInputTokens: 550_000, rotate: true, rawInputIncludesCached: false },
{ adapterType, inputTokens: 50_000, cachedInputTokens: 449_999, rotate: false, rawInputIncludesCached: false },
{ adapterType, inputTokens: 0, cachedInputTokens: 500_000, rotate: true, rawInputIncludesCached: false },
{ adapterType, inputTokens: 400_000, cachedInputTokens: 350_000, rotate: false, rawInputIncludesCached: undefined },
{ adapterType, inputTokens: 500_000, cachedInputTokens: 350_000, rotate: true, rawInputIncludesCached: undefined },
]))("counts cached prompt input for $adapterType session rotation ($inputTokens + $cachedInputTokens; raw includes cache: $rawInputIncludesCached)", async ({ adapterType, inputTokens, cachedInputTokens, rotate, rawInputIncludesCached }) => {
const { companyId, agentId } = await seedCompanyAndAgent({ heartbeatConfig: {
sessionCompaction: { enabled: true, maxSessionRuns: 0, maxRawInputTokens: 500_000, maxSessionAgeHours: 0 },
} });
const sessionId = randomUUID();
await db.update(agents).set({ adapterType }).where(eq(agents.id, agentId));
await db.insert(agentRuntimeState).values({ companyId, agentId, adapterType, sessionId });
await db.insert(heartbeatRuns).values({ companyId, agentId, status: "succeeded", sessionIdAfter: sessionId,
usageJson: { ...(rawInputIncludesCached === false ? { rawInputIncludesCached } : {}), inputTokens, cachedInputTokens, outputTokens: 10, rawInputTokens: inputTokens, rawCachedInputTokens: cachedInputTokens },
resultJson: { summary: "Continue from this completed turn" }, startedAt: new Date(), finishedAt: new Date(),
});
mockAdapterExecute.mockResolvedValueOnce({ exitCode: 0, signal: null, timedOut: false, errorMessage: null,
provider: "test", model: "test-model", costUsd: 0, usageComplete: true, usageBasis: "per_run",
usage: { inputTokens: 1, cachedInputTokens: 2, outputTokens: 3 },
});
const run = await heartbeat.wakeup(agentId, { source: "on_demand", triggerDetail: "manual" });
expect(run).not.toBeNull();
expect(await waitForCondition(() => Promise.resolve(countExecuteCallsForRun(run!.id) === 1))).toBe(true);
const [context] = mockAdapterExecute.mock.calls.find(([context]) => context?.runId === run!.id)!;
expect(context.runtime.sessionId).toBe(rotate ? null : sessionId);
if (rotate) {
expect(context.context.paperclipSessionRotationReason).toContain((inputTokens + (rawInputIncludesCached === false ? cachedInputTokens : 0)).toLocaleString("en-US"));
expect(context.context.paperclipSessionHandoffMarkdown).toContain("Continue from this completed turn");
} else expect(context.context.paperclipSessionRotationReason).toBeUndefined();
await heartbeat.drainActiveRunExecutions();
const [finished] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run!.id));
expect(finished.usageJson).toMatchObject({ rawInputTokens: 1, rawCachedInputTokens: 2, rawInputIncludesCached: false });
});
it("skips wakes before queueing when per-agent daily cost cap is reached", async () => {
const { companyId, agentId } = await seedCompanyAndAgent({
heartbeatConfig: {
server/src/__tests__/heartbeat-task-drain-admission-release.test.ts
+2
View File
@@ -2,6 +2,7 @@ import { randomUUID } from "node:crypto";
import { eq, sql } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest";
import {
costEvents,
activityLog,
agents,
agentRuntimeState,
@@ -51,6 +52,7 @@ describeEmbeddedPostgres("heartbeat task-drain admission release", () => {
await db.delete(heartbeatRunEvents);
await db.delete(activityLog);
try {
await db.delete(costEvents);
await db.delete(heartbeatRuns);
return;
} catch (error) {
server/src/__tests__/heartbeat-workspace-branch-containment.test.ts
+2
View File
@@ -7,6 +7,7 @@ import { promisify } from "node:util";
import { and, eq, inArray } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
import {
costEvents,
activityLog,
agentRuntimeState,
agentWakeupRequests,
@@ -179,6 +180,7 @@ async function deleteHeartbeatRunsForCleanup(db: Db) {
await db.delete(heartbeatRunEvents);
await db.delete(activityLog);
try {
await db.delete(costEvents);
await db.delete(heartbeatRuns);
return;
} catch (error) {
server/src/__tests__/heartbeat-workspace-busy.test.ts
+2
View File
@@ -5,6 +5,7 @@ import path from "node:path";
import { and, eq } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
import {
costEvents,
agents,
agentRuntimeState,
agentWakeupRequests,
@@ -151,6 +152,7 @@ describeEmbeddedPostgres("shared-workspace run serialization", () => {
}
async function cleanupFixtureOnce() {
await db.delete(costEvents);
await db.delete(activityLog);
await db.delete(environmentLeases);
await db.delete(issueComments);
server/src/__tests__/heartbeat-workspace-finalize-branch.test.ts
+2
View File
@@ -7,6 +7,7 @@ import { promisify } from "node:util";
import { and, asc, eq } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
import {
costEvents,
activityLog,
agentRuntimeState,
agentTaskSessions,
@@ -121,6 +122,7 @@ async function deleteHeartbeatRowsAfterActivityLogDrains(db: Db) {
await db.delete(activityLog);
await db.delete(heartbeatRunEvents);
try {
await db.delete(costEvents);
await db.delete(heartbeatRuns);
await db.delete(agentWakeupRequests);
return;
server/src/__tests__/heartbeat-worktree-suppression.test.ts
+2
View File
@@ -2,6 +2,7 @@ import { randomUUID } from "node:crypto";
import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest";
import { eq, sql } from "drizzle-orm";
import {
costEvents,
activityLog,
agents,
agentWakeupRequests,
@@ -52,6 +53,7 @@ describeEmbeddedPostgres("heartbeat worktree suppression", () => {
await db.delete(heartbeatRunEvents);
await db.delete(activityLog);
try {
await db.delete(costEvents);
await db.delete(heartbeatRuns);
return;
} catch (error) {
server/src/__tests__/issue-monitor-scheduler.test.ts
+2
View File
@@ -3,6 +3,7 @@ import { eq, sql } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest";
import { PROVIDER_QUOTA_MONITOR_SERVICE_NAME } from "@paperclipai/shared";
import {
costEvents,
activityLog,
agentRuntimeState,
agentWakeupRequests,
@@ -111,6 +112,7 @@ describeEmbeddedPostgres("issue monitor scheduler", () => {
await db.delete(environmentLeases);
await db.delete(workspaceRuntimeServices);
await db.delete(issues);
await db.delete(costEvents);
await db.delete(heartbeatRuns);
await db.delete(agentWakeupRequests);
await db.delete(agentRuntimeState);
server/src/__tests__/issue-onboarding-first-task-routes.test.ts
+2
View File
@@ -4,6 +4,7 @@ import request from "supertest";
import { and, eq } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
import {
costEvents,
activityLog,
agentRuntimeState,
agents,
@@ -94,6 +95,7 @@ describeEmbeddedPostgres("issue create onboarding first-task routes", () => {
await db.delete(issueThreadInteractions);
await db.delete(issueComments);
await db.delete(heartbeatRunEvents);
await db.delete(costEvents);
await db.delete(heartbeatRuns);
await db.delete(agentWakeupRequests);
await db.delete(agentRuntimeState);
server/src/__tests__/issue-recovery-actions.test.ts
+2
View File
@@ -7,6 +7,7 @@ import request from "supertest";
import { and, eq, sql } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
import {
costEvents,
agents,
agentRuntimeState,
authUsers,
@@ -154,6 +155,7 @@ describeEmbeddedPostgres("issue recovery actions", () => {
await db.delete(issueComments);
await db.delete(environmentLeases);
await db.delete(activityLog);
await db.delete(costEvents);
await db.delete(heartbeatRuns);
await db.delete(agentWakeupRequests);
await db.delete(environments);
server/src/__tests__/issue-watchdogs-routes.test.ts
+2
View File
@@ -4,6 +4,7 @@ import request from "supertest";
import { and, eq } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
import {
costEvents,
activityLog,
agentRuntimeState,
agentWakeupRequests,
@@ -80,6 +81,7 @@ describeEmbeddedPostgres("issue watchdog routes", () => {
await db.delete(activityLog);
await db.delete(issueComments);
await db.delete(heartbeatRunEvents);
await db.delete(costEvents);
await db.delete(heartbeatRuns);
await db.delete(agentWakeupRequests);
await db.delete(agentRuntimeState);
server/src/__tests__/low-trust-red-team-routes.test.ts
+2
View File
@@ -6,6 +6,7 @@ import { WebSocketServer } from "ws";
import { and, eq } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest";
import {
costEvents,
activityLog,
authUsers,
agentWakeupRequests,
@@ -103,6 +104,7 @@ async function deleteHeartbeatRunsAndWakeupsAfterActivityLogDrains(db: Db) {
await db.delete(activityLog);
try {
await db.delete(heartbeatRunEvents);
await db.delete(costEvents);
await db.delete(heartbeatRuns);
await db.delete(agentWakeupRequests);
return;
Loaded 100 of 152 files, more files were not shown because too many files have changed in this diff. Show more