mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-29 21:07:40 +01:00
DottaandPaperclip
9b7d5b35eb
## Thinking Path
> - Paperclip coordinates agent work through Runner.
> - Scheduled Product E2E campaigns need measured cost and time
forecasts.
> - The saved forecast samples came from October 21.
> - The October 27 qualifications retain final usage for all 19 model
requests.
> - The Claude measurements are hosted; the Codex measurements are
local.
> - This change refreshes the samples and keeps measurement hosts
separate.
> - Operators can inspect the latest evidence without treating it as
current qualification or an invoice.
## Linked Issues or Issue Description
Refs #17127 and #17946.
**What happened?**
The scheduled forecast still uses the older pilot amounts. It reports no
hosted measurement and does not record the measurement host per sample.
**What did you expect to happen?**
The forecast must retain the latest complete model receipts and their
source hashes. It must keep local and hosted timing cohorts separate. A
changed definition must remain unqualified.
**Steps to reproduce**
Prepare the daily or weekly forecast from the default branch. Inspect
its saved measurement date, totals, and hosted fields.
## What Changed
- Refresh four samples from two intact request ledgers and four retained
Product results.
- Record measurement hosts, final request counts, and original campaign
outcomes.
- Add historical hosted totals. Current hosted totals still require
matching evidence for every selected cell.
- Keep hosts in separate per-cell cohorts. Preserve definition drift and
unknown slow-tail estimates.
- Document the model-cost scope and timing limits.
## Verification
- Forecast and scheduled-preflight tests: 33 passed.
- The extraction replays both hash-checked ledgers, requires final
receipts, and checks the retained pricecards. The four samples cover 19
final requests.
- Daily historical model subtotal: $0.428650100; summed attempt time:
107.286 seconds.
- Weekly historical model subtotal: $0.793676500; summed attempt time:
158.093 seconds. The Codex samples are local.
- Full Product support suite at the initial head: 4,510 passed, seven
failed, three skipped. A clean 6b9e5f3a baseline with the same installed
dependencies reproduces the same seven failures: two Copilot image
assertions, four workflow-selection shell assertions, and one database
startup test.
- Product typecheck also fails on that clean baseline because the shared
installed Runner workspace exports are stale. The fresh CI build remains
the typecheck gate.
- No provider request, new live campaign, budget change, or schedule
activation is part of this PR.
## Risks
- These are successful-attempt measurements, not a prediction of failure
frequency or a campaign invoice.
- The feedback source fingerprint changed after the samples.
Current-compatible totals remain unknown; no historical fingerprint is
rewritten.
- Hosted worker cases passed, but their original workflow failed during
report dependency setup. That outcome stays visible.
- Hosted detection deliberately accepts only the measured GitHub Ubuntu
environment. Other host labels remain unqualified until declared with
their own evidence.
- The weekly selection still has no complete hosted timing measurement.
Infrastructure, storage, and observability charges stay separate from
model spend.
## Model Used
OpenAI Codex, GPT-6 based, with tool execution and extended reasoning.
The exact serving model ID and context window are not exposed in this
session.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
99 lines
5.0 KiB
JSON
99 lines
5.0 KiB
JSON
{
|
|
"version": 1,
|
|
"recordedAt": "2026-10-28T03:23:41.620Z",
|
|
"costStatus": "estimated",
|
|
"sampleScope": "successful Product attempts with complete terminal model receipts; original failed campaign outcomes retained",
|
|
"host": "mixed; see each sample measurementHost",
|
|
"samples": [
|
|
{
|
|
"executionId": "core-compatibility.runner-acpx-claude.local.message-marker",
|
|
"attemptId": "c718184ebbdde012b1523317a800f6fb",
|
|
"sourceRevision": "bf3ef12f3a78d0b15a32ad57f7e16520fbf6cd5e",
|
|
"definitionHash": "19541f579a34848b2cab664296eb30773c883416bc5ea62cc34c71c6d415ccaa",
|
|
"knownModelUsd": "0.124193300",
|
|
"attemptDurationMs": 52545,
|
|
"model": "claude-sonnet-5",
|
|
"environmentId": "local",
|
|
"runtimeMode": "native",
|
|
"route": "braintrust",
|
|
"priceVersion": "anthropic-standard-2026-10-20",
|
|
"measurementHost": "github-hosted-ubuntu-latest",
|
|
"campaignOutcome": "failed_report_dependency_setup",
|
|
"finalModelRequests": 5,
|
|
"measuredAt": "2026-10-27T20:24:56.120Z",
|
|
"resultSha256": "34d2c8b0b2aa39c563397eb402e981a7617ea6adbd9fdc1d0c59f852a356390c",
|
|
"ledgerSha256": "b94a48f61471050559d5cd217ed9d3e2270e7d342b69ad4caa236ae1bbfc916f",
|
|
"ledgerFinalHash": "eb4a8aff44281c8f2ff420f137496e58513beb9afabee1cce2a560ebb056e2ac"
|
|
},
|
|
{
|
|
"executionId": "core-compatibility.runner-codex.local.message-marker",
|
|
"attemptId": "1021b4ba8fdd0aa68df04c023fc997c2",
|
|
"sourceRevision": "bf3ef12f3a78d0b15a32ad57f7e16520fbf6cd5e",
|
|
"definitionHash": "19541f579a34848b2cab664296eb30773c883416bc5ea62cc34c71c6d415ccaa",
|
|
"knownModelUsd": "0.173863000",
|
|
"attemptDurationMs": 20216,
|
|
"model": "gpt-5.6-sol",
|
|
"environmentId": "local",
|
|
"runtimeMode": "native",
|
|
"route": "braintrust",
|
|
"priceVersion": "openai-standard-2026-10-20",
|
|
"measurementHost": "local-docker-linux-arm64",
|
|
"campaignOutcome": "passed",
|
|
"finalModelRequests": 4,
|
|
"measuredAt": "2026-10-27T20:59:06.423Z",
|
|
"resultSha256": "4f31f46c7ae78a47a8157e19657dcd4794bb96e651e5114ce57652eadece24e6",
|
|
"ledgerSha256": "5dc4529f3f3e4173615f0bbc31a65394c3fc94677f2debb32e06c0bb5bb48405",
|
|
"ledgerFinalHash": "481b0aa38bb45d5308ffdbc73b7bdbb2827ca427c1894430900a59395ce1e9a1"
|
|
},
|
|
{
|
|
"executionId": "feedback-persistence.runner-acpx-claude.local.complaint-and-suggestion",
|
|
"attemptId": "6d01c1d288747c84c0f2424e4e7a8042",
|
|
"sourceRevision": "bf3ef12f3a78d0b15a32ad57f7e16520fbf6cd5e",
|
|
"definitionHash": "df18788acadb987274bec4c1e4598147232bdee7db9b4848ceb7d7906ae31a4c",
|
|
"knownModelUsd": "0.304456800",
|
|
"attemptDurationMs": 54741,
|
|
"model": "claude-sonnet-5",
|
|
"environmentId": "local",
|
|
"runtimeMode": "native",
|
|
"route": "braintrust",
|
|
"priceVersion": "anthropic-standard-2026-10-20",
|
|
"measurementHost": "github-hosted-ubuntu-latest",
|
|
"campaignOutcome": "failed_report_dependency_setup",
|
|
"finalModelRequests": 5,
|
|
"measuredAt": "2026-10-27T20:23:36.857Z",
|
|
"resultSha256": "481ab702917e83f520c597a1e2b88ce69e6a34dcd21e1b3961d7aa728547f42c",
|
|
"ledgerSha256": "b94a48f61471050559d5cd217ed9d3e2270e7d342b69ad4caa236ae1bbfc916f",
|
|
"ledgerFinalHash": "eb4a8aff44281c8f2ff420f137496e58513beb9afabee1cce2a560ebb056e2ac"
|
|
},
|
|
{
|
|
"executionId": "feedback-persistence.runner-codex.local.complaint-and-suggestion",
|
|
"attemptId": "f88c1b5843c4772ac7082781053611b2",
|
|
"sourceRevision": "bf3ef12f3a78d0b15a32ad57f7e16520fbf6cd5e",
|
|
"definitionHash": "df18788acadb987274bec4c1e4598147232bdee7db9b4848ceb7d7906ae31a4c",
|
|
"knownModelUsd": "0.191163400",
|
|
"attemptDurationMs": 30591,
|
|
"model": "gpt-5.6-sol",
|
|
"environmentId": "local",
|
|
"runtimeMode": "native",
|
|
"route": "braintrust",
|
|
"priceVersion": "openai-standard-2026-10-20",
|
|
"measurementHost": "local-docker-linux-arm64",
|
|
"campaignOutcome": "passed",
|
|
"finalModelRequests": 5,
|
|
"measuredAt": "2026-10-27T20:58:30.021Z",
|
|
"resultSha256": "d0520306ac87e321196a8ba4746ab9e674177a453093e23dd250ede853df1a5c",
|
|
"ledgerSha256": "5dc4529f3f3e4173615f0bbc31a65394c3fc94677f2debb32e06c0bb5bb48405",
|
|
"ledgerFinalHash": "481b0aa38bb45d5308ffdbc73b7bdbb2827ca427c1894430900a59395ce1e9a1"
|
|
}
|
|
],
|
|
"limitations": [
|
|
"One successful sample per cell; no measured slow-tail distribution or predicted failure rate",
|
|
"Gateway estimates include setup, background and disconnected-client requests with terminal usage",
|
|
"Attempt time includes setup, execution and teardown; it excludes workflow queue, installation and compilation",
|
|
"Claude samples ran on GitHub hosted Ubuntu; Codex samples ran in a local Linux ARM64 container",
|
|
"The hosted campaign failed only during report dependency setup; original workflow and grader outcomes remain retained",
|
|
"Provider usage exports match; dollars remain estimates pending final monthly invoices and non-model reconciliation",
|
|
"No change to conservative per-request reservations, approved budgets or recurrence gates"
|
|
]
|
|
}
|