Files
PaperClipAI/tests
DottaandPaperclip 64a42d2cf0 fix(evals): honor the Daytona setup attempt deadline (#17947)
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/110/changes/paperclip/110
<!-- entire-trail-link-end -->

## Thinking Path

> - Budgeted Product E2E cases must attribute asynchronous hello probes
to setup.
> - Daytona can take longer than local startup to provision an isolated
environment.
> - The setup helper imposed a 45-second local limit on both
environments.
> - A qualified remote marker stopped while the agent was still
verifying.
> - Remote readiness should use its existing finite attempt deadline.

## Linked Issues or Issue Description

Refs #17936.

**What happened?** The bounded Daytona marker campaign stopped at the
setup helper after 45 seconds while the agent was still verifying. The
case permits a 15-minute attempt and the environment allows up to five
minutes to provision. It recorded zero model requests; retained cleanup
passed and a provider-side company-label query confirmed no sandbox
remained.

**Expected behavior** Daytona readiness can finish within the existing
attempt budget. Local setup keeps the current short limit; observed
errors, holds and identity changes still stop either environment
immediately.

**Steps to reproduce** Run a budgeted native marker against a cold
Daytona environment whose readiness takes longer than 45 seconds.

## What Changed

- Pass the execution environment to the budgeted setup helper.
- Use the existing finite attempt deadline for Daytona; retain the
45-second local limit.
- Set a five-minute saved-account hello-probe allowance only for
explicit budgeted Daytona fixtures, matching the environment
provisioning allowance. A later qualification exposed the inner probe’s
separate 90-second timeout.
- Keep the gateway in the setup phase until the same agent is ready.
Model limits, attempt limits, cleanup and automatic retries are
unchanged.
- Update the operations guide with the two timing windows.

## Verification

- Focused setup and operations regression suites: 48 passed.
- Eval TypeScript check passed.
- A fixture regression verifies the five-minute probe allowance is
remote-only and preserves gateway credential routing. The timing
regression advances the observed clock past 45 seconds: Daytona reaches
ready while local setup still stops. Existing tests retain
identity/error/hold and attempt-deadline guards.
- Fresh real Daytona qualification follows this correction with one
attempt and the existing $20 stage cap.

## Risks

A failed remote setup can now consume more of its existing attempt
window. It cannot extend that deadline, increase model allowance, or
automatically retry. This change does not claim the prior failed
campaign passed.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository inspection and
code execution. The exact serving model ID and context window are not
exposed to this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 17:52:46 -05:00
..