Commit Graph
2 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 0c04ff76eb fix(native-runtime): a follow-up that arrives as a warm session closes continues the task (#21279)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native agent session on a remote sandbox stays warm after a run,
and an idle timeout closes it after about 5 minutes
> - When a person's follow-up comment arrived just as that idle close
began, the new run failed with `native_session_supervisor_busy`
> - The idle close of the finished owner could also fail its final
checkpoint with `native_runner_checkpoint_owner_lost`, because the task
already pointed at the queued follow-up run
> - Then the retry was held for workspace repair and the task stayed
blocked; the environment could not be deleted (409) because its warm
runner could not be stopped
> - This pull request makes the follow-up wait for the idle close that
is in flight, and lets the finished owner save its final checkpoint
while its successor still waits in the queue
> - The benefit is that a follow-up that lands at the warm session's
idle close continues the task with no person involved

## Linked Issues or Issue Description

Refs #18011 (the checkpoint ownership fence this change extends).

**What happened?**

A follow-up comment arrived 295–340 s after the previous run ended,
while the warm session's idle close was in progress. The new run failed
with `native_session_supervisor_busy`. In another case, the idle close
failed its final checkpoint with `native_runner_checkpoint_owner_lost`,
the next run failed
`native_instruction_materialization_handoff_unavailable`, and the retry
was cancelled with `execution_reconciliation_required`. The task stayed
blocked. A hosted instance showed the same class as "previous run's
session on this sandbox could not be retired".

**Expected behavior**

The follow-up run waits for the idle close (inside the existing bounded
release wait) and then starts. The finished owner saves its final
checkpoint and closes. The task completes hands-off.

**Steps to reproduce**

1. Run an opencode agent on a Daytona sandbox with a warm session (idle
timeout 300 s).
2. Let a run finish.
3. Post a comment on the task 298–306 s after the run ended.

**Paperclip version or commit**

master `444dc18d98`.

**Deployment mode**

Local server with Daytona remote sandboxes (restart-chaos harness, light
lane). The same class was seen on a hosted staging instance.

## What Changed

- `claimWarmNativeInstructionCopy` now also waits when an idle close is
in flight for the session scope (`closingWarmNativeSessions`). The wait
uses the same deadline as the existing release wait
(`warmSessionReleaseWait`), so it adds no new wait and nothing before
`turn.start` on the normal path.
- `validateNativeRunnerCheckpointOwner` accepts the finished owner's
final checkpoint (accepted result, no new work) when
`issues.executionRunId` points at a successor run that is `queued` or
`scheduled_retry` for the same company and task and has no
`native_run_finalizations` row.
- Tests: a server test for a follow-up that arrives while the idle
timeout closes the warm session; four new scenarios (`queued_successor`,
`scheduled_retry_successor` × `runner.suspend`/`turn.stop`) and four new
negatives (`queued_wrong_issue`, `queued_new_work` × both commands) in
the early-retirement checkpoint matrix.
- Restart-chaos harness:
`RESTART_CHAOS_MIX_FOLLOW_UP_AFTER_END_MS=<min>-<max>` and
`RESTART_CHAOS_MIX_FOLLOW_UP_COUNT` post the staging-mix follow-up a set
delay after the task's last run ended, so a cell can aim the follow-up
at the warm idle close. The restart-chaos README describes both
variables. A failed post is retried with the same `clientRequestId`, the
grade check `after_end_follow_ups_served` needs an agent reply after
each post and the ledger marker, and the task budget includes the
planned waits.
- Harness: the wave database cleanup
(`tests/runner-e2e/controller-database.ts`) no longer waits forever when
a group signal (for example `timeout` or Ctrl-C) already stopped the
embedded Postgres. Before, `embedded-postgres` `stop()` waited for an
exit event that had already fired, and an interrupted wave hung after
the Daytona R6 sweep.

## Verification

- Fix 1, red without the product hunk: `native-session-executor.test.ts
-t "idle timeout is closing"` fails with `promise rejected "Error:
native_session_supervisor_busy"`. Green with the fix: 1 passed.
- Fix 2, red without the product hunk:
`native-runner-restart-recovery.integration.test.ts` fails 4 of 208
(`queued_successor` and `scheduled_retry_successor` ×
`runner.suspend`/`turn.stop`) with
`native_runner_checkpoint_owner_lost`. Green with the fix: 208 passed,
14 skipped. The two new negative cases pass in both states.
- Harness hang fix, red without the fix: the new
`controller-database.test.ts` case ("finishes cleanup when the database
already stopped because a group signal reached it") times out on cleanup
(`expected 'hung' to be 'cleaned'`). Green with the fix: 5 passed, 1
skipped.
- After the rebase onto master `7154030ef6`, both server test files
together: 1639 passed, 21 skipped.
- Restart-chaos harness tests (`staging-mix`, `staging-mix-retry`,
`observer`): 75 passed, including a test for the after-end retry, the
`after_end_follow_ups_served` grade check and the added budget. `tsc
--noEmit` passes for `server`, `tests/runner-e2e` and
`tests/restart-chaos`.
- `pnpm verify` exits 2 because `tests/restart-chaos/observer.ts` is a
global CI input. Required CI supplies the broad coverage.
- Daytona cell on this branch (4 opencode tasks, warm sandboxes, 1 vCPU,
follow-up comment posted 300.8–308.2 s after each task's last run ended,
image built for master `444dc18d98`): 4/4 tasks done hands-off, 8/8 runs
succeeded, 0 Retry banners, 0 `owner_lost` / `supervisor_busy` / "could
not be retired" / `execution_reconciliation_required` lines in the
server log, environment delete clean, warm-shutdown grade pass, 0
sandboxes left. The harness exited by itself after 1076 s.
- Same cell before the fix (master, 8 tasks): 1/8 tasks blocked
(`native_session_supervisor_busy` →
`native_runner_checkpoint_owner_lost` →
`execution_reconciliation_required`). With fix 1 only: 1/8 still blocked
(`owner_lost` on the idle close). With both fixes: 8/8 done and 0 failed
runs (cut by the 20-minute cap before cleanup), then 4/4 above.
- The cell grade `T:completion_once` fails on all 4 tasks by design: the
timed follow-up comment arrives after the task is done, so the agent
posts a second completion comment.

## Risks

- Fix 1 can delay a follow-up by up to the existing warm release wait
(30 s) while an idle close finishes. Before this change, that run
failed. If the close fails or the wait expires, the claim returns null
and the run uses a fresh instruction copy, as before.
- Fix 2 widens the checkpoint ownership fence from #18011. It does not
reopen what #18011 closed:
- The exemption applies only to the finished owner's final save: the
checkpoint must hold an accepted result and no new work, so no provider
turn can be granted through it.
- The successor must still be unclaimed (`queued` or `scheduled_retry`)
and must have no native coordinator (`native_run_finalizations`) row. No
controller has started it, so nothing can have consumed or depended on
the predecessor's checkpoint. Once the successor is claimed, the
existing preparing/claimed rules from #18011 apply unchanged.
  - The successor must belong to the same company and the same task.
- Cases that must still reject, with tests in the same matrix:
- `queued_wrong_issue`: a queued successor for a different task is
rejected with `native_runner_checkpoint_owner_lost` and the stored
checkpoint does not change.
- `queued_new_work`: a `turn.start` (new work) while the successor is
queued is rejected with `native_runner_checkpoint_owner_lost` and the
stored checkpoint does not change.
- All earlier rejections stay: unclaimed native coordinator,
dispatching, expired or foreign preparation, foreign native preparation,
wrong task, new work, unaccepted result, replaced controller.
- Not changed: an owner whose close fails still blocks environment
delete (409). That is a separate product decision.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use (shell, file edit, test runs). Earlier commits on
this branch were written by the same model in an earlier session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:42:50 -06:00
DottaandPaperclip 6987c2f20b test(upgrades): qualify controller replacement with immutable artifacts and phase evidence (#17455)
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/105/changes/paperclip/105
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip needs evidence that an admitted execution survives a
main-server replacement.
> - Runner mechanics and server adoption belong to #17097 and #17439.
> - A healthy HTTP endpoint does not prove recovery of the original
execution.
> - Qualification must identify the serving artifact, natural ownership,
authenticated attachment and resumed processing separately.
> - It must also retain failed runs and measure normal-operation costs.
> - This PR extends the existing qualification machinery and acceptance
record.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Operators can assess upgrade continuity from phase evidence tied to
exact artifacts and profiles.

**Current behavior**

A same-artifact restart or synthetic executor pass cannot establish
different-artifact upgrade and rollback support. Idle SQL gates also do
not measure active-run capacity.

**Proposed behavior**

Use immutable A/B artifacts and strict phase receipts through the
existing supervisor. Preserve failed evidence and add measured
before/after workload costs.

Builds on merged #17097 and #17439. The base is now `master`. #17456
follows for review convenience. This PR lands qualification machinery
and the evidence record. The remaining hosted gates stay open for
deployment qualification; merging this machinery does not claim those
experiments passed.

## What Changed

- Extend the main-server supervisor and installed-artifact verification
for actual serving revisions, forward migrations and compatible
rollback.
- Distinguish HTTP readiness, natural ownership, authenticated
attachment, resumed processing and backlog settlement.
- Retain the original reconnect grace and execution identity checks.
Never substitute a new task or edited lease for continuity.
- Reuse real-harness and Daytona qualification paths, with exact profile
and artifact provenance.
- Bind current Copilot attestation inputs in the Daytona image
fingerprint.
- Keep one acceptance record, including both failed file-only stress
phases and the final passing bounded-pool comparison.

## Verification

After retargeting to master, all 158 focused qualification cases across
13 files pass, with E2E typecheck. The first 12-file batch has four
report-lock test failures and 133 passes. Master #17811 had committed
the exact already-approved output lock, while that fixture assumed HEAD
must be the uncorrected input. The fixture now checks both pinned forms,
executes the committed-lock path, and reconstructs the pinned input to
retain all rejection tests. The production helper, policy hashes, patch
and dependency resolution are unchanged. The corrected batch explicitly
includes both process-preservation suites; the prior 146-case receipt
has its own source and selection.

Current candidate `8f1d6ec02f` integrates master `09ad9adce2`, including
the merged server slice. Its diff contains only qualification code and
documentation. The [qualification
record](https://github.com/paperclipai/paperclip/blob/8f1d6ec02f5bf6c90e7b27da8156953cabd9a180/doc/plans/2026-10-22-hot-upgrade-qualification.md)
is the acceptance ledger and retains source-attributed failures and
hosted limits.

On the eighth integration, all 146 cases across 12 focused qualification
files pass, including report-lock preparation, workflow security and
workflow targeting. E2E typecheck passes. The earlier 118-case result
remains attributed to its prior source. The first sandbox invocation has
117 passes and one failure because process inspection was denied during
inert fixture cleanup. The permitted repeat passes all 118 and a
targeted process check finds no remaining fixture daemon. The earlier
96-case receipt remains source-attributed in the ledger. The latest two
review regressions first fail: an intermediate replacement can cite
another runner/session despite a valid final step, and malformed receipt
JSON aborts evidence packaging. Each replacement now binds
run/company/issue/runner/session, and corrupt receipts retain explicit
invalid metadata plus sanitized diagnostics while other logs and
receipts still package. These checks prove qualification machinery with
injected supervisor operations, configuration subprocesses and real
filesystem packaging. They do not qualify a hosted harness.

Three request-reader cases, 14 artifact/configuration/cache/history
cases, and three lost-partial-receipt cases fail before their respective
corrections. The final collector preserves sanitized startup, readiness
and recovery phase receipts through packaging and temporary-root
deletion. Earlier embedded-Postgres initialization, report-catalog
drift, CI scheduling/cleanup failures and fixture collisions remain
recorded with their corrections or limits. The previous Greptile service
error is retained as historical review evidence, not a current review
pass.

The record includes prior source `9bb9d3417e`'s actual Linux
contention/fairness successes and the ordinary-warm regression proof
from the server slice. The historical 300-session component campaign
passes four after phases and retains earlier burst failures. Those
component costs do not establish hosted capacity. The hosted API-only
pair has zero agents/viewers, fails both absolute latency gates and the
candidate memory gate, and does not complete the active-agent/viewer
matrix.

The strict ten-agent same-artifact restart remains attributed to its
original artifact. It does not qualify this updated stack. Actual
A→B/B→A, the normal drain-free Cloud rollout, and other hosted harnesses
remain open. No new provider turn, deployment or hosted qualification
was run during merge preparation.

Prior source `57d3767eac` jobs 122590643944, 122590643983 and
122590644161 ended after runner shutdown/cancellation without final
Vitest results. The shutdown cause is unproven. No retry was issued
before this source was superseded by the required server recovery fix.
These attempts remain failures/incomplete evidence. Fresh exact-head CI,
legacy status, completed Greptile 5/5, resolved feedback and normal
merge eligibility remain required.

## Risks

Different-artifact A→B/B→A, the normal drain-free Cloud rollout path,
other hosted harnesses and the complete concurrency/viewer matrix remain
open. Frozen branch builds are development-build evidence. This PR owns
qualification and evidence; production defects must be fixed in their
earlier owning slice.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository tools and code
execution. The runtime does not expose a more specific model ID or
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 21:58:09 -05:00