Commit Graph
4 Commits
Author SHA1 Message Date
DottaandPaperclip 5842185e4f fix: surface native cleanup quarantine and add chat qualification evals (#13775)
## Thinking Path

> - Paperclip lets people manage AI agents and their work.
> - Agent Chat needs reliable native execution before native runners
become the onboarding default.
> - Existing stories covered idle reassignment and controller restart,
but not an executing worker handoff or worker process loss.
> - Status answer tests also need to reject stale claims and invented
facts.
> - This pull request adds six opt-in full-stack cells with independent
state assertions and retained evidence.
> - The probes exposed a misleading Retry across server projection and
recovery-banner paths; the fix reports the blocked recovery honestly.
> - The tests preserve failures without changing recovery policy,
production prompts, or onboarding defaults.

## Linked Issues or Issue Description

Refs: #13762. Related: #13765 (Retry targets the latest failed attempt),
#13753 (task context ownership), #13746 (native recovery work).

## What Changed

- Add active reassignment with saved draft and plan preservation,
old-worker cancellation, and successor completion checks.
- Preserve recovery-needed projection when native cleanup fails before
its coordinator exists, refuse a generic retry that would immediately
fail again, and replace the recovery banner's misleading Retry with
Inspect run.
- Add verified local worker process loss with a required successful
continuation; retain a failing qualification result when recovery is
unavailable, while independently verifying the UI/API refuse doomed
retries.
- Add two-turn factual answer checks for current blockers, stale claims,
inactive backlog work, and unknown facts. Retain prose for separate
semantic review.
- Add positive and negative oracle calibration and document fault
isolation, cleanup, billing, and qualification limits.

## Verification

- Eval TypeScript check passes.
- All 442 eval support tests pass locally. The 89 focused server tests
and server typecheck pass. Six recovery-banner UI tests and token gates
pass.
- Initial new-cell campaign:
https://github.com/paperclipai/paperclip/actions/runs/35657128077. All
six results are retained; four failed on fixture-contract issues and two
exposed real worker cleanup quarantine.
- All 26 existing native onboarding cells:
https://github.com/paperclipai/paperclip/actions/runs/35656761484 (26/26
passed on master 846336e5a, all cleanup passed).
- Intermediate handoff/fault campaign:
https://github.com/paperclipai/paperclip/actions/runs/35657945095 (four
retained failures: two overly strict draft oracles, two real crash
quarantines).
- Final active handoff:
https://github.com/paperclipai/paperclip/actions/runs/35659014397 (2/2
passed on cf6d4ae3a; both cleanup passed).
- Clarified answer-quality fixtures:
https://github.com/paperclipai/paperclip/actions/runs/35658262695 (2/2
passed on 4a26f10be; both cleanup passed; all four answers semantically
reviewed).
- Quarantine guard regression campaign:
https://github.com/paperclipai/paperclip/actions/runs/35659580100 (both
API requests correctly refused with 409/no second run, but exposed a
separate misleading Retry in the recovery banner and a fixture wait on a
non-admitted run).
- Final quarantine guard verification:
https://github.com/paperclipai/paperclip/actions/runs/35661067305
(147e42f7e: both providers verify Inspect run/no Retry, HTTP 409, one
retained run, unchanged saved plan, and successful disposable cleanup.
Both evals intentionally remain red with
`worker_crash_recovery_unqualified`; no successful continuation exists).
The preceding campaign 35658772755 never ran provider cases because
GitHub artifact finalization returned HTTP 403.
- Full repository CI passes on 147e42f7e: typecheck, tests, build, and
browser gates. One unchanged local-service-supervisor readiness test
failed initially; its six-test file passed in isolation and the failed
shard passed on its single rerun. Latest-head rollup: 54 successful, 2
intentionally skipped, no failed or pending checks. Greptile is 5/5 with
zero unresolved findings.
- See tests/runner-e2e/QUALIFICATION-2026-09-21.md for retained attempts
and semantic review. Published reports:
[onboarding](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-35656761484-1/),
[handoff](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-35659014397-1/),
[grounded
answers](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-35658262695-1/),
[crash guards and unqualified
recovery](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-35661067305-1/).

## Risks

- Paid cells are explicit-only and local-only. The fault fixture signals
only the exact native run PID after checking its identity.
- Live worker-loss probes currently fail on cleanup quarantine for both
providers. The eval must remain red until there is a usable recovery,
even when preservation and refusal checks pass. Verified cleanup with a
fresh attempt versus exact-session resume remains a product decision.
- Structured facts alone do not qualify prose quality; semantic review
remains separate.
- Onboarding uses the existing runtime switch after the real wizard and
before provider execution. Native UI selection and public defaults
remain unchanged.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, tool use, and code
execution. The exact served snapshot and context-window size are not
exposed in this session.

## Checklist


- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-21 18:13:01 -05:00
DottaandPaperclip 0e9b24c821 fix: resume subscription comment wakes and preserve retry status (#13433)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Concurrent task runs can share a managed AI subscription with one
credential lease.
> - #13438 added durable retries for busy subscriptions and task-lock
checks.
> - A comment wake can run for an agent who is not the task assignee.
Such a run never owns the task lock, so the new checks suppress its
retry.
> - This pull request preserves those comment wakes while keeping the
lock checks for assignee runs.
> - It also shows the subscription wait in task status and preserves the
failure count through the database projection.

## Linked Issues or Issue Description

Refs #13438.

**What happened?**

When a subscription is busy, a non-assignee comment wake is cancelled
without a successor. Repeated subscription waits also display an
inflated attempt count because the execution query omits the preserved
failure count.

**Expected behavior**

An eligible comment wake waits and resumes without claiming the
assignee's task lock. The task shows “Waiting for AI subscription”.
Waiting does not consume provider-failure retries. Assignee retries
still stop when the task lock is cleared or transferred.

**Steps to reproduce**

1. Configure an agent with a managed subscription and concurrent runs.
2. Hold its credential lease in one run.
3. Mention the agent on a task assigned to another actor.
4. Release the lease and inspect whether the comment wake has a
scheduled retry.

The test uses a real embedded Postgres database and a real credential
lease. Provider execution uses a fixture. The reassignment test applies
a database mutation after the real checkout. No live provider account is
needed.

**Paperclip version or commit**

Based on `f912ecaac` from #13438. Before the reconciliation fix, the
rebased regression at `2063cfd13` failed because the comment wake had no
scheduled retry. The original configuration failure was reproduced on
`5282cabde` before #13438 merged.

**Deployment mode**

Built from source with embedded Postgres for local verification.

## What Changed

- Record non-assignee comment-wake authority at admission while holding
the task and run locks. A later reassignment cannot grant this
exception. Preserve it through repeated subscription waits.
- Keep master’s lock checks for assignee retries, including the check
inside the scheduling transaction.
- Show the subscription wait and preserve its failure count in the
execution projection query.
- Limit pre-provider wait receipts to fresh executions. A persisted
native execution input must retain its recovery path.
- Add lease, retry, projection, cancellation, reassignment, pause,
revocation, service-recreation, and ownership regression coverage.
- Use master’s 60–120 second retry interval and document the resulting
behavior.

## Verification

- Red: after rebase, the comment-wake test failed with no scheduled
retry; the other 50 contention and retry-scheduling tests passed.
- Red/green: reassigning the task immediately after real checkout
reproduced an unwanted successor for both assignment and comment wakes.
Both tests pass after recording authority at admission.
- Green: all 285 targeted tests pass across 12 suites, including
#13438’s four cancellation-race phases, its hiring/connection suite,
run-dispatch integration tests, and both reassignment regressions.
- `pnpm -r typecheck` passes after rebase.
- `pnpm build` passes on the final revision.
- The full general and serialized test suites pass in [CI run
34899514681](https://github.com/paperclipai/paperclip/actions/runs/34899514681)
on `99f1e4c1d`. Full-suite verification ran in CI; local verification
used the 285 targeted tests.
- All 32 active PR checks pass, including build, typecheck, native
runner verification, canary dry run, and all browser shards. Two
Storybook checks are skipped by path filters.
- Greptile reviewed `99f1e4c1d` at 5/5 with no outstanding findings. All
review threads are resolved.
- `git diff --check` and `node scripts/check-module-boundaries.mjs`
pass.

## Risks

- The non-assignee exception requires authority recorded under admission
locks or its server-created subscription retry. Tests cover
caller-supplied flags, assignment and comment reassignment races,
repeated comment waits, and assignee lock protections.
- A wait uses master’s 60–120 second interval. A long-held lease can
produce multiple wait records.
- Existing native sessions can have provider effects. They must not
receive a fresh-execution receipt that permits replay.
- No schema migration or credential permission change is required.

## Model Used

OpenAI Codex, model `gpt-6-astra`, with repository search, code editing,
shell execution, and test tools. The session does not expose its
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-14 16:58:37 -05:00
DottaandPaperclip 6809314a3f fix: recover sandbox workspace setup and retries (#13353)
## Thinking Path

> - Paperclip manages AI agents and their tasks.
> - Sandbox tasks need a workspace and a provider session before work
can start.
> - A selected Git subfolder is a valid workspace, but it is not a
repository fetch source.
> - A resumed sandbox must keep one resource identity when the provider
fills in its region.
> - Workspace reuse does not prove that a provider session has started.
> - A recorded continuation must not leave an old failure blocking Retry
for a newer attempt.
> - This pull request fixes those setup and recovery boundaries while
preserving ownership checks.

## Linked Issues or Issue Description

**What happened?**

A Daytona task failed before provider startup when its project folder
was inside a parent Git repository. After the folder was repaired, Retry
resumed the same sandbox and verified its workspace sentinel, but
workspace preparation reported that the lease was no longer active. A
later attempt could fail because the resumed workspace had no provider
checkpoint. The old run also kept a recovery projection after the server
recorded its explicit successor, hiding Retry for the new failure.

**Expected behavior**

Sync the selected folder without importing parent files or history. Keep
the existing sandbox identity stable. Allow a new provider session only
with proof that its exact session has never started. Let the current
failed attempt retain Retry once the old recovery has a recorded
successor.

**Steps to reproduce**

1. Select a subfolder of a Git repository as a project workspace and
start a Daytona task.
2. Leave the target region unset. Release its reusable sandbox after
setup fails, then resume and realize the workspace.
3. Retry a provider setup that failed before any session directory or
checkpoint was created.
4. Record an explicit successor for a native failure, fail that
successor during setup, and inspect Retry in the task thread.

**Paperclip version or commit**

Reproduced against `8d1f0c20a` with new failing regressions before each
production fix.

**Deployment mode**

Local controller with a Daytona environment. Automated tests use
deterministic provider fixtures and real filesystem operations.

Related: #13338, #13163, #13264, #13349. The open work-folder stack in
#13264 includes a broader fresh-session authority change. This patch
addresses the independently reproduced startup failure with existing
durable bootstrap proof and atomic directory creation. It does not
include the work-folder migration or credential changes from that stack.

## What Changed

- Classify only the selected Git repository root as a fetch source. Sync
subfolders as directories and preserve their enclosing Git ignore rules
on upload and restore. Recognize the shared scheduler’s completed
non-repository result so ordinary folders still sync; timeout,
cancellation, and output-limit failures remain closed.
- Remove the placement target from Daytona account cache identity. Keep
API endpoint, credential digest, company, environment, driver, and
sandbox ID boundaries.
- Preserve closed-lease admission until a sentinel-verified resume
reopens that same resource.
- Show the already-recorded explicit successor of a resolved recovery.
Keep the old failure evidence and unresolved holds. No historical status
writes occur.
- Permit a resumed workspace to create a new session directory only with
matching durable identity, zero connections and events, untouched
bootstrap commands, no backup, and an absent remote session. Claim the
directory atomically. Existing, partial, or ambiguous state still blocks
startup.

## Verification

- Final head `c43c7403f`: [CI completed
successfully](https://github.com/paperclipai/paperclip/actions/runs/34733846820/attempts/2),
with 32 successful checks and 2 conditional skips. This includes every
server, UI, package, serialized-route, browser, native-runner,
typecheck, and build gate. Greptile reviewed the same head at [5/5 with
no unresolved
findings](https://github.com/paperclipai/paperclip/pull/13353#issuecomment-5650270168).
- New regressions failed before each of the four production fixes.
Git/archive/restore suites: 148 passed. The scheduler-wrapped non-Git
regression also failed before its fix; 135 affected Git/sync/Codex tests
then passed.
- Daytona plugin: 230 passed, 6 opt-in live tests skipped. Native
executor, projection, and TaskChatThread: 494 passed, including
existing/partial state, wrong identity, prior connections or turns,
backups, unavailable proof, and unresolved recovery controls.
- Local full-repository typecheck, build, and token gates passed on the
final head. Local CLI: 485 passed. Complete single-worker package rerun:
3,224 passed, 19 skipped.
- Local verification is an aggregate with recorded retries, not one
pristine green invocation: the general-server run began on `e721a920a`
and finished with 12,002 passed, 3 failed, 70 skipped. Its real Codex
scheduler failure is fixed above; the socket and workspace-runtime
timeout failures passed unchanged in focused reruns. Both Inbox failures
passed unchanged in the full 27-test Inbox file; database/shared-package
failures passed in the single-worker package rerun. The supplemental
local serialized-route rerun remains in progress; all five corresponding
final-head CI lanes passed.
- The first final-head CI attempt hit a Daytona fixture-readiness race
and a signoff-browser heartbeat receipt timeout. One supported unchanged
failed-job rerun passed both and the aggregate gates. Live combined
user-journey verification is tracked in the related follow-up; this PR's
provider tests use deterministic fixtures and real filesystem checks.

## Risks

- A selected subfolder uses directory sync and does not carry parent Git
history. Its ignored files stay local.
- The target region remains a creation setting and part of workspace
reuse policy; it does not split the account identity of an existing
sandbox.
- Incomplete or conflicting provider state still fails closed. This
change does not erase a session, infer completed work, bypass a user
decision, or replay uncertain actions.
- A later remote setup failure can leave a claimed partial session
directory. It remains blocked rather than being overwritten.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository tools, and code
execution. The exact hosted model ID and context-window size are not
exposed in this task.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-12 22:19:42 -05:00
DottaandPaperclip 35fdc0c66b fix: make task recovery durable and preserve current requests (#13075)
Make task recovery durable and preserve the latest user request across native and legacy continuations. Keep routine recovery quiet and prevent replay when action outcomes are uncertain.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-09 09:14:25 -05:00