Commit Graph
6 Commits
Author SHA1 Message Date
DottaandPaperclip f5e5b05a33 Preserve native workspace recovery across Stop and sandbox reuse (#18332)
Land the thirteen-file native recovery fix independently from #18245.

Retain verified Stop evidence and queued user comments. Retire stale
preparation ownership atomically and require paired sandbox lease
handoff receipts before workspace recovery.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-30 10:57:53 -05:00
DottaandPaperclip ed8fae8dbb fix(workspaces): preserve completed shallow sandbox exports (#18221)
## Thinking Path

> - Paperclip coordinates agent work and preserves its results.
> - Remote execution copies completed workspace changes back to the
host.
> - An agent can fetch a published branch with shallow Git history.
> - A full Git bundle omits the shallow boundary metadata and can fail
host connectivity checks.
> - This pull request carries the boundaries through the full-bundle
fallback and imports through normal Git fetch.
> - Recovery must also recognize completed native work on a reusable
lease without treating it as ordinary executor reuse.
> - Completed work can be saved without another provider turn, while
existing host history remains connected.

## Linked Issues or Issue Description

**What happened?**
Sandbox finalization can fail with a missing parent commit after an
agent checks out a separately fetched shallow branch. Re-exporting a
full bundle repeats the failure because Git bundles do not encode
shallow boundaries.

**Expected behavior**
Import the completed branch and saved files. Preserve existing host
ancestry and concurrent host edits. Keep normal Git connectivity and
workspace merge checks.

**Steps to reproduce**
1. Prepare a sandbox from one host commit.
2. Fetch a different published branch with depth one and check it out.
3. Save a result file and restore the sandbox workspace.

**Paperclip version or commit**
Reproduced against master before this repair. The regression fails with
the missing-parent error before the production change.

Related public context: #15268 covers rewritten history integration.
This repair handles shallow bundle transport before integration.
Searches found no equivalent open repair.

## What Changed

- Export shallow boundaries beside the full bundle when the delta import
lacks history.
- Unpack into a temporary shared bare repository. Keep parent
connections already available on the host and import with Git's shallow
fetch protocol.
- Reject malformed boundary values and retain existing connectivity,
branch, and concurrent-work merge guards. Keep boundary probes within 15
seconds without consuming the separate 60-second Git import budget.
- Let source-bound preservation recover reusable leases only for the
finalizer-selected, accepted successful native result and matching
completion contract. Retain provider-stop, physical ownership and
host-writer checks; ordinary reusable sources remain blocked.
- Classify invalid Git metadata separately from omittable unsafe
archives. Keep native finalization failed and preserve service-owned
file recovery; carry the sender's dedicated rejection across the command
boundary without transient retries.
- Share the failure allowlist across bounded run/activity projections
and cancelled-preservation selection. Migration `0416` rebuilds the
matching partial index; task chat retains the non-retryable recovery
notice.
- Add real Git, filesystem sandbox and database recovery regressions.
Update execution semantics and affected verification maps.

## Verification

- Red: real filesystem sandbox restore failed with the same
missing-parent/revision-walk error before the export repair.
- Green: 240 focused Git, workspace restore, sandbox transport and
native-finalization tests pass. Coverage includes both transfer paths,
malformed/oversized metadata, a genuinely shallow host with its original
boundary intact, and concurrent host work with both merge parents
verified.
- Red/green: a 14-second simulated probe left the final real Git fetch
only 45,893 ms before the budget split. It now retains over 55 seconds
after the probes; a 16-second probe exhausts its own budget before host
import.
- Red/green: an oversized read progress report was retried three times
before its distinct metadata classification. It now aborts the first
read before payload consumption, leaves the host tip unchanged and does
not retry.
- Red/green: the real metadata error was swallowed by best-effort export
and native finalization incorrectly returned success. Both regressions
now pass; all 240 focused Git/restore/native-finalization tests pass. A
PostgreSQL/filesystem fault-injection test also verifies retained files,
failed run, service-owned recovery barrier, no preservation receipt and
no new wake.
- Red/green: duplicate metadata produced 47 Git process starts before
deduplication; the bounded-work regression now passes. Oversized byte
and boundary inputs fail before Git probes.
- Recovery red: both reusable lease policies exhausted recovery despite
a selected accepted successful result. The assembled
PostgreSQL/filesystem path now passes for retained and both reusable
policies, including idempotent re-admission after exhausted automatic
recovery, saved files, no agent wake, and once-only existing-result
finalization.
- Source verification controls reject ordinary reusable leases,
rejected/failed/unselected results, another physical allocation owner,
and an active host writer without changing recorded state.
- The 78-test PostgreSQL/filesystem recovery/source suite passed before
the metadata classification correction. Its new persisted
metadata-failure regression also passes. Full workspace typecheck and
build pass at final head `69f10daab2`.
- `node scripts/check-maps.mjs` and `git diff --check` pass.
- Full local `pnpm test:run` attempts did not finish. They are not
counted as passes. Early focused runs exceeded 5-second and 30-second
fixture deadlines; the final serial suite passed all 176 tests with a
30-second per-test budget in 56.8 seconds.
- Red/green: both bounded PostgreSQL projections dropped the new code in
oversized results. All 19 source/projection cases now pass, plus an
actual index-predicate test that excludes a metadata repair hold and
still selects ordinary cancelled work.
- Red/green: an oversized sender `.git/shallow` lost its error type
before host validation. The real transport test now verifies one typed
rejection, no shallow-payload read, and an unchanged host tip. All 465
command/sandbox/task-chat tests pass, and token/migration gates pass.
- Full typecheck/build pass after these final contract corrections at
`69f10daab2`; all 20 PostgreSQL source/projection/index contracts pass.
All current-head CI checks passed; Greptile is 5/5 with no unresolved
review threads. Merged #18230 supplies the CI-cache and Linux
shell-fixture baseline repairs; their duplicate changes have been
removed from this PR. The final Runner lane passed after one justified
rerun of an unchanged authentication fixture timeout (all 11 fixture
tests also pass locally). Previous CI failures and incomplete local runs
remain recorded; live deployment and retained-run recovery remain
separate verification.

## Risks

- The fallback creates a temporary shared bare clone. Metadata is
limited to 64 KiB and 256 unique boundaries, duplicate/parent probes are
cached, and Git import has a 60-second command budget plus a separate
15-second boundary-probe budget. It runs only after a missing-history
import failure.
- Incorrect boundaries could hide real host ancestry. The importer
retains available parent connections, and regressions cover that case.
- The full-bundle fallback can transfer a bundle twice. This preserves
the existing fast delta path; removing that extra transfer is outside
this correctness repair.
- Preservation eligibility expands only for completed native exports on
reusable leases. It creates no provider turn and leaves ordinary
reusable leases ineligible. Database/filesystem regressions cover result
selection and competing physical/host ownership.
- The migration rebuilds one partial index; no tables, columns or user
records change. There are no agent execution admission, model, trust,
recovery permission, workflow or lockfile changes.

## Model Used

OpenAI Codex, GPT-6, with reasoning, code execution, and browser tools.
The exact serving variant and context-window size are not exposed in
this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-29 20:02:53 -05:00
DottaandPaperclip bd382eee53 fix(recovery): explain retained source verification failures (#17773)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Workspace recovery must prove the identity of retained work before
it uses an allocation.
> - The verifier rejects several different contracts with the same
`source_binding_invalid` code.
> - A saved Retry can therefore stop before file preservation without
enough evidence for a safe repair.
> - Operators need a bounded observation of the selected source without
another attempt or weaker guards.
> - This pull request records closed failure reasons and adds an
explicit read-only inspection on the run page.
> - The benefit is a precise next diagnostic step while the saved work,
intent, and ownership fences remain intact.

## Linked Issues or Issue Description

Refs: #17741, #17053. This is a diagnostic follow-up. It does not change
recovery admission or the retained-source acceptance policy.

**What happened?**

A retained-source recovery exhausted in `verify_execution` with only
`source_binding_invalid`. Run and task diagnostics could not distinguish
an invalid reference, lease policy, source binding, or reopened
allocation. The normal recovery-actions read can also revalidate
recovery state, so it was not a safe substitute for a current read-only
inspection.

**Expected behavior**

Record an allowlisted reason at the rejected guard. Allow a reader to
inspect one exact task/run/action without changing its attempts, saved
input, receipts, or ownership. Clearly separate the recorded failure
from current identity observations.

**Steps to reproduce**

1. Prepare a terminal native run with a retained-source action and a
lease policy of `reuse_by_environment`.
2. Attempt source verification. The guard must still reject that policy.
3. Open the failed run and choose Inspect recovery, then Inspect current
source. The diagnostic should name `lease_policy_not_retained` and the
allowlisted policy.

## What Changed

- Split existing source-binding guards into closed reason codes. Keep
each acceptance predicate and physical recovery action unchanged.
- Persist validated, path-free reason/policy detail with future system
recovery failures. Keep historical generic records compatible.
- Add an explicit query on the existing task recovery-actions GET. Exact
company/task/run/action selection bypasses read revalidation. Run
selection lists at most 20 actions; inspection stops before per-lease
lookup when a record has more than eight sources.
- Add a manual inspection control on native export-failure run pages.
Keep resolved retained actions inspectable. Show recorded failure and
current recorded-source identity separately.
- Update execution semantics and feature/access recipes. The shared type
is the normalized server/UI contract; it contains no provider payload or
secret.

## Verification

- RED: the prior verifier failed both reason assertions for rejected
warm policy and foreign-run lease binding. GREEN: the final focused
source/HTTP, system-state, and UI suites pass 23 tests. The UI follow-up
passes 8 tests after the review fixes.
- HTTP tests prove exact selection, cross-company denial,
resolved-action access, reused-allocation rejection, redaction, and
unchanged actions, runs, leases, tasks, wakeup receipts, workspace
operations, and activity. A nine-source case proves bounded inspection
while the existing worker verifier still accepts all nine valid sources.
- Review follow-up: show allowlisted saved detail beside a changed
current observation, preserve all listed actions, and follow
exact-action selection in verified/limited/error tests. The prior
component fails the saved/current drift regression in an in-memory RED
control; the current component passes. UI typecheck/build pass again.
- `pnpm -r typecheck` and `pnpm build` pass. Token gates, maps, module
boundaries, and `git diff --check` pass.
- Rebased onto master `46de15ff4b` without conflicts. Range-diff proves
both diagnostic commits are unchanged. On new head
`a7e2bc87fcea75e76b6b736304d1287257f71174`, all 23 focused tests, full
typecheck/build, token gates, maps, and module checks pass again. Both
fresh reviews are 5/5 with zero unresolved threads. Final exact-head
GitHub status is CLEAN: 55 success, 2 skipped, 1 neutral, no pending
checks or failures. The [ci / verify
aggregate](https://github.com/paperclipai/paperclip/actions/runs/40387002325/job/121802415824)
passed in eight seconds. All test/build/typecheck/E2E lanes passed.
- Existing full workspace recovery suite: 56/56 pass (103.5 seconds)
before the byte-equivalent rebase. `pnpm test:run` hit repeated embedded
PostgreSQL initialization failure before database-suite assertions and
was stopped. No full local suite pass is claimed. The owned external
`_test` database provided the focused HTTP/source verification. The
prior browser attempt also stalled at Network.enable before its click;
that attempt did not establish browser acceptance.
- Current-head local browser acceptance passes through the actual run
page: Inspect recovery lists the exact action, Inspect current source
shows recorded generic failure separately from the current rejected
policy, and reload resets inspection to the explicit control. Explicit
selection works again after reload. The app used isolated data, an empty
inherited environment, a paused synthetic agent, and no provider
credentials. Final state remains failed/exhausted with zero attempts,
zero wakeup receipts, zero workspace operations, and unchanged original
generic failure. The local app and temporary tab were stopped. This
proves the diagnostic journey, not the deployed source's actual lease
policy or its recovery.
- Final merge verification at
`a7e2bc87fcea75e76b6b736304d1287257f71174`: GitHub reports CLEAN and
mergeable. Both repository-required checks, `ci / verify` and `ci /
e2e`, pass, as does the exact-head Greptile check (5/5, zero unresolved
threads). The separate [Entire
trail](https://entire.com/gh/paperclipai/paperclip/trails/7) has no
recorded reviewer approval; no such approval is claimed. It is not a
configured GitHub merge gate.
- Initial local setup failures were distinct from product assertions:
localhost sandbox EPERM required scoped local access; a browser harness
import path was corrected. A test UUID type inference error and
historical native-export UI selection gap were corrected before the
green run. The live task remains blocked; this diagnostic does not claim
it recovered.

## Risks

- This change improves evidence. It does not prove provider termination,
local descriptor integrity, or successful file preservation, and it does
not repair the original source.
- Current inspection spans multiple database reads. It is an
observation, not a historical cause or execution authority. It performs
no provider/filesystem calls and starts no retry or executor.
- The optional persisted detail is backward compatible. Unknown values
and extra fields are excluded. Oversized source lists remain uninspected
through this explicit diagnostic path; physical recovery rules do not
change.

## Model Used

OpenAI Codex based on GPT-6, with code execution and browser tools. The
exact serving model ID and context window were not exposed in this
session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 08:21:28 -05:00
DottaandPaperclip 0373f9f261 fix(recovery): wake retained workspace preservation after Retry (#17741)
## Thinking Path

> - Paperclip lets people manage AI agents and their saved work.
> - A stopped run can owe workspace preservation before saved input can
continue.
> - Retry saved a recovery request but did not wake the worker that owns
preservation.
> - Older recovery records could also take the bounded scan slots before
a due request.
> - A second Retry could change the revision while the physical owner
was still saving files.
> - This pull request wakes a separate worker after commit, selects due
work first, and keeps the active owner's revision.
> - Saved work can progress without replaying the stopped provider turn
or blocking other finalization.

## Linked Issues or Issue Description

**What happened?**

An operator approved a saved design and selected Retry after native
workspace finalization failed. The API accepted one canonical request.
The source remained scheduled at zero preservation attempts for more
than 30 minutes. No new provider turn started. A local regression also
found that repeated Retry during export invalidated the physical owner's
completion revision.

**Expected behavior**

Committed Retry must wake bounded source preservation. Due recovery must
progress ahead of passive cleanup obligations and unrelated history.
Duplicate requests must keep one physical attempt. Saved input must
still pass normal admission after verified recovery.

**Steps to reproduce**

1. Retain a terminal run with an exhausted workspace recovery and saved
authorized input.
2. Let the delivery coordinator become quiet, then request Retry.
3. Observe the recovery state and physical preservation receipt. Repeat
Retry while the export is held.

**Paperclip version or commit**

Observed on served commit `4b89eec644d6e1a99d58926e77ae58927c3fa87b`.
Local baseline: `a1f77821b9cd5c9c846fdf7a0fa5cc620ad2b6bb`.

Related work: Refs #17053, #17267, #17094. This change repairs explicit
Retry delivery. It is separate from the run instruction projection in
#17724 and historical unsafe-export discovery in #17687.

## What Changed

- Notify a dedicated workspace recovery duty from the transaction that
saves Retry, recovery admission, adoption, or recovery progress.
- Reconstruct work and saved retry deadlines at startup. Fill each free
export slot independently, track physical work for idle drain, and join
the dedicated worker's exports on shutdown.
- Drain one bounded pass through completed cleanup and saved-input
obligations. Restart that pass after committed changes without
repeatedly rechecking paused input.
- Share source eligibility between deadline selection and candidate
consumption. Keep resolved files-only obligations and ignore unrelated
resolved or cancelled history.
- Select due unfinished preservation before passive historical records.
Rotate past source locks held by other workers and retain the indexed
source lookup.
- Bind the canonical Retry intent without changing an active physical
attempt's generation or revision. Preserve that binding across later
progress writes.
- Add real producer, physical preservation, startup, deadline, idle, and
independent finalization regressions. Document the delivery contract.

## Verification

- Red controls: Retry without a notification stayed scheduled; duplicate
Retry invalidated the active export; whole-batch waiting stranded a
third Retry; a scan without cursor rotation starved a source behind two
actual advisory locks. Removing unfinished-first selection left HTTP
Retry scheduled behind five real recovered sources still owing cleanup.
The fixture explicitly verifies their eligibility with the worker's
actual query filter.
- Further red controls: the previous worker visits only two of three
eligible completed sources. Disabling the committed-change cursor reset
misses a new earlier source arriving while two callbacks are held. Both
pass with the final worker.
- Green: the public Retry HTTP route, registered production worker, real
file copyback, duplicate canonical receipts, and public recovery receipt
reload passed on an owned local fixture. Its agent is paused to keep
this files-only check from starting a provider turn.
- Full workspace recovery suite passed 56 tests, including the
foreign-lock regression.
- Execution-control lifecycle suite passed 24 tests. It covers idle SQL,
stored deadlines, cancelled and resolved history, independent
finalization, and shutdown. One, three, and seven paused completed
sources are each visited in a bounded pass, then stay deferred without a
successor or SQL during 90 seconds of fake-clock idle time. A committed
earlier source is evaluated when it arrives during two held callbacks.
- Both GitHub and Telegram `automatic_running` continuation regressions
passed with the new canonical intent binding.
- `pnpm -r typecheck` passed.
- `pnpm build` passed.
- `node scripts/check-maps.mjs` and `node
scripts/check-module-boundaries.mjs` passed.
- Local `pnpm test:run` could not complete: embedded PostgreSQL
repeatedly failed to start. The owned run was stopped after this setup
failure repeated. Latest-head CI supplies broad coverage.
- Earlier setup failures included a localhost sandbox denial and loss of
the owned disposable Docker database. These were separate from the
product red controls above.
- Local database tests use an explicitly named disposable PostgreSQL
`_test` database. That opt-in database is reset before the suite.
Default embedded test behavior is unchanged.
- Exact-head CI is green at `ef7dcf1ba1060864c9d216c33b51f4d9f8f57fbd`;
Greptile and Ponytail both rate it 5/5, with no unresolved review
threads. Earlier CI caught the indexed discovery assertion and two exact
chat intent-binding regressions; all three were fixed and pass locally
and in CI. A separate chat shard lost contact with its self-hosted
runner without a product assertion result. A later shared runner
shutdown interrupted several jobs; one infrastructure rerun passed
before the final test-and-worker follow-up triggered a fresh green run.
- No staging deployment is part of this PR. The original approval
remains saved; deployed recovery still needs acceptance after rollout.

## Risks

- More recovery writes now wake a bounded maintenance duty. Completed
history must not keep it active. Idle and deadline tests cover this
boundary.
- Resolved files-only obligations must remain eligible without granting
replay. The startup regression checks that path.
- Provider stop proof, copyback ownership, source binding, retry
budgets, and normal execution admission remain required. There is no
schema or public API change.

## Model Used

- OpenAI Codex based on GPT-6, with code execution and browser tools.
The exact serving model ID and context window are not exposed in this
session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-26 10:28:12 -05:00
DottaandPaperclip eaacadeba3 fix(recovery): let preservation owners reopen retained sources (#17053)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Recovery must save retained workspace files before ordinary
execution can resume.
> - The environment runtime blocks access to retained sources to prevent
unsafe reuse or deletion.
> - The recovery worker also used that ordinary resume path.
> - The guard returned no lease, so recovery reported an invalid source
binding before it could save files.
> - This pull request gives the physical preservation owner a separate,
source-bound resume path.
> - Ordinary execution and destructive cleanup keep their existing
guards.

## Linked Issues or Issue Description

Refs #16965.

**What happened?**

A retained workspace with a workspaceRestoreRecovery marker could not
recover through the real environment runtime. The runtime returned null
before calling the provider. Recovery classified this as
source_binding_invalid. Earlier recovery fixtures replaced the runtime
facade, so they did not exercise this guard.

**Expected behavior**

The verified preservation owner can reopen its exact stopped allocation
to save files. Other callers remain blocked. Recovery preserves newer
local files and does not start another agent or repeat a completed
external action.

**Steps to reproduce**

1. Seed a terminal source run with a retained sandbox, a recovery
action, and the workspaceRestoreRecovery lease marker.
2. Give the source a verified stop receipt.
3. Call recoverRetainedWorkspace through environmentRuntimeService.
4. Before this change, the recovery action returns to blocked with
source_binding_invalid. No files are copied.

**Paperclip version or commit**

Reproduced on master at 65f59e339a. The
same failure was observed in a served build containing #16965.

## What Changed

- Add an internal preservation resume method. It validates the exact
company, task, run, action, lease, environment, current physical owner,
and stop-only cleanup claim before provider access.
- Keep ordinary resume and destructive cleanup blocked for protected
sources.
- Run all retained-workspace fixtures through the assembled environment
runtime. Add a failing-before/passing-after protected-source regression
and four rejection cases.
- Use the same runtime path and protection marker in the opt-in Daytona
interruption probe. Allow a verified immutable snapshot UUID as its
input.
- Document the authority boundary and verification recipes.

## Verification

- Red: the protected-source regression failed with expected recovered,
received blocked. It recorded source_binding_invalid before provider
resume.
- Green: 235 tests passed across workspace recovery, durable task Retry,
and environment runtime. These include saved files, newer local files,
bounded retries, competing owners, and no extra run or wake.
- pnpm -r typecheck and pnpm build passed. The initial sandboxed
typecheck could not open a tsx IPC pipe; the same command passed with
local process access.
- node scripts/check-maps.mjs and git diff --check passed.
- Live Daytona qualification passed using the same immutable snapshot ID
as the affected deployment. Export was interrupted after a real
download; a second controller restored the files and verified one
original agent start, one simulated external action, one saved snapshot,
and a stopped retained source. The disposable sandbox was deleted after
verification.
- Greptile reviewed 804dfe24ef at 5/5 with
no actionable findings or unresolved review threads.
- Full current-commit CI passed:
https://github.com/paperclipai/paperclip/actions/runs/39540771747.
Required ci / verify and ci / e2e passed. Superagent, Socket, and Snyk
checks passed.
- Full local pnpm test:run did not pass: 22,992 tests passed, one
managed-run identity test timed out at 60 seconds, and 100 tests skipped
across the first general-server phase. The command stopped before its
later phases. The unchanged agent-identity.test.ts suite then passed all
11 tests in isolation in 15.53 seconds. The original timeout remains
recorded; it is not reclassified as a full local pass. The same commit's
complete CI test matrix passed.
- The optional Paperclip Review completed with a neutral advisory 5/5 on
the same head and no actionable findings. It inspected all eight changed
files and connected code. It did not independently run tests or the
provider probe.
- Hosted ordinary Retry recorded one durable intent and no successor run
while file preservation remained blocked. The existing published review
stayed unchanged. Hosted recovery remains unverified until this
follow-up is deployed. A Retry acknowledgement alone does not prove
recovery.

## Risks

- Recovery can reopen a retained sandbox. The dedicated path requires
the current physical ownership claim and the exact source's stop-only
intent. It does not clear the ordinary protection marker.
- Provider failures retain the source and use the existing bounded
recovery budget. The provider cannot substitute another allocation.
- The live probe uses a disposable sandbox and simulated
agent/external-effect counters. It does not prove a live LLM or GitHub
review journey.
- No schema, public API, permission, budget, pause, or approval changes.

## Model Used

OpenAI GPT-6 through Codex, with repository inspection, code execution,
browser verification, and test tools. The exact served model variant and
context window are not exposed by this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (focused suites; full suite
still running)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-21 07:27:45 -05:00
DottaandPaperclip d33b40a5c3 fix: recover saved work and provide reliable task Retry (#16965)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The control plane must retain work when an execution or workspace
export fails.
> - Retained-workspace holds can leave saved input waiting with no
usable Retry action.
> - Users cannot verify process ownership or repair provider
allocations.
> - This pull request gives system recovery a durable budget and gives
each deliberate Retry a durable intent.
> - Safe recovery can preserve files, settle an accepted result, or
prepare an independent workspace.
> - One task state keeps infrastructure diagnostics out of the ordinary
conversation while preserving real permission and approval gates.

## Linked Issues or Issue Description

**What happened?**

A task with a retained workspace can show recovery, failed-run and
saved-message warnings at the same time. Historical runs without native
workspace descriptors cannot use the native restore path. Old recovery
metadata can prevent a new attempt even when an independent workspace
would be safe.

**Expected behavior**

Paperclip owns bounded infrastructure retries and file preservation. A
user can request another attempt with ordinary Retry. Thinking appears
immediately. The service preserves completed effects and current user
controls. It does not claim that starting fresh recovered the old files.

**Steps to reproduce**

1. Retain a failed run and its workspace after export or process-stop
failure.
2. Save a follow-up while the recovery hold is active.
3. Open the task and try to continue without infrastructure credentials
or operator certification.

Refs #16906, #16835, #16883. This extends their recovery and receipt
contracts. Related work: #16962 and #16963.

## What Changed

- Add task-level Retry with company-scoped authorization, all-status
idempotency and durable admission before physical work. Keep exact-run
Retry compatible.
- Add typed recovery phases, finite persisted backoff, historical
adoption and durable local failure diagnostics.
- Preserve native workspaces or verified historical snapshots. Admit an
independent fresh workspace only after execution containment. Keep
old-source preservation and cleanup obligations separate.
- Carry completed publication and API receipts forward. Fence uncertain
mutation retries across replacement runs. Preserve connected-chat
identity and authorization.
- Re-enter normal dispatch for saved input. Reuse accepted results
through the existing finalizer.
- Use one task interruption projection across the thread, composer and
task list. Show optimistic Thinking, ordinary Retry and genuine user
gates.
- Add PostgreSQL, filesystem, API, browser-fixture and controlled
provider coverage. Document the behavior and update the feature/access
maps.

## Verification

- All GitHub Actions CI jobs pass on
`6d8c6d27d50586d8548b31d03b47f6d73a0ee11d`: typecheck, all
server/chat/workspace and serialized suites, Runner checks, eight
browser shards, canary and build. The required merge gate passes.
- Greptile reviewed the final commit at 5/5, with no actionable findings
or unresolved review threads. Its two earlier P1 findings are fixed.
- Full recursive typecheck and build pass locally on the final commit.
All 16 task Retry cases pass against isolated real PostgreSQL. The
unrelated-success case and both stale Retry cases fail on the previous
code and pass after the fix.
- The corrected gateway fixture passes its uncertain-mutation and
independent-read case. Four real HTTP API cases pass for definite
403/422 rejection, completed-effect reuse, and uncertain-effect fences
across deliberate and automatic retries. These focused gateway/API
checks used a temporary database lifecycle harness on an existing
healthy local PostgreSQL process. The standard repository helper is
unchanged.
- Earlier qualification covers 291 task UI cases, saved-file copyback
and ancestry, current Agent Chat ownership races, nine chat Retry API
browser cases, and six assembled browser recovery fixtures. A real
API/PostgreSQL/native-runner journey showed Thinking in 25 ms despite a
delayed response, admitted one successor in a separate workspace,
retained original files, and survived reload.
- The controlled Daytona test interrupts export, joins the controller,
restarts it in a separate process, and verifies one agent start, one
external sentinel operation, retained file contents and no premature
cleanup. It does not establish abrupt-kill or live LLM/GitHub
acceptance.
- The full local `pnpm test:run` was attempted and was not green: an
OAuth row-lock timing assertion failed, and a later suite exhausted
macOS PostgreSQL semaphores. The ten OAuth timing cases passed on rerun
against a separate database. Later embedded database initialization
failures remain recorded; the final CI matrix passes.
- Security scanning passed on production-code commit
`63953edf069b111c6becc3144dffa6deb52f68b5`. Its only successor changes
one test-fixture identity line. The final scan is neutral/inconclusive
because its pinned merge-base checkout failed or timed out. Both rerun
API requests returned 404, so no fresh successful scan is claimed.
Socket and Snyk checks pass on the final head.
- The optional Paperclip Review remains queued behind an earlier run in
workspace finalization. Its prior completed assessment was incomplete
because a comparison tool was blocked by the platform filter. It is not
counted as a completed code review. GitHub does not require this check
for merge.
- Hosted acceptance remains pending deployment. Local and CI results do
not establish recovery of a hosted task or its retained files.

## Risks

- Recovery changes span admission, physical ownership and dispatch.
Incorrect evidence could permit overlapping work. Existing physical
locks and authoritative stop requirements remain in place.
- The migration adds concurrent indexes. It requires the existing
top-level migration runner and retries incomplete index creation safely.
- A fresh attempt can complete while old-source preservation remains
unresolved. The service retains that obligation and does not report
those files as recovered.
- Unknown tool mutations remain blocked until their authoritative
receipt owner resolves them. Generic upstream tools have no universal
result-query protocol.
- Recovery and diagnostics stay inside the stack. No Cloud incident
endpoint, notification worker or new outbound reporting channel is
included.

## Model Used

OpenAI Codex, GPT-6 family. The runtime does not expose a more specific
model version or context-window size. The model used reasoning, shell
execution, source editing and browser tools. No model subagents were
used for this implementation.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (focused suites; the full
local run limitation is documented above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-20 19:58:53 -05:00