mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-08 21:35:36 +01:00
DottaandClaude Opus 5.5
a994a648f1
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run in a Daytona sandbox saves its workspace after the provider turn. This save is "copy-back". > - On staging, about 20 runs kept a saved result, but copy-back failed three times. The issues became blocked with `native_workspace_sync_out_retry_exhausted`. > - #19152 added a redacted cause for a sandbox command that exits with an error. Three gaps stayed open: a provider that refuses the command still gives a bare code, the run does not show the cause, and the failed save operation stays "running". > - The first failure also had the wrong name. The runner kept the result but did not prove a durable suspension. The executor reports this as `native_session_lease_lost`, and the workspace operation shows only that text. > - This pull request names each of these failures with closed, redacted labels. It also ends a failed save operation as "failed". > - The benefit is that operators can see why copy-back failed and which save is still active. Retries and saved results do not change. ## Linked Issues or Issue Description Refs #19152, #19145. These are the earlier copy-back diagnostics and retry-scheduling repairs. This PR does not duplicate them. **What happened?** A native task ran in a Daytona sandbox and its result was accepted. The run then failed with `native_workspace_sync_out_retry_exhausted`. The run had no error text. The automatic copy-back retries each failed in under one second, with only `workspace_sync_out_failed`. The final `remote_workspace_snapshot` operation stayed "running" after the retries stopped. The first `workspace_finalize` operation showed only `native_session_lease_lost`. The run log showed that the runner shutdown timed out before durable suspension. The lease was not lost. **Steps to reproduce** 1. Run a native task in a Daytona sandbox until its result is accepted. 2. Make the sandbox provider refuse the copy-back command (for example, a lease that is no longer active), so that no exit status comes back. 3. Let the automatic retries stop. Then read the run and its workspace operations. **Expected behavior** The run, the coordinator, and the failed operations name a redacted cause. The failed save operation is "failed" with a finish time. **Actual behavior** The run has no cause. The save operation stays "running". A provider refusal keeps only the bare `workspace_sync_out_failed` code. ## What Changed - `remote-workspace-transport.ts`: the provider can throw instead of returning an exit status for the copy-back helper, the capture, or the archive download. `remoteWorkspaceFailureCause` then returns `remote_workspace_operation_failed: operation=<op> transport=<label>[ code=<ERRNO>]`. The label is from a closed list (for example `sandbox_lease_inactive`, `permission_denied`, `sandbox_not_running`, `transport_timeout`, `transport_closed`, `daytona_sandbox_not_found`, `provider_error`). Provider text is never copied into the cause. The thrown error does not change, so preparation errors, `classifyNativeWorkspaceFailure`, and other callers keep their contracts. Archive-safety and Git-metadata refusals get no label, because they have their own diagnostics. - `remote-workspaces.ts`: when the last active invocation of a save fails, the save operation becomes "failed" with `exitCode`, `finishedAt`, a redacted `stderrExcerpt`, and `metadata.lastFailure`. A retry of the same boundary sets it to "running" again. A success sets it to "succeeded". After the final save commits, a failed save is retired for export cleanup in the same way as a running save. - `remote-workspace-snapshot-retention.ts`: a failed save that is not retired keeps its snapshot pinned, as it did while it was "running". Its retry can still use an uploaded snapshot. - `native-run-finalizer.ts` and `native-workspace-finalizer.ts`: the redacted cause of each copy-back attempt goes to the coordinator `failureDetail.failureCause` and the run `resultJson.failureCause`. When retries stop, the run error adds `Last cause: <cause>`. The server checks the cause against a fixed grammar again before it writes it. A cause with a path or other text is dropped. - `native-post-result-failure.ts` (new), `native-session-executor.ts`, `runtime-execution.ts`: a failure after an accepted result gets the label `runner_settlement_incomplete` when the runner did not prove a durable suspension. The failed `workspace_finalize` operation records this closed diagnostic as `failureDiagnostic`. The coordinator records it as `native_post_result_failure: category=<label>`. The control code `native_session_lease_lost` and its fencing do not change. ## Verification - Red, before the fix (the new tests run against `master` source): - 4 provider-refusal cases and the archive download failure: no redacted cause was available, so the copy-back recorded only the bare code. - Failed save, with only the transport fix applied: `- "status": "failed"` / `+ "status": "running"`. - Run cause after retries stop: `expected null to be 'remote_workspace_operation_failed: …'`. - Green, after the fix: `cd server && npx vitest run src/services/remote-workspace-transport.test.ts src/services/remote-workspace-save-concurrency.test.ts src/__tests__/native-workspace-finalization-work.test.ts src/services/native-runtime/native-post-result-failure.test.ts src/services/remote-workspaces.integration.test.ts src/services/remote-workspace-snapshot-retention.test.ts` passes. - Safety cases: - The failed save is retried on the same boundary operation, records "succeeded", and keeps the saved file bytes. - `daytona_sandbox_not_found` is still classified as `workspace_sync_out_unrecoverable`. - An archive-safety refusal keeps its exact text. - A forged cause with a path never reaches the run or the coordinator. - A run-level timeout cause (`remote_workspace_operation_timed_out: capture (configured timeout 300000ms)`) reaches the run and the coordinator. - `heartbeat-native-remote-failure-retention.test.ts` passes: a preparation failure keeps its exact run error. - The existing integration tests for retired saves, poisoned reservations, duplicate invocations, and snapshot retention pass. They prove export cleanup and identity protection. - `tsc --noEmit` for `server` reports no errors. - Not done: a live Daytona copy-back. This PR adds diagnostics and corrects the operation state. It does not change when a retry runs. ## Risks - Thrown copy-back errors do not change. Only the recorded cause is new. The `daytona_sandbox_not_found` and archive-safety classifications are tested. - A failed save operation is now "failed" before the final save commits. Retirement, export cleanup, and snapshot pinning now also select these failed rows. They select only rows with `lastFailure` and no retirement receipt. - Privacy: each new cause is a closed label or a server-built code. The server checks it again with an anchored grammar before it writes the run. No sandbox output, path, or message text is stored. - The retry count, the retry delay, and the classification of each failure do not change. No provider turn is added. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Provider: Anthropic. Model: Claude Opus 5.5 (`claude-opus-5-5`), in the Claude Code agent harness. - Capabilities used: tool use, shell and code execution, repository search, read-only browser reads of staging, and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
25 lines
1.8 KiB
TypeScript
25 lines
1.8 KiB
TypeScript
import { describe, expect, it } from "vitest";
|
|
import { NativeSessionCloseUnrecoverableError } from "../../vendor/paperclip-runner/index.js";
|
|
import { nativePostResultFailureCategory, readNativePostResultFailureDiagnostic } from "./native-post-result-failure.js";
|
|
|
|
describe("native post-result failure diagnostic", () => {
|
|
it("names a runner that kept the result but did not prove its suspension", () => {
|
|
const unsettled = new NativeSessionCloseUnrecoverableError({ runnerSuspended: false, providerDrained: false }, true);
|
|
expect(nativePostResultFailureCategory(unsettled)).toBe("runner_settlement_incomplete");
|
|
expect(nativePostResultFailureCategory(new AggregateError([unsettled], "close and receipt failed")))
|
|
.toBe("runner_settlement_incomplete");
|
|
expect(nativePostResultFailureCategory(Object.assign(new Error("missing"), { code: "ENOENT" }))).toBe("io_not_found");
|
|
expect(nativePostResultFailureCategory(new Error("SYNTHETIC other failure"))).toBe("execution_failure");
|
|
});
|
|
|
|
it("reads only the closed labels from an error or its cause", () => {
|
|
const diagnostic = { stage: "post_result_execution", category: "runner_settlement_incomplete", resultAlreadyAccepted: true };
|
|
const control = Object.assign(new Error("native_session_lease_lost"), { diagnostic });
|
|
expect(readNativePostResultFailureDiagnostic(control)).toEqual(diagnostic);
|
|
expect(readNativePostResultFailureDiagnostic(new Error("wrapped", { cause: control }))).toEqual(diagnostic);
|
|
expect(readNativePostResultFailureDiagnostic(Object.assign(new Error("native_session_lease_lost"),
|
|
{ diagnostic: { ...diagnostic, category: "/SYNTHETIC/private" } }))).toBeNull();
|
|
expect(readNativePostResultFailureDiagnostic(new Error("native_session_lease_lost"))).toBeNull();
|
|
});
|
|
});
|