Files
PaperClipAI/server
DottaandPaperclip e0276e0166 fix(workspaces): identify remote command timeouts (#18827)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Remote workspaces prepare repositories and save files through
sandbox commands.
> - Each command has a timeout and a known operation.
> - The transport discarded that information when a command timed out.
> - A failed clone then appeared as the same generic error as other
failures.
> - This pull request keeps the operation and timeout in a fixed
diagnostic.
> - Operators can identify the failed stage without exposing command
output.

## Linked Issues or Issue Description

**What happened?**

A remote workspace setup command reached its 300-second transport limit
before provider work started. The workspace layer reported only
`remote_workspace_operation_failed`. That message concealed the timeout
and the affected operation. A later full clone on the same sandbox image
passed in 107 seconds, so the original clone delay remains unexplained.
This PR repairs the loss of diagnostic information. It does not claim to
prevent the underlying delay.

**Expected behavior**

A transport timeout must remain a failure. Its diagnostic must identify
the operation and configured limit. It must exclude raw output and must
not automatically repeat a command whose remote outcome is unknown.

**Steps to reproduce**

1. Return `timedOut: true` from the sandbox command boundary during
workspace preparation or cleanup.
2. Observe that the base implementation reports only
`remote_workspace_operation_failed`.
3. Apply this change. Preparation reports
`remote_workspace_operation_timed_out: prepare (configured timeout
300000ms)`. Cleanup retains its separate 10-second limit.

**Paperclip version or commit**

The regression was reproduced on master base
`367f29478d14003cc5c0f161b7ce8f3164993634`. The fixed current head is
`be45ce6594254bd6b94023a550d8b655dd704bb0`.

**Deployment mode**

Hosted Paperclip with native Runner and Daytona sandbox execution. Local
acceptance also uses disposable PostgreSQL and filesystem commands.

Related work: #18403 introduced remote workspace authority. #18705 added
a fixed full-disk diagnostic. I searched recent clone and
remote-workspace PRs and found no duplicate timeout-reporting repair.
This is a defect fix and does not add a roadmap feature.

## What Changed

- Give each internal workspace command a closed operation name: prepare,
repository, archive helper, capture, restore, or cleanup.
- Preserve that operation and the configured limit in a typed error when
the transport reports a timeout. Do not claim a measured elapsed time.
- Keep other failure diagnostics and exact safe-message checks
unchanged.
- Preserve typed cleanup timeouts in the operation cleanup journal while
retaining generic diagnostics for untrusted errors.
- Add preparation, cleanup, and real-PostgreSQL lifecycle regressions.
Document the timeout and the need to reconcile saved receipts before
recovery.

## Verification

- Red: both transport regressions failed on the base implementation. A
separate real-PostgreSQL cleanup regression reproduced the review
finding: the operation journal replaced a typed timeout with a generic
error.
- Green: all 62 remote transport and archive tests and all 50
real-PostgreSQL workspace lifecycle cases passed. The lifecycle suite
ran with an explicit disposable database; it was not skipped. Command:
`pnpm exec vitest run --project @paperclipai/server
server/src/services/remote-workspace-transport.test.ts
server/src/services/remote-workspace-snapshot-archive.test.ts`.
- Safety: a timeout with exit code zero still fails. Raw stdout and
stderr stay out of the error. A timeout cannot claim the fixed
unfinished-Git diagnostic. No command retry occurs. Existing saved files
remain unchanged. Existing archive, restore, receipt, symlink,
credential-isolation, and full-disk controls pass. The cleanup lifecycle
cases retain the acknowledged snapshot, preserve working files, avoid
duplicate uploads, record only a safe typed timeout, and settle cleanup
on the next normal reconciliation. An untrusted error that mimics the
timeout prefix stays generic.
- Live acceptance: production transport code ran against actual Daytona
SDK 0.203.0 timeouts on the incident image, with 4 CPUs, 8 GiB RAM, and
a 10 GiB disk. A controlled one-second SDK limit produced the generic
baseline error and the fixed exact preparation timeout. The current
`be45ce6594` code was rechecked for both preparation and cleanup. Each
returned the typed error and its own configured limit (300 seconds and
10 seconds) while the controlled SDK fault remained one second. Each
case made one command call. Each disposable sandbox was deleted before
the next case.
- The first live-probe attempt used an incorrect error-name check. It
failed its acceptance check and remains recorded. The probe was
corrected to use the same SDK timeout base class as production. Its
first sandbox was deleted. This was a probe setup failure, not red
evidence for the repair.
- A separate same-image probe ran the real full repository preparation
successfully in 107 seconds. The affected command path is unchanged
except for error reporting. The original task was retried once through
the product on deployed source
`adb49a44346849e751548b2fb640693dbfad4b6f`. The requested memo and its
first document revision persisted after reload and matched the
downloaded attachment. The run succeeded, the task reached Done, its
execution lock cleared, and final workspace save and result commit
succeeded. The diagnostic PR is not deployed.
- Recursive typecheck and build passed on `be45ce6594`. `git diff
--check` passed. The first full-suite attempt stopped before tests
because the offline install had skipped an embedded PostgreSQL native
postinstall. The package's official symlink hydration fixed `initdb`.
- The next Mac run passed 350 chat cases and hit one unchanged 15-second
chat deadline. That case then passed in isolation, and all six full chat
shards passed. The broader Mac matrix was later stopped as partial when
CI results became available. No full single-command Mac pass is claimed.
- Full Vitest file coverage uses the exact successful CI merge tree,
`d264fd98e7fb6eff4a9e6b00d402f88894cc0e57`. Its canonical inventory has
2,899 files. Passing CI covered 2,057 files. The remaining 842 ran
locally through the standard exact-file harness. The inventory and
selection checks show no unassigned files. All four changed source/test
files are byte-identical between the PR head and that merge tree.
- The first remaining-file run passed 771 files. The database group
passed 68 files and reported four deadline/readiness failures in three
unchanged files. Those same three files passed all 12 cases when run
sequentially on Node 24.22.2, matching CI's major version. Assertions,
readiness limits, and test deadlines did not change. The first attempt
used Node 26.4.0 with a parallel UI workload; this does not isolate the
contribution of runtime version versus load. Both attempts remain
recorded.
- [GitHub
CI](https://github.com/paperclipai/paperclip/actions/runs/41699549115)
is green on current head `be45ce6594`: `ci / verify`, typecheck, build,
all selected server/chat/workspace groups, Runner checks, and all eight
browser shards passed. Greptile is 5/5 with no open threads. The prior
P2 was resolved by removing new code comments.
- Structured Paperclip review remains queued because its own previous
accepted result exhausted workspace export recovery. A fresh
current-head request is saved, but normal Retry returns HTTP 409
(`accepted_result_without_new_input`). Its advisory cleanup finding is
fixed in `be45ce6594` and covered by red/green lifecycle evidence. This
check is informational. The branch requires `ci / verify` and `ci /
e2e`, and both pass. The independent review-service limitation remains
recorded; this PR is not deployed.

## Risks

Timeout error text changes. Consumers that compare the old generic
message may need to accept the new diagnostic. No such production
comparison was found in this repository. API shapes, deadlines, retries,
ownership, credentials, and model selection do not change. A timeout
still does not prove that the remote process stopped or that a save
failed before publication. Existing recovery must reconcile that state.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, code execution, and browser
tools. The host does not expose a more specific model ID or
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-03 14:13:16 -06:00
..