PaperClipAI

Jannik/PaperClipAI

0 Fork 0

mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-10 01:50:13 +01:00

Commit Graph

Author	SHA1	Message	Date
DottaandClaude Opus 5.5	c97c0dd248	fix(workspaces): give run preparation its own source lock scope (#19734 ) <!-- Write all pull request text in Simplified Technical English (ASD-STE100): short sentences, one instruction per sentence, simple approved vocabulary, and the active voice. --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A run prepares its task workspace under per-source advisory locks (`workspace-source-lock.ts`). The locks live in a transaction on a dedicated connection. An `AsyncLocalStorage` scope lets nested calls reuse that transaction. > - Timers and detached promises keep the async context that created them. A native recovery timer or dispatch can start a run from inside another operation's source scope. > - When that run prepares its workspace, it sees the other scope as "held". If that scope already committed, its dedicated connection is closed. The lock query then fails with `Failed query: select pg_try_advisory_xact_lock(...)` and cause `CONNECTION_CLOSED`. If that scope is still open, the run's lock is tied to the other transaction and is released early. > - In a restart chaos wave, a reattached follow-up run failed setup this way 50 ms after "Recovering the existing native runner process". The setup-failure log did not show the driver cause. > - This pull request makes only a live scope count as held, and makes run workspace preparation take its locks in an independent scope. > - The benefit is that a recovered or dispatched run gets its own source lock, and a lost-connection failure is diagnosable from the log. ## Linked Issues or Issue Description What happened? After a control-plane restart, a native run was reattached (`reattach_remote_runner`). Its workspace preparation failed at once with `Failed query: select pg_try_advisory_xact_lock(hashtextextended($1, 0)) as acquired params: isolated_repository_source:<company>:<issue>`. The run ended `failed`/`setup_failed`. The stack showed the "already held scope" branch of `withWorkspaceSourceLocks`, not a new acquisition. The same failure occurred earlier in the same wave on a different issue without a restart. Steps to reproduce 1. Call `withIsolatedRepositorySourceLock` for issue A. In its action, start a detached promise that later calls `withIsolatedRepositorySourceLock` for issue B. 2. Let the outer action return so its transaction commits. 3. Let the detached call run. It fails with `Failed query ... pg_try_advisory_xact_lock`, cause `CONNECTION_CLOSED`. Expected behavior Work that starts after a source scope ends, or independent work such as a run's preparation, acquires its own locks on its own connection. Environment Restart chaos campaign on Daytona (OpenCode, workload C1, fault K1, comment trigger), embedded Postgres. Reproduced locally on embedded Postgres by the new tests. Related: #14910 (deferred comment wake reopens a closed issue). This PR does not change wake promotion. ## What Changed - `server/src/services/workspace-source-lock.ts`: a scope has an `active` flag. The flag is cleared when its transaction ends. `withWorkspaceSourceLocks`, `retainManagedProjectSource`, `retainCurrentWorkspaceSource`, and `assertCurrentWorkspaceSourceHeld` use only a live scope. - `server/src/services/workspace-source-lock.ts`: new `withIndependentWorkspaceSourceLocks`. It acquires locks in a new scope and ignores an inherited one. - `server/src/services/heartbeat/workspace-preparation.ts`: run workspace preparation uses `withIndependentWorkspaceSourceLocks`. - `server/src/services/heartbeat/run-completion.ts`: the "heartbeat execution setup failed" log records a bounded driver `causeCode` (for example `CONNECTION_CLOSED` or a SQLSTATE). It does not log the cause text. - Tests: two embedded-Postgres tests in `isolated-remote-workspace-source-lock.test.ts`, one unit test in `run-completion.test.ts`. ## Verification - `cd server && npx vitest run src/services/isolated-remote-workspace-source-lock.test.ts` — 4 passed. - Red (old scope rule): "gives work detached from a finished scope its own source lock" failed with `Failed query: select pg_try_advisory_xact_lock(...)`, cause `CONNECTION_CLOSED`. This is the production error. - Red: "keeps an independent preparation's lock after the scope that started it commits" failed because the lock was released with the other transaction. - `cd server && npx vitest run src/services/heartbeat/run-completion.test.ts` — 95 passed. - `cd server && npx vitest run src/services/remote-workspace-online-source.test.ts src/services/heartbeat/workspace-preparation.test.ts src/services/isolated-repository-rehome.test.ts src/services/isolated-task-directory-retention.test.ts src/services/isolated-task-directory.test.ts src/__tests__/execution-workspace-reopen.test.ts` — all passed. - `cd server && npx tsc --noEmit` — clean. - No live Daytona replay was run. ## Risks - Low. A nested call inside a live scope behaves as before. - A call that inherited an ended scope now opens its own dedicated connection and lock transaction. Before, it failed. - Run preparation never joins another operation's lock scope. No caller relies on that today. If one did, it would now take a second, independent lock and could see `workspace_source_busy` instead of silently sharing. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edit, embedded Postgres tests). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 23:08:57 -06:00
DottaandPaperclip	0b266bd084	fix(workspaces): keep source preparation locks active (#16956 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Each task prepares its workspace before it starts the provider. > - Source preparation holds a transaction-scoped advisory lock on a dedicated database connection. > - Preparation can use other connections and the filesystem for longer than PostgreSQL allows that lock connection to stay idle. > - PostgreSQL then closes the connection and releases the lock while preparation still runs. > - This pull request reuses the workspace lock keepalive during source preparation. > - Slow preparation retains exclusion and returns its original result or error. ## Linked Issues or Issue Description What happened? Tasks failed during workspace preparation with `CONNECTION_CLOSED` at the configured 60-second idle transaction timeout. No provider had started. A live retry showed the source-lock transaction idle while authorization reads ran on another connection. Its advisory-lock key matched the task's source-preparation key. PostgreSQL closed that transaction at the timeout. The same defect remains on current master. A real PostgreSQL regression also shows that another preparation can enter after the timeout while the original callback still runs. Expected behavior Source preparation must keep its advisory lock until the callback and any pending keepalive query finish. The ordinary pool must remain usable. The original preparation result or failure must survive. Steps to reproduce 1. Set `idle_in_transaction_session_timeout` to one second in a disposable PostgreSQL database. 2. Call `withIsolatedRepositorySourceLock` and keep its callback active for longer than one second without querying the lock connection. 3. Try to acquire the same source lock from another operation. 4. On the old code, the second operation enters and the first transaction fails with `CONNECTION_CLOSED`. Paperclip version or commit Reproduced on source at `cde944ed75648adfb569e86133adc8b118b55dd6`. The repair is rebased onto `d855e550ad`. Additional context Refs #16340. That change introduced the keepalive for workspace copyback. This PR applies the existing helper to the source-preparation transaction. ## What Changed - Read the configured idle timeout in the same query that acquires the source lock. - Keep that dedicated connection active during source preparation with the existing helper. - Add two real PostgreSQL regressions for exclusion, company independence, a one-slot ordinary pool, nested acquisition, cleanup, and preservation of callback errors. - Update the execution contract and workspace verification maps. ## Verification - Red before the production edit: both new tests failed. A competing operation entered after the lock expired. A callback failure was replaced by `CONNECTION_CLOSED`. - Green after the repair: 86 tests passed across the source lock, source admission, keepalive, and heartbeat workspace preparation suites. - `pnpm -r typecheck`, `pnpm build`, and `node scripts/check-maps.mjs` passed on final commit `2b5000ce25`. - `git diff --check` passed. The lockfile is unchanged. - After rebase, the same 86 focused tests passed. The two source-lock tests passed again after the review cleanup change. - Live acceptance on `7df94d7a06`: a fresh local instance on macOS arm64, Node v26.4.0, native Codex `gpt-6.1-sol`, and permission mode `never`. PostgreSQL had a one-second idle transaction timeout. A temporary Git wrapper delayed repository reads by four seconds. Workspace preparation after environment acquisition took 8,297 ms. The real task completed, wrote and attached the correct line-count/SHA-256 report, reached `done` with native phase `committed`, and cleared its execution lock. The attachment matched the workspace file and survived browser reload. No dedicated lock connections remained. - The live probe initially used `gpt-5.4`, which the ChatGPT account rejected. Selecting the reported task model, `gpt-6.1-sol`, resolved that readiness failure. The local Claude CLI has no login, so no live Claude run is claimed. - Full CI passed on `2b5000ce25`: all general, chat, workspace, serialized-server, Runner, and E2E test shards, plus typecheck, build, and canary packaging. Both Greptile and Paperclip Review rated this commit 5/5 with no open findings. The Paperclip check concluded neutral; two unrelated visual checks were skipped. - The duplicate local `pnpm test:run` serial invocation was stopped with SIGINT after the complete CI test shards passed. It did not finish locally; no local full-suite pass is claimed. Local proof consists of the focused real PostgreSQL tests, typecheck, build, map checks, and the real task above. - The final commit changes test teardown only; production code matches the live acceptance commit. - Initial sandbox runs could not open test database ports or build IPC sockets. The same commands ran with local process access. The frozen install also found an existing manifest/lockfile mismatch; dependency resolution was local and the lockfile was restored. ## Risks - The lock connection sends one small query per keepalive interval. The helper derives that interval from the configured timeout and caps it at 30 seconds. It does no polling when the timeout is disabled. - The helper joins an in-flight query before the transaction ends. A slow database can therefore delay cleanup. - This change does not disable database deadlines, retry preparation, or recover an unexpected connection loss. It preserves existing admission checks and company/source lock keys. - Deployment and recovery of the reported tasks remain separate from code verification. ## Model Used OpenAI GPT-6 through Codex, with repository tools, code execution, and browser verification. The exact served model variant and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-20 07:01:17 -05:00

Author

SHA1

Message

Date

DottaandClaude Opus 5.5

c97c0dd248

fix(workspaces): give run preparation its own source lock scope (#19734 )

<!-- Write all pull request text in Simplified Technical English
(ASD-STE100): short sentences, one instruction per sentence, simple
approved vocabulary, and the active voice. -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A run prepares its task workspace under per-source advisory locks
(`workspace-source-lock.ts`). The locks live in a transaction on a
dedicated connection. An `AsyncLocalStorage` scope lets nested calls
reuse that transaction.
> - Timers and detached promises keep the async context that created
them. A native recovery timer or dispatch can start a run from inside
another operation's source scope.
> - When that run prepares its workspace, it sees the other scope as
"held". If that scope already committed, its dedicated connection is
closed. The lock query then fails with `Failed query: select
pg_try_advisory_xact_lock(...)` and cause `CONNECTION_CLOSED`. If that
scope is still open, the run's lock is tied to the other transaction and
is released early.
> - In a restart chaos wave, a reattached follow-up run failed setup
this way 50 ms after "Recovering the existing native runner process".
The setup-failure log did not show the driver cause.
> - This pull request makes only a live scope count as held, and makes
run workspace preparation take its locks in an independent scope.
> - The benefit is that a recovered or dispatched run gets its own
source lock, and a lost-connection failure is diagnosable from the log.

## Linked Issues or Issue Description

**What happened?**
After a control-plane restart, a native run was reattached
(`reattach_remote_runner`). Its workspace preparation failed at once
with `Failed query: select
pg_try_advisory_xact_lock(hashtextextended($1, 0)) as acquired params:
isolated_repository_source:<company>:<issue>`. The run ended
`failed`/`setup_failed`. The stack showed the "already held scope"
branch of `withWorkspaceSourceLocks`, not a new acquisition. The same
failure occurred earlier in the same wave on a different issue without a
restart.

**Steps to reproduce**
1. Call `withIsolatedRepositorySourceLock` for issue A. In its action,
start a detached promise that later calls
`withIsolatedRepositorySourceLock` for issue B.
2. Let the outer action return so its transaction commits.
3. Let the detached call run. It fails with `Failed query ...
pg_try_advisory_xact_lock`, cause `CONNECTION_CLOSED`.

**Expected behavior**
Work that starts after a source scope ends, or independent work such as
a run's preparation, acquires its own locks on its own connection.

**Environment**
Restart chaos campaign on Daytona (OpenCode, workload C1, fault K1,
comment trigger), embedded Postgres. Reproduced locally on embedded
Postgres by the new tests.

Related: #14910 (deferred comment wake reopens a closed issue). This PR
does not change wake promotion.

## What Changed

- `server/src/services/workspace-source-lock.ts`: a scope has an
`active` flag. The flag is cleared when its transaction ends.
`withWorkspaceSourceLocks`, `retainManagedProjectSource`,
`retainCurrentWorkspaceSource`, and `assertCurrentWorkspaceSourceHeld`
use only a live scope.
- `server/src/services/workspace-source-lock.ts`: new
`withIndependentWorkspaceSourceLocks`. It acquires locks in a new scope
and ignores an inherited one.
- `server/src/services/heartbeat/workspace-preparation.ts`: run
workspace preparation uses `withIndependentWorkspaceSourceLocks`.
- `server/src/services/heartbeat/run-completion.ts`: the "heartbeat
execution setup failed" log records a bounded driver `causeCode` (for
example `CONNECTION_CLOSED` or a SQLSTATE). It does not log the cause
text.
- Tests: two embedded-Postgres tests in
`isolated-remote-workspace-source-lock.test.ts`, one unit test in
`run-completion.test.ts`.

## Verification

- `cd server && npx vitest run
src/services/isolated-remote-workspace-source-lock.test.ts` — 4 passed.
- Red (old scope rule): "gives work detached from a finished scope its
own source lock" failed with `Failed query: select
pg_try_advisory_xact_lock(...)`, cause `CONNECTION_CLOSED`. This is the
production error.
- Red: "keeps an independent preparation's lock after the scope that
started it commits" failed because the lock was released with the other
transaction.
- `cd server && npx vitest run
src/services/heartbeat/run-completion.test.ts` — 95 passed.
- `cd server && npx vitest run
src/services/remote-workspace-online-source.test.ts
src/services/heartbeat/workspace-preparation.test.ts
src/services/isolated-repository-rehome.test.ts
src/services/isolated-task-directory-retention.test.ts
src/services/isolated-task-directory.test.ts
src/__tests__/execution-workspace-reopen.test.ts` — all passed.
- `cd server && npx tsc --noEmit` — clean.
- No live Daytona replay was run.

## Risks

- Low. A nested call inside a live scope behaves as before.
- A call that inherited an ended scope now opens its own dedicated
connection and lock transaction. Before, it failed.
- Run preparation never joins another operation's lock scope. No caller
relies on that today. If one did, it would now take a second,
independent lock and could see `workspace_source_busy` instead of
silently sharing.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edit, embedded Postgres tests).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

2026-11-06 23:08:57 -06:00

DottaandPaperclip

0b266bd084

fix(workspaces): keep source preparation locks active (#16956 )

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Each task prepares its workspace before it starts the provider.
> - Source preparation holds a transaction-scoped advisory lock on a
dedicated database connection.
> - Preparation can use other connections and the filesystem for longer
than PostgreSQL allows that lock connection to stay idle.
> - PostgreSQL then closes the connection and releases the lock while
preparation still runs.
> - This pull request reuses the workspace lock keepalive during source
preparation.
> - Slow preparation retains exclusion and returns its original result
or error.

## Linked Issues or Issue Description

**What happened?**

Tasks failed during workspace preparation with `CONNECTION_CLOSED` at
the configured 60-second idle transaction timeout. No provider had
started. A live retry showed the source-lock transaction idle while
authorization reads ran on another connection. Its advisory-lock key
matched the task's source-preparation key. PostgreSQL closed that
transaction at the timeout.

The same defect remains on current master. A real PostgreSQL regression
also shows that another preparation can enter after the timeout while
the original callback still runs.

**Expected behavior**

Source preparation must keep its advisory lock until the callback and
any pending keepalive query finish. The ordinary pool must remain
usable. The original preparation result or failure must survive.

**Steps to reproduce**

1. Set `idle_in_transaction_session_timeout` to one second in a
disposable PostgreSQL database.
2. Call `withIsolatedRepositorySourceLock` and keep its callback active
for longer than one second without querying the lock connection.
3. Try to acquire the same source lock from another operation.
4. On the old code, the second operation enters and the first
transaction fails with `CONNECTION_CLOSED`.

**Paperclip version or commit**

Reproduced on source at `cde944ed75648adfb569e86133adc8b118b55dd6`. The
repair is rebased onto `d855e550ad`.

**Additional context**

Refs #16340. That change introduced the keepalive for workspace
copyback. This PR applies the existing helper to the source-preparation
transaction.

## What Changed

- Read the configured idle timeout in the same query that acquires the
source lock.
- Keep that dedicated connection active during source preparation with
the existing helper.
- Add two real PostgreSQL regressions for exclusion, company
independence, a one-slot ordinary pool, nested acquisition, cleanup, and
preservation of callback errors.
- Update the execution contract and workspace verification maps.

## Verification

- Red before the production edit: both new tests failed. A competing
operation entered after the lock expired. A callback failure was
replaced by `CONNECTION_CLOSED`.
- Green after the repair: 86 tests passed across the source lock, source
admission, keepalive, and heartbeat workspace preparation suites.
- `pnpm -r typecheck`, `pnpm build`, and `node scripts/check-maps.mjs`
passed on final commit `2b5000ce25`.
- `git diff --check` passed. The lockfile is unchanged.
- After rebase, the same 86 focused tests passed. The two source-lock
tests passed again after the review cleanup change.
- Live acceptance on `7df94d7a06`: a fresh local instance on macOS
arm64, Node v26.4.0, native Codex `gpt-6.1-sol`, and permission mode
`never`. PostgreSQL had a one-second idle transaction timeout. A
temporary Git wrapper delayed repository reads by four seconds.
Workspace preparation after environment acquisition took 8,297 ms. The
real task completed, wrote and attached the correct line-count/SHA-256
report, reached `done` with native phase `committed`, and cleared its
execution lock. The attachment matched the workspace file and survived
browser reload. No dedicated lock connections remained.
- The live probe initially used `gpt-5.4`, which the ChatGPT account
rejected. Selecting the reported task model, `gpt-6.1-sol`, resolved
that readiness failure. The local Claude CLI has no login, so no live
Claude run is claimed.
- Full CI passed on `2b5000ce25`: all general, chat, workspace,
serialized-server, Runner, and E2E test shards, plus typecheck, build,
and canary packaging. Both Greptile and Paperclip Review rated this
commit 5/5 with no open findings. The Paperclip check concluded neutral;
two unrelated visual checks were skipped.
- The duplicate local `pnpm test:run` serial invocation was stopped with
SIGINT after the complete CI test shards passed. It did not finish
locally; no local full-suite pass is claimed. Local proof consists of
the focused real PostgreSQL tests, typecheck, build, map checks, and the
real task above.
- The final commit changes test teardown only; production code matches
the live acceptance commit.
- Initial sandbox runs could not open test database ports or build IPC
sockets. The same commands ran with local process access. The frozen
install also found an existing manifest/lockfile mismatch; dependency
resolution was local and the lockfile was restored.

## Risks

- The lock connection sends one small query per keepalive interval. The
helper derives that interval from the configured timeout and caps it at
30 seconds. It does no polling when the timeout is disabled.
- The helper joins an in-flight query before the transaction ends. A
slow database can therefore delay cleanup.
- This change does not disable database deadlines, retry preparation, or
recover an unexpected connection loss. It preserves existing admission
checks and company/source lock keys.
- Deployment and recovery of the reported tasks remain separate from
code verification.

## Model Used

OpenAI GPT-6 through Codex, with repository tools, code execution, and
browser verification. The exact served model variant and context window
are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-10-20 07:01:17 -05:00