PaperClipAI/doc/workspace-restore-locks.md

# Workspace restore locks

Workspace restore and agent-file collection serialize writes to each canonical
target directory. Their lock files live in
`<instance root>/locks/directory-merge`, outside the writable target. All writers
must use the same instance root and a filesystem with reliable SQLite file
locking. Network filesystems that do not provide that locking are not supported.

Each target has a permanent `<hash>.lock.sqlite` file. An open SQLite
`BEGIN IMMEDIATE` transaction holds its reserved file lock for the entire write
operation. Contenders retry without blocking the Node event loop, up to the
existing 30-second limit. Closing the connection releases the lock; the operating
system also releases it when the process exits or crashes. Independent targets
use different files and can proceed concurrently.

This uses the same built-in `node:sqlite` dependency as workspace manifests.
See [SQLite file locking](https://www.sqlite.org/lockingv3.html) for the reserved
lock contract. No application database or schema migration is involved.

The adjacent `<hash>.lock.owner.json` records a PID and creation time only for
bounded timeout diagnostics. Missing, stale, or incorrect metadata cannot grant
or retain ownership. PID reuse, PID namespaces, and wall-clock changes do not
decide whether a writer owns the lock.

A timeout error (`ERR_WORKSPACE_RESTORE_LOCK_TIMEOUT`) carries that holder
evidence. `directoryMergeLockTimeoutDiagnosticsFor` reduces it to the closed
agent-directory operation label, the owner state (`alive`, `dead`, `unknown`,
`missing`, or `invalid`), whether this process knows of a local holder, whether
the owner is this process or predates it, the owner's age, and the contender's
wait in milliseconds (each capped at seven days). Unknown labels and invalid
values are omitted. The record never includes the owner's PID or file paths.
Run-failure reports use it for their `restoreLock*` fields. When the server
reports an uncaught timeout to Sentry, the event gets this record as an
event-local `directory_merge_lock` context and an
`error_code=workspace_restore_lock_timeout` tag. The error message, which names
the lock file, and default grouping are unchanged. An owner much older than the
contender's wait points to a stuck holder; a recent owner points to contention.
Reporting does not change acquisition, timeouts, or retries.

An unchanged returned snapshot still acquires the target writer lock and runs
both authorization/publication hooks. If every returned path, type, file hash,
file mode and symlink target matches the baseline, there is no incoming delta.
The merge then skips the host snapshot scan and leaves concurrent host edits
untouched. Changed exports retain the full conflict preflight. Checkpoint and
archive validation still run before the merge.

**Never delete, replace, or move a `.lock.sqlite` file while an instance can
write to it.** Its stable inode is part of the locking contract. Replacing it
could let two writers lock different files for the same target. Files remain
after release, including for targets that no longer exist. Their diagnostic
owner sidecars are normally removed after release.

## Upgrade from directory locks

Older versions created `<hash>.lock/owner.json` and checked only whether the
recorded PID existed. A server restart could reuse that PID and leave an orphaned
lock permanently protected. Those records cannot establish a process lifetime or
PID namespace, so the new implementation never guesses that a legacy holder is
dead. An existing legacy directory continues to block admission.

Do not run old and new lock protocols concurrently against the same instance
root. An old process does not participate in the SQLite lock protocol.

1. Drain and stop **all** old server and worker processes that can write to the
   instance root, including processes on other hosts or in other containers.
2. Preserve any unfinished-run evidence needed for recovery. After all writers
   are stopped, move leftover legacy `<hash>.lock/` directories to an operator
   scratch directory outside the lock root. Do not remove `.lock.sqlite` files.
3. Start all writers on the new version. Verify that a run completes both the
   provider turn and file collection/restore. A successful model response alone
   does not prove that its local file changes were saved.

The same drain requirement applies to rollback. Existing permanent SQLite files
can remain on disk; older versions ignore them. Do not infer that a legacy lock
is safe to remove from its age, an absent PID, or a successful task response.

This change prevents new orphaned ownership. It cannot recover file changes
that an earlier failed collection discarded.

## Required remote restore failures

The copy-back first repairs itself. While the run still holds its merge
baseline, a workspace copy-back that fails before writing any host file is
retried up to three times in total, with a short backoff. This covers dropped
transfers, provider errors, and a merge lock held by another restore. An unsafe
archive, a permission failure, or any failure after the merge started writing
host files is never retried. Only a copy-back that still fails reaches the
repair hold below. Adapter environment Tests restore into a throwaway temp
workspace, so they opt out of these retries and report a failure at once.

When a legacy sandbox run cannot copy required workspace files back, the server
keeps the original allocation for repair. It writes the failed run, the exact
lease retention intent, and a board-owned repair hold in one transaction. The
hold survives reassignment and conversation resets and coexists with newer
recovery incidents; it never supersedes them.
Cleanup stops that allocation and verifies its provider receipt. It does not
use ordinary ephemeral destruction or return the source to warm reuse. Stop
failures use the existing cleanup retry and backoff policy.

The retained source uses `retain_on_failure`. A stopped source can still incur
provider storage charges. It stays available until an operator recovers or
explicitly discards it; elapsed time never proves that its files were copied.
The task's `GET /api/issues/:id/recovery-actions` response lists these repair holds
alongside any newer active incident, including the original lease IDs. Inspect
those exact leases through `GET /api/environment-leases/:leaseId`,
confirm that cleanup stopped them, recover the missing files, and record
`workspaceRepairEvidence` through the existing recovery-action resolution API.
Send `POST /api/issues/:id/recovery-actions/resolve` with the exact recovery action
ID, `outcome: "restored"`, the task's **unchanged**
`sourceIssueStatus`, and `executionReconciliation` for the recorded source run.
Only a board operator can record this repair. It preserves the current task
owner, conversation generation, status, and execution locks. It does not queue a
continuation or replay the stopped run; after repair, any new work still follows
the normal admission rules.
The original legacy run may be `succeeded` while required files still need
repair. Verified file-only repair accepts that terminal result and keeps it
unchanged. It still requires the retained-source record, exact remote stop
receipts, and recovered-file evidence. A successful result does not permit a
normal execution retry, and a queued or running source cannot be repaired this
way. Later saved messages can continue through ordinary admission after the
file hold clears.
The task shows a compact operator-attention notice and an Inspect run link.
Diagnostics and repair guidance stay under Recovery details. It does not ask the
task user to certify recovered infrastructure files. Operators use the guarded
resolution API with recovered-file evidence; the API verifies the original stop
receipts and returns the exact action and lease IDs. Provider access may require
an instance administrator or Cloud support.
Then use the provider's operator console to remove that exact retained
allocation when its files are no longer needed. Verify its immutable provider
allocation ID against the recorded lease before removal. The ordinary reusable
sandbox cleanup does not select `retain_on_failure` sources; do not use
environment deletion as proof that this allocation was removed. Keep the
environment and its provider credentials until recovery is complete. Recording a
repair decision does not automatically destroy it.

A new message, Retry, or `/new` cannot clear this workspace repair hold. Ordinary
local lock-timeout retries and runs without a required remote copy-back failure
keep their existing policy. This change preserves future failures; it cannot
recover files from an allocation that was already destroyed.