PaperClipAI/doc/workspace-restore-locks.md

Workspace restore locks

Workspace restore and agent-file collection serialize writes to each canonical target directory. Their lock files live in <instance root>/locks/directory-merge, outside the writable target. All writers must use the same instance root and a filesystem with reliable SQLite file locking. Network filesystems that do not provide that locking are not supported.

Each target has a permanent <hash>.lock.sqlite file. An open SQLite BEGIN IMMEDIATE transaction holds its reserved file lock for the entire write operation. Contenders retry without blocking the Node event loop, up to the existing 30-second limit. Closing the connection releases the lock; the operating system also releases it when the process exits or crashes. Independent targets use different files and can proceed concurrently.

This uses the same built-in node:sqlite dependency as workspace manifests. See SQLite file locking for the reserved lock contract. No application database or schema migration is involved.

The adjacent <hash>.lock.owner.json records a PID and creation time only for bounded timeout diagnostics. Missing, stale, or incorrect metadata cannot grant or retain ownership. PID reuse, PID namespaces, and wall-clock changes do not decide whether a writer owns the lock.

Never delete, replace, or move a .lock.sqlite file while an instance can write to it. Its stable inode is part of the locking contract. Replacing it could let two writers lock different files for the same target. Files remain after release, including for targets that no longer exist. Their diagnostic owner sidecars are normally removed after release.

Upgrade from directory locks

Older versions created <hash>.lock/owner.json and checked only whether the recorded PID existed. A server restart could reuse that PID and leave an orphaned lock permanently protected. Those records cannot establish a process lifetime or PID namespace, so the new implementation never guesses that a legacy holder is dead. An existing legacy directory continues to block admission.

Do not run old and new lock protocols concurrently against the same instance root. An old process does not participate in the SQLite lock protocol.

1. Drain and stop all old server and worker processes that can write to the instance root, including processes on other hosts or in other containers.
2. Preserve any unfinished-run evidence needed for recovery. After all writers are stopped, move leftover legacy <hash>.lock/ directories to an operator scratch directory outside the lock root. Do not remove .lock.sqlite files.
3. Start all writers on the new version. Verify that a run completes both the provider turn and file collection/restore. A successful model response alone does not prove that its local file changes were saved.

The same drain requirement applies to rollback. Existing permanent SQLite files can remain on disk; older versions ignore them. Do not infer that a legacy lock is safe to remove from its age, an absent PID, or a successful task response.

This change prevents new orphaned ownership. It cannot recover file changes that an earlier failed collection discarded.

Required remote restore failures

The copy-back first repairs itself. While the run still holds its merge baseline, a workspace copy-back that fails before writing any host file is retried up to three times in total, with a short backoff. This covers dropped transfers, provider errors, and a merge lock held by another restore. An unsafe archive, a permission failure, or any failure after the merge started writing host files is never retried. Only a copy-back that still fails reaches the repair hold below. Adapter environment Tests restore into a throwaway temp workspace, so they opt out of these retries and report a failure at once.

When a legacy sandbox run cannot copy required workspace files back, the server keeps the original allocation for repair. It writes the failed run, the exact lease retention intent, and a board-owned repair hold in one transaction. The hold survives reassignment and conversation resets and coexists with newer recovery incidents; it never supersedes them. Cleanup stops that allocation and verifies its provider receipt. It does not use ordinary ephemeral destruction or return the source to warm reuse. Stop failures use the existing cleanup retry and backoff policy.

The retained source uses retain_on_failure. A stopped source can still incur provider storage charges. It stays available until an operator recovers or explicitly discards it; elapsed time never proves that its files were copied. The task's GET /api/issues/:id/recovery-actions response lists these repair holds alongside any newer active incident, including the original lease IDs. Inspect those exact leases through GET /api/environment-leases/:leaseId, confirm that cleanup stopped them, recover the missing files, and record workspaceRepairEvidence through the existing recovery-action resolution API. Send POST /api/issues/:id/recovery-actions/resolve with the exact recovery action ID, outcome: "restored", the task's unchanged sourceIssueStatus, and executionReconciliation for the recorded source run. Only a board operator can record this repair. It preserves the current task owner, conversation generation, status, and execution locks. It does not queue a continuation or replay the stopped run; after repair, any new work still follows the normal admission rules. The board currently shows repair guidance and an Inspect run link, not a repair form. Recording the decision is an operator API/runbook workflow; the recovery API response contains the exact action and lease IDs. Provider access may require an instance administrator or Cloud support. This change does not add a repair UI. Then use the provider's operator console to remove that exact retained allocation when its files are no longer needed. Verify its immutable provider allocation ID against the recorded lease before removal. The ordinary reusable sandbox cleanup does not select retain_on_failure sources; do not use environment deletion as proof that this allocation was removed. Keep the environment and its provider credentials until recovery is complete. Recording a repair decision does not automatically destroy it.

A new message, Retry, or /new cannot clear this workspace repair hold. Ordinary local lock-timeout retries and runs without a required remote copy-back failure keep their existing policy. This change preserves future failures; it cannot recover files from an allocation that was already destroyed.