Merge qualified remote workspace storage assembly for finite boot acceptance

* commit '343f1e57122f061cb229b8403fe2f522df178aea':
  feat(workspaces): keep remote files off the webserver disk (#18403)

doc/DATABASE.md

@@ -946,6 +946,44 @@ and process-start evidence proves the prior controller is gone, or when the
 lease expires. Recovery generation changes do not increment the independent
 provider-attempt counter.
 
+## Experimental remote workspace snapshots
+
+Migration [0421_bright_skaar.sql](../packages/db/src/migrations/0421_bright_skaar.sql)
+adds the company-scoped `execution_workspace_snapshots` ledger. It records the
+workspace, source operation/run, parent, generation, writer epoch, encrypted
+object receipt, hashes, manifest digest, and retention state. Integrity and key
+fields remain nullable while an upload is pending. Upload attempts record their
+candidate object keys before I/O so a lost reply can be reconciled and abandoned
+objects can be reclaimed. Object existence alone is not a committed save.
+
+`execution_workspaces.metadata.remoteWorkspace`, schema
+`paperclip.remote-workspace.v1`, holds the source, epoch, committed snapshot ID,
+remote allocation/path, writer run, and dirty flag. Paths in this record belong
+to the remote allocation. Save publication checks the current writer/epoch and
+atomically commits the ledger row, authority pointer, and operation receipt.
+`workspace_operations` journals preparation, save, restore, and local import;
+large file bodies stay outside the database. Ledger identities deliberately
+survive source-record deletion so object cleanup can finish.
+
+Snapshots stream through authenticated encryption into S3. Portable archives
+preserve supported files, modes, empty directories, confined symlinks, and Git
+checkout/index/object state, including unpushed history. Linked worktrees and
+object alternates are normalized into standalone Git storage. The versioned
+exclusion policy omits runtime transport material and proven rebuildable
+dependencies; unsupported entries fail the save.
+
+The current snapshot and active run/restore/recovery/unfinished-operation
+references pin saved data. Unreferenced superseded versions receive a seven-day
+grace. Retention uses committed producer notifications and actual deadlines;
+pinned history and an empty queue do not cause recurring scans. Deletion fences
+new adoption, rechecks references, and removes every recorded upload candidate
+and winning object within the company/workspace prefix.
+
+Database backups contain snapshot receipts, not S3 payloads or encryption keys.
+Disaster recovery requires the database, object store, and original master key.
+Unsaved live sandbox edits remain outside that saved-version guarantee. See
+[setup and explicit local import](DEVELOPING.md#experimental-native-remote-workspace-storage).
+
 ## Chat communication snapshots
 
 Chat communication guidance uses two additive columns: endpoint

doc/DEVELOPING.md

@@ -1629,6 +1629,84 @@ that the whole web server is disposable or that all execution and recovery paths
 survive web-server replacement. Run-log/trace storage and producer retention are
 separate changes.
 
+### Experimental native remote workspace storage
+
+`experimental.remoteWorkspaceStorageEnabled` is a managed feature, on by default
+for Cloud and self-hosted instances. New Paperclip Runner sandbox workspaces
+enroll automatically when durable storage is configured and the fresh source is
+supported. Existing host workspaces and configured local folders retain their
+legacy behavior until explicitly imported; the default does not convert them.
+The local, SSH, and plugin environment drivers, legacy adapters, and Dot keep
+their existing paths.
+An explicit stored `false` disables new enrollment, and a managed setting takes
+precedence over the stored choice. Enrolled workspaces retain remote authority
+when enrollment is disabled; losing a storage prerequisite fails their admission
+instead of falling back to a controller checkout.
+
+After enrollment, roll back only to binaries that understand the same remote
+workspace authority and saved-version format. Older local-only binaries are
+unsupported rollback targets.
+
+This mode requires an external PostgreSQL `DATABASE_URL`, configured S3 storage,
+and the same encryption master key on every replacement server. The managed
+profile requires `PAPERCLIP_SECRETS_MASTER_KEY` as described above. Self-hosted
+instances may instead preserve their existing configured
+`PAPERCLIP_SECRETS_MASTER_KEY_FILE` across replacements. An embedded database,
+local-disk object storage, or a newly generated replacement key cannot provide
+this durability contract. New workspaces continue through the legacy path when
+these prerequisites are absent. Validate S3 put/read/delete and database access
+with the deployment credentials before enrollment.
+Snapshot expiry also needs `s3:ListBucketVersions` on the bucket and
+`s3:DeleteObjectVersion` on the snapshot object prefix. S3-compatible endpoints
+must support version listing. Retention verifies that every version and delete
+marker is gone. Denied permissions and object locks keep a retry obligation.
+
+The sandbox holds current working files. PostgreSQL records its location,
+writer epoch, and committed snapshot; encrypted immutable snapshots live in S3.
+Preparation creates a saved baseline, and native finalization saves the stopped
+writer's files before releasing its persistence obligation. Replacing a
+webserver reconnects to the recorded allocation without copying files back to
+a controller checkout. Saved versions restore in a new sandbox without the old
+host, sandbox, or Git remote. Files changed since the last save still depend on
+the live sandbox: losing that allocation can lose those edits. There is no
+periodic workspace backup in this release.
+
+Existing local task workspaces require an explicit import. Run from a source
+checkout with the deployment's database, S3, and key configuration loaded:
+
+```sh
+pnpm --filter @paperclipai/server exec tsx scripts/migrate-remote-workspaces.ts \
+  --company-id <company-uuid> --workspace-id <workspace-uuid> --dry-run
+pnpm --filter @paperclipai/server exec tsx scripts/migrate-remote-workspaces.ts \
+  --company-id <company-uuid> --workspace-id <workspace-uuid> --apply
+```
+
+Dry-run is the default and reports eligibility and blocking reasons without
+importing. Apply retains the source folder, uploads an encrypted snapshot,
+independently restores it for verification, rechecks source/writer state, and
+atomically activates remote authority. Active execution, unresolved operations
+or recovery, unsafe paths, and explicitly concurrent shared writers block
+import. An interrupted apply retains its journal and admission hold; retry the
+same command to reconcile it before starting another writer.
+
+Supported sources include empty task folders, repository URLs, and managed
+mutable repositories from the same project. Configured local folders use the
+legacy path until imported. V1 rejects read-only cross-project referenced folders
+and custom cleanup/teardown commands on remote close. Automatic Core services, manual workspace jobs, services,
+and runtime commands are also rejected before any local directory access; run
+those commands through an agent task. One writer owns each logical
+workspace; separate workspaces can run concurrently. The Files panel identifies
+**Live workspace** and **Saved copy** with its save time. An unavailable dirty
+allocation is not silently replaced with an older saved copy.
+
+Use the focused `remote-workspace-*` and `remote-workspaces.integration` suites
+for storage, import, retention, and lifecycle regression checks. These checks do
+not qualify a deployed Runner/provider or a zero-outage webserver deploy. Keep
+the existing persistent volume until the other instance data paths and deploy
+handoff are separately qualified. See the
+[implementation contract](SPEC-implementation.md#experimental-native-remote-workspace-authority-2026-10-30)
+and [snapshot storage contract](DATABASE.md#experimental-remote-workspace-snapshots).
+
 ## Agent Artifact Uploads
 
 When an agent generates a file that a board user or reviewer should inspect as

doc/SPEC-implementation.md

@@ -2599,6 +2599,83 @@ Agents inspect/select workspaces and request contained repository preparation th
 
 See [the layer architecture](plans/2026-10-09-task-workspaces.md) for resolution rules, module ownership, persistence versions, rollout, and verification.
 
+### Experimental native remote workspace authority (2026-10-30)
+
+The managed feature `remoteWorkspaceStorageEnabled` defaults on for Cloud and
+self-hosted instances. New native Paperclip Runner sandbox workspaces use remote
+authority automatically when durable storage is configured and their fresh
+source is supported. Existing host workspaces and configured local folders keep
+their legacy path until explicitly imported. An explicit stored `false` disables
+new enrollment; managed configuration can override the stored choice. External
+PostgreSQL records workspace identity, location, writer epoch, and the committed
+saved version; S3 holds encrypted portable snapshots. A stable deployment
+encryption key is required. The webserver uses disposable transfer/restore
+scratch instead of a durable task checkout. New workspaces without these
+prerequisites retain the legacy path. The local, SSH, and plugin environment
+drivers, legacy adapters, and Dot retain their existing behavior.
+
+One writer owns a logical workspace. Preparation establishes a saved baseline;
+finalization saves supported changes before settling persistence. A webserver
+replacement reconnects to the existing allocation. A committed saved version can
+restore elsewhere without the old sandbox or Git remote, including supported
+dirty Git state and unpushed commits. Changes after the last save still depend
+on the live allocation; this release provides no periodic backup of those edits.
+Failed saving retains the source and recovery obligation before destructive
+cleanup. File browsing distinguishes live bytes from a saved copy and its time.
+After a final save, older failed copies can expire when all their save attempts
+have stopped. Cleanup retains copies whose upload completion is unknown. This
+includes operations from older builds without save-attempt records. A final
+save alone does not prove that an older upload has stopped.
+
+V1 supports plain task folders, Git sources, and managed mutable repositories
+from the same project. It rejects read-only cross-project referenced folders,
+explicit concurrent shared writers, and custom cleanup/teardown commands on
+remote close. Manual workspace runtime commands, jobs, and services are also
+unsupported in V1 and must fail before local filesystem work. Agent tasks remain
+the supported command path. Disabling new enrollment preserves existing authority
+and recovery. An enrolled workspace whose persistence prerequisites are lost
+must fail admission rather than fall back to a controller checkout.
+
+An imported saved Git workspace can resume from its saved files after its old
+local repository is removed. Local repository paths remain invalid for new
+remote acquisition. Saves reject unfinished Git operations, including rebase,
+cherry-pick, revert, merge, and bisect. The source files remain protected. Finish
+or abort the Git operation before saving. V1 does not claim a complete saved
+copy while these continuation records cannot be restored safely.
+
+Saved-copy browsing restores and verifies the complete archive for each request.
+Availability batches share one verified access per execution workspace within
+the request and allow at most eight remote workspace accesses. Each path still
+checks current authority. V1 has no extraction cache across requests. Large saved workspaces can therefore need
+substantial transfer time and temporary disk space for each browse request.
+Saved Files access admits at most two open extractions per process. It limits an
+encrypted archive to 512 MiB, its manifest to 4 MiB, and extraction to 10,000
+entries. It reserves at most 2 GiB of temporary space across those accesses and
+checks free disk space with a 256 MiB margin. Capacity failures stop before an
+object download. Cleanup releases capacity after it removes the owned files.
+These limits apply to saved-copy file access. Agent workspace saves and restores
+retain their existing archive limits.
+Each managed repository preparation also saves a complete workspace archive.
+These saves preserve each preparation boundary; V1 does not combine them or use
+incremental uploads.
+
+Snapshot retention removes all stored object versions and delete markers before
+it records deletion. S3 credentials need `s3:ListBucketVersions` and
+`s3:DeleteObjectVersion`, in addition to normal object access. S3-compatible
+endpoints must support object-version listing. Permission errors
+and object locks retain the cleanup obligation for retry. Other storage deletes
+keep their existing behavior unless they explicitly request all versions.
+
+Ordinary sandbox Runner tasks retain the existing API upload restriction:
+`call_api` cannot read a workspace file through `files[].path`. Use
+`register_deliverable` to register the file, then pass its `artifactId` to
+`call_api`. Plugin task providers keep their existing file-read capability.
+
+This feature does not remove the instance volume or qualify HTTP traffic drain,
+active provider handoff, or other instance data paths. See
+[setup/import](DEVELOPING.md#experimental-native-remote-workspace-storage) and the
+[snapshot storage contract](DATABASE.md#experimental-remote-workspace-snapshots).
+
 ## Company fast-response service
 
 Companies may configure an optional shared API connection and model for short acknowledgements of accepted human turns. Fast responses use direct SDK inference alongside normal agent work, with a three-second admission deadline, bounded prompts/output, current authorization and independent `fast_response` accounting. They are server-marked platform receipts attributed to the agent, never completion, interaction or recovery evidence. Company-sponsored unlinked external senders are explicit; linked-user permission denials are never bypassed. See [fast-responses.md](fast-responses.md) for provider, delivery, history and endpoint contracts.