Bound reverse ancestor discovery before materializing adjacency, deduplicate cycles and emit one canonical affected-issue batch per write. Overflow delegates complete company reconstruction to existing paging instead of dropping eligibility or creating per-ancestor worker debt. Prove real high-fanin producer bursts, PostgreSQL execution limits, canonical cyclic batches and far ancestor archive deadlines.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Keep bounded returned child facts independent of graph node and edge admission, and use the existing live-work predicate for running status, active runs and pending waits. Cover the opposite query ordering with 1,999 explicit blockers, fresh retention checks, no archive/outbox and unchanged SQL/resource bounds.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Reuse bounded timezone formatters and upcoming openings only within their exact searched closed ranges. Short-circuit empty windows while preserving the existing cold DST search and precise instant eligibility. Verify cost bounds, cache eviction, timezone transitions and twenty-card ordinary source delivery.
Co-Authored-By: Paperclip <paperclip@paperclip.ing>
Preserve company reconstruction for graph, run and eligibility changes. A real PostgreSQL producer regression verifies one assignment check among1000 open tasks, with broader eligibility reconstruction still intact.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Coalesce committed company hints in a fixed one-second window without flooring real deadlines. Verify one hundred ordinary issue/comment writes, company isolation, deadline preemption, retained hints and settled SQL quietness. Await actual evaluation I/O in fake-clock acceptance instead of jumping across connection timers.
Co-Authored-By: Paperclip <paperclip@paperclip.ing>
Revalidate the exact selected plugin before result persistence even when an allowed fallback still supports the object. Preserve due debt and token ownership so a fresh allowed resolver can proceed.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Use a dedicated advisory-lock connection so a one-slot write pool can finish evaluation and result writeback. Record internal deadline provenance so startup recovers lost membership hints without advancing saved policy or retry debt.
Co-Authored-By: Paperclip <paperclip@paperclip.ing>
Load company-partitioned decision/recovery links, run/agent visibility and newer-run probes once per page; share one blocker graph reconstruction and exact SQL timestamps across page entries. Keep fresh per-entry archive locks and revalidation. Add full100-source query budgets and late visibility/supersession safety controls.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
* origin/master:
fix(evals): support official TestOps job metadata (#18385)
fix(config): require shared storage for managed cloud (#18350)
feat(skills): make managed content independent of web-server disk (#18334)
Add a portable agent question and answer resumption journey (#18346)
fix(ci): isolate test selection and require current CI CODEOWNERS (#18376)
fix(evals): capture TestOps registration after watch (#18375)
fix(runner): allow session reset after confirmed task retirement (#18373)
fix(evals): batch task search result readback across renders (#18370)
fix(ui): keep task plugin views available for manual access (#18363)
test: wait for the evidence expiry worker to settle (#18241)
# Conflicts:
# server/src/services/company-skills.ts
# server/src/services/native-runtime/paperclip-runner-tool-authority.ts
Reuse the provider-owned request cancellation contract from merge-confirmation work. Keep host signals out of plugin RPC payloads, preserve refresh debt on shutdown, and target identities or known objects for source writes.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
* origin/master:
fix(evals): support official TestOps job metadata (#18385)
fix(config): require shared storage for managed cloud (#18350)
feat(skills): make managed content independent of web-server disk (#18334)
Add a portable agent question and answer resumption journey (#18346)
fix(ci): isolate test selection and require current CI CODEOWNERS (#18376)
fix(evals): capture TestOps registration after watch (#18375)
fix(runner): allow session reset after confirmed task retirement (#18373)
fix(evals): batch task search result readback across renders (#18370)
fix(ui): keep task plugin views available for manual access (#18363)
test: wait for the evidence expiry worker to settle (#18241)
Batch review and confirmation eligibility per company, scope decision and budget hints, and keep child blocker relation intents inside their owning transaction. Cover ordinary child, budget and execution producers plus full candidate-page query bounds; update startup/import fixtures for source-driven retention.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Bound GitHub object and repository probes, keep readiness independent of external reads, and page pending cards through the owning parser.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
* origin/master:
fix(config): require shared storage for managed cloud (#18350)
feat(skills): make managed content independent of web-server disk (#18334)
Add a portable agent question and answer resumption journey (#18346)
fix(ci): isolate test selection and require current CI CODEOWNERS (#18376)
fix(evals): capture TestOps registration after watch (#18375)
fix(runner): allow session reset after confirmed task retirement (#18373)
fix(evals): batch task search result readback across renders (#18370)
fix(ui): keep task plugin views available for manual access (#18363)
test: wait for the evidence expiry worker to settle (#18241)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents use installed skills and exact version pins during execution.
> - Managed skills still depended on mutable directories on the web
server.
> - A replacement server could lose acknowledged edits or remove an
unassigned skill whose directory was absent.
> - This pull request makes immutable versions the authority for managed
content and installed repository reads.
> - The server can rebuild delivery files from durable content after
replacement.
> - This establishes skills readiness for ephemeral web servers. General
server draining and workspace recovery remain separate work.
## Linked Issues or Issue Description
**Subsystem affected**
Skills services, shared contracts, database receipts, Skill Studio, and
runtime delivery.
**Problem or motivation**
A fresh web server must discover, read, edit, and deliver protected
installed skills using the same database, object storage, secrets, and
supported application artifacts. It must not need its predecessor's
files or process memory.
**Proposed solution**
Use `managed` ownership with a null source locator. Read installed
content by immutable version. Prepare packages before a short
publication transaction. Persist execution selections before
materialization. Recover exact pinned runtime assets and reject deletion
while active or recovering pins exist.
**Alternatives considered**
Reconstructing a mutable installed directory preserves the old
source-of-truth ambiguity. Permanent dual writes create two competing
authorities. This change uses one managed model after a bounded legacy
conversion.
**Roadmap alignment**
This improves the existing Skills Manager, Studio, version history, and
Cloud execution capabilities. The maintainer supplied the implementation
contract. This does not add a separate skills product or deploy fleet
changes.
Related work: #18082 supplies immutable package storage. #17273
addresses edits outside the API for linked local skills; this PR keeps
that source category separate. #2648 is a closed earlier
ephemeral-filesystem proposal.
## What Changed
- Add managed ownership and focused content publication and execution
materialization modules. Keep the archive format and package storage
tables.
- Publish managed create, save, delete, fork, rename, restore, and
catalog content with version checks. Add durable create receipts.
Preserve file-save receipt responses.
- Return file version identities. Keep Studio drafts after a conflict
and provide an explicit reload action. A delayed reload cannot overwrite
newer typing or navigation.
- Persist exact runtime selections, including empty selections. Rebuild
pinned assets and verify their admitted digests on a replacement server.
- Preserve source access on forks and disconnected copies. Keep the
existing workspace policy category for managed skills.
- Add bounded conversion, eligibility, and independent verification
phases. Load persisted Cloud configuration in the standalone CLI without
seeding or changing it. Reject partial current versions against
installed inventories; recover only complete saved content or the exact
source. Document storage migration blockers and the compatible rollback
floor.
- Add separate-process PostgreSQL/S3 qualification and update domain,
retention, policy, and UI regressions.
- Pass the plugin worker manager into Studio execution, admit restricted
`skill_test` mode through native Runner contracts, and preserve admitted
scope across native and REST tools. Native tests cannot create ordinary
tasks, browse company data, or use external tools.
- Select the exact durable Studio revision even when unassigned or Beta
skills is off. Persist Studio output and audit atomically with native
task completion, preserving the result on replay.
## Verification
- Final revision `13e8af480b88ba6fea5737bac4bd81104d604792` passed full
CI: all 49 workflow jobs completed successfully ([run
41127396098](https://github.com/paperclipai/paperclip/actions/runs/41127396098)).
This includes typecheck, test suites, Runner checks, browser checks and
build. Greptile review 15 is 5/5 on that same revision, with zero
unresolved threads. The PR is ready for review.
- Local workspace typecheck and build passed after integrating master.
Focused checks passed: 33 execution/completion cases, 52 Runner HTTP
cases, 16 managed-content cases, 17 Studio cases, the new real-process
CLI regression, map checks and UI token gates. The CLI regression proves
inspection does not seed missing Cloud configuration. Server typecheck
passed after that fix.
- The independent-process PostgreSQL/S3 matrix passed through normal
HTTP skill routes. It covers A → B → C replacement, lost commit
responses, upload crashes, concurrent saves, binary contents and
executable delivery. Metadata listing and warm selected delivery retain
zero object downloads. Local timing measurements are recorded; they do
not establish hosted latency.
- Hosted A → B → C passed on the owned disposable test tenant with
external Postgres, S3 and a real remote Paperclip Computer. Predecessor
temporary homes were absent. C read and executed B's acknowledged
managed save. New runs used repository v1 while v2 was installed. Hashes
matched an independently checked exact upstream commit.
- Compatible image rollback to `6518b323` passed. Studio retained the
unassigned fork's C save. A fresh remote run executed the newer managed
content and historical repository pin. The actual shell event confirmed
expected hashes, stdout, exit 0 and mode 0555.
- Runtime candidate `9200e3fcb4` deployed successfully ([Cloud run
41124147876](https://github.com/paperclipai/paperclip-cloud/actions/runs/41124147876)).
The final Studio test delivered only the unassigned fork's exact v3,
executed its script and persisted a successful output snapshot.
Independent command-event inspection confirmed all three hashes, stdout,
exit 0 and mode 0555. All 14 installations were version-backed.
Independent verification checked all 25 versions: 23 remote objects, two
bundled database snapshots, zero blockers.
- The deployed image is `9200e3fcb4`. Later master integration, the
client-only delayed-reload draft guard and the CLI bootstrap have
separate automated proof. The CLI bootstrap was also exercised against
the tenant's stored configuration and storage through the same read-only
services. This PR does not claim that the deployed image is the final PR
revision.
- Live Studio checks exposed worker wiring, admission, access scope,
exact selection and completion-publication gaps. They are fixed. Failed
attempts remain in the evidence. The full local suite exceeded 80
minutes under host contention and was stopped. Later combined/isolated
local attempts also timed out; they are not counted as passes. Full CI
covers the integrated suites. Earlier CI found an obsolete restart-scope
fixture, which was corrected and passed all 52 HTTP cases.
- Reproduction and migration:
[`doc/skill-ephemeral-web-servers.md`](doc/skill-ephemeral-web-servers.md).
Source revisions, deployments, versions, run IDs, hashes and failure
history:
[`doc/plans/2026-10-30-ephemeral-web-skills.md`](doc/plans/2026-10-30-ephemeral-web-skills.md).
## Risks
- Legacy conversion requires stopped admission and retired
filesystem-authoritative writers. Pre-conversion binaries are not
supported rollback targets afterward.
- Cloud readiness requires remote object storage. Local provider
receipts and provider mismatches remain blockers until objects are
copied and verified.
- Linked local sources retain filesystem semantics and are outside the
cloud guarantee.
- Cold reads still fetch the existing archive format. Warm runtime
preparation retains the verified-cache path.
- Bundled reconciliation preserves an installed head. Existing release
snapshots remain selectable. An older server must not replace a newer
head with its shipped files.
- No production volumes, fleet defaults, or general deployment
coordination change in this PR.
## Model Used
OpenAI Codex, GPT-6 family, with reasoning, repository tools, code
execution, and browser use. The exact backend model ID and
context-window size are not exposed in this session. No subagents were
used.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Operators use run logs and restricted provider traces to inspect
agent work.
> - Local log and trace files depend on the web server's persistent
volume.
> - That dependency prevents overlapping volume-backed Railway
deployments.
> - This PR reuses the instance S3 store for batched log and trace
payloads with conditional publication.
> - Existing Postgres rows retain their contracts; native export
recovery reuses events already committed before Runner acknowledgement.
> - S3-configured servers automatically copy and verify retained history
before a later deployment cutover.
## Linked Issues or Issue Description
**Subsystem affected**
Server run logs, restricted provider traces, storage providers, and
migration tooling.
**Problem or motivation**
Replacing the web server and its disk must preserve committed log output
and retained traces. The current S3 mirror can repeatedly upload a
growing file. Moving those payloads into Postgres would add database
writes, WAL, replication, and backup work.
**Proposed solution**
Reuse the existing instance storage configuration, S3-compatible
provider and credential chain. Store new run logs under run-logs/ and
opted-in raw traces under provider-traces/, below the existing prefix.
Cloud keeps its existing bucket and bucket-scoped credentials per stack.
Add no new configuration switches. Store immutable payload segments and
bounded binary range indexes directly in S3. Publish each batch with an
ETag-conditional head. Keep existing database ownership and lifecycle
records. Recover canonical native exports from their existing validated
control event source, checkpointing export progress in S3. Add no new
payload writes to Postgres. Preserve local defaults and migrate settled
history with verified byte counts and digests.
**Alternatives considered**
Postgres payload chunks add control-plane load. Repeated whole-file
mirrors amplify transfer work. The first index prototype left superseded
index nodes; the final binary index keeps all normally published nodes
reachable.
**Roadmap alignment**
This is a storage prerequisite for replacing the main web/API server. It
does not enable deployment overlap, detach a volume, or change native
Runner protocol acknowledgements. The maintainer approved this
implementation scope. ROADMAP.md has no duplicate implementation item.
Related PR: #16770 improves the existing in-flight mirror. This PR
preserves that legacy backend and uses the existing instance S3 profile
for new journals.
## What Changed
- Automatically select S3 segments for new logs and opted-in traces when
instance file storage uses S3. Reuse bucket, endpoint, region, prefix,
path style and credentials. Remove separate backend switches. Keep
local-disk instances local.
- Share the existing storage configuration resolver with product files.
Preserve config-file settings, environment precedence and the existing
root prefix. Keep pre-existing run-log mirror settings for legacy
history.
- Add bounded buffering, conditional ownership fencing, immutable range
reads, stable byte offsets, and completion hashing.
- Dispatch all existing log consumers through the recorded backend.
Coalesce output progress writes and publish live log messages after
storage receipts.
- Recover interrupted canonical native exports from their existing
durable source. Exclude separate credential, cleanup, and reprojection
records. Replay missing callback ranges and deduplicate late delivery.
Repair settled archives after controller lease expiry. Bound shared
repair work and stop it when all readers cancel. Serve the published
prefix when best-effort repair fails.
- Add encrypted optional trace channels, replacement reads, durable
deletion tombstones, and deadline-driven local/external expiry cleanup
independently of heartbeat scheduling. Stream historical source and
remote verification. Propagate worker cancellation through trace open,
conditional-write probing, upload, restore and reads. Fence deletion
even before the first object head exists; abort and drain owned
captures.
- Retry transient storage errors, distinguish 409 from failed
preconditions, and backpressure bounded log admission. Track interrupted
trace imports explicitly so ordinary local deletion stays independent of
S3.
- Automatically migrate retained history through the existing delivery
coordinator. Process one log and one trace per batch. Retry unresolved
history with backoff and stop scans after completion. Keep the
inventory/import command as optional diagnosis and repair. Verify source
and remote bytes before switching references. Verify finalized native
size and digest before setting its export cursor. Require ordered source
coverage when completeness metadata is missing. Preserve original trace
expiry. Block active, missing, corrupt, and unproven sources.
- Preserve identity and run-secret redaction in live and recovered
native logs. Register runtime log redaction secrets once through the
existing encrypted registry. Keep agent private keys in their existing
owner.
- Add storage, migration, retention, UTF-8, and route-privacy coverage,
plus operating documentation and qualification tooling.
- Add no schema, payload table, per-record SQL, production dependency,
or lockfile change.
## Verification
- Final fixes at `b3a44c898e` passed all 116 selected startup,
retention, attribution and cancellation controls. The disabled-scheduler
cleanup and concurrent-repair defects reproduced red before the fixes. A
timed-out repair still prevents later storage reads.
- The merged source passed 124 storage and real-Postgres Cloud
configuration tests. These include automatic history migration, existing
environment/file vault keys, source-directory removal, five real-SDK
stalled-request cancellation cases, shared capability-probe cancellation
and multi-chunk trace ordering. Saved Cloud S3 settings work after the
legacy config file is gone.
- Workspace typecheck and full build passed at the merged parent
`19f702d780`. Server typecheck and build passed again at `b3a44c898e`.
- Actual app acceptance: finish a process-agent run through the normal
API, stage its retained local-file input, then restart. Automatic
migration reports one migrated item and zero blockers. Remove the
original web home and restart at `b3a44c898e` with an empty home, the
existing S3 profile and the existing vault key file. The normal log API
preserves all five producer lines and exactly 788 bytes, with SHA-256
`74d9ae469da28bfe89b9b50a0e132aec24fe9e6d686496ed5e4ffa6167151036`. No
log-specific storage settings are present.
- Actual expiry acceptance: with heartbeat scheduling disabled, a trace
remains unexpired after startup and later reaches its fixture deadline.
The worker marks the row deleted and tombstones both S3 channel heads.
The retained run log stays byte-identical. The process fixture emits no
provider protocol frames, so nonempty encrypted payloads are qualified
separately through the real SDK round trips.
- Isolation tests use identical run identities in two stack buckets.
Cleanup in one existing prefix leaves another prefix, another company
and product files intact. Read-only Cloud source inspection confirms a
bucket and bucket-scoped credentials per stack. Four existing Cloud
storage environment-contract tests passed. No Cloud provisioning changed
and no new live vendor permission probe was performed.
- Earlier qualification passed 1,014 affected storage/native tests and
27 privacy-route tests with isolated Docker Postgres. An abrupt process
test commits 1,000 records, receives receipts, dies by SIGKILL and loses
its directory; another process reads every byte and continues. The real
Rust Runner experiment acknowledges 32 records while the export cursor
is zero, then continues the same Runner PID after controller loss. All
64 records export once with exactly 64 existing control rows. Repeated
snapshot/Stop commands completed in 162/81 ms.
- Emulator workload: 1,000 burst records used two batch PUTs. Eight
steady streams produced 240 records with 104 conditional PUTs and 72
GETs over 6.9 seconds. These are synthetic operation counts, not fleet
sizing or provider prices.
- Failure evidence is retained. Compact native events over 64 KiB,
missing file-key support, disabled-scheduler retention and repair
failure handling all reproduced before their fixes. Fixture repairs
addressed cold module setup, localhost/IPC permissions, the S3
bucket-name limit and a mode-0700 crash-test directory. Full local `pnpm
test:run` was stopped after host PostgreSQL semaphore exhaustion;
affected suites used isolated Docker databases instead. Native package
library symlinks were restored with the package's standard hydration
helper.
- The `19f702d780` browser session-recovery case timed out while saving
a task title. All eight browser shards passed on `b3a44c898e`, including
that case, with no UI behavior or assertion change. Fresh-head CI passed
at `b3a44c898e`: 55 successful checks, three skipped checks and one
neutral check. The branch is conflict-free.
- Map and module-boundary checks passed. Current-head Greptile is 5/5
with zero unresolved review threads.
Reproduce the opt-in storage suites with
`PAPERCLIP_LOG_TEST_S3_ENDPOINT` and a dedicated
`PAPERCLIP_RUN_EVIDENCE_MIGRATION_DATABASE_URL` ending in `_test`. See
`doc/run-evidence-storage.md` for operating limits, automatic migration,
optional diagnostics and rollout qualification.
## Risks
- Keep the existing per-stack bucket, prefix and vault key stable across
replacements. Relocating that storage is a separate whole-instance
operation.
- This release automatically changes new log storage for existing
S3-configured instances. Local-disk instances remain local. Canonical
native source replay is qualified with a deterministic real Runner;
actual remote execution and production-provider qualification remain
rollout gates. Generic server-only diagnostics and optional raw trace
tails without a surviving source can be incomplete after a crash. This
does not establish full zero-downtime deployment.
- Before deploying this release, qualify the actual S3 provider. It must
enforce conditional writes and pass read, version, deletion, workload,
and latency qualification. A capability probe rejects providers which
ignore preconditions. Moto is emulator evidence only.
- Dirty output is bounded and unconfirmed until S3 publication. Storage
failure leaves persistence incomplete while task results and usage can
settle.
- Trace expiry is not yet represented by Cloud's external wake contract.
Outstanding trace deadlines conservatively block sleep.
- Automatic history copying requires the original source volume. Keep it
until all retained resources pass verified import and clean-filesystem
reads. Stop old binaries from creating new local-file runs. Missing or
corrupt history remains blocked and visible in server logs. Rollback
requires a compatible reader against the same external state. Ordinary
object deletion does not erase retained historical object versions or
backups.
## Model Used
OpenAI Codex, `gpt-6.1-sol`, with reasoning, tool use, code execution,
and browser verification. The runtime did not expose a context-window
size.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
