Distinguish a budget or agent refusal saved on the existing receipt from genuine recovery failure. Preserve HTTP409 messages and details, grant each new wait reason its existing finite grace, and retain failure backoff for unsaved or unexpected errors. Prove actual Retry admission,25-minute SQL quiet, independent warm producer-to-process release, missing/cancelled receipts and real PostgreSQL write failure.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Bound reverse ancestor discovery before materializing adjacency, deduplicate cycles and emit one canonical affected-issue batch per write. Overflow delegates complete company reconstruction to existing paging instead of dropping eligibility or creating per-ancestor worker debt. Prove real high-fanin producer bursts, PostgreSQL execution limits, canonical cyclic batches and far ancestor archive deadlines.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Keep bounded returned child facts independent of graph node and edge admission, and use the existing live-work predicate for running status, active runs and pending waits. Cover the opposite query ordering with 1,999 explicit blockers, fresh retention checks, no archive/outbox and unchanged SQL/resource bounds.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Reuse bounded timezone formatters and upcoming openings only within their exact searched closed ranges. Short-circuit empty windows while preserving the existing cold DST search and precise instant eligibility. Verify cost bounds, cache eviction, timezone transitions and twenty-card ordinary source delivery.
Co-Authored-By: Paperclip <paperclip@paperclip.ing>
Preserve company reconstruction for graph, run and eligibility changes. A real PostgreSQL producer regression verifies one assignment check among1000 open tasks, with broader eligibility reconstruction still intact.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Reconcile failed ephemeral Cloud leases through a signed cleanup-only absence observation when local historical inspection is unavailable. Preserve failure outcomes, exact resource bindings, retention and row-version fences; grant no run-continuation authority.
All 54 current-head checks/statuses passed. Greptile is 5/5, Medium, with no findings or unresolved threads. The 38 focused tests pass; both new reconciliation cases fail without the fallback. Isolated baseline and branch idle suites each pass all 225 cases. The original broader local teardown timeout is retained in the PR, and the full local run remains in progress.
Cloud protocol support is already serving in staging. Exact candidate composition and hosted provider sleep/wake qualification follow this merge. Production has not been changed.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Release workflows publish npm packages and multi-platform images.
> - A master push can finish its image before the npm canary dispatch
starts.
> - The dispatch currently builds that same source again on two runners.
> - This PR verifies and reuses the signed master image by digest.
> - Missing images retain the native build path. Channel promotion still
follows npm.
## Linked Issues or Issue Description
**What existing behavior does this improve?**
The Docker workflow dispatched after npm canary publication.
**Current behavior**
Every canary dispatch builds both architectures, even when master has
already published the same source. This consumes runner capacity and
delays the image channel update.
**Proposed behavior**
Resolve the full-SHA master image. Verify its platform index, digest and
signed source identity. Retag that immutable digest. Build normally when
the image does not exist.
**Reason and benefit**
Remove duplicate image builds while keeping missing-image recovery and
the current npm channel check. A recent duplicate used about 25
runner-minutes. The saving depends on whether the master image exists.
## What Changed
- Add a canary-only reuse job before the native matrix.
- Require signed provenance from the canonical Docker workflow on master
at the exact source and signer commit. Read the proof from GHCR.
- Reuse the validated digest or build both platforms when the full-SHA
image is absent. Stop on registry and verification errors.
- Keep the serialized npm dist-tag promotion after either path. Keep
other build lanes independent.
- Add failure tests, register them in CI and update release
documentation.
## Verification
- Passed 603 tests: `node --test '.github/scripts/tests/*.test.mjs'
scripts/standard-image-contract.test.mjs
scripts/reuse-canary-image.test.mjs
scripts/__tests__/release-verify-workflow.test.mjs
scripts/cloud-source-verification.test.mjs`.
- Passed `actionlint -shellcheck='' .github/workflows/docker.yml
.github/workflows/pr-trusted.yml` and `git diff --check`.
- Full actionlint reports the same six existing shellcheck findings as
the unchanged master workflow. No new findings.
- A published master image passed the exact provenance command. This was
a read-only check. No images were retagged.
- Gitleaks and manual diff inspection found no secrets or private
references.
- Attempted `pnpm -r typecheck`, `pnpm test:run` and `pnpm build`. They
cannot start because this isolated checkout has no installed
dependencies (`tsc` and `tsx` are absent). Repository CI passed on
`3c46f0152d4db94a7f393bc79f466aa6c2c745e2` after the documented
browser-test rerun. All checks are complete. The original failure and
prior baseline occurrence remain recorded in the PR comments; its root
cause remains unknown.
## Risks
- The image can appear just before its attestation. Verification then
fails closed. Rerun after the master producer completes.
- Reuse preserves the first build's resolved dependencies and tool
versions. It intentionally does not rebuild the same source to refresh
floating inputs.
- A skipped or unfinished master image still needs a full build. No
speedup is claimed for that path.
- End-to-end reuse and runner savings require observation after merge.
No live publication was performed.
## Model Used
OpenAI `gpt-6.1-sol`, through the Codex harness in T3 Code, with xhigh
reasoning effort and shell/code tools. The context window size is not
exposed by this runtime.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Co-authored-by: Paperclip <noreply@paperclip.ing>
Coalesce committed company hints in a fixed one-second window without flooring real deadlines. Verify one hundred ordinary issue/comment writes, company isolation, deadline preemption, retained hints and settled SQL quietness. Await actual evaluation I/O in fake-clock acceptance instead of jumping across connection timers.
Co-Authored-By: Paperclip <paperclip@paperclip.ing>
## Thinking Path
> - Paperclip lets people direct agents while their work is active.
> - Product E2E checks must prove that saved user input reaches the
intended work.
> - The existing active chat story depends on a file in a local agent
workspace.
> - That setup cannot qualify an attached deployment with remote agent
execution.
> - This change adds one shared active follow-up journey through the
normal task UI and API.
> - Independent native operation, saved input, final reply, and cleanup
checks make the result inspectable across targets.
## Linked Issues or Issue Description
**What existing behavior does this improve?**
Product E2E coverage for a user follow-up sent while a native agent is
working.
**Current behavior**
The classic chat story holds work open with a local file. The shared
deployment-targeted harness has no active follow-up journey. A running
heartbeat alone cannot prove that work was active when input arrived.
**Proposed behavior**
Run one bounded command through the real native provider. Observe its
active execution before and after one follow-up is saved. Require one
final reply with the late requirement, correct task and agent identity,
an empty input queue, and visible readback after reload. Use the same
story on managed and attached targets.
## What Changed
- Add a shared `followup-while-running` journey with normal task
creation and reply submission.
- Require a bound native turn and pending shell execution at the input
boundary. Retain a closed receipt with source identity and a row hash.
- Accept one steered work run or one normal successor (1–2 native runs).
Preserve every other journey's declared run contract.
- Reuse the saved native final reply observer, run binding checks, usage
collection, target lifecycle and private evidence path.
- Accept canonical builtin/process execution events independent of
provider tool display names; calibrate against real OpenCode, Codex and
ACP mappers. Reject Paperclip semantic tool names across native
wrappers/prefixes and unnamed or ambiguous builtin calls.
- Recognize only the observed legacy PRP v1
builtin/bash/unknown/readOnly=false command shape and fail promptly when
owned work finishes before a live boundary can be observed.
- Ready the task page/composer before the observed command boundary, use
a bounded 90-second command, and preserve active-after-persistence
proof.
- Add adversarial oracle and retained-result calibration. Register the
journey, include its sources in the definition hash, and document its
coverage.
- Keep six real browser document-readiness checks and verify
shared-deadline behavior with a deterministic clock/state test that
rejects timeout-reset mutations.
## Verification
Exact current head `e72e20ad111bd94728d38606015d1fdac3c92929` passed
[full CI run
41142552195](https://github.com/paperclipai/paperclip/actions/runs/41142552195):
all 49 workflow jobs reached accepted terminal results, including
typecheck, build, every test lane and aggregate verify. All 59 reported
checks are terminal: 55 success, three expected skips and one neutral
5/5 Paperclip review. Snyk passed. [Fresh Greptile
review](https://github.com/paperclipai/paperclip/pull/18345#issuecomment-6528913235)
explicitly names this exact head and gives 5/5. No review threads remain
open. The PR is mergeable.
These source gates do not establish a live QA pass at this head. Fresh
protected QA at this exact source and the separately recorded deployed
application/runtime remains required before primary merge and daily
admission.
Historical live QA staging **PASS** on qualified commit
`af9e091b15b5e24b6b2cc1b424523430cc2ab024`: 6/6 checks, all five phases
including cleanup, two native work runs, and the configured
model/environment check passed. The [sanitized qualification
report](https://pages.paperclip.ing/evals-qa-smoke-41122620146-1/)
records that exact measurement.
Final integration head `e72e20ad111bd94728d38606015d1fdac3c92929` merges
exact actual squash `18b1637b6371a6a6adade857e2a73654ee626fef` from
[#18254](https://github.com/paperclipai/paperclip/pull/18254) into
previous head `e54d64743964b1a19d165537bebbe62c50389a23`. The merge had
no conflicts. All 18 files in this PR's change set, the complete 45-file
shared journey directory, and six shared native/evidence helpers retain
their exact previous-head Git blobs: 59 scoped files in total. The
production clarification helper remains blob
`800a747284db19b8484650b6759e8bd1f356839d`. Follow-up 1–2 runs, question
1–3 runs, older exact counts, owned native event checks,
active-before/after persistence proof, cleanup/accounting and feature
mappings remain unchanged.
Source integration head `e55d74491f5bcd99c1455e9ace02bd7ad7319fae`
integrates exact actual squash
`8595efc20bda3ce20a9fbbf34a5d156a6fcf6d57` from
[#18385](https://github.com/paperclipai/paperclip/pull/18385). The merge
had no conflicts. All 17 files in this PR's change set and six shared
native/evidence helpers retain their exact Git blobs from previous head
`162e341cda5b627639b486b4019a1675bb018ea2`. This preserves the follow-up
and question readiness checks, feature mappings, six closed follow-up
check IDs, immutable before/after active-operation proof, follow-up 1–2
runs, question 1–3 runs and older exact counts.
Previously reviewed head `e54d64743964b1a19d165537bebbe62c50389a23` adds
a separate test-only reliability repair on top of that source
integration. It replaces one scheduler-sensitive browser case with a
deterministic Vitest clock and locator-state model that calls the actual
production helper. The six other real Chromium checks remain
byte-for-byte unchanged and pass **6/6**. Focused unit suites pass
**51/51**, including the deterministic deadline case. This is six
browser checks plus focused unit coverage, not seven browser checks.
The model makes the panel ready at 100 ms, the supplied answer ready at
600 ms, and full valid content ready at 1,300 ms. It requires rejection
at the shared 1,000 ms deadline and records immutable rejection state
before simulating a 10-second scheduler stall. Both restarted-timeout
mutations fail: resetting all waits wrongly resolves at 1,300 ms;
resetting matcher timeouts while keeping the final real deadline guard
rejects only at 1,300 ms with valid content present. The genuine helper
rejects at 1,000 ms with the answer still incomplete. The production
helper is unchanged, Git blob
`800a747284db19b8484650b6759e8bd1f356839d`. That two-file test-only
patch changed no journey, application, run, accounting, model or runtime
behavior. Its six-browser/51-unit results are prior-head evidence; the
complete patch is unchanged in the new integration.
The final source also retains the actual merged question-primary squash
`d9354bbfc35ef74333e90e596ce21dc8e59149b9` from
[#18346](https://github.com/paperclipai/paperclip/pull/18346). Later
commits `409117451da260a03fc28a3107c7808ecbe68f8f` and
`ea3e7b1d1bed562351fcf4722c7175001b987b59` bound question document
readiness to one campaign deadline and test that waits cannot restart
it. Commit `162e341cda5b627639b486b4019a1675bb018ea2` adds
question/follow-up report feature mappings and coverage. These files are
unchanged by the final merge.
The imported upstream source has real application changes:
[#18334](https://github.com/paperclipai/paperclip/pull/18334) makes
managed skill content durable and extends native skill-test contracts;
[#18350](https://github.com/paperclipai/paperclip/pull/18350) requires
managed Cloud shared storage and bootstrap prerequisites. #18385 updates
TestOps job metadata and repairs their combined maintenance fixture. The
new merge additionally imports
[#18383](https://github.com/paperclipai/paperclip/pull/18383) at
`81ecaf0654fe79a2a2dbda6588f033e39ee610e4`, which bounds failed
lifecycle verification;
[#18357](https://github.com/paperclipai/paperclip/pull/18357) at
`45a3a92e6dc52d20923b9aa34a22fee2ec5443f9`, which persists
sandbox-creation ownership and updates cleanup/adapter/plugin contracts;
and #18254 at `18b1637b6371a6a6adade857e2a73654ee626fef`, which accepts
a valid late Antigravity authorization URL while native authentication
drains. This PR's diff against exact upstream
`18b1637b6371a6a6adade857e2a73654ee626fef` contains only journey tests,
report coverage and documentation. Harness source and deployed
application/runtime identity are separate inputs. This source
integration does not deploy the imported application changes or claim
the QA target runs this application revision.
This final SHA has not been measured by the historical run above:
**fresh QA at this exact final head is required before primary merge**,
followed by normal TestOps verification/admission. Fresh exact-head CI
and review have passed as recorded above. The live measurement must
record the actual separately deployed QA application/runtime identity.
Seven whole-file blobs remain identical to the qualified SHA. All paths
are relative to `tests/runner-e2e/`:
| Journey/helper | Unchanged Git blob |
| --- | --- |
| `product-journeys/active-followup-evidence.ts` |
`83a5d9fab52abb9849a40e83eca6b6a9c29bf587` |
| `product-journeys/active-followup.test.ts` |
`728160c610c18292d2cf06d4a87d25a8719f1734` |
| `native-task-reply.ts` | `59d11f1a178dad9db3da8fa2342ff93643cb9922` |
| `native-event-envelope.ts` |
`305b0b456c59c511ee328e529ad574c0583075a0` |
| `run-observations.ts` | `08c8ad184a821a7cd536926b3d7e3d1156a1e3da` |
| `user-actions.ts` | `e9611348dc41e17899367dab2e3b36d3d1be9480` |
| `stop-redirect-flow.ts` | `4580bd605a425bc72511e02d988492acbec8d60e` |
The follow-up preservation comparison against historical `af9` has three
explicit exceptions; no whole-file identity is claimed for them. The
separately documented question readiness and report additions above also
require final-source verification:
1. `product-journeys/agent-flows.ts` adds `question-answer-resume` to
the `AgentJourney` type exclusion, alongside follow-up. Its remaining
contents and runtime helper bodies are byte-for-byte unchanged.
Qualified blob `a0afaa968c1d854476159fef21de7b6dd50a4a98`; final blob
`0cc797c0ae594e7769338e63a80c4f67e29e4a54`.
2. The common run-count helper is centralized in `success-contract.ts`
and re-exported by the catalog. It accepts follow-up **1–2**,
clarification **1–3**, and each older paid journey's exact count. The
follow-up catalog entry, maximum of two runs, six required checks,
native dispatch, collection and retained-run contract remain
byte-for-byte unchanged. The helper's text differs; its preserved count
semantics are verified explicitly.
3. `product-journeys/active-followup-flow.ts` changes exactly one
composer readiness assertion from Playwright's implicit five-second
default to an explicit 30-second timeout, matching `submitTaskReply` for
attached targets. Prompts, actions, native evidence, before/after
persistence proof and grading otherwise compare byte-for-byte unchanged.
Qualified blob `64392329198a1a4d9828e32834d4507e175216e6`; final blob
`2746b5ba89e2af26b4097ba7897913186786ac5d`. This correction is why a
fresh final-head QA run is required.
- **310 focused tests passed at current final head
`e72e20ad111bd94728d38606015d1fdac3c92929`**: 286
journey/evidence/catalog/report/shared-deadline tests in 14 files and 24
native authentication/real-child stream-order tests in two files. Both
invocations used Node24 and one Vitest worker. The earlier `e55`
integration also passed 365 focused tests, including 40 Cloud
configuration and 40 TestOps metadata cases; those results remain
prior-source evidence.
- **157 explicit count assertions passed again at the current final
head**, including both interactive ranges, older exact counts,
invalid/fractional/non-number values and catalog/contract export
identity. The maximum bounds and six closed follow-up check IDs also
matched.
- Maps and diff checks passed. The official QA runtime-import preflight
passed on Node24 at the current final head without network access. It
uses a dummy adapter and no credentials. It does not prove target
connectivity, sign-in, model readiness or deployed application/runtime
identity. Those require the normal protected QA run. No model/provider
work or optional Runner release was dispatched.
- Current local runtime dependencies use ignored isolated
workspace-source views. No shared dependencies or compilers were
changed. A previous local typecheck attempt hit stale reused Runner
declarations. No full repository typecheck, test run or build was
repeated locally; the successful exact-head full CI run above now
supplies these gates.
- CI on previous head `162e341cda5b627639b486b4019a1675bb018ea2` failed
the Cloud/local maintenance fixture in [server shard
9](https://github.com/paperclipai/paperclip/actions/runs/41133716802/job/124331643100),
plus dependent verify. The actual tested merge combined that head with
master `198218bf554877fe3b7a83742eefb3b8f12392c6`. A Cloud child
selected local-disk storage and hit the new shared-storage policy. This
was a deterministic upstream fixture mismatch; no unchanged failed-job
retry was used.
- The actual #18385 repair configures a valid Cloud child, verifies
local bytes independently, checks the expected provider mismatch and
retains exact stored-config readback. Its own final head
`b76d5dd40d7711f1b5f6b6316b1114c0aae02d57` passed 59 checks, including
[server shard
9](https://github.com/paperclipai/paperclip/actions/runs/41133876636/job/124331450635).
That prior result supports the repair; it does not replace this final
head's fresh CI. Separate repair candidate #18392 and its
source/evidence are retained as superseded history.
- A local attempt to run the database fixture could not start PostgreSQL
because the host exhausted macOS semaphores. A first setup attempt also
failed package resolution before test execution. Skipped database cases
and these setup failures are not passes. No kernel settings, shared
processes or databases were changed.
- The complete previous-head CI run
[41138618869](https://github.com/paperclipai/paperclip/actions/runs/41138618869)
finished with one underlying failure: [Runner Vitest shard
1](https://github.com/paperclipai/paperclip/actions/runs/41138618869/job/124347859445)
expected native Antigravity login success but observed failure at
`antigravity-auth.test.ts:101`. The lane had one failed test, 1,994
passing tests and eight skipped tests. Aggregate verify failed only
because of this lane. All other 57 checks were accepted, including
build, typecheck, all server/browser suites, Runner shard 2, Snyk and
exact-head 5/5 reviews. No unchanged retry was requested.
- The independently reviewed #18254 repair removes an invalid cross-pipe
ordering assumption: a valid URL on stderr can arrive after the
authenticate response on stdout during drain. Its real-child regression
forces both orders. The current final source imports that actual squash;
the 24 focused authentication cases pass here. Fresh full CI and
exact-head review have now passed for this source, as recorded above.
The earlier failed result is not relabeled.
- The original live compatibility failure is retained: an older PRP v1
pack labeled builtin `bash` as operation `unknown`, causing the observer
to miss the command and never send the follow-up. Its exact shape was
reproduced red before repair. Generic unknown operations, semantic
tools, wrappers, read-only work and foreign provenance remain rejected.
The historical qualified pass used the repaired oracle; no failed result
was relabeled.
- Earlier CI on `6fad965d69` hit an unchanged heartbeat test's
three-second adapter-call wait; its original failure and one narrow
rerun remain recorded. The qualified SHA subsequently passed CI. An
earlier report-catalog subprocess failure is retained. The final focused
Node24 run now includes that suite and passes after the documented
report feature mappings.
## Risks
The fixture asks the native agent to run one bounded `sleep 90` command.
The observer must see actual active native execution; a fast or
noncompliant turn fails rather than passing from elapsed time. A
completed result requires one or two bounded native work runs, source
attribution, configured model/environment, and collection. Unknown
provider prices stay unknown. This change does not alter production
execution or grant new access.
## Model Used
OpenAI Codex, GPT-6 family, with reasoning and code execution. The host
does not expose an exact underlying model version or context-window
size.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with available version and
capability details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs; existing
follow-up execution fixes do not provide this deployment-targeted test
- [x] I have described the issue in-PR following the enhancement
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal
ticket or instance-derived details
- [x] I have run the new focused tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation
- [x] I have considered and documented risks
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
Use a dedicated advisory-lock connection so a one-slot write pool can finish evaluation and result writeback. Record internal deadline provenance so startup recovers lost membership hints without advancing saved policy or retry debt.
Co-Authored-By: Paperclip <paperclip@paperclip.ing>
Load company-partitioned decision/recovery links, run/agent visibility and newer-run probes once per page; share one blocker graph reconstruction and exact SQL timestamps across page entries. Keep fresh per-entry archive locks and revalidation. Add full100-source query budgets and late visibility/supersession safety controls.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip manages companies and the agents that do their work.
> - Remote execution creates provider resources that need a durable
owner.
> - The server could create a sandbox before it saved its lease.
> - A failed lease write could leave cleanup data only on the old web
server disk.
> - This change saves a company-scoped operation before creation and
transfers ownership atomically.
> - Recovery finds the original Daytona resource or records an explicit
manual reconciliation outcome.
## Linked Issues or Issue Description
**What happened?**
A sandbox could exist without a lease after a database failure or
company rebind. Failed cleanup recording could then create a local
fallback file. Replacing that web server could lose the cleanup
reference.
**Expected behavior**
A failed admission write makes no provider call. An interrupted creation
retains a committed operation. Recovery must not create a second sandbox
or destroy an attached lease.
**Steps to reproduce**
1. Run the disposable PostgreSQL cases in
`server/src/__tests__/sandbox-creation-operations.test.ts`.
2. Reject the lease insert after the controlled provider creates its
resource.
3. Start a new receiver process with a different empty web disk. It must
find the same resource and create one cleanup owner.
Related ownership report: Refs #14672. This change addresses new
allocations before lease persistence. It does not replace existing lease
recovery.
## What Changed
- Add an idempotent migration and a small creation-operation table.
Retain original company, environment and run identities across deletion.
- Commit creation intent before run, ad-hoc lease, and plugin task
replacement calls. Fence duplicate requests and ambiguous lease commits.
- Retain a known returned task resource ID in the existing operation
before attachment. Preserve manual reconciliation and never delete an
ambiguously attached resource.
- Keep company and run authority checks and atomic lease transfer with
the lease owner.
- Reuse Daytona names, labels, read-only inspection and claimed cleanup.
Keep provider payload handling in the provider.
- Retain nonsecret credential identity hashes. Refuse recovery lookup
under a changed credential value.
- Record manual reconciliation for missing/deleted or explicitly
inactive saved credentials, versions, and provider vaults. Keep
temporary resolution failures retryable. Never reactivate a disabled
credential.
- Use transaction signals and concrete deadlines. Failed records back
off independently. Manual reconciliation stays durable and quiet.
- Remove new orphan-file fallback writes from the covered paths. Keep
legacy file replay and its tests for cutover.
- Replace obsolete fallback-producer assertions with normal-entrypoint
durability tests. Keep cleanup, diagnostics and warm handoff coverage.
## Verification
- [Independent current-head
verification](https://github.com/paperclipai/paperclip/pull/18357#issuecomment-6534744320)
passed 190/190 combined tests, 40/40 incoming TestOps tests, full
workspace typecheck, and build. PR head
`958af986130f25e136065ad30d1fdd843c46255c` has tree
`e6c3e5b02182d5941a70ccf5bf55638df3c2e28d`, identical to tested combined
commit `b9d7a9c2e9b30f0153685fb24a5f70b56132201f`. Real PostgreSQL,
compiled Rust Runner and controlled object providers were used;
external-S3 integration remains outside this local proof.
- Current head `958af986130f25e136065ad30d1fdd843c46255c` normally
merges master `8595efc20bda3ce20a9fbbf34a5d156a6fcf6d57`. The sole
conflict was the maintenance test: a parameterized case now preserves
both legacy-local `local_storage` and valid shared-S3
`provider_mismatch` checks, including the parent local receipt, exact
version identity, missing-snapshot/no-seed refusal, and unchanged
persisted row. The three master TestOps files remain intact. No
production, schema, migration, or package changes were made after the
fully qualified prior head `0b4d05505e`. Independent review approved the
resolution; the full managed-skill-content file and Cloud guard controls
pass 58/58 (105.11 seconds). [Required PR CI passed on this exact merge
head, first
attempt](https://github.com/paperclipai/paperclip/actions/runs/41136987412):
54 successful checks and three intended skips. The additional Paperclip
review is neutral at 4/5; two external Superagent checks (Contributor
trust and Security Scan) remain in progress with zero annotations and no
assessment, so they are not passing scan evidence. Greptile reviewed
exact head `958af986130f25e136065ad30d1fdd843c46255c` at 5/5; the final
thread audit has zero unresolved threads. All completed evidence below
retains its recorded revision scope. The additional Paperclip Review on
`958af98613` is neutral at 4/5 with a Ship verdict and no must-fix or
should-fix item. Its optional observation is that a generic create
rejection remains pending/manual even when an operator believes
allocation did not occur. This change preserves uncertainty because the
generic create-error contract does not prove provider absence; no new
error classification or provider-specific compensation was added.
- [Independent final-source
verification](https://github.com/paperclipai/paperclip/pull/18357#issuecomment-6534328247)
passed all 189 real-PostgreSQL/Runner and controlled-fixture cases, full
workspace typecheck, and build. The integration commit
`c8411a8528386f4824b340ec5fdaaed1efb9ea69` has the same source tree as
final PR head `0b4d05505e` (tree
`bda65efc14b4c904d0f589154db77bb160641c8c`). Controlled object-provider
coverage passed; the separately guarded external-S3 suite was not run
because the bounded local MinIO pull was unavailable. This includes the
two final compatibility fixes.
- The first final-head CI attempt exposed one E2E timing failure in
`disconnect-write-recovery.spec.ts:181`: the title stayed unchanged
through the 15-second session-recovery poll. The same original case
passed in 13.8 seconds on immediately prior head `7ec06f77b7`, and all
UI/session/E2E source is identical across those heads. The failure log
and CI artifacts are retained; no cause is claimed. One failed-job-only
retry is authorized on this evidence, without changing a timeout,
assertion, or production behavior. [The single failed-job retry
completed
successfully](https://github.com/paperclipai/paperclip/actions/runs/41133619710/attempts/2):
all 35 E2E shard cases passed (6.6 minutes), including the original
session-recovery case in 14.0 seconds. The initial failure remains
disclosed and its cause remains unproven.
- Final head `0b4d05505ee0aa72f2981cdff74761406960ec73` adds two
independently reviewed integration fixes after the complete 7ec CI
matrix exposed two deterministic failures. Server 9/12 failed legacy
local-storage maintenance because the CLI reached the new Cloud server
startup guard; server 5/12 found the Daytona package version did not
match its bumped manifest. The failed run is retained at
https://github.com/paperclipai/paperclip/actions/runs/41131204203. It is
not passing evidence and was not blindly retried.
- Maintenance now constructs the existing explicit storage dependency
after loading the required persisted Cloud snapshot, and passes it to
both maintenance owners. It can verify legacy local storage before
provider cutover while normal Cloud server startup remains guarded. The
original case reproduced red locally, then the complete
managed-skill-content and managed-cloud-prerequisites files passed 57/57
(32.05 seconds), including missing-snapshot refusal, unchanged stored
configuration, and existing server guard controls. Daytona package
version now matches manifest 0.1.10; the complete bundled package
identity file passed 14/14 (3.98 seconds), and explicit
manifest/lifecycle controls passed 2/2 (328 intentionally filtered; 3.42
seconds). An initial workspace-filter command matched no project and is
not counted as pass evidence. The final diff changes no sandbox runtime
or schema behavior. [Full CI is green on this exact final
head](https://github.com/paperclipai/paperclip/actions/runs/41133619710/attempts/2):
56 successful checks, three intended skips, one additional
incomplete/neutral Paperclip Review, and no pending or failed checks.
The neutral review is not a passing assessment. [Greptile reviewed the
exact final head at
5/5](https://github.com/paperclipai/paperclip/pull/18357#issuecomment-6529396969),
with no new actionable issues and all review threads resolved.
- Final combined head `7ec06f77b7c14a745c569d18401f16df3d26b626`
includes master managed-skills migration 0417 and shared-storage
prerequisites from `198218bf55` through normal merges. Sandbox creation
moves to generated `0418_curved_lady_mastermind.sql`; its idempotent SQL
is byte-for-byte unchanged. The regenerated 0418 snapshot contains both
sandbox creation operations and the master skill mutation receipts, and
points to the master 0417 snapshot. Master SQL and journal history
remain intact. Exactly five snapshots remain, 0414 through 0418. The
only sandbox-specific changes since the approved credential policy are
migration metadata/number and the matching test/document references; the
sandbox runtime files are unchanged.
- [Independent final combined
verification](https://github.com/paperclipai/paperclip/pull/18357#issuecomment-6533960026)
passed on integration revision `e6f05b4cf0cd`, containing final sandbox
head `7ec06f77b7` and actual master `198218bf55`: migration/declaration
checks, all 158 real-PostgreSQL/Runner and controlled-fixture cases (251
seconds), full workspace typecheck (81 seconds), and build (35 seconds).
- Final combined migration proof: fresh real PostgreSQL has both tables.
A separate original-0417 sandbox table with a retained pending row
survives two applications of the renamed 0418 SQL unchanged, with the
expected primary/index and foreign-key structure. Migration
numbering/safety checks pass. The normal runtime migration reapply case
passes 1/1 (20 intentionally filtered cases; 46.69 seconds). The first
temporary qualification harness stopped before any database assertion
because its external `.ts` file selected CommonJS; the `.mts` harness
corrected that setup and passed. Independent review approved the exact
metadata and reference diff. Earlier full suite and live-provider
results below retain their stated revision scope.
- Credential-policy head `c606d473b03709f27577a55a1dfc0b0d3b4359fd`
handles the remaining inactive-credential review warning. The existing
boundary accepts only concrete secret-owner 422 codes and exact
no-details disabled/coming-soon vault messages. Unknown 422,
authorization errors, provider wrappers, and temporary database/provider
failures still propagate. Independent source review approved this
three-file correction. No generic resolver, schema, provider protocol,
retry cap, or activation behavior changed.
- Creation/cleanup qualification at `c606d473b0`: all 39 cases pass
against real PostgreSQL (81.27 seconds). The new cases use actual
disabled/archived saved secrets, a valid disabled version, and provider
vaults created ready before normal API updates disable them or mark them
coming soon. Each remains manual through restart/repeated scans, with
zero provider lookup and no retry deadline. The independent plugin idle
blocker remains intact. The temporary 503 control retains retry
ownership and later recovers one cleanup owner. Before this correction,
all five inactive-state cases remained pending; that failed evidence is
retained. The prior 18-case known-ID task proof below belongs to
`237144fcf6`, whose helper and task path are unchanged by this
correction.
- Known-ID head `237144fcf69a249c0e0310563428c0f55ed6d55c` also
addresses the independent task-replacement reliability warning. The
existing creation operation now saves only the known nonsecret returned
resource ID before expiry validation and attachment. The write is scoped
to operation/company/original environment and allows only a missing or
matching ID. It changes no ownership state and copies no resolved
configuration, credentials, or provider payload. Independent source
review approved this four-file fix.
- The complete real-PostgreSQL task suite passes 18/18 (9.52 seconds).
Rejected attachments retain the ID through manual reconciliation and
repeated scans. A real committed attachment followed by an injected lost
response retains the same active lease and attached operation, with no
release or second create. Same-ID retries work; conflicting IDs and
foreign companies cannot change it. The initial three regression cases
failed because the ID was absent, before the fix.
- The shipped Daytona manifest does not declare task support, so it
cannot enter this generic replacement path. The dispatcher requires both
declared task support and an environmentTask worker method. Generic
drivers without creation lookup still use manual reconciliation, with a
known returned ID retained when the database write can commit. No
automatic lookup capability or unconditional compensation was added.
- Previous boundary-fix head `8ba77def151ea85eeae2db7693254db7f87673b0`
fixes the fresh review finding about missing credentials. Only known
secret-owner 404 results at the recovery secret-resolution boundary
select the existing manual outcome. Provider inspection and transient
database/provider errors remain retryable. Independent source review
approved this change.
- Secret-boundary focused acceptance at `8ba77def15`: all 34 cases pass
against isolated real PostgreSQL. The complete creation suite passes
16/16 (100.75 seconds); the complete cleanup coordinator suite passes
18/18 (99.83 seconds). The new cases use normal saved-secret creation
and deletion, confirm zero provider lookup, restart/repeated scans, no
automatic deadline, preserved independent plugin idle blocking, and no
additional idle blocker after the controlled worker supplies its quiet
proof. A temporary 503 resolution failure preserves pending ownership
and later recovers one cleanup owner. No additional live provider
resource was created.
- Failed attempts remain part of the evidence. The initial red run
reproduced the pending-operation bug and also exposed a numeric-deadline
test harness mistake, which was corrected. A later combined run reported
an existing replacement-receiver failure and a new idle-fixture
assertion failure; it was interrupted while the next test file remained
in imports. An isolated diagnostic confirmed that the manual state was
correct but the installed fixture plugin correctly blocked sleep without
its independent quiet proof. The corrected test retains that blocker
assertion before supplying the controlled worker witness. The unchanged
replacement-receiver case passes in the final complete 16-case run; its
earlier failure is not counted as a pass or assigned a proven cause. No
timeout or production idle guard was relaxed.
- Greptile reviewed previous combined head
`7ec06f77b7c14a745c569d18401f16df3d26b626` at 5/5, with no unresolved
review threads. Its CI produced the two concrete integration failures
addressed above. The earlier CI and live evidence below remain
historical evidence, not a claim that the current full gates have
completed.
- Master integration update: merged `dbff3747b8` and then latest master
`76fde891cc` without rewriting branch history. Only `doc/DATABASE.md`
conflicted; the sandbox ownership, object-backed run evidence, and cloud
configuration sections all remain. The master merges changed no sandbox
implementation, migration, or provider code. Migration numbering/safety
checks pass and exactly five schema snapshots remain. At merge head
`f5d23d4a59828835d01c150f83fa2645fb2d3bec`, the two focused
creation/coordinator suites pass all 32 cases against isolated real
PostgreSQL (64.57 seconds). Those merge-head checks precede the
missing-credential fix below. The earlier live Daytona proof still
covers the unchanged normal creation/lookup/teardown path; the new
missing-credential branch is qualified with real PostgreSQL and
controlled provider hooks.
The optional Superagent Security Scan was inconclusive on prior head
`7ec06f77b7`: its pinned PR merge-base checkout failed or timed out,
with a neutral conclusion and zero annotations. An earlier scan could
not start within 120 seconds; a bounded rerun request returned HTTP 404.
These scanner failures are not passing security-scan evidence. On final
head `0b4d05505e`, the additional Paperclip Review check is neutral: its
assigned review execution did not complete and produced no validated
assessment. Fresh Greptile 5/5 and independent source review are
separate evidence. Previous-head required CI, independent review, and
the live Daytona qualification below passed.
- [Complete previous-head
CI](https://github.com/paperclipai/paperclip/actions/runs/41105726052)
passed at `0b5cbb763ab8faa25aa0ff1d8a405cdc0d3db409`: 53 successful
checks and three intended skips, including typecheck, full tests, build
and E2E. Greptile is 5/5 on this head and both original findings are
resolved.
- Initial head: 200 PostgreSQL runtime, creation and coordinator tests
passed. The corrected head also passes all 32 creation and coordinator
tests, including deletion before/during lookup, a failed lookup after
deletion, restart, settled polling and actual idle-safety checks. Cases
cover failed admission, failed lease writes, concurrent recovery, lost
provider replies, ambiguous transfer retries, company rebind,
environment deletion, stopped runs, lookup backoff, restart, drain and
settled SQL quiet.
- Separate receiver processes: passed. A controlled HTTP provider
retains the resource while three independent receiver processes use two
empty web disks. Recovery creates one cleanup owner and never repeats
creation.
- Fresh PostgreSQL migration and two additional applications: passed.
The pending creation row survives unchanged.
- Server TypeScript and SDK/shared build checks pass after the
corrections. Migration numbering/safety checks passed. Exactly five
schema snapshots remain.
- Daytona adapter: all 330 tests passed, including stable-name lookup,
ownership checks, deletion and the updated manifest version. An earlier
run exposed the version expectation and it was corrected. Previous-head
CI passed.
- Live Daytona acceptance passed on `0b5cbb763a`: isolated real
PostgreSQL, normal runtime acquisition through independent receiver
processes, actual Daytona plugin/SDK 0.203.0, one disposable Ubuntu
24.04 sandbox (1 CPU / 1 GiB RAM / 1 GiB disk). A forced lease insert
failure retained the committed operation; a fresh receiver with empty
disk found the same sandbox, repeated recovery retained one cleanup
owner, and the actual claimed cleanup sweep expired the lease and
resolved the operation. Independent SDK lookup returned typed not-found;
ordinary recorded-ID inspection returned destroyed. This successful
attempt made one create, one recovery lookup, and one teardown. Two
earlier failed harness/provider-confirmation attempts are disclosed in
the [live acceptance
record](https://github.com/paperclipai/paperclip/pull/18357#issuecomment-6530313054).
The coordinator independently queried all three exact resource IDs
through the SDK and confirmed typed not-found for each.
- Complete CI-affected heartbeat/plugin warm-task suites passed 21/21
after test corrections. An unchanged workspace exposure case passed in
isolation after the prior CI shard reported an unhandled readiness
rejection. Complete previous-head CI and refreshed review passed.
## Risks
- Deleted companies retain an explicit manual reconciliation outcome.
Recovery does not retry missing credentials or insert a lease for a
deleted company.
- Only Daytona declares automatic creation lookup. Other uncertain
providers retain `needs_reconciliation`. A missing creation name alone
cannot prove that a late allocation will never appear.
- Pending operations use the existing process-local signal and startup
recovery model. This does not add cross-replica wake delivery or prove
deployed sleep.
- Legacy direct provider probes and custom-image setup use separate
lifecycles. They are outside this replacement of the run-lease
orphan-file fallback. The normal Test path that acquires a runtime lease
is covered.
- Existing orphan files still require the separate cutover inventory and
settlement. This change does not delete them.
- The schema change is additive. The new SQL can be reapplied.
Historical SQL and journal entries are preserved; only the oldest
generated snapshot is pruned.
## Model Used
OpenAI `gpt-6.1-sol`, with tool use, code execution and repository
review. The harness did not report the context-window size.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
The required branch status checks (`ci / verify`, `ci / e2e`) and all
test/build jobs passed on this head. The non-required Superagent
Contributor trust and Security Scan checks are still running without
findings; they are not claimed as passes. No required check or branch
rule is overridden.
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
Failed lifecycle verification could allocate another test environment every minute indefinitely. Count failed host verification tests separately from claims, persist a repair wait for confirmed configuration failures or three generic failed tests, and clear it through authorized retry or execution configuration changes.
Preserve typed temporary execution-target failures through adapter checks so timeouts, transport failures and interrupted probes keep automatic recovery. Retain existing readiness, authorization, termination and cleanup gates. Document the policy and add real PostgreSQL persistence and actual OpenCode adapter regressions.
Verified 398 focused tests (one Windows-only test skipped on macOS), repository-wide typecheck and build, structural checks, fully green current-head CI and Greptile 5/5 with no unresolved comments.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
* origin/master:
fix(evals): support official TestOps job metadata (#18385)
fix(config): require shared storage for managed cloud (#18350)
feat(skills): make managed content independent of web-server disk (#18334)
Add a portable agent question and answer resumption journey (#18346)
fix(ci): isolate test selection and require current CI CODEOWNERS (#18376)
fix(evals): capture TestOps registration after watch (#18375)
fix(runner): allow session reset after confirmed task retirement (#18373)
fix(evals): batch task search result readback across renders (#18370)
fix(ui): keep task plugin views available for manual access (#18363)
test: wait for the evidence expiry worker to settle (#18241)
# Conflicts:
# server/src/services/company-skills.ts
# server/src/services/native-runtime/paperclip-runner-tool-authority.ts
* origin/master:
fix(evals): support official TestOps job metadata (#18385)
fix(config): require shared storage for managed cloud (#18350)
feat(skills): make managed content independent of web-server disk (#18334)
Add a portable agent question and answer resumption journey (#18346)
fix(ci): isolate test selection and require current CI CODEOWNERS (#18376)
fix(evals): capture TestOps registration after watch (#18375)
fix(runner): allow session reset after confirmed task retirement (#18373)
fix(evals): batch task search result readback across renders (#18370)
fix(ui): keep task plugin views available for manual access (#18363)
test: wait for the evidence expiry worker to settle (#18241)
Batch review and confirmation eligibility per company, scope decision and budget hints, and keep child blocker relation intents inside their owning transaction. Cover ordinary child, budget and execution producers plus full candidate-page query bounds; update startup/import fixtures for source-driven retention.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Dotta