Files
PaperClipAI/server
Nicky LeachandClaude Sonnet 5 9effe51b63 fix(server): let the cloud-harness sandbox environment self-heal past operator-drift protection (#13177)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Every cloud-harness-managed stack gets one platform-owned "Paperclip
Computer" sandbox environment, reconciled from
`PAPERCLIP_MANAGED_CONFIG` on boot
> - That reconciler deliberately refuses to overwrite a row it
classifies as operator-modified, to protect a self-hosted operator's
hand-edited environment (#10979)
> - But for the cloud-harness-managed row specifically, no operator has
any path to hand-edit it at all — so a hash mismatch there can only be
drift between two platform-driven reconciliation passes, never a real
customization
> - Roughly 10 staging stacks got stuck on a broken sandbox image
because of exactly this: a `sandbox_image` campaign correctly delivered
a fixed snapshot, but the reconciler classified the row as
operator-modified and silently skipped applying it
> - This pull request adds an explicit `platformFullyManaged` flag so
the cloud-harness caller can assert that guarantee and let its own drift
self-heal, without weakening the protection for every other caller
(self-hosted kubernetes-execution-mode, tests, admin routes) where an
operator genuinely can edit the row
> - The benefit is that a sandbox-image rollout can no longer get
silently stuck fleet-wide, while self-hosted operator customization
keeps exactly the protection #10979 built

## Linked Issues or Issue Description

No public issue exists for this internal-instance-discovered bug;
opening directly per CONTRIBUTING.md path B, following the bug report
template fields.

**What happened?**
After a `sandbox_image` campaign delivered a fixed Daytona snapshot
fleet-wide, ~10 of 79 active staging stacks kept booting agents against
the old, broken snapshot. Their `PAPERCLIP_MANAGED_CONFIG` env var and
the reconciler's own bookkeeping (`built_in_managed_resources`) both
correctly showed the new snapshot — but `environments.config.snapshot`,
the field the runtime actually reads to acquire a sandbox lease, was
never updated on those rows.

**Expected behavior**
A `sandbox_image` campaign (or any `PAPERCLIP_MANAGED_CONFIG` delivery)
to the cloud-harness-managed sandbox environment should always converge
that row's `config` to the newly-desired value, since no operator can
have a competing edit to protect.

**Steps to reproduce**
1. Boot a cloud-harness-managed stack; let the reconciler create the
managed sandbox row and record its stock hash in
`built_in_managed_resources`.
2. Somehow cause the row's live content hash to no longer match the
recorded binding hash without an operator ever touching it (in the
field, this happened via drift between two platform-driven
reconciliation passes carried out across a catalog-version bump — the
exact trigger wasn't fully pinned down, but is irrelevant to the fix).
3. Deliver a new `PAPERCLIP_MANAGED_CONFIG` (e.g. via a `sandbox_image`
campaign).
4. Observe `ensureManagedSandboxEnvironment` classify the row
`operator_modified` and skip writing `config`, even though
`updateAvailable: true` is reported and the binding itself already
advanced to the new stock hash.

**Paperclip version or commit**
`master` as of this PR.

**Deployment mode**
Cloud-managed stacks with `enableManagedSandboxOnly` declared (any
Paperclip Cloud–provisioned staging or production stack).

Related PR for context (not a duplicate — this is additive to it, not a
revert): #10979, which introduced the `operator_modified` classification
this PR narrowly opts the cloud-harness path out of.

## What Changed

- `server/src/services/environments.ts`: added `platformFullyManaged?:
boolean` to `ManagedSandboxEnvironmentInput`. When set, a plain
content-hash mismatch against a real prior binding (i.e.
`operator_modified` that isn't an archive-reaffirmation) is reclassified
as `stock_update_available` before the skip-vs-apply branch, so it flows
through the normal update path instead of being frozen.
- `server/src/services/managed-environments.ts`: pass
`platformFullyManaged: true` from both `ensureManagedSandboxEnvironment`
call sites — the main boot ensure and the provider-recovery reactivation
path. These are the *only* two callers driven by
`PAPERCLIP_MANAGED_CONFIG`; `ensureKubernetesEnvironment` (self-hosted
`kubernetes-execution-mode` bootstrap) and every other caller are
untouched and keep the original protective default.
- `server/src/services/managed-environments.test.ts`: updated the two
`toHaveBeenCalledWith` assertions that now include the flag.
- `server/src/__tests__/environment-service.test.ts`: two new tests —
one confirming the bypass applies drift under `platformFullyManaged`,
one confirming archive-reaffirmation still wins even under the flag.

Archive-reaffirmation is deliberately *not* bypassed even under
`platformFullyManaged`: a `sandbox_image` update must never resurrect a
row something else deliberately kept archived after Paperclip's own
provider-unavailability archival. That's a distinct, still-real signal,
orthogonal to config drift.

## Verification

- `vitest run` on `managed-environments.test.ts` and
`managed-resource-drift.test.ts`: 26/26 pass, including the two updated
assertions.
- `environment-service.test.ts` — the file both new tests live in, and
the file holding the two pre-existing tests this change must not regress
("classifies operator drift, preserves the row, and exposes the pending
stock update" and "preserves an existing unmanaged sandbox row holding
the desired name") — requires a real embedded-Postgres instance
(`describeEmbeddedPostgres`) not available in the sandbox this was
developed in; `getEmbeddedPostgresTestSupport()` reports unsupported
there, so the whole file is skipped locally. I traced the reconciliation
logic by hand against all four relevant tests (the two new ones plus the
two pre-existing ones) line by line to confirm the expected outcomes,
but **CI running this suite for real is the actual gate here**, not this
description — please don't merge on a green run of everything else alone
if this suite doesn't show as executed.
- `tsc --noEmit`: zero errors in any of the four touched files. The
pre-existing ~229 errors elsewhere in `server` are unrelated
missing-module issues from packages needing a build step first,
confirmed unchanged by this diff.
- Manually reproduced the underlying bug against real staging data (a
`paperclip-cloud`-managed stack whose `environments` row was stuck
exactly this way) before writing the fix, and confirmed via direct SQL
inspection that the recorded `built_in_managed_resources` baseline
already held the correct desired snapshot on every affected stack — i.e.
the reconciler already *knew* the right answer, it was just refusing to
apply it. That data point is what ruled out "the campaign didn't
actually deliver the update" as the cause.

## Risks

- Scope is intentionally narrow: only the two
`PAPERCLIP_MANAGED_CONFIG`-driven call sites pass the new flag; every
other caller of
`ensureManagedSandboxEnvironment`/`ensureKubernetesEnvironment` is
byte-for-byte unchanged. The two pre-existing regression tests that
specifically cover self-hosted operator-edit protection don't pass this
flag and are unmodified.
- The main residual risk is the unresolved root cause of *why* the hash
drifted in the first place (a race between two close-together
reconciliation passes, or a catalog-version-dependent change to what
gets hashed, most likely) — this PR makes that drift self-healing rather
than fixing whatever produces it. If the drift is being caused by a
genuine concurrency bug (rather than an expected, occasional side effect
of a stock-field/catalog-version change), that bug still exists and
could recur; it just no longer gets stuck when it does.
- Low risk of behavior change for real self-hosted deployments: none of
them can reach the new code path, since only the two now-flagged call
sites exist inside `managed-environments.ts`, itself gated to
`PAPERCLIP_MANAGED_CONFIG` (which self-hosted
`kubernetes-execution-mode` explicitly refuses to run alongside — see
the existing mutual-exclusivity check this PR does not touch).

## Model Used

Claude Sonnet 5 (`claude-sonnet-5`), via Claude Code, with tool use
(file edits, shell/git, `gh` CLI, direct Postgres inspection of live
staging data via `psql`/`pg`, Railway SSH for on-host diagnosis). No
extended-thinking mode. Standard Claude Code context window.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass — see Verification: the
file holding the four load-bearing tests can't run in this sandbox (no
embedded-Postgres support); traced by hand instead, CI is the real gate
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes — none
applicable beyond the inline doc comments this PR adds
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green — pending CI run on this PR
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups —
pending review
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-10 19:23:13 -07:00
..