fix(environments): let same-run recovery create a replacement once its sandbox is proven stopped (#21223)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent runs execute on environment leases, and `createOwnedSandbox`
records each sandbox creation so that a lost reply never creates a
second sandbox
> - The record key is company, environment, run and request key. The
plugin request key is the environment fingerprint, so it is the same for
every attempt of one run
> - A same-run recovery can need a new environment after the first lease
is closed with successful cleanup and a termination receipt
> - The old record is still `attached` to the closed lease, so the
recovery fails with "Sandbox creation has already transferred ownership"
before the provider is called
> - This pull request lets the creation fence accept the same stop proof
that recovery already requires. A creation record whose sandbox is
proven stopped becomes resolved, and a resolved record no longer holds
its key
> - The benefit is that same-run recovery gets one replacement sandbox,
while unconfirmed cleanup still blocks a second sandbox

## Linked Issues or Issue Description

Fixes #21214

Refs #20755. That pull request fixes the same failure with a different
rule. It retries under a new key generation (`<key>#2`, `#3`, ...) when
the recorded lease has ended, and it accepts any ended status, including
`failed`, without a termination receipt. A lease whose cleanup is not
confirmed can then produce a second sandbox while the first one can
still run. This pull request keeps the original key and requires the
same stop proof that Core uses elsewhere
(`hasRemoteTerminationReceipt`). It does not add a key generation.

## What Changed

- `server/src/services/sandbox-creation-operations.ts`: when
`createOwnedSandbox` finds the key held by an `attached` operation whose
lease is not active, it checks the same stop proof that same-run
recovery requires before it acquires:
`hasRemoteTerminationReceipt(resolveRemoteTerminationLease(lease))`, for
the sandbox that this operation created (`creationProviderLeaseId`, or
the lease's `providerLeaseId`).
- With that proof, it runs the same admission transaction one more time.
That transaction first marks the old operation `resolved` (`WHERE id AND
state = 'attached' AND lease_id`), and then inserts the new operation.
  - Without that proof, it throws exactly as before.
- The check covers every source of a stop receipt: `releaseLease`, the
Runner `complete` operation (which writes receipts with direct SQL for
reusable leases), stop-and-retain settlement, and the receipt that
`resolveRemoteTerminationLease` derives for an expired reusable lease.
- A task replacement can leave an earlier operation attached to a lease
that now holds another sandbox. That operation is not proven and keeps
its key.
- Concurrent recoveries serialize on the row update. The second one
updates nothing, finds the first one's unresolved operation through the
partial index, and replays or refuses as before. There is one provider
call.
- Migration `0475_sandbox_creation_owner_unresolved.sql` recreates
`sandbox_creation_operations_owner_idx` under the same name as a partial
unique index (`state <> 'resolved'`). The drop and the create are in one
migration transaction. The Drizzle schema matches.
- The conflict lookup in `createOwnedSandbox` ignores resolved rows. An
active attached lease is still replayed. Pending and
`needs_reconciliation` rows still refuse. `releaseLease` does not
change.
- `doc/DATABASE.md` and `doc/idle-environment-cleanup.md` describe the
operation lifecycle.

## Verification

- `pnpm --filter @paperclipai/db generate`
- `pnpm --filter @paperclipai/db typecheck` and `pnpm --filter
@paperclipai/server typecheck`
- `vitest run src/migration-snapshot-drift.test.ts` (db)
- `vitest run src/__tests__/sandbox-creation-operations.test.ts`: 48/48.
With the service change reverted, the 5 replacement tests fail.
- The 9 other server test files that read `sandbox_creation_operations`,
plus `environment-runtime.test.ts`: 564/564
- `pnpm check:migrations` passes

New tests in `sandbox-creation-operations.test.ts`:

- A stop receipt from `releaseLease` (sandbox driver, and the plugin
driver through `acquireRunLease` in the #21214 sequence), from the
`complete` operation's direct write on a retained reusable lease, and
derived only from the destroyed successor of an expired reusable lease
each let the next creation resolve the old operation and create exactly
one replacement. A retry replays that replacement. These tests fail
before the fix.
- Failed cleanup, success without a receipt, a receipt for another lease
or another provider task, a retained lease, and pending cleanup each
still throw "transferred ownership" with no provider call.
- An earlier task's operation that a task replacement left attached
keeps its key.
- Two concurrent recoveries make one provider call. This test fails
before the fix.
- Reapplying migration 0418 keeps the partial index.

I did not run `pnpm verify` or the full suite locally. CI runs them.

## Risks

- Migration: the drop and the rebuild of the index take an exclusive
lock on `sandbox_creation_operations`. The table has one small row for
each sandbox creation, so the lock is short.
- Rolling deploy: the migration is marked `online_compatible`. The
previous release uses an untargeted `ON CONFLICT DO NOTHING`, which also
matches a partial unique index. Its conflict lookup has no state filter,
so it can find a resolved row and throw. That fails closed and never
creates a second sandbox.
- A legacy runner-task operation without `creationProviderLeaseId`
matches any lease it is attached to. Runner-task operations record this
field before they attach, so this is only a theoretical gap.
- An operation stays `attached` until the next creation with the same
key checks it. This is the same as before when no recovery comes.
- Stop-and-retain settlement is not tested directly. The shared receipt
check covers it.
- The fence is unchanged for every case without a stop receipt.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, sub-agents) and extended thinking. One sub-agent
wrote the change. A second session reviewed it before this pull request.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Nicky LeachandClaude Opus 5.5 authored and GitHub committed 2026-11-12 20:24:34 -08:00
1 parent 4dc5492c2d
commit 717aae060a
8 files changed
+806 -10

No files matched your search

+7 -2
View File
@@ -2422,8 +2422,13 @@ the optional live environment FK uses SET NULL. Company/run deletion must not
erase an outstanding remote-resource obligation.
The unique company/environment/run/request index fences duplicate creation and
ambiguous successful lease commits. Normal lease attachment and operation transfer
share a transaction. Recovery transfers one claimed cleanup lease after provider
ambiguous successful lease commits. Since migration 0475 it covers only operations
that are not `resolved`. A later attempt for the same run and request that meets an
`attached` operation whose lease is closed checks the same stop proof same-run
recovery requires (a termination receipt for that sandbox, stored or derived from a
destroyed reusable successor); with it, one transaction resolves the operation and
records the replacement.
Normal lease attachment and operation transfer share a transaction. Recovery transfers one claimed cleanup lease after provider
ownership lookup. The pending deadline index drives the existing cleanup worker.
`needs_reconciliation` is a durable manual outcome, not an automatic retry or an
idle blocker. See [cleanup ownership](idle-environment-cleanup.md).
+12 -2
View File
@@ -70,8 +70,18 @@ writers and another server process are not covered by these process-local signal
A creation operation retains company, original environment, run, request identity,
provider, and nonsecret configuration references. Its environment FK uses SET NULL;
company/run/task identifiers remain after those records are removed. One operation
owns each company/environment/run/request. A retry of an attached operation returns
its active lease. Pending or unresolved ownership never authorizes another create.
owns each company/environment/run/request until it is resolved. A retry of an
attached operation returns its active lease. Pending or unresolved ownership never
authorizes another create. An attached operation resolves when its lease is proven
stopped, checked at the next creation with the same key: the closed lease must
carry the stop proof same-run recovery requires (`remoteExecutionHasStopped`), a
termination receipt for the sandbox this operation created, whether release, the
Runner `complete` operation, or a destroyed reusable successor supplied it. That
creation resolves the operation and records its replacement in one transaction,
so concurrent recoveries make one provider call. A recovered cleanup operation
resolves when its cleanup lease settles. Failed cleanup, success without a
receipt, a receipt for another sandbox, a retained lease, and pending cleanup keep
the operation and its key.
Normal success transfers operation ownership and the lease in one transaction.
Current company binding and run authority are checked again at attachment.
Declared secret-reference fields retain only their references and nonsecret value
@@ -0,0 +1,3 @@
-- paperclip:migration-compat online_compatible: The new index is the same unique key restricted to rows whose state is not 'resolved', so every write the previous release can make still satisfies it, and that release's untargeted ON CONFLICT DO NOTHING matches a partial unique index as well. The drop and rebuild share one transaction, so no write sees the key unfenced; the table holds one small row per sandbox creation, so the exclusive lock lasts only for that index build.
DROP INDEX "sandbox_creation_operations_owner_idx";--> statement-breakpoint
CREATE UNIQUE INDEX "sandbox_creation_operations_owner_idx" ON "sandbox_creation_operations" USING btree ("company_id","original_environment_id","run_id","request_key") WHERE "sandbox_creation_operations"."state" <> 'resolved';
@@ -1,6 +1,6 @@
{
"id": "f0e17e47-6217-4080-b6b6-9d83b33e9ced",
"prevId": "ac25715a-74c6-4988-8525-63ad14781c74",
"id": "ee1f1f66-0cbe-4b18-b364-5e96708a1f0b",
"prevId": "a7108924-4d75-41e2-9139-f43f9396c02d",
"version": "7",
"dialect": "postgresql",
"tables": {
@@ -22342,6 +22342,18 @@
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"dispatching_at": {
"name": "dispatching_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"dispatch_lease_expires_at": {
"name": "dispatch_lease_expires_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
}
},
"indexes": {
@@ -53674,6 +53686,12 @@
"primaryKey": false,
"notNull": true
},
"pending_delete_object_key": {
"name": "pending_delete_object_key",
"type": "text",
"primaryKey": false,
"notNull": false
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
@@ -53697,6 +53715,21 @@
"concurrently": false,
"method": "btree",
"with": {}
},
"runner_api_response_reservations_asset_idx": {
"name": "runner_api_response_reservations_asset_idx",
"columns": [
{
"expression": "asset_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {
@@ -53909,6 +53942,7 @@
}
],
"isUnique": true,
"where": "\"sandbox_creation_operations\".\"state\" <> 'resolved'",
"concurrently": false,
"method": "btree",
"with": {}
@@ -57502,6 +57536,12 @@
"notNull": true,
"default": "'{}'::jsonb"
},
"ui_meta": {
"name": "ui_meta",
"type": "jsonb",
"primaryKey": false,
"notNull": false
},
"risk_level": {
"name": "risk_level",
"type": "text",
@@ -59627,6 +59667,253 @@
"checkConstraints": {},
"isRLSEnabled": false
},
"public.tool_mcp_app_views": {
"name": "tool_mcp_app_views",
"schema": "",
"columns": {
"id": {
"name": "id",
"type": "uuid",
"primaryKey": true,
"notNull": true,
"default": "gen_random_uuid()"
},
"company_id": {
"name": "company_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"invocation_id": {
"name": "invocation_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"issue_id": {
"name": "issue_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"run_id": {
"name": "run_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"agent_id": {
"name": "agent_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"responsible_user_id": {
"name": "responsible_user_id",
"type": "text",
"primaryKey": false,
"notNull": false
},
"connection_id": {
"name": "connection_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"catalog_entry_id": {
"name": "catalog_entry_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"application_name": {
"name": "application_name",
"type": "text",
"primaryKey": false,
"notNull": true
},
"tool_name": {
"name": "tool_name",
"type": "text",
"primaryKey": false,
"notNull": true
},
"upstream_tool_name": {
"name": "upstream_tool_name",
"type": "text",
"primaryKey": false,
"notNull": true
},
"resource_uri": {
"name": "resource_uri",
"type": "text",
"primaryKey": false,
"notNull": true
},
"tool_input": {
"name": "tool_input",
"type": "jsonb",
"primaryKey": false,
"notNull": true,
"default": "'{}'::jsonb"
},
"tool_result": {
"name": "tool_result",
"type": "jsonb",
"primaryKey": false,
"notNull": true
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {
"tool_mcp_app_views_issue_idx": {
"name": "tool_mcp_app_views_issue_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "issue_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "created_at",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
},
"tool_mcp_app_views_connection_idx": {
"name": "tool_mcp_app_views_connection_idx",
"columns": [
{
"expression": "connection_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {
"tool_mcp_app_views_company_id_companies_id_fk": {
"name": "tool_mcp_app_views_company_id_companies_id_fk",
"tableFrom": "tool_mcp_app_views",
"tableTo": "companies",
"columnsFrom": [
"company_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"tool_mcp_app_views_invocation_id_tool_invocations_id_fk": {
"name": "tool_mcp_app_views_invocation_id_tool_invocations_id_fk",
"tableFrom": "tool_mcp_app_views",
"tableTo": "tool_invocations",
"columnsFrom": [
"invocation_id"
],
"columnsTo": [
"id"
],
"onDelete": "set null",
"onUpdate": "no action"
},
"tool_mcp_app_views_issue_id_issues_id_fk": {
"name": "tool_mcp_app_views_issue_id_issues_id_fk",
"tableFrom": "tool_mcp_app_views",
"tableTo": "issues",
"columnsFrom": [
"issue_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"tool_mcp_app_views_run_id_heartbeat_runs_id_fk": {
"name": "tool_mcp_app_views_run_id_heartbeat_runs_id_fk",
"tableFrom": "tool_mcp_app_views",
"tableTo": "heartbeat_runs",
"columnsFrom": [
"run_id"
],
"columnsTo": [
"id"
],
"onDelete": "set null",
"onUpdate": "no action"
},
"tool_mcp_app_views_agent_id_agents_id_fk": {
"name": "tool_mcp_app_views_agent_id_agents_id_fk",
"tableFrom": "tool_mcp_app_views",
"tableTo": "agents",
"columnsFrom": [
"agent_id"
],
"columnsTo": [
"id"
],
"onDelete": "set null",
"onUpdate": "no action"
},
"tool_mcp_app_views_connection_id_tool_connections_id_fk": {
"name": "tool_mcp_app_views_connection_id_tool_connections_id_fk",
"tableFrom": "tool_mcp_app_views",
"tableTo": "tool_connections",
"columnsFrom": [
"connection_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"tool_mcp_app_views_catalog_entry_id_tool_catalog_entries_id_fk": {
"name": "tool_mcp_app_views_catalog_entry_id_tool_catalog_entries_id_fk",
"tableFrom": "tool_mcp_app_views",
"tableTo": "tool_catalog_entries",
"columnsFrom": [
"catalog_entry_id"
],
"columnsTo": [
"id"
],
"onDelete": "set null",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.tool_mcp_gateway_tokens": {
"name": "tool_mcp_gateway_tokens",
"schema": "",
@@ -60341,6 +60628,334 @@
"checkConstraints": {},
"isRLSEnabled": false
},
"public.tool_mcp_skill_files": {
"name": "tool_mcp_skill_files",
"schema": "",
"columns": {
"id": {
"name": "id",
"type": "uuid",
"primaryKey": true,
"notNull": true,
"default": "gen_random_uuid()"
},
"company_id": {
"name": "company_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"connection_id": {
"name": "connection_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"file_uri": {
"name": "file_uri",
"type": "text",
"primaryKey": false,
"notNull": true
},
"digest": {
"name": "digest",
"type": "text",
"primaryKey": false,
"notNull": true
},
"size": {
"name": "size",
"type": "integer",
"primaryKey": false,
"notNull": true
},
"content_base64": {
"name": "content_base64",
"type": "text",
"primaryKey": false,
"notNull": true
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {
"tool_mcp_skill_files_digest_idx": {
"name": "tool_mcp_skill_files_digest_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "connection_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "file_uri",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "digest",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": true,
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {
"tool_mcp_skill_files_company_id_companies_id_fk": {
"name": "tool_mcp_skill_files_company_id_companies_id_fk",
"tableFrom": "tool_mcp_skill_files",
"tableTo": "companies",
"columnsFrom": [
"company_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"tool_mcp_skill_files_connection_id_tool_connections_id_fk": {
"name": "tool_mcp_skill_files_connection_id_tool_connections_id_fk",
"tableFrom": "tool_mcp_skill_files",
"tableTo": "tool_connections",
"columnsFrom": [
"connection_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.tool_mcp_skills": {
"name": "tool_mcp_skills",
"schema": "",
"columns": {
"id": {
"name": "id",
"type": "uuid",
"primaryKey": true,
"notNull": true,
"default": "gen_random_uuid()"
},
"company_id": {
"name": "company_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"connection_id": {
"name": "connection_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"skill_uri": {
"name": "skill_uri",
"type": "text",
"primaryKey": false,
"notNull": true
},
"name": {
"name": "name",
"type": "text",
"primaryKey": false,
"notNull": true
},
"description": {
"name": "description",
"type": "text",
"primaryKey": false,
"notNull": true,
"default": "''"
},
"frontmatter": {
"name": "frontmatter",
"type": "jsonb",
"primaryKey": false,
"notNull": true
},
"manifest": {
"name": "manifest",
"type": "jsonb",
"primaryKey": false,
"notNull": false
},
"manifest_digest": {
"name": "manifest_digest",
"type": "text",
"primaryKey": false,
"notNull": false
},
"status": {
"name": "status",
"type": "text",
"primaryKey": false,
"notNull": true,
"default": "'pending_review'"
},
"status_reason": {
"name": "status_reason",
"type": "text",
"primaryKey": false,
"notNull": false
},
"approved_manifest_digest": {
"name": "approved_manifest_digest",
"type": "text",
"primaryKey": false,
"notNull": false
},
"allowed_tools_approved": {
"name": "allowed_tools_approved",
"type": "boolean",
"primaryKey": false,
"notNull": true,
"default": false
},
"reviewed_at": {
"name": "reviewed_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"reviewed_by_user_id": {
"name": "reviewed_by_user_id",
"type": "text",
"primaryKey": false,
"notNull": false
},
"last_seen_at": {
"name": "last_seen_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
},
"updated_at": {
"name": "updated_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {
"tool_mcp_skills_connection_uri_idx": {
"name": "tool_mcp_skills_connection_uri_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "connection_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "skill_uri",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": true,
"concurrently": false,
"method": "btree",
"with": {}
},
"tool_mcp_skills_company_status_idx": {
"name": "tool_mcp_skills_company_status_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "status",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {
"tool_mcp_skills_company_id_companies_id_fk": {
"name": "tool_mcp_skills_company_id_companies_id_fk",
"tableFrom": "tool_mcp_skills",
"tableTo": "companies",
"columnsFrom": [
"company_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"tool_mcp_skills_connection_id_tool_connections_id_fk": {
"name": "tool_mcp_skills_connection_id_tool_connections_id_fk",
"tableFrom": "tool_mcp_skills",
"tableTo": "tool_connections",
"columnsFrom": [
"connection_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.tool_oauth_states": {
"name": "tool_oauth_states",
"schema": "",
+7
View File
@@ -3312,6 +3312,13 @@
"when": 1794524253117,
"tag": "0474_milky_wolf_cub",
"breakpoints": true
},
{
"idx": 475,
"version": "7",
"when": 1794532772498,
"tag": "0475_sandbox_creation_owner_unresolved",
"breakpoints": true
}
]
}
@@ -23,5 +23,8 @@ export const sandboxCreationOperations = pgTable("sandbox_creation_operations",
updatedAt: timestamp("updated_at", { withTimezone: true }).notNull().defaultNow(),
}, table => ({
pendingDeadline: index("sandbox_creation_operations_pending_deadline_idx").on(table.nextAttemptAt).where(sql`${table.state} = 'pending'`),
owner: uniqueIndex("sandbox_creation_operations_owner_idx").on(table.companyId, table.originalEnvironmentId, table.runId, table.requestKey),
// A resolved operation's sandbox is proven stopped, so it no longer holds its
// creation key: a later attempt for the same run and request may create again.
owner: uniqueIndex("sandbox_creation_operations_owner_idx").on(table.companyId, table.originalEnvironmentId, table.runId, table.requestKey)
.where(sql`${table.state} <> 'resolved'`),
}));
@@ -18,6 +18,8 @@ import * as secretOwner from "../services/secrets.js";
import { readIdleSleepSafety } from "../services/idle-sleep-safety.js";
import { nextEnvironmentCleanupAt } from "../services/environment-cleanup-state.js";
import { readProcessStartedAt } from "../services/hot-restart.js";
import { hasRemoteTerminationReceipt, remoteTerminationReceipt, resolveRemoteTerminationLease } from "../services/remote-execution-termination.js";
import { createReusableLeaseHandoff } from "../services/environment-lease-handoff.js";
import type { PluginWorkerManager } from "../services/plugin-worker-manager.js";
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_HISTORY_TEST_DATABASE_URL");
@@ -164,6 +166,11 @@ describe("durable sandbox creation through the runtime entry point", () => {
}
expect(await db.select().from(sandboxCreationOperations)).toEqual([before]);
expect(creates).toBe(1);
// The later migration narrowed the key to unresolved operations; a reapply keeps that.
const indexes = await db.execute(sql`select indexdef from pg_indexes where indexname = 'sandbox_creation_operations_owner_idx'`);
expect([...indexes].map(row => String((row as { indexdef: unknown }).indexdef))).toEqual([
expect.stringContaining("WHERE (state <> 'resolved'::text)"),
]);
});
it("fails before the external action when the durable admission write fails", async () => {
await rejectWrites("sandbox_creation_operations", "reject_fixture_creation");
@@ -178,6 +185,129 @@ describe("durable sandbox creation through the runtime entry point", () => {
expect(await db.select().from(environmentLeases)).toHaveLength(1);
expect(await nextEnvironmentCleanupAt(db)).toBeNull();
});
describe("same-run recovery after the attached lease closes", () => {
type Lease = Awaited<ReturnType<typeof acquire>>["lease"];
const stopReceipt = (lease: Lease) => remoteTerminationReceipt(lease, { providerLeaseId: lease.providerLeaseId, state: "stopped" });
const operations = () => db.select().from(sandboxCreationOperations);
async function acquirePlugin() {
await db.update(environments).set({ driver: "plugin", config: { pluginKey: "paperclip.creation-fixture", driverKey: "creation-fixture", driverConfig: {} } }).where(eq(environments.id, environmentId));
const [plugin] = await db.select().from(plugins).where(eq(plugins.id, pluginId));
await db.update(plugins).set({ manifestJson: { ...plugin.manifestJson, environmentDrivers: [{ driverKey: "creation-fixture", kind: "environment_driver", displayName: "Computer", creationOperationVersion: 1, configSchema: { type: "object" } }] } }).where(eq(plugins.id, pluginId));
return runtime().acquireRunLease({ companyId, environment: (await environmentService(db).getById(environmentId))!,
heartbeatRunId: runId, issueId: null, persistedExecutionWorkspace: null });
}
/** The closing write left the operation attached; the next creation with the
* same key proves the stop, resolves it, and creates exactly one replacement. */
async function expectOneReplacement(first: { lease: Lease }, acquireLease: () => Promise<{ lease: Lease }>) {
const [before] = await operations();
expect(before).toMatchObject({ state: "attached", leaseId: first.lease.id });
const second = await acquireLease();
expect(second.lease.id).not.toBe(first.lease.id);
expect(second.lease.providerLeaseId).not.toBe(first.lease.providerLeaseId);
expect(creates).toBe(2);
expect(await operations()).toEqual(expect.arrayContaining([
expect.objectContaining({ id: before!.id, state: "resolved", leaseId: first.lease.id }),
expect.objectContaining({ state: "attached", leaseId: second.lease.id, runId, requestKey: before!.requestKey }),
]));
// A retry of the replacement replays it instead of creating again.
expect((await acquireLease()).lease.id).toBe(second.lease.id);
expect(creates).toBe(2);
}
it.each([["sandbox", acquire], ["plugin", acquirePlugin]] as const)("creates one replacement once a stop receipt closes the %s lease", async (_driver, acquireLease) => {
const first = await acquireLease();
expect(await environmentService(db).releaseLease(first.lease.id, "released", { cleanupStatus: "success", remoteExecutionTermination: stopReceipt(first.lease) }))
.toMatchObject({ status: "released" });
await expectOneReplacement(first, acquireLease);
});
it("creates one replacement after the Runner complete operation stops a retained reusable lease", async () => {
const first = await acquirePlugin();
await db.update(environmentLeases).set({ status: "retained", leasePolicy: "reuse_by_environment", releasedAt: null })
.where(eq(environmentLeases.id, first.lease.id));
// The same write the `complete` operation makes for each reusable owner of a
// stopped task: no releaseLease call.
const [owner] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, first.lease.id));
await db.update(environmentLeases).set({ status: "released", releasedAt: new Date(), cleanupStatus: "success",
metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || ${JSON.stringify({
remoteExecutionTermination: remoteTerminationReceipt(owner!, { providerLeaseId: owner!.providerLeaseId, state: "stopped" }),
runnerTaskExecutionStopped: true })}::jsonb` }).where(eq(environmentLeases.id, first.lease.id));
await expectOneReplacement(first, acquirePlugin);
});
it("creates one replacement when only the destroyed successor proves an expired reusable lease stopped", async () => {
const first = await acquire();
const [{ agentId }] = await db.select({ agentId: agents.id }).from(agents).where(eq(agents.companyId, companyId));
const issueId = randomUUID(), successorRunId = randomUUID();
await db.insert(issues).values({ id: issueId, companyId, title: "Reused sandbox", status: "done" });
await db.insert(heartbeatRuns).values({ id: successorRunId, companyId, agentId, invocationSource: "on_demand", status: "succeeded" });
const acquiredAt = new Date(Date.now() - 2000), handoffAt = new Date(Date.now() - 1000);
const [previous] = await db.update(environmentLeases).set({ status: "expired", leasePolicy: "reuse_by_environment", issueId,
acquiredAt, releasedAt: handoffAt, cleanupStatus: "success", metadata: sql`${environmentLeases.metadata} || ${JSON.stringify({ agentId })}::jsonb` })
.where(eq(environmentLeases.id, first.lease.id)).returning();
const successor = { ...previous!, id: randomUUID(), heartbeatRunId: successorRunId, acquiredAt: handoffAt,
status: "released" as const, releasedAt: new Date(), metadata: { agentId } };
const handoff = createReusableLeaseHandoff(previous!, successor);
await db.update(environmentLeases).set({ metadata: sql`${environmentLeases.metadata} || ${JSON.stringify({ reusableLeaseSuccessor: handoff })}::jsonb` })
.where(eq(environmentLeases.id, first.lease.id));
await db.insert(environmentLeases).values({ ...successor, metadata: { agentId, reusableLeasePredecessor: handoff,
remoteExecutionTermination: remoteTerminationReceipt(successor, { providerLeaseId: successor.providerLeaseId, state: "destroyed" }) } });
const [stored] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, first.lease.id));
// No receipt is stored on the expired lease; recovery computes it from the successor.
expect(hasRemoteTerminationReceipt(stored!)).toBe(false);
expect(hasRemoteTerminationReceipt(await resolveRemoteTerminationLease(db, stored!))).toBe(true);
await expectOneReplacement(first, acquire);
});
it.each([
["failed cleanup", "failed", { cleanupStatus: "failed", receipt: true }],
["cleanup success without a receipt", "released", { cleanupStatus: "success", receipt: false }],
["a receipt for another lease", "released", { cleanupStatus: "success", receipt: "foreign" }],
["a receipt for another provider task", "released", { cleanupStatus: "success", receipt: "other-task" }],
["a retained lease", "retained", { cleanupStatus: "success", receipt: true }],
["pending cleanup", "pending_cleanup", { cleanupStatus: "failed", receipt: false }],
] as const)("keeps the creation key after %s", async (_case, status, options) => {
const first = await acquire();
const receipt = options.receipt === "foreign" ? stopReceipt({ ...first.lease, id: randomUUID() })
: options.receipt === "other-task" ? { ...stopReceipt(first.lease), providerLeaseId: "other-task" }
: options.receipt ? stopReceipt(first.lease) : undefined;
await environmentService(db).releaseLease(first.lease.id, status, { cleanupStatus: options.cleanupStatus,
...(receipt ? { remoteExecutionTermination: receipt } : {}) });
await expect(acquire()).rejects.toThrow("Sandbox creation has already transferred ownership");
expect(await operations()).toEqual([expect.objectContaining({ state: "attached", leaseId: first.lease.id })]);
expect(creates).toBe(1);
});
it("keeps an earlier task's operation when the lease now holds a replacement task", async () => {
const first = await acquire();
await db.update(sandboxCreationOperations).set({ metadata: sql`metadata || '{"creationProviderLeaseId":"earlier-task"}'::jsonb` });
await environmentService(db).releaseLease(first.lease.id, "released", { cleanupStatus: "success", remoteExecutionTermination: stopReceipt(first.lease) });
await expect(acquire()).rejects.toThrow("Sandbox creation has already transferred ownership");
expect(await operations()).toEqual([expect.objectContaining({ state: "attached", leaseId: first.lease.id })]);
expect(creates).toBe(1);
});
it("lets only one of two concurrent recoveries create the replacement", async () => {
const first = await acquire();
await environmentService(db).releaseLease(first.lease.id, "released", { cleanupStatus: "success", remoteExecutionTermination: stopReceipt(first.lease) });
// Hold whichever recovery reaches the provider until the other one has settled.
let settleOther!: () => void;
const otherSettled = new Promise<void>(resolve => { settleOther = resolve; });
beforeReply = () => otherSettled;
const outcomes = [acquire(), acquire()].map(attempt => attempt.then(
value => ({ value, error: undefined }), (error: unknown) => ({ value: undefined, error })));
const loser = await Promise.race(outcomes);
settleOther();
const settled = await Promise.all(outcomes);
expect(loser.error).toEqual(expect.objectContaining({ message: "Sandbox creation requires reconciliation before another attempt" }));
expect(settled.filter(outcome => outcome.value)).toHaveLength(1);
expect(creates).toBe(2);
expect((await operations()).filter(operation => operation.state !== "resolved")).toEqual([
expect.objectContaining({ state: "attached", leaseId: settled.find(outcome => outcome.value)!.value!.lease.id }),
]);
expect(await operations()).toHaveLength(2);
});
});
it("recovers a failed lease write with a fresh receiver and empty web disk", async () => {
await rejectWrites("environment_leases", "reject_fixture_lease");
await expect(acquire()).rejects.toThrow();
@@ -1,14 +1,15 @@
import { collectSecretRefPaths, isUuidSecretRef, parseSecretRefBindingObject, readConfigValueAtPath } from "./json-schema-secret-refs.js";
import { createHash, randomUUID } from "node:crypto";
import { and, eq, lte, or, sql } from "drizzle-orm";
import { and, eq, lte, ne, or, sql } from "drizzle-orm";
import { currentControllerProcess } from "./controller-process-identity.js";
import { companies, environments, heartbeatRuns, sandboxCreationOperations, type Db } from "@paperclipai/db";
import { companies, environmentLeases, environments, heartbeatRuns, sandboxCreationOperations, type Db } from "@paperclipai/db";
import { readEnvironmentCreationCleanupError, type PluginEnvironmentLease, type PluginEnvironmentLeaseObservation } from "@paperclipai/plugin-sdk";
import { assertEnvironmentCompanyBinding, environmentService } from "./environments.js";
import { ENVIRONMENT_CLEANUP_RETRY_MS, recoverableSandboxCreationCondition } from "./environment-cleanup-state.js";
import { logger } from "../middleware/logger.js";
import { notifyEnvironmentCleanup, writeEnvironmentCleanup } from "./environment-cleanup-notifications.js";
import { crashPoint } from "@paperclipai/shared/crash-points";
import { hasRemoteTerminationReceipt, resolveRemoteTerminationLease } from "./remote-execution-termination.js";
export type SandboxCreationOperation = typeof sandboxCreationOperations.$inferSelect;
@@ -70,9 +71,14 @@ export async function createOwnedSandbox(
const runId = owner.heartbeatRunId ?? randomUUID();
const settleAt = new Date(Date.now() + Math.max(CREATION_SETTLE_FLOOR_MS, (owner.timeoutMs ?? 0) + 60_000));
const creationController = await currentControllerProcess();
const operation = await db.transaction(async tx => {
const admit = (stopped?: SandboxCreationOperation) => db.transaction(async tx => {
await notifyEnvironmentCleanup(tx, { companyId: owner.companyId, runId: owner.heartbeatRunId ?? undefined });
await assertEnvironmentCompanyBinding(tx as unknown as Db, owner.environmentId, owner.companyId);
// Only one concurrent attempt resolves the stopped operation; any other one
// then meets the replacement below and is fenced exactly as before.
if (stopped) await tx.update(sandboxCreationOperations).set({ state: "resolved", updatedAt: new Date() }).where(and(
eq(sandboxCreationOperations.id, stopped.id), eq(sandboxCreationOperations.state, "attached"),
eq(sandboxCreationOperations.leaseId, stopped.leaseId!)));
const [inserted] = await tx.insert(sandboxCreationOperations).values({
id, companyId: owner.companyId, environmentId: owner.environmentId, originalEnvironmentId: owner.environmentId,
runId, requestKey: owner.requestKey ?? "run", issueId: owner.issueId, executionWorkspaceId: owner.executionWorkspaceId,
@@ -85,10 +91,13 @@ export async function createOwnedSandbox(
const [existing] = await tx.select().from(sandboxCreationOperations).where(and(
eq(sandboxCreationOperations.companyId, owner.companyId), eq(sandboxCreationOperations.originalEnvironmentId, owner.environmentId),
eq(sandboxCreationOperations.runId, runId), eq(sandboxCreationOperations.requestKey, owner.requestKey ?? "run"),
ne(sandboxCreationOperations.state, "resolved"),
));
if (!existing) throw new Error("Sandbox creation ownership was not recorded");
return existing;
});
let operation = await admit();
if (operation.id !== id && await createdSandboxStopped(db, operation)) operation = await admit(operation);
if (operation.id !== id) {
if (operation.state !== "attached" || !operation.leaseId) throw new Error("Sandbox creation requires reconciliation before another attempt");
const existing = await environmentService(db).getLeaseById(operation.leaseId);
@@ -116,6 +125,20 @@ export async function createOwnedSandbox(
}
}
/** An attached operation is finished once its closed lease carries the same
* stop proof same-run recovery requires (remoteExecutionHasStopped) for the
* sandbox this operation created. A task replacement can leave an earlier
* operation attached to a lease that now holds another sandbox; that operation
* is not proven and keeps its key. */
async function createdSandboxStopped(db: Db, operation: SandboxCreationOperation): Promise<boolean> {
if (operation.state !== "attached" || !operation.leaseId) return false;
const [lease] = await db.select().from(environmentLeases).where(and(
eq(environmentLeases.id, operation.leaseId), eq(environmentLeases.companyId, operation.companyId)));
if (!lease || lease.status === "active" || !lease.providerLeaseId) return false;
const created = operation.metadata.creationProviderLeaseId ?? lease.providerLeaseId;
return created === lease.providerLeaseId && hasRemoteTerminationReceipt(await resolveRemoteTerminationLease(db, lease));
}
/** Keep an already returned resource identity even when task attachment fails.
* This enriches the existing owner; it never transfers or reopens ownership. */
export async function recordSandboxCreationResource(db: Db, input: {