Jannik/PaperClipAI
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-10 20:46:28 +01:00
Code Issues Packages Projects Releases Wiki Activity
4,296 Commits 1,737 Branches 2,411 Tags
f06005ef27fa2ddd955ff1d330557986c4eabd35
Commit Graph
4 Commits
Author SHA1 Message Date
Devin Foley 547463d3a2 refactor(environments): make execution environments instance-scoped (#8375) 
## Thinking Path

> - Paperclip is the control plane for AI-agent companies, so execution
environment selection has to stay inspectable and predictable across
companies, agents, and runs.
> - The environment subsystem decides where an agent heartbeat actually
runs and how remote sandbox state is realized and restored.
> - That subsystem previously mixed company-scoped environment catalogs
with issue-level environment stamping, so a reassigned issue could keep
executing in the previous assignee's sandbox.
> - That behavior breaks the control-plane contract: changing the
assignee should change the executing agent/environment path unless there
is an explicit current override.
> - Fixing it cleanly required more than a narrow patch; the environment
model had to move to instance scope with a single inherited default and
per-agent override semantics.
> - This pull request rewires the schema, server/API surface, runtime
resolution, and UI around that model, then adds regression coverage for
cross-company inheritance and per-agent isolation.
> - The benefit is that environment choice now follows the approved
instance/agent configuration path instead of stale issue state, while
shared environments only need to be configured once per instance.

## Linked Issues or Issue Description

- No directly matching public GitHub issue or PR was found while
searching for this refactor.

### What happened?

Reassigning work between agents with different execution environments
could keep running in the previous sandbox because environment choice
was stamped onto the issue and outranked the current assignee. The same
subsystem also forced environment catalogs to be duplicated per company
even though the underlying execution environments were instance-wide
resources.

### Expected behavior

Execution should resolve through the current instance and agent
configuration path, with one instance-scoped environment catalog, one
instance default, optional per-agent override, and no stale issue-level
environment authority surviving reassignment.

### Steps to reproduce

1. Configure two agents to use different execution environments.
2. Assign an issue to the first agent so the issue records execution
state in that environment.
3. Reassign the same issue to the second agent and run another
heartbeat.
4. Observe that the pre-fix runtime can still sync or execute in the
original sandbox instead of the second agent's environment.

### Paperclip version or commit

Current `master` before this PR.

### Deployment mode

Self-hosted server.

### Installation method

Built from source (`pnpm dev` / `pnpm build`).

### Agent adapter(s) involved

- Claude Code
- Not adapter-specific (core bug in environment authority / resolution)

### Database mode

External Postgres.

### Access context

Both board reassignment and agent heartbeats were involved.

## What Changed

- Moved environments and their default selection contract to instance
scope in DB/shared types, including the migration that dedupes legacy
per-company environments and seeds the instance local default.
- Reworked environment CRUD/auth flows to use instance-scoped APIs and
added route/service coverage for instance-level environment management.
- Changed runtime resolution to prefer `agent default -> instance
default -> built-in local`, removed issue-level environment stamping
from the active execution path, and isolated sandbox/plugin leases by
`(executionWorkspaceId, agentId)`.
- Added environment env-var runtime precedence so environment-provided
values act as the baseline for agent execution.
- Moved the environment UI into instance settings and updated agent
configuration surfaces to reflect inherit/override behavior.
- Added regression coverage for instance-default inheritance across
companies and for the new runtime resolution behavior.
- Fixed a rebase-only duplicate `enableTaskWatchdogs` flag regression in
instance settings types/validators/services so the branch typechecks
cleanly on current `master`.
- Updated stale server tests so CI matches the shipped instance-scoped
environment contract.

## Verification

- `git diff --check`
- `pnpm --filter @paperclipai/shared typecheck`
- `pnpm --filter @paperclipai/db typecheck`
- `pnpm exec vitest run
server/src/__tests__/environment-runtime-driver-contract.test.ts
server/src/__tests__/agent-permissions-routes.test.ts
server/src/__tests__/environment-routes.test.ts
server/src/__tests__/environment-instance-routes.test.ts
server/src/__tests__/execution-workspace-policy.test.ts
server/src/__tests__/heartbeat-plugin-environment.test.ts
server/src/__tests__/instance-settings-routes.test.ts`

## Risks

- The migration changes environment scope and dedupes existing rows, so
installs with unusual legacy environment combinations should be reviewed
carefully during upgrade.
- Remote execution behavior now depends on instance-default inheritance
semantics instead of issue-level stamping, so any remaining code paths
that still assume issue-scoped environment authority would surface as
follow-up bugs.
- This PR includes both server/runtime behavior and UI relocation, so
reviewers should watch for authorization edge cases around instance
settings and environment management.

> I checked [`ROADMAP.md`](ROADMAP.md). This work fits the existing
Cloud / Sandbox agents direction as a bug-fix/refactor to current
behavior, not a new parallel product surface.

## Model Used

- OpenAI Codex coding agent in this Paperclip/Codex session; GPT-5-class
tool-using model with code execution and shell access. The exact backend
model ID is not exposed to the session runtime.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] If this change affects the UI, I have included before/after
screenshots
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
 2026-06-20 09:42:53 -07:00
Devin FoleyandPaperclip f3e01c63bd fix(environments): partial unique index to dedup managed sandbox rows (#8247) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The `server/services/environments.ts` module lazily provisions a
managed Kubernetes sandbox environment for each company on first
heartbeat. Idempotency relies on `ensureKubernetesEnvironment` returning
the single managed row per company.
> - The `(company_id, driver)` unique index in the `environments` schema
is partial on `driver='local'` only, so two concurrent callers (e.g.
simultaneous first heartbeats from a freshly synced tenant) can both
insert a `driver='sandbox'` row before either sees the other.
> - The function tried to converge after the race by re-reading, picking
the oldest managed row as winner, and deleting the loser. Under
autocommit + read-committed, each post-insert SELECT is a fresh snapshot
— A may not see B's row, B may not see A's, so both pick their own and
neither deletes. Two rows survive.
> - The same race fired in CI as `expected 2 to be 1` at
`environment-service.test.ts:293`, gating multiple unrelated PRs on
retries.
> - This pull request encodes the operator-level invariant ("at most one
Paperclip-managed sandbox row per company") at the DB layer with a
partial unique index, then switches `ensureKubernetesEnvironment` to
`INSERT … ON CONFLICT DO NOTHING` keyed on that index. Losers re-read
the surviving row.
> - The benefit is the race is impossible by construction — no
application-side convergence loop, no test flake, and any future
`ensureXyzSandboxEnvironment` that sets `managedByPaperclip=true`
inherits the invariant for free.

## Linked Issues or Issue Description

Paperclip issue: PAPA-783 — implement managed-sandbox dedup fix (phase 2
of the approved plan on the parent flake-investigation issue).

This is the phase-2 fix for the flaky `environmentService > deduplicates
concurrent managed Kubernetes environment creation` test introduced by
`4ad94d0bd` (PR #7938). Failing CI runs since then on at least PRs
#7595, #8233, #8215, #8212. The plan was reviewed and approved on the
parent issue before implementation.

Closely related (not duplicates):
- PR #7938 — introduced the test and the in-process convergence loop
being replaced here.
- PR #7595, PR #8233, PR #8215, PR #8212 — downstream PRs affected by
the flake; one of them will be rebased onto this fix as the acceptance
gate.

## What Changed

- `packages/db/src/schema/environments.ts`: added
`environments_company_managed_sandbox_idx`, a partial unique index on
`(company_id) WHERE driver='sandbox' AND
(metadata->>'managedByPaperclip')::boolean = true`. The umbrella
`managedByPaperclip` predicate covers any current or future
Paperclip-managed sandbox flavor without needing a new index per
provider.
- `packages/db/src/migrations/0102_managed_sandbox_dedup_index.sql`:
one-shot dedup `DELETE` keeping the oldest managed-sandbox row per
`company_id` (scoped to `driver='sandbox' AND managedByPaperclip=true`),
`RAISE NOTICE` if any duplicates were removed, then `CREATE UNIQUE INDEX
IF NOT EXISTS environments_company_managed_sandbox_idx`. `CONCURRENTLY`
is omitted because the codebase's migration runner wraps each file in a
transaction (see `applyPendingMigrationsManually`); the table holds 1–3
rows per company, so the short lock is acceptable and consistent with
every other migration in the repo.
- `server/src/services/environments.ts`: `ensureKubernetesEnvironment`
now uses `INSERT … ON CONFLICT DO NOTHING` keyed on the new index. On
conflict it re-reads the surviving managed-sandbox row and returns it.
Drops the post-insert convergence (re-read by `createdAt ASC, id ASC`,
delete the loser) and the trailing comment that flagged "until a partial
unique index is added via migration" as the proper long-term fix.
- Unused `asc` import removed from
`server/src/services/environments.ts`.

## Verification

Local (matches the success criteria in the issue body):

```
$ cd server
$ passes=0; for i in $(seq 1 20); do
    pnpm vitest run src/__tests__/environment-service.test.ts -t "deduplicates concurrent" \
      && passes=$((passes+1)) || break
  done; echo "$passes/20"
20/20

$ passes=0; for i in $(seq 1 10); do
    pnpm vitest run src/__tests__/environment-service.test.ts \
      && passes=$((passes+1)) || break
  done; echo "$passes/10"
10/10
```

Adversarial fan-out stress (temporarily bumped `Array.from({ length: 8
}, …)` to `length: 32` on both dedup tests; reverted before commit):

```
$ # both dedup tests fan-out-of-32, 10 iterations
10/10
```

Two ensure paths exist in the parent plan, but only
`ensureKubernetesEnvironment` is on `master`.
`ensureManagedSandboxEnvironment` (referenced by the approved plan as
commit `dce9a9622`) lives only on an unmerged feature branch, not
master. The plan's helper-extraction and symmetric dedup test for that
path are deferred to whichever PR lands the second ensure path — it
inherits the same DB invariant by setting `managedByPaperclip=true`.
Discrepancy was flagged on the issue thread before implementation.

Typecheck:

```
$ pnpm -C server typecheck
ok
```

## Risks

Low risk.

- **Migration safety.** `IF NOT EXISTS` on the index makes the migration
idempotent. The dedup `DELETE` is bounded to rows matching the
managed-sandbox predicate; in production this should be a no-op (no race
has been reported in the wild — only in CI). On dev/CI DBs that already
accumulated duplicates, the migration emits a `NOTICE` reporting the
count.
- **No `CONCURRENTLY`.** The migration runner wraps each `.sql` file in
a transaction, which is incompatible with `CREATE INDEX CONCURRENTLY`.
The `environments` table holds 1–3 rows per company and the row count is
bounded by company count; the short ACCESS EXCLUSIVE during `CREATE
UNIQUE INDEX` is acceptable here and matches every other index migration
in the repo.
- **Predicate scope.** The partial index predicate matches exactly the
rows that `ensureKubernetesEnvironment` writes (`driver='sandbox'` with
`metadata.managedByPaperclip=true`). Tenant-created sandbox envs (via
`svc.create`) do not set this marker and are not covered — no false
positives, no surprise constraint violations on unrelated inserts.

## Model Used

Claude (Anthropic), `claude-opus-4-7`. Tool use: code edit + bash +
filesystem search; no extended-thinking mode.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] If this change affects the UI, I have included before/after
screenshots
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-06-17 16:09:45 -07:00
Devin Foley e4995bbb1c Add SSH environment support (#4358) 
## Thinking Path

> - Paperclip orchestrates AI agents for zero-human companies
> - The environments subsystem already models execution environments,
but before this branch there was no end-to-end SSH-backed runtime path
for agents to actually run work against a remote box
> - That meant agents could be configured around environment concepts
without a reliable way to execute adapter sessions remotely, sync
workspace state, and preserve run context across supported adapters
> - We also need environment selection to participate in normal
Paperclip control-plane behavior: agent defaults, project/issue
selection, route validation, and environment probing
> - Because this capability is still experimental, the UI surface should
be easy to hide and easy to remove later without undoing the underlying
implementation
> - This pull request adds SSH environment execution support across the
runtime, adapters, routes, schema, and tests, then puts the visible
environment-management UI behind an experimental flag
> - The benefit is that we can validate real SSH-backed agent execution
now while keeping the user-facing controls safely gated until the
feature is ready to come out of experimentation

## What Changed

- Added SSH-backed execution target support in the shared adapter
runtime, including remote workspace preparation, skill/runtime asset
sync, remote session handling, and workspace restore behavior after
runs.
- Added SSH execution coverage for supported local adapters, plus remote
execution tests across Claude, Codex, Cursor, Gemini, OpenCode, and Pi.
- Added environment selection and environment-management backend support
needed for SSH execution, including route/service work, validation,
probing, and agent default environment persistence.
- Added CLI support for SSH environment lab verification and updated
related docs/tests.
- Added the `enableEnvironments` experimental flag and gated the
environment UI behind it on company settings, agent configuration, and
project configuration surfaces.

## Verification

- `pnpm exec vitest run
packages/adapters/claude-local/src/server/execute.remote.test.ts
packages/adapters/cursor-local/src/server/execute.remote.test.ts
packages/adapters/gemini-local/src/server/execute.remote.test.ts
packages/adapters/opencode-local/src/server/execute.remote.test.ts
packages/adapters/pi-local/src/server/execute.remote.test.ts`
- `pnpm exec vitest run server/src/__tests__/environment-routes.test.ts`
- `pnpm exec vitest run
server/src/__tests__/instance-settings-routes.test.ts`
- `pnpm exec vitest run ui/src/lib/new-agent-hire-payload.test.ts
ui/src/lib/new-agent-runtime-config.test.ts`
- `pnpm -r typecheck`
- `pnpm build`
- Manual verification on a branch-local dev server:
  - enabled the experimental flag
  - created an SSH environment
  - created a Linux Claude agent using that environment
- confirmed a run executed on the Linux box and synced workspace changes
back

## Risks

- Medium: this touches runtime execution flow across multiple adapters,
so regressions would likely show up in remote session setup, workspace
sync, or environment selection precedence.
- The UI flag reduces exposure, but the underlying runtime and route
changes are still substantial and rely on migration correctness.
- The change set is broad across adapters, control-plane services,
migrations, and UI gating, so review should pay close attention to
environment-selection precedence and remote workspace lifecycle
behavior.

## Model Used

- OpenAI Codex via Paperclip's local Codex adapter, GPT-5-class coding
model with tool use and code execution in the local repo workspace. The
local adapter does not surface a more specific public model version
string in this branch workflow.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] If this change affects the UI, I have included before/after
screenshots
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] I will address all Greptile and reviewer comments before
requesting merge
 2026-04-23 19:15:22 -07:00
Devin Foley 13551b2bac Add local environment lifecycle (#4297) 
## Thinking Path

> - Paperclip orchestrates AI agents for zero-human companies.
> - Every heartbeat run needs a concrete place where the agent's adapter
process executes.
> - Today that execution location is implicitly the local machine, which
makes it hard to track, audit, and manage as a first-class runtime
concern.
> - The first step is to represent the current local execution path
explicitly without changing how users experience agent runs.
> - This pull request adds core Environment and Environment Lease
records, then routes existing local heartbeat execution through a
default `Local` environment.
> - The benefit is that local runs remain behavior-preserving while the
system now has durable environment identity, lease lifecycle tracking,
and activity records for execution placement.

## What Changed

- Added `environments` and `environment_leases` database tables, schema
exports, and migration `0065_environments.sql`.
- Added shared environment constants, TypeScript types, and validators
for environment drivers, statuses, lease policies, lease statuses, and
cleanup states.
- Added `environmentService` for listing, reading, creating, updating,
and ensuring company-scoped environments.
- Added environment lease lifecycle operations for acquire, metadata
update, single-lease release, and run-wide release.
- Updated heartbeat execution to lazily ensure a company-scoped default
`Local` environment before adapter execution.
- Updated heartbeat execution to acquire an ephemeral local environment
lease, write `paperclipEnvironment` into the run context snapshot, and
release active leases during run finalization.
- Added activity log events for environment lease acquisition and
release.
- Added tests for environment service behavior and the local heartbeat
environment lifecycle.
- Added a CI-follow-up heartbeat guard so deferred issue comment wakes
are promoted before automatic missing-comment retries, with focused
batching test coverage.

## Verification

Local verification run for this branch:

- `pnpm -r typecheck`
- `pnpm build`
- `pnpm exec vitest run server/src/__tests__/environment-service.test.ts
server/src/__tests__/heartbeat-local-environment.test.ts --pool=forks`

Additional reviewer/CI verification:

- Confirm `pnpm-lock.yaml` is not modified.
- Confirm `pnpm test:run` passes in CI.
- Confirm `PAPERCLIP_E2E_SKIP_LLM=true pnpm run test:e2e` passes in CI.
- Confirm a local heartbeat run creates one active `Local` environment
when needed, records one lease for the run, releases the lease when the
run finishes, and includes `paperclipEnvironment` in the run context
snapshot.

Screenshots: not applicable; this PR has no UI changes.

## Risks

- Migration risk: introduces two new tables and a new migration journal
entry. Review should verify company scoping, indexes, foreign keys, and
enum defaults are correct.
- Lifecycle risk: heartbeat finalization now releases environment leases
in addition to existing runtime cleanup. A finalization bug could leave
stale active leases or mark a failed run's lease incorrectly.
- Behavior-preservation risk: local adapter execution should remain
unchanged apart from environment bookkeeping. Review should pay
attention to the heartbeat path around context snapshot updates and
final cleanup ordering.
- Activity volume risk: each heartbeat run now logs lease acquisition
and release events, increasing activity log volume by two records per
run.

## Model Used

OpenAI GPT-5.4 via Codex CLI. Capabilities used: repository inspection,
TypeScript implementation review, local test/build execution, and
PR-description drafting.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] If this change affects the UI, I have included before/after
screenshots (N/A: no UI changes)
- [x] I have updated relevant documentation to reflect my changes (N/A:
no user-facing docs or commands changed)
- [x] I have considered and documented any risks above
- [x] I will address all Greptile and reviewer comments before
requesting merge
 2026-04-22 20:07:41 -07:00