mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-12 18:06:11 +02:00
dcac49a4fd
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Isolated workspaces give each task a safe and reproducible checkout. > - The existing setup cloned the development database before an agent needed to run the app. > - This made worktree creation slower and heavier for tasks that never start a service. > - Runtime services already use one server start path for heartbeat, operator, and startup recovery flows. > - This pull request moves heavy setup to that start path and keeps worktree creation lean. > - The benefit is faster isolated workspace creation with the same reliable runtime setup when a service starts. ## Linked Issues or Issue Description Related pull request: #10652 covers the initial deferred database-seeding slice. This pull request supersedes it with end-to-end runtime provisioning and safe cleanup. **What existing behavior does this improve?** This improves isolated worktree creation, runtime service startup, and isolated instance cleanup. **Subsystem affected** Cross-cutting: CLI worktree setup, server runtime orchestration, shared workspace contracts, and development scripts. **Current behavior** Paperclip seeds an isolated development database during worktree creation. It can also leave an isolated instance directory after workspace teardown. This work happens even when no runtime service starts. **Proposed behavior** Paperclip creates the worktree with a lean eager setup. It runs an idempotent runtime provision command before the first managed service spawn. Concurrent starts share one provision attempt. Teardown removes the isolated instance safely. **Reason and benefit** Many agent tasks only edit and test code. They do not need a running Paperclip instance. Deferring the database seed reduces workspace startup cost while preserving automatic setup for tasks that start the app. **Breaking changes** None. The new runtime provision command is optional. Existing workspace behavior is unchanged when it is absent. ## What Changed - Split Paperclip worktree setup into a lean eager script and an idempotent runtime provision script. - Added `runtimeProvisionCommand` to project, issue, realized workspace, and persisted workspace contracts. - Added a per-workspace provision mutex before local service spawn for heartbeat, operator, and startup recovery flows. - Added a persisted `provisioning` service state and the `workspace_runtime_provision` operation phase. - Kept provision time outside the service readiness timeout and made failed attempts visible and retryable. - Reclaimed isolated instance data during safe workspace teardown. - Serialized deferred database seeding across processes and bound teardown to the instance root captured in persisted workspace metadata. - Added tests for config flow, concurrency, retry, no-op behavior, readiness timing, scripts, CLI commands, and cleanup. - Documented the eager and runtime provisioning contracts. ## Verification - `pnpm -r typecheck` - `pnpm build` - `pnpm test:run` (server: 3,201 passed; UI: 3,345 passed; the CLI phase exposed one environment-sensitive AWS doctor assertion because the agent runtime injects static AWS credentials) - `env -u AWS_ACCESS_KEY_ID -u AWS_SECRET_ACCESS_KEY pnpm exec vitest run cli/src/__tests__/secrets.test.ts -t 'passes AWS doctor checks when non-secret provider config is present'` - Focused runtime tests cover serialized provisioning, retry after stderr failure, absent-command no-op behavior, operation logging, persisted state order, and readiness timeout exclusion. - Focused CLI and cleanup tests cover concurrent seed serialization, stale-lock fail-closed behavior, persisted instance ownership, and rewritten sibling pointers. ## Risks - A faulty runtime provision script blocks service startup. Paperclip records stderr, marks the service failed, and retries on the next start. - Concurrent service requests share an in-process provision attempt, while the seed command uses an atomic filesystem lock across processes. A stale lock fails closed and requires an operator to verify no seed is running before removing it. - Isolated instance cleanup is destructive. The cleanup service validates ownership and path containment before removal. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI Codex, `gpt-5.6-sol`, with agentic reasoning, tool use, and code execution. The service does not expose the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
100 lines
7.1 KiB
Markdown
100 lines
7.1 KiB
Markdown
---
|
|
title: Execution Workspaces And Runtime Services
|
|
summary: How project runtime configuration, execution workspaces, and issue runs fit together
|
|
---
|
|
|
|
This guide documents the intended runtime model for projects, execution workspaces, and issue runs in Paperclip.
|
|
|
|
Paperclip now presents this as a workspace-command model:
|
|
|
|
- `Services` are long-running commands that stay supervised.
|
|
- `Jobs` are one-shot commands that run once and exit.
|
|
- Raw runtime JSON is still available for advanced config, but it is no longer the primary mental model.
|
|
|
|
## Project runtime configuration
|
|
|
|
You can define how to run a project on the project workspace itself.
|
|
|
|
- Project workspace runtime config describes the services and jobs available for that project checkout.
|
|
- This is the default runtime configuration that child execution workspaces may inherit.
|
|
- Defining the config does not start anything by itself.
|
|
|
|
## Runtime control: manual and heartbeat-driven
|
|
|
|
Workspace commands can be controlled manually from the UI, and heartbeat runs also start services automatically.
|
|
|
|
- Project workspace services are started and stopped from the project workspace UI, and project jobs can be run on demand there.
|
|
- Execution workspace services are started and stopped from the execution workspace UI, and execution-workspace jobs can be run on demand there.
|
|
- Heartbeat runs also auto-start the workspace's runtime services at the beginning of an issue run. `ensureRuntimeServicesForRun` (`server/src/services/workspace-runtime.ts`, called from `server/src/services/heartbeat.ts`) starts each service whose desired state resolves to `running` — which is the default when no explicit per-service desired state is set. A running service that matches an existing reuse key is reused rather than restarted.
|
|
- You can opt a service out of that auto-start by setting its desired state to `stopped`/`manual` in the runtime config; those services stay UI-controlled.
|
|
- Paperclip does not automatically restart workspace services on server boot — services only come back up when the next run (or a manual start) brings them up.
|
|
|
|
## Execution workspace inheritance
|
|
|
|
Execution workspaces isolate code and runtime state from the project primary workspace.
|
|
|
|
- An isolated execution workspace has its own checkout path, branch, and local runtime instance.
|
|
- The runtime configuration may inherit from the linked project workspace by default.
|
|
- The execution workspace may override that runtime configuration with its own workspace-specific settings.
|
|
- The inherited configuration answers "which commands exist and how to run them", but any running service process is still specific to that execution workspace.
|
|
|
|
## Issues and execution workspaces
|
|
|
|
Issues are attached to execution workspace behavior, not to automatic runtime management.
|
|
|
|
- An issue may create a new execution workspace when you choose an isolated workspace mode.
|
|
- An issue may reuse an existing execution workspace when you choose reuse.
|
|
- Multiple issues may intentionally share one execution workspace so they can work against the same branch and running runtime services.
|
|
- Running an issue auto-starts the workspace's `running`-desired runtime services for the duration of the run (see "Runtime control" above); it does not stop them when the run ends unless they are ephemeral and no other run holds a lease.
|
|
|
|
## Execution workspace lifecycle
|
|
|
|
Execution workspaces are durable until a human closes them.
|
|
|
|
- The UI can archive an execution workspace.
|
|
- Closing an execution workspace stops its runtime services and cleans up its workspace artifacts when allowed.
|
|
- Shared workspaces that point at the project primary checkout are treated more conservatively during cleanup than disposable isolated workspaces.
|
|
|
|
## Resolved workspace logic during heartbeat runs
|
|
|
|
Heartbeat resolves a workspace for the run (code location and session continuity) and also brings up that workspace's runtime services.
|
|
|
|
1. Heartbeat resolves a base workspace for the run.
|
|
2. Paperclip realizes the effective execution workspace, including creating or reusing a worktree when needed.
|
|
3. Paperclip persists execution-workspace metadata such as paths, refs, and provisioning settings.
|
|
4. Heartbeat passes the resolved code workspace to the agent run.
|
|
5. Heartbeat calls `ensureRuntimeServicesForRun` to start the workspace's `running`-desired runtime services, running the lazy runtime provision command first if one is configured and has not yet run (see "Lazy runtime provisioning" below).
|
|
|
|
## Lazy runtime provisioning
|
|
|
|
Some workspaces need heavy one-time setup — seeding a database, warming caches — before their runtime services can start. That work can be deferred to the first runtime-service start instead of running eagerly during workspace preparation.
|
|
|
|
- Configure a **runtime provision command** on the project's workspace strategy (Project properties → execution workspace), or override it per execution workspace on the workspace's Configuration tab.
|
|
- When set, workspace preparation stays lean and the command runs exactly once, immediately before the first runtime-service start for that workspace. Leaving it empty keeps the legacy eager path (all setup during workspace provisioning).
|
|
- The command's outcome is recorded as a `workspace_runtime_provision` operation on the execution workspace and surfaced on the workspace detail page:
|
|
- **Deferred** — configured but not yet run (no runtime service has started yet).
|
|
- **Provisioned at <time>** — the command completed successfully.
|
|
- **Provisioning failed** — the command failed; the workspace detail links to the runtime logs for the failing operation.
|
|
- While the command runs, the runtime service shows a **Provisioning…** state before it transitions to starting/running.
|
|
|
|
## Cross-run persistence (no-remote-git contract)
|
|
|
|
Code state moves between runs through the local execution-workspace cwd alone — not through a git remote.
|
|
|
|
- Each run's prepare step bundles the local worktree to the run's remote dir over ssh, with no `git remote` configured.
|
|
- The adapter's restore step at the end of the run writes any new remote commits back into the local worktree directly.
|
|
- Adapters must never `git push` from runtime code, and must never assume a remote exists.
|
|
- A failed restore is a run-level error and records `workspace_finalize=failed` on the execution workspace, which gates dependent issue wakes until the next successful finalize.
|
|
|
|
The invariant is enforced by the "no-remote-git contract" case in `packages/adapter-utils/src/ssh-fixture.test.ts`, which asserts a remote-only commit reaches the local worktree with no remote configured at any point.
|
|
|
|
## Current implementation guarantees
|
|
|
|
With the current implementation:
|
|
|
|
- Project workspace command config is the fallback for execution workspace UI controls.
|
|
- Execution workspace runtime overrides are stored on the execution workspace.
|
|
- Heartbeat runs auto-start the workspace's `running`-desired runtime services (via `ensureRuntimeServicesForRun`); services set to `stopped`/`manual` stay UI-controlled.
|
|
- A configured runtime provision command runs once, lazily, before the first runtime-service start.
|
|
- Server startup does not auto-restart workspace services.
|