mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-23 17:33:51 +02:00
989ee668e08b4161ff7cfdedb5f19ccfafe740e4
5
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Nicky LeachandPaperclip
|
989ee668e0 |
Support provider-owned fresh retry workspaces (#17252)
## Thinking Path > - Paperclip coordinates AI agents and retains their work across failures. > - A retry can require a new workspace while preserving the failed attempt. > - Core rejects this recovery strategy for provider-owned storage. > - Providers should implement the required outcome using their own storage design. > - This pull request adds a semantic fresh-preparation contract and delegates recovery preparation. > - The benefit is that a supported provider can run a fresh retry without losing prior work. ## Linked Issues or Issue Description **What happened?** A retry that selects `fresh_isolated_workspace` fails with `fresh_workspace_storage_unsupported` for provider-owned storage. **Expected behavior** Core asks a capable provider to prepare new storage and attach verified recovery files. The failed workspace remains intact. **Steps to reproduce** Run a task in a task-capable plugin environment. Let the task fail without a verified workspace preservation receipt. Retry the task so Core selects a fresh attempt. **Paperclip version or commit** |
||
|
Nicky LeachandPaperclip
|
e5351b240a |
Verify native Runner setup through plugin tasks (#17029)
## Thinking Path > - Paperclip manages work for AI agents. > - An environment plugin can own a Runner task. > - The merged launch-preparation contract lets the execution owner build the provider configuration. > - Core still needs a task process owner and a setup test that uses that configuration. > - This PR connects the task API to the Runner process-owner contract. > - Operators can test a plugin environment before Core sends it task work. ## Linked Issues or Issue Description **Subsystem affected** Runner, plugin SDK, and agent environment tests. **Problem or motivation** A task-only environment cannot use the shell-based setup path. Core must verify the Runner hello exchange and process cleanup through dedicated task operations. **Proposed solution** Add a lease-bound task driver and use it for a bounded setup probe. Keep heartbeat admission and task execution in #16978. This PR follows the merged launch-preparation contract in #16997. **Alternatives considered** A provider declaration alone cannot prove that Runner and the provider connect. A shell bridge would give the plugin an unnecessary command interface. ## What Changed - Pass structured Runner launch data to process owners. Support an external connection and explicit detach. - Add typed launch authentication checks and checkpoint operations. Check lease, run, project, and retained-task authority. - Connect launch preparation, PRP, termination proof, and credential handoff to the task driver. Reject local environment overrides during installed-account setup. Pass saved-account identity and credential home directly from Core. - Run a short-lived setup probe through the existing agent environment test API. Require a verified provider hello and cleanup. Keep saved Copilot account checks metadata-only through a dedicated task operation. - Update the Copilot source identity because the shared transport source changed. Keep provider behavior and the earlier identity fixture. ## Verification - CI passes on `f987adec7`: 55 successful checks and 5 expected skips. Standard Greptile and Apex are both 5/5 with no open findings. The branch merges cleanly with current master. - Repository typecheck and build pass after the rebase onto the workspace-provider contract. Server typecheck and build also pass after the final setup fix. - The 110 focused lease, SDK, setup, Copilot, and route cases pass after the rebase. All 14 setup cases pass after the final change. These include a real local Runner with a test provider. The saved-agent model-discovery case also passes. - Apex found that Core's relative workspace marker failed the driver path check, and non-Codex sessions omitted the observed model. Both failures were reproduced and fixed. The probe leaves physical workspace selection to plugin preparation. The transport reads each provider's observed model and exposes it to the driver. All 14 setup cases, 6 transport peer cases, and both existing real Runner/OpenCode cases pass. Runner build and server typecheck pass. - The coverage audit removed two redundant cases, repeated fixture code, and two extra real Runner launches. It kept authorization, credential selection, cancellation, lost replies, termination proof, checkpoint identity, and provider setup checks. - The Copilot source-identity check passes. The added-line secret and PII scan passes. The unmerged v52 attestation was regenerated after the shared transport fix. - CI on `13faeff02` caught one stale Copilot digest assertion after attestation regeneration. The literal now matches the candidate attestation, and all 49 installation cases pass. That CI lane otherwise passed 1,818 cases. - An earlier CI run hit an unchanged UI toast timing test. The separate fix is tracked in #16952. Current CI passes that shard. CI also caught a stale model-discovery assertion in this change; the assertion is fixed. - The local full-suite run was stopped when master changed. The final CI run supplies exact-head full-suite validation. No live Cloud or provider test was run. ## Risks - Native hello tests require a saved Core account or the installed account. Local environment overrides are unsupported for installed-account probes and fail before the provider runs. - A native task provider must declare launch preparation and authentication-check version 1. It must keep the checked configuration fixed through submission. Core rejects providers that cannot make this guarantee. - A lost task reply does not prove process termination. Core retains grants until the provider proves termination and Core saves required state. - This PR does not enable native heartbeat dispatch through plugin environments. Cloud deployment and live provider validation remain separate. ## Model Used OpenAI Codex, GPT-6. The exact model ID and context window are not exposed in this session. Tool use includes code editing, local tests, and the GitHub CLI. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal issue details - [x] I have run focused tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandPaperclip
|
8b4ab65b1c |
Add plugin-owned project storage and task workspaces (#17001)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Tasks can use isolated repository workspaces. > - Core currently prepares those repositories on its own filesystem. > - A storage provider may only expose repositories inside remote agent computers. > - Project storage and task workspace preparation must have one provider owner. > - This pull request adds that contract and keeps execution as a separate plugin interface. > - Core keeps authorization, durable task bindings, and lifecycle fences. ## Linked Issues or Issue Description **Subsystem affected** Plugin SDK, project workspace policy, heartbeat preparation, execution leases, and workspace lifecycle. **Problem or motivation** Core cannot prepare a task workspace when its repository storage is unavailable on the Core host. An execution lease must not be required just to prepare persistent storage. **Proposed solution** Let one plugin provide project resource lifecycle and workspace prepare, inspect, restore, and remove behavior. Core reserves a task workspace before it calls the plugin. The plugin returns an opaque durable reference. The matching execution plugin receives that reference on its task RPC. **Alternatives considered** Preparing repositories on Core would require a provider mount on Core. Preparing them only during task launch would tie persistent workspace lifecycle to execution leases and agent availability. **Roadmap alignment** This extends the remote provider model in ROADMAP.md. Related execution work is in #16978. This PR does not replace that task bridge. Native execution of managed storage requires the bridge; this change fails closed when it is absent. ## What Changed - Add one storage provider declaration with supported modes/strategies, a capability, a validated RPC contract, and a test harness. - Persist provider ownership, repository revision, and execution-workspace identity before external effects. Reconcile retries with a stable operation ID. - Prepare provider storage before environment acquisition. Keep Core filesystem preparation out of this path. - Pin the durable reference to the matching execution plugin and validate the lease binding before task dispatch. - Route inspection, reopen, and removal through the existing lifecycle process. Keep local data when remote recovery is uncertain. - Keep the application database pool free during lifecycle locks. Preserve pending workspace selections, validate source edits before retirement, and retry unfinished closed storage cleanup. - Protect provider metadata and live repository sources from caller edits. Storage edits retire only closed records with confirmed removal; retired records cannot reopen. ## Verification - Current head `8214b3ca5`: all 25 provider lifecycle tests pass with `maxConnections=1`. They cover preparation, inspection, reopening, removal, a paused provider with responsive ordinary SQL and a competing source edit, pending selection preservation, rejected source edits, cleanup retry scheduling, and providers that cache operation results, and committed cleanup notifications for storage intent and recovered state. - Final frozen-tree lifecycle run on `8214b3ca5`: all 318 tests passed in 10 files, covering workspace services, reopen, compaction, task binding, project sources, heartbeat preparation, display caching, and source locks. - Direct server `tsc --noEmit` passes. The SDK build and all 128 SDK tests passed during initial verification; later SDK changes are documentation only. - Full CI on the previous head `ec5c7209a` passed all 55 applicable checks; 2 conditional checks skipped. Standard Greptile reviewed current head `342e6d9c3` at 5/5 with no unresolved threads. All 55 applicable CI checks passed on `342e6d9c3`; 2 conditional checks skipped. The Slack browser and CLI startup jobs passed on their single unchanged-head rerun. Its Apex review returned 4/5 with one pending-restoration issue. Commit `1b58943b1` reconciles that durable intent before cleanup, advances the generation without reopening the Core row, and blocks direct removal or retirement while intent remains unresolved. Standard Greptile reviewed `1b58943b1` at 5/5 with no new findings. All 54 applicable CI checks passed on `1b58943b1`; 3 conditional checks skipped. The chat job passed on its single unchanged-head rerun. Apex then returned 4/5 for a missing cleanup wake after recovery. Commit `8214b3ca5` atomically registers cleanup intent before storage intent and recovery-result writes. Its provider test observes real committed notification hints and the visible advanced generation. Standard Greptile reviewed `8214b3ca5` at 5/5 with no new findings. All 55 applicable current-head CI checks passed on the first run; 2 conditional checks skipped. Final Apex verification completed at 5/5 on this exact head: 38 files reviewed, no new comments, and no unresolved threads. The PR is ready for review and mergeable. - The first Apex review found five issues after a standard 5/5: application-pool starvation, stale pending selections, retirement during rejected source edits, lost cleanup retries, and reused inspection identities. Commits `6d72e7fd4` and `342e6d9c3` fix each with regression coverage. Archive lock waits use dedicated connections, and source edits fail promptly if the lifecycle is busy. Retirement compares normalized values. The normalized-source regression initially failed with the raw-value comparison; after moving the check after normalization, a clean provider-suite rerun passed all 24 tests at that stage. Those threads are resolved; the earlier Apex 0/5 is not a passing acceptance result. - Earlier local integration attempts could not bind localhost inside the sandbox. The same suites passed with permission to start local PostgreSQL. - The earlier broad local `pnpm test:run` overlapped edits and reported stale-revision and checkout-path/Git-fixture failures. It was not a final-tree result. Full CI subsequently passed; the original local failure remains part of verification history. - Local repository-wide typecheck/build cannot complete because `cargo` is unavailable. Direct server TypeScript checks and affected tests pass; full Rust and build validation comes from CI. - An earlier browser run on `ff918c5a8` failed in the unchanged case-filter journey: expected “Done,” received “All statuses,” with an announcement handler recorded during the assertion. The cause is not confirmed. [The failed job](https://github.com/paperclipai/paperclip/actions/runs/39500611857/job/118896954508) remains verification history. The unchanged shard passed in the next full CI run on `ec5c7209a`; no UI or fixture change was made. - Current-head browser shard 2 failed waiting five seconds for the mocked Slack setup Success heading. This PR changes neither that UI nor its fixture. [The failed job](https://github.com/paperclipai/paperclip/actions/runs/39506070513/job/118916021232) remains verification evidence. The cause is not confirmed; the single unchanged-shard rerun passed; no UI or fixture change was made, and the original cause remains unconfirmed. - The workspace test shard passed its 3,978 general tests, then its CLI company-import/export fixture timed out after 30 seconds waiting for startup health before creating any company or workspace. [The failed job](https://github.com/paperclipai/paperclip/actions/runs/39506070513/job/118916021855) remains verification history. The child reached external-PostgreSQL startup without exiting or reporting an exception. Its cause is not confirmed; the single unchanged-job rerun passed, including the CLI fixture. No startup or fixture change was made, and the original cause remains unconfirmed. - The new cached-removal regression initially failed because its fake worker advertised availability for unrelated fixture plugins, so another workspace changed its fake checkout state. Worker availability now matches the fixture plugin; the complete 25-test provider suite passes. - On `1b58943b1`, the chat shard timed out polling for one conversation after an authorized GitHub description edit. [The failed job](https://github.com/paperclipai/paperclip/actions/runs/39510521657/job/118930193418) remains evidence. The exact case passes locally against PostgreSQL (1 passed, 1,380 unrelated cases skipped), and this PR changes neither that test nor its handler. The original cause is unconfirmed; the single unchanged-job CI rerun passed without a code or fixture change. The original cause remains unconfirmed. - The diff was checked for credentials, private references, and generated artifacts. ## Risks This is an additive provider contract with no database migration. Existing projects keep their local behavior unless their policy selects a storage plugin. A selected provider must be ready and enabled. Unknown or unsafe cleanup fails closed. Managed native execution depends on #16978. Core refuses a host Runner fallback. Providers must fence lifecycle generations, preserve ambiguous retries, and report remote recovery honestly. Provider-specific live storage and guest mount qualification remain separate from these Core contract tests. ## Model Used OpenAI Codex (GPT-6), with repository inspection, code execution, and tests. The exact model variant was not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked existing issues or described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket id - [x] I have run tests locally and they pass (affected suites; broad-run limits are documented above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandPaperclip
|
dc6dd23ea4 |
Add plugin Runner preparation task contract (#16993)
## Thinking Path > - Paperclip manages work for AI agents. > - Environment plugins can own Runner tasks. > - Core selects provider policy and runtime assignments. > - The execution environment must resolve its own paths and configuration. > - The task API needs a preparation exchange before execution. > - This PR adds that exchange under the existing lease authority. ## Linked Issues or Issue Description **Problem or motivation** The environment task API can start a Runner. It cannot return a prepared provider configuration from Core requirements. **Proposed solution** Add a versioned `prepare` request and `prepared` result. Keep the provider configuration bound to its task and requirements. Check the same company, run, project, and retained-session authority as submission. **Alternatives considered** A shell preparation command would expose execution details in the API. Provider-specific ownership flags would spread this contract across backends. **Roadmap alignment** This extends the existing plugin and remote execution APIs. It does not add a separate execution system. Refs #16977, #15406. The prerequisite #16977 is merged. This PR now targets master. Runner transport and provider implementations follow separately. ## What Changed - Add strict launch requirements and prepared result schemas. - Add the preparation capability version and saved configuration ID. - Apply existing lease and resource checks to preparation. - Document idempotency, cancellation, recovery, and cleanup requirements. ## Verification - `pnpm -r typecheck` passed. - Plugin SDK build passed. - All 55 Dot Runner and native Codex server integration tests passed with Cargo available on PATH. - All 18 environment task runtime tests passed. These include preparation dispatch, inactive leases, wrong run identity, unauthorized projects, result identity, and unknown command rejection. - The broader root suite on the parent refactor finished with 22,420 passing tests, seven failed tests in unchanged Git and diagnostic code, and two suite setup errors because Cargo was absent from its PATH. This PR does not claim a passing full root suite. - No live provider calls or deployment were run. - Standard Greptile and Apex each scored commit `d3ac15906240b7066f5ff505ed2ffd9ef5dd43c1` 5/5. There are no unresolved review threads. - All 55 CI checks passed. Two optional checks were skipped. The branch is ready and mergeable against master. - A fresh master fetch and merge-base check passed immediately before the Apex request. ## Risks - This defines a new optional worker operation. Existing providers keep their current operations. - A provider must validate policy and resource bindings. Schema validation alone does not validate a concrete provider descriptor. - This PR defines the exchange. It does not connect native execution to a preparation provider. ## Model Used - OpenAI Codex, GPT-6. Reasoning and repository tools. The exact serving model identifier and context size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandPaperclip
|
33cc567f75 |
Add lease-scoped Runner task hooks to environment plugins (#15406)
## Thinking Path > - Paperclip coordinates AI agents and their runs. > - Environment plugins own remote capacity. > - A plugin needs a typed API to start and stop its installed Runner. > - Core must derive authority from the saved lease. > - A lost reply or controller restart must not lose cleanup authority. > - This PR adds the worker contract and the host task service. > - Native execution and Runner changes are separate PRs. ## Linked Issues or Issue Description **Subsystem affected** Plugin SDK and environment lease services. **Current behavior** Environment plugins have no typed Runner task hook. Core cannot dispatch these tasks or recover their credential handoff through a saved lease. **Proposed behavior** Add validated task operations, worker negotiation, company and project checks, retained-session authorization, and durable cleanup. Keep credentials out of lease metadata. **Additional context** This PR is the first part of a smaller series. Runner installed-home support is in #16977. Native execution and setup are in #16978. Both are stacked drafts. This PR does not enable native plugin execution, change default environments, add a route, or add a database migration. ## What Changed - Add `supportsTasks` and `onEnvironmentTask` with seven typed operations. - Validate worker requests and responses. Reject shell commands and file transfer operations. - Pin dispatch and cleanup to the saved company, lease, plugin, and driver. - Authorize projects and retained sessions. Validate the current run separately from the retained PRP identity. - Save dispatch before the provider call. Fence admission during cleanup. Require observed process termination. - Save account writeback authority on the lease. Recover Codex refreshes through the existing account and secret services. - Document the task API. Keep the Runner, heartbeat, setup, workspace, and provider changes in later PRs. ## Verification - Full `pnpm -r typecheck` and `pnpm build` passed on the separated API scope. - 201 focused tests passed: SDK worker dispatch, task validation, PostgreSQL task lineage, credential recovery, and existing environment cleanup. - The lineage test covers a third run, repeated same-run replacements, a stale submitting run, and exact termination receipts. - Initial commands hit sandbox socket/IPC restrictions. The permitted runs executed PostgreSQL and passed. A typecheck attempt lacked Cargo in PATH; the run with the installed toolchain passed. - The earlier combined implementation had a full local run with 22,286 passing tests, 120 skips, and eight failed assertions. Task fixture failures were fixed before this split. Remaining host Git and temporary-path assertions are recorded in the previous PR revision. This split does not claim a fresh full local suite pass. - The standard Greptile review scored this API head 5/5 with no unresolved threads. CI is green. The single Apex review also scored this head 5/5 with no actionable findings. - Fresh API CI lost a self-hosted runner during server shard 9. GitHub reported lost communication, with no failed test step. The affected jobs passed on retry without code changes. - CI on the prior combined head failed an unchanged workspace dev-server readiness case. No unrelated test fix is included here. Fresh CI is required for this smaller head. - This scope does not run a provider or a Cloud test. The user owns deployment and the intermediate Cloud test. ## Risks - The API is opt-in. Native plugin execution requires the later integrations. - Providers must make task submission idempotent and prove descendant termination. - A failed status read or credential export keeps the lease available for cleanup. - Codex refresh writeback requires the same active grant, secret reference, and session epoch. Revocation or account replacement discards the old refresh. ## Model Used OpenAI GPT-6 through Codex. Used source inspection, code execution, tests, and GitHub tools. The exact served model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run local checks and recorded failures and their results above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |