4 Commits

Include renames

Author	SHA1	Message	Date
Devin FoleyandPaperclip	3ebd7bc0c9	test: isolate accounting fixtures and include all adapter suites (#14989) ... Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-06 18:13:46 -07:00
Dotta	4c349fe6b7	feat(runtime): managed Tailscale HTTPS lifecycle, durable runtime leases, and bounded control recovery (#11525) ... <!-- Simplified Technical English (ASD-STE100). --> > **Stacked pull request.** This targets #11524. Merge #11524 first. Review only the second commit, `feat(runtime): managed Tailscale HTTPS lifecycle...`. ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Paperclip starts and supervises managed runtime services, so an agent's branch can be previewed while the agent works > - The previous pull request added the host broker, the shared contract, and the database columns, but no code used them > - A managed runtime can only be exposed over HTTPS if it holds a stable loopback port pair for the whole life of the service. The current control path cannot promise this: two controls can race the same execution workspace, a stranded control can stay `running` forever, and a start can adopt a port it does not own > - This pull request adds the HTTPS lifecycle and the control-path hardening that the lifecycle depends on > - The benefit is that a managed preview becomes reachable from another device, and a managed control now always reaches a terminal state ## Linked Issues or Issue Description No public GitHub issue exists. The change follows the feature request template. **Subsystem affected** Managed workspace runtime services, workspace operations, the execution workspace routes, and the workspace runtime UI. **Problem or motivation** A managed runtime service is reachable only on loopback, so a preview cannot be opened from a phone or a second computer. Exposing it safely needs an exclusively held port pair. Three existing gaps block that. Overlapping controls can race the same workspace. A control whose owner dies stays `running` and blocks the lane forever. Port allocation does not confirm that the process holding a port is the process Paperclip spawned. **Proposed solution** Add the exposure lifecycle on top of the broker from #11524: reserve before spawn, expose after readiness, validate the public URL, and remove on stop. In the same change, make managed controls mutually exclusive per workspace, give each control a durable issue-owned lease and a terminal state, and verify port ownership before use. **Alternatives considered** - Add HTTPS exposure without the control hardening. This was rejected because a raced or stranded control makes exposure point at the wrong process. - Guard the lane with an in-memory lock only. This was rejected because the lock does not survive a server restart, so the lane can be lost or double-claimed. - Trust the requested bind address. This was rejected because a checkout that predates managed HTTPS overwrites `PAPERCLIP_BIND` from its own `--bind` argument, and then binds the wildcard address. **Roadmap alignment** This completes the managed workspace runtime capability that already exists. It adds no new product surface beyond the HTTPS link. **Additional context** This is the second of three pull requests. The third adds central mediation of leased port pairs. ## What Changed Exposure lifecycle: - Add the server-side broker client and the exposure lifecycle manager. The manager reserves the mapping before spawn, exposes after backend readiness, validates the public URL, and removes the mapping on stop. - Default managed worktree runtimes to `tailscale_https`, read exposure intent from legacy `expose` blocks, and backfill runtimes that are still HTTP-only. - Verify listener ownership for the app port and its Vite HMR companion before the broker is asked to expose anything. An unrelated listener on either port fails the start closed. - Force the loopback bind through argv instead of environment hints. Leave a non-Paperclip service's `--bind` argument alone. - Probe loopback for readiness instead of the public URL, and give Vite HMR its own loopback-bound server in middleware mode. - Preserve operator-declared Serve mappings across the managed lifecycle, so cleanup never removes a mapping that Paperclip did not create. - Name which listener predicate denied an expose, so an operator can act on the message. Control-path hardening: - Make `start`, `stop`, `restart`, and job `run` mutually exclusive per execution workspace. An overlap gets `409 workspace_runtime_control_in_progress`, and authorization is still checked first. - Take a durable exclusivity lease on the execution workspace, owned by the controlling issue. A different issue gets `409 workspace_runtime_lease_conflict` before any operation is recorded. Board and operator actions bypass the lease. - Give every control a terminal state. Each control stamps its owning process and pid, heartbeats while it runs, and has a wall-clock ceiling. Recovery of a stranded control uses a compare-and-swap on `updated_at`, so a live owner is never stolen. - Bound readiness probes, verify allocated port ownership on POSIX and Windows, harden sibling port allocation, and reconcile desired runtimes on server startup. - Surface exposure state and bounded runtime errors in the workspace runtime UI. - Record the new behavior in `doc/DEVELOPING.md`. ## Verification Focused checks, all run on this branch: - `npx tsc --noEmit -p server/tsconfig.json` — 139 errors, exactly the count on `master`. All 139 come from the unbuilt `@paperclipai/plugin-sdk` package. - `pnpm --filter @paperclipai/ui typecheck` — clean. - Server suites, 177 tests pass across 9 files: `workspace-runtime.test.ts`, `workspace-runtime-leases.test.ts`, `workspace-runtime-control-recovery.test.ts`, `execution-workspace-runtime-control-conflict.test.ts`, `execution-workspace-runtime-lease-route.test.ts`, `workspace-operations-reconciliation.test.ts`, `workspace-runtime-start-terminality.test.ts`, `app-hmr-port.test.ts`, and `workspace-runtime-ready-comment.test.ts`. - Exposure unit suites, 77 tests pass: `src/services/runtime-exposure/` and `workspace-runtime-exposure-backfill.test.ts`. - UI: `WorkspaceRuntimeControls.test.tsx` and `WorkspaceServiceControlBar.test.tsx` — 34 tests pass. **One suite is red on the development host and is expected to be green in CI.** `server/src/services/workspace-runtime-exposure.test.ts` has 10 failures on the machine used to write this branch. The cause is host contamination, not the code. That machine already runs an HTTPS canary that holds ports 42000, 42001, 52000, and 52001 on a tailnet address. The suite allocates from the same range, so the new listener-ownership check correctly reports: ``` listener_ownership_mismatch — port 42000 is bound to 100.123.243.20, 127.0.0.1, fd7a:115c:a1e0:0:0:0:dd3a:f314 ... instead of loopback only ``` A CI runner has no listener on those ports, so the check sees loopback only and the suite passes. Please confirm this from the CI result on this pull request rather than from a local run on a host that already exposes a managed runtime. This is a real weakness of the current test fixture, and the third pull request in the series removes it by allocating the pair through a central mediator instead of a stubbed availability check. `workspace-runtime-https-live-exercise.test.ts` needs a live `tailscale` host and was not run locally. ## Risks - This is the behavior-bearing pull request of the three, so it carries the most risk. - Two new `409` responses appear on managed control routes. A caller that assumed a control always starts must handle a conflict. Board and operator actions are deliberately exempt, so an agent lease cannot lock an operator out. - Managed worktree runtimes now default to `tailscale_https`. If the host has no working broker, the start fails closed and reports the exposure failure instead of silently serving plain HTTP. This is intended, and it is the reason the failure message names the denying predicate. - Startup reconciliation touches persisted runtime rows. It is scoped to desired state and does not resurrect a service that never came up. - The lease has a 30-minute time to live and explicit release paths, so a crashed owner cannot hold a lane forever. - No migration runs in this pull request. The tables and columns land in #11524. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. ## Model Used Claude Opus 5 (`claude-opus-5`), 1M context window, extended thinking, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass, with the one host-contaminated suite explained above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge	2026-08-17 06:23:33 -04:00
3 people	ad961227f5	feat(secrets): add user-specific runtime secrets (#8825) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agent runs often need provider credentials, API tokens, and other environment-bound secrets. > - Company-level secrets work for shared credentials, but they do not model values that should differ by human operator. > - Without a user-scoped model, a run can dispatch without knowing whether the responsible human has supplied the needed value. > - Paperclip also needs run attribution to make those user-scoped runtime checks deterministic and auditable. > - This pull request adds user-specific secret definitions, per-user values, environment bindings, responsible-user attribution, and runtime resolution gates. > - The benefit is that teams can define the secret once, let each user provide their own value, and block runs before dispatch when required user secrets or active definitions are unavailable. ## Linked Issues or Issue Description Refs #224 Refs #6057 This PR implements user-specific secret support as a core secret-management capability rather than a one-off adapter setting. It is related to existing public work on company secrets UI and runtime secret refs, but is distinct because the value is owned by the responsible user and resolved at run dispatch time. Related PR search before opening found existing secrets work such as #1550, #8256, #8614, #8634, and #8647; none of those add the full user-secret definition/value/runtime gate covered here. ## What Changed - Added user-secret definitions and per-user "My secrets" values, keeping stored values out of access metadata. - Added `user_secret_ref` environment bindings and UI affordances to pick them alongside existing secret refs. - Added responsible-user runtime resolution so user-secret refs resolve against the human responsible for the run. - Added pre-dispatch missing-secret gates so runs fail before adapter dispatch when required user values are absent or definitions are inactive. - Added low-trust allowlist hardening for user-secret runtime access. - Added issue, routine, run, and agent API key responsible-user attribution and fail-closed dispatch behavior when attribution cannot be resolved. - Added denial-copy mapping so responsible-user authorization failures surface as actionable run outcomes instead of opaque setup failures. - Added OpenAPI documentation for the user-secret routes. - Rebases cleanly on current `master`; migrations were renumbered incrementally as `0128_user_specific_secrets`, `0129_agent_api_key_responsible_user`, and `0130_run_responsible_user_invariant` after upstream `0126`/`0127` migrations. - Removed previously committed local design screenshots so the PR contains code/docs/tests only. ## Verification - PASS: PR head `2527febd106bcf3ca264ca0da7fca491084192d6` is based on `paperclipai/paperclip:master`. - PASS: `git diff --check` - PASS: `git diff --name-only public/master...HEAD | rg '^(pnpm-lock\\.yaml|\\.github/workflows/|screenshots/)' || true` produced no files. - PASS: migration journal audit confirmed unique indexes through `130` with tail entries `0126_issue_comment_derived_attribution`, `0127_environment_custom_images_instance_scoped`, `0128_user_specific_secrets`, `0129_agent_api_key_responsible_user`, and `0130_run_responsible_user_invariant`. - PASS: `pnpm --filter @paperclipai/ui typecheck` - PASS: `pnpm --filter @paperclipai/server typecheck` - PASS: `pnpm --filter @paperclipai/server exec vitest run src/__tests__/heartbeat-responsible-user-invariant.test.ts` - PASS: `pnpm --filter @paperclipai/server exec vitest run src/__tests__/heartbeat-active-run-output-watchdog.test.ts src/__tests__/heartbeat-stale-queue-invalidation.test.ts src/__tests__/heartbeat-workspace-finalize-branch.test.ts src/__tests__/issue-monitor-scheduler.test.ts` - PASS: `pnpm --filter @paperclipai/server exec vitest run src/__tests__/heartbeat-comment-wake-batching.test.ts src/__tests__/heartbeat-retry-scheduling.test.ts src/__tests__/heartbeat-accepted-plan-workspace-refresh.test.ts src/__tests__/heartbeat-plugin-environment.test.ts` - PASS: `pnpm --filter @paperclipai/server exec vitest run src/__tests__/low-trust-red-team-routes.test.ts` - PASS: `pnpm --filter @paperclipai/server exec vitest run src/__tests__/secrets-service.test.ts` (55 tests) - PASS: `pnpm vitest run server/src/__tests__/secrets-routes.test.ts server/src/__tests__/secrets-service.test.ts` (89 tests after final Greptile cleanup fixes) - PASS: `pnpm --filter @paperclipai/server exec vitest run src/__tests__/heartbeat-issue-liveness-escalation.test.ts` (17 tests after the final rebase CI fix) - PASS: focused server Vitest batches covering heartbeat recovery, project env, plugin env, routines, low-trust, pipelines, monitors, watchdog, and stale queue paths. - PASS: GitHub checks are green on `2527febd106bcf3ca264ca0da7fca491084192d6`, including Typecheck + Release Registry, Build, General tests, serialized server suites, e2e, Canary Dry Run, verify, security checks, and Greptile Review. - PASS: Greptile Review completed successfully on `2527febd106bcf3ca264ca0da7fca491084192d6` with Confidence Score 5/5, and GraphQL review-thread audit returned zero unresolved non-outdated threads. ## Risks - Runtime behavior now depends on a run having a correct responsible user; missing or incorrect responsibility assignment can block runs before adapter dispatch. - `user_secret_ref` bindings intentionally expose metadata without values, but UI/API callers may need to handle the new binding kind explicitly. - External secret providers and IAM policies are not automatically provisioned by this PR; operators still need to configure provider-side access for non-local vaults. - The PR is broad across db/shared/server/UI/runtime paths, so release validation should include both API and UI secret workflows before merge. - The migration renumbering is intentionally incremental after upstream migrations; the branch migrations use guarded column/table/index/constraint creation so users who tested the older draft numbering should not hit duplicate DDL for the existing objects. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI Codex, GPT-5-based coding agent (`gpt-5`), Codex local adapter with shell/tool use and code execution. Context window and internal reasoning mode are not exposed by the runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-05 05:58:20 -05:00
DottaandPaperclip	9a8d219949	[codex] Stabilize tests and local maintenance assets (#4423) ... ## Thinking Path > - Paperclip orchestrates AI agents for zero-human companies > - A fast-moving control plane needs stable local tests and repeatable local maintenance tools so contributors can safely split and review work > - Several route suites needed stronger isolation, Codex manual model selection needed a faster-mode option, and local browser cleanup missed Playwright's headless shell binary > - Storybook static output also needed to be preserved as a generated review artifact from the working branch > - This pull request groups the test/local-dev maintenance pieces so they can be reviewed separately from product runtime changes > - The benefit is more predictable contributor verification and cleaner local maintenance without mixing these changes into feature PRs ## What Changed - Added stable Vitest runner support and serialized route/authz test isolation. - Fixed workspace runtime authz route mocks and stabilized Claude/company-import related assertions. - Allowed Codex fast mode for manually selected models. - Broadened the agent browser cleanup script to detect `chrome-headless-shell` as well as Chrome for Testing. - Preserved generated Storybook static output from the source branch. ## Verification - `pnpm exec vitest run src/__tests__/workspace-runtime-routes-authz.test.ts src/__tests__/claude-local-execute.test.ts --config vitest.config.ts` from `server/` passed: 2 files, 19 tests. - `pnpm exec vitest run src/server/codex-args.test.ts --config vitest.config.ts` from `packages/adapters/codex-local/` passed: 1 file, 3 tests. - `bash -n scripts/kill-agent-browsers.sh && scripts/kill-agent-browsers.sh --dry` passed; dry-run detected `chrome-headless-shell` processes without killing them. - `test -f ui/storybook-static/index.html && test -f ui/storybook-static/assets/forms-editors.stories-Dry7qwx2.js` passed. - `git diff --check public-gh/master..pap-2228-test-local-maintenance -- . ':(exclude)ui/storybook-static'` passed. - `pnpm exec vitest run cli/src/__tests__/company-import-export-e2e.test.ts --config cli/vitest.config.ts` did not complete in the isolated split worktree because `paperclipai run` exited during build prep with `TS2688: Cannot find type definition file for 'react'`; this appears to be caused by the worktree dependency symlink setup, not the code under test. - Confirmed this PR does not include `pnpm-lock.yaml`. ## Risks - Medium risk: the stable Vitest runner changes how route/authz tests are scheduled. - Generated `ui/storybook-static` files are large and contain minified third-party output; `git diff --check` reports whitespace inside those generated assets, so reviewers may choose to drop or regenerate that artifact before merge. - No database migrations. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI Codex coding agent based on GPT-5, with shell, git, Paperclip API, and GitHub CLI tool use in the local Paperclip workspace. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] If this change affects the UI, I have included before/after screenshots - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] I will address all Greptile and reviewer comments before requesting merge Note: screenshot checklist item is not applicable to source UI behavior; the included Storybook static output is generated artifact preservation from the source branch. --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-04-24 15:11:42 -05:00