mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-01 15:35:12 +01:00
d6dfbb495db44b60faa37416675e8ffbb9cc10d0
7
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Nicky LeachandPaperclip
|
9f4a956b90 |
Add durable agent runtime synchronization hooks (#16499)
## Thinking Path > - Paperclip manages agents and their runtime configuration. > - Plugins can maintain external agent computers. > - Configuration and file saves currently miss durable plugin delivery. > - A transient activity notification cannot recover after interruption. > - This pull request adds an opt-in agent runtime resource stream. > - The existing dispatcher delivers and acknowledges its committed work. > - Plugins can prepare computers when configuration changes. ## Linked Issues or Issue Description Related: #15631 and #16203 establish agent lifecycle and native durable delivery. **Problem** Plugins cannot reliably react to agent configuration and instruction changes without polling or waiting for execution. **Proposed solution** Add an opt-in `agent_runtime` stream to the existing resource journal. Record changes at the database mutation and managed file audit boundaries. Wake the dispatcher with content-free commit notifications. Provide scoped runtime inputs to the selected lifecycle plugin. **Alternatives considered** Route-only activity events miss service and background writers. Scheduled scans delay changes and repeatedly query unchanged state. **Roadmap alignment** This extends the existing plugin and agent configuration work. ## What Changed - Add durable update/delete events for agent runtime state, including an existing-agent baseline. - Capture configuration changes, instruction edits, deletion, restore, and managed run writeback. - Add commit notifications and a reconnect scan. Register one listener for configured, enabled runtime subscribers. Release it when none remain. - Keep HTTP startup independent of queued plugin callbacks. - Add `ctx.agents.getRuntime`. Require lifecycle capability, invocation company, enabled plugin, and owning lifecycle selection. - Keep legacy agent/project subscriptions isolated. Manual inbox reads retain both legacy streams. - Resolve recovered instruction paths without reading or hashing bundle contents. Use the company reporting graph for runtime health. - Repair missed file events on identical save/deletion retries after a journal failure; preserve content tokens and conditional-write checks. - Correlate file audits with runtime revisions to deduplicate recorded retries without a new table. Bound the lookup with the existing company/timestamp index. - Document the contract, migration, access boundaries, and session-connection requirement. ## Verification - Rebased the full stack onto origin/master after #16507. - Final replayed stack: `pnpm -r typecheck` and `pnpm build` passed. - All 132 affected runtime, skill, connector, publication, instruction, managed-agent, and delivery tests passed after the latest-master rebase. Workspace typecheck and build passed. - Journal failure injection verifies recovery on identical file writes, deletions, and instruction retries from an acknowledged inbox. - Deduplication follow-up: 80 runtime/instruction/delivery tests and server typecheck passed. Repeated retries create no audit/events; old hashes returning after an intervening commit still repair delivery. Built-in/native instruction writer regressions passed all 56 tests. - PR 1 follow-up tests cover valid/cyclic reporting chains, listener registration failure, multiple subscribers, global/company disable, re-enable, unload, and startup with blocked callbacks. - Database coverage includes rollback, journal failure, no-op suppression, file saves/deletion, ordering/replay, owner/company denial, independent-client commits, and real listener loss/reconnect. - SDK/shared suites passed 1,059 tests during implementation. CLI/delivery regressions also passed. - Migration safety, map checks, diff whitespace, and local secrets/PII scan passed. - Hosted CI and security checks are fully green at `b296405e2`. Standard Greptile and Apex both returned 5/5 with no unresolved findings. - The broad local suite encountered an unchanged diagnostic test that rejects the word "private" from an error stack. This worktree lives under /private/tmp. The same unchanged diagnostic suite passed all 108 tests in the original checkout. Hosted full-suite checks provide exact-head verification. ## Risks - Apply migration 0347 before enabling a runtime subscriber. Runtime notifications require session-level PostgreSQL LISTEN. Transaction-mode poolers are unsupported for this opt-in feature. - Database triggers share the mutation transaction. Journal failure prevents configuration commits. Trigger work adds a journal insert to relevant changes. - Filesystem publication and SQL commit are separate boundaries. Direct filesystem edits do not emit hooks. External clients must fence stale revisions and retain conditional-write checks. - The reader returns only that plugin's own instance secrets. Keep them out of UI data, logs, plugin state, and agent files. - No private transport integration or live remote computer qualification is included. ## Model Used OpenAI Codex, GPT-6, with reasoning, repository inspection, code execution, and testing. The exact deployment model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal Paperclip ticket id - [x] I have run tests locally and they pass (focused suites; broad local limitation documented above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandPaperclip
|
db65a0863e |
Deliver durable resource lifecycle events to plugins without polling (#16203)
## Thinking Path > - Paperclip manages AI agents and their work. > - Plugins keep external resources aligned with Paperclip projects and agents. > - The durable resource journal stores changes, but its plugin interface requires polling. > - Project deletion has no journal entry, so consumers must scan for missing projects. > - This PR delivers committed journal events through a native plugin callback and records deletion. > - Plugins can recover unfinished work without periodic change scans. ## Linked Issues or Issue Description Related: #15343 introduced the durable inbox. #15631 added required agent lifecycle participation. This change keeps both contracts and adds automatic delivery for the resource inbox. **What happened?** Plugins must schedule reads of the durable inbox. A deleted project has no durable deletion event. Consumers must scan saved bindings to detect its removal. **Expected behavior** Core should deliver committed resource events and replay unfinished work. Empty queues should have no polling timer. Project removal should leave a durable event. **Steps to reproduce** 1. Install a plugin that consumes project lifecycle events. 2. Create or update a project. The plugin must call `listLifecycle` to discover it. 3. Delete the project. The journal has no deletion entry. ## What Changed - Add `resourceLifecycle` to the manifest and `onResourceLifecycle` to the SDK. Require `events.subscribe` and a configured, enabled company scope. - Reuse transaction-aware work signals and the delivery coordinator. Wake after commit, on startup and plugin activation, and after configuration changes. - Acknowledge only a matching successful result. Retry unfinished or interrupted work. Retain deferred events without an idle retry timer. - Wake again after successful worker configuration application, and retain failed scans through idle drain (both Apex findings fixed). - Keep ordering per resource and deliver past failed or deferred resources. Keep the manual inbox methods for existing consumers. - Record project deletion in the removal transaction. Extend the journal constraint with a generated migration. ## Verification - Apex follow-up: 129 focused worker, delivery, coordinator, and application tests passed; server typecheck passed. Regression coverage verifies configuration-application races and failed scans across idle drain. - Focused lifecycle, transaction coordinator, manifest, and SDK dispatch suites: 105 tests passed, including single-connection saves, redacted failure diagnostics, wakeups retained during idle drain, and per-event retry deadlines. - Complete SDK suite plus delivery database tests: 128 tests passed. - Shared, database, and SDK typechecks passed. Direct server `tsc --noEmit` passed against rebuilt SDK declarations. - Full local `pnpm test:run` finished with 19,491 passed, 148 skipped, and 6 failed tests across 5 files (plus two suite setup errors). This run overlapped branch updates and is not final-head evidence: failures included missing Cargo, local Git default-branch assumptions, and stale in-process lifecycle modules. Final Apex-fix commit `75d028788` passed all CI gates, including the full test shards, and received Apex 5/5 with no unresolved comments. - `pnpm -r typecheck` and `pnpm build` were attempted. Both stopped because Cargo is unavailable locally. CI supplies the Rust toolchain. - Local installation used the repository's documented lockfile fallback because master has dependencies missing from its lockfile. No dependency changes are included here. ## Risks - Delivery is at least once. Plugins must re-read current state and make external effects idempotent. - Signals are process-local, like the other delivery queues. Startup recovers committed rows. Independent database writers or multiple active server processes require an explicit ownership and wake protocol. - Apply migration `0340_open_warlock.sql` before running code that records project deletion. Earlier deletions cannot be reconstructed. - Automatic delivery selects plugins with company configuration rows. `deferred` waits for a later startup, configuration change, or other work notification. - This does not replace required agent lifecycle participation or establish that external mounts are unused. ## Model Used OpenAI Codex (GPT-6), with tool-assisted code editing, code inspection, and test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandPaperclip
|
2dd9811e8b |
Own agent hiring and termination in a lifecycle module (#15631)
## Thinking Path > - Paperclip manages AI agents and their work. > - Agent creation, approval, pause, and termination change whether an agent can work. > - These changes currently use separate service and database paths. > - Some agents need setup after their record exists and before their harness can run. > - This PR adds one module that owns those state changes and their transactions. > - The module records progress and retries incomplete work after a server restart. ## Linked Issues or Issue Description **Problem or motivation** A hire can become active before its execution environment is ready. Direct agent state writes also bypass one common lifecycle boundary. Termination does not have a durable step for resource cleanup. **Proposed solution** Use agent lifecycle commands for hiring, approval, pause, resume, and termination. Keep proposed hires pending until approval. Prepare each approved hire, test its saved harness configuration, and then permit work. Wait for required plugin work before completing a step. **Alternatives considered** A general database commit hook would expose transaction details to callers. A timer alone would delay each new hire. This module owns each command transaction and starts its worker after commit. A periodic scan recovers work after a process stops. **Roadmap alignment** This change adds the lifecycle boundary for the existing agent and plugin systems. It does not add a user interface or a provider implementation. It replaces the lifecycle delivery approach in #15391. Related proposals include #11475 and #13099; this change does not add their external executor protocols. ## What Changed - Add durable approval, preparation, verification, pause, resume, stop, and cleanup states. - Keep agent creation, lifecycle state, configuration writes, and agent record deletion behind `modules/agent-lifecycle`. Keep validation, credential rules, reads, onboarding, invitations, general approvals, and company data deletion in their services. - Commit a hire before the remaining onboarding or invitation work. Serialize requests with the existing dedicated database connection helper. Reuse the hire after a failed attempt. - Keep accepted work in the run queue during setup. Require execution and cleanup evidence before completing pause or termination. - Retain the original credential owner when approving migrated hires. - Start lifecycle work after each command commits. Use database leases and revision checks for retries and late results. - Add a generic required plugin method and capability. Store the required plugin IDs for each agent. Track host and plugin completion separately. - Keep harness verification and credential rules in services. Supply their results to lifecycle through explicit ports. Reject failed logins and incomplete probes. - Remove reverse imports from lifecycle adapters into services. Share record queries and validation below both layers. - Use one hire decision path for agent IDs and approval IDs. Resolve the approval and agent change in one transaction. Expose policy reconciliation on the command object. - Save configuration and verification invalidation through one module operation. Ordinary commands own their transaction. Restrict the shared transaction integration to its listed callers. Reject identity, company, accounting, timestamp, and unknown fields. - Require completed termination before deleting agent records. The company deletion service calls the module through a separate deletion interface. - Save an approved secret before applying its agent binding. A failed binding leaves the secret approved. A retry reuses it. - Remove old permission and metadata export files. Move hire configuration rules out of lifecycle persistence. - Report failed worker steps and scans with operation context and known error codes. Omit provider messages, stacks, and configuration values. Document ordinary application use and extension rules. - Recover expired native owners only after execution stops. Retry cleanup of failed retained leases. Wait for setup in the live smoke script. - Keep manual, budget, and company pause reasons separate. Guard execution status writes with the ready state. - Add a migration, boundary checks, regression tests, and a lifecycle guide. Keep the current UI. - Limit periodic policy checks to 100 eligible agents per minute. Continue from the last agent ID. Exclude pending hires and agents in termination or a final state. Keep explicit company and budget updates immediate. ## Verification - Latest head `ccc3c86b0` aligns the shared status update schema with the PATCH route. All 15 shared tests, 104 server route/OpenAPI tests, 2 CLI tests, and shared/server/CLI TypeScript checks pass. The Greptile comment is fixed. CI is running; no new Apex review was requested for this small fix. - Rebase `75708ac94` rebases onto master at `2eb5187e4`. The backup test uses the shared 90-second timeout from master; the older timeout commit was removed. The affected backup test passes. - Agent budget edits now pass the target agent ID through enforcement. All 50 affected tests, server TypeScript, and 11 boundary tests pass. Both configuration update paths have regression coverage. CI is running on the latest head. No new Apex review was requested for these small fixes. - Rebase `9b568666a` preserves the run-control extraction and delivery-work startup changes from master. All 559 affected tests, server TypeScript, and 11 boundary tests pass. CI and Apex review are running on the final head. - The policy scan fix passes 56 lifecycle and diagnostic tests, 24 policy and caller tests, and 11 boundary tests. A regression test checks the page limit, interval, excluded states, and cursor wraparound. - Rebase `fd4698750` passes all 611 tests in eight affected lifecycle, route, and heartbeat suites. Server TypeScript and all 11 module boundary tests pass. The company deletion rollback and concurrent hire tests are included. - Follow-up `00aac7646` fixes the failed decision deletion test and the public update schema. All 126 affected server tests, 8 shared tests, 2 CLI tests, and 11 boundary tests pass. Shared, server, CLI, and UI TypeScript checks pass. - Server TypeScript passes: `pnpm --filter @paperclipai/server exec tsc --noEmit`. - All 11 boundary tests pass: `node --test scripts/check-module-boundaries.test.mjs`. The repository boundary scan also passes. - All 68 lifecycle, failure diagnostic, and harness verification tests pass. Tests reject forbidden fields through the root command, combined transition command, and restricted transaction integration. They also check invalid fields returned by configuration preparation. - Failure tests cover host work, plugin work, claims, renewals, deferrals, and scans. Log tests check known error codes and exclusion of private error contents. - All 355 tests in 10 existing caller suites pass. These tests cover secret approval, pairing, pending hires, credentials, company import, built-in agents, managed agents, instructions, and skills. - The branch is rebased onto `origin/master` at `2eb5187e4`. The queue guards and cancellation lock fix use the extracted heartbeat queue. Agent cancellation uses the extracted run-control file. The source diff scan found no secrets or private URLs. - Earlier checks cover approval retries, rollback, late verification results, policy holds, deletion rollback, concurrent company deletion, and queue admission during setup. - Before the rebase, CI passed on `00aac7646` and Greptile Apex rated it 5/5 with no open comments. CI and a new Apex review are pending for the rebased commit. The branch has no merge conflicts. - The full local build and native Dot suite were not run. The host has no Cargo. No live provider test was run. ## Risks - Background policy recovery takes more than one minute when more than 100 agents are eligible. Explicit company and budget updates still reconcile immediately. - The restricted configuration integration keeps an agent reference and its dependent records in one transaction. It cannot change a budget or invoke a lifecycle transition. A periodic scan starts pending setup after that transaction commits. - Secret approval and binding use separate transactions. A failed binding can leave an approved secret with no agent binding. - New approved hires remain unavailable until setup and verification complete. Existing clients see `paused` during these steps. This PR does not change the UI. - Migration `0320_daily_onslaught.sql` maps existing statuses and pause reasons. It does not create external resources or claim that an existing harness passed verification. - Use a coordinated server update for this migration. Older server versions can write legacy status without the new lifecycle checks. - A failed onboarding or invitation request can leave a committed hire. A retry reuses that hire and completes the remaining records. - Deletion requires completed termination. Company deletion can require another request while cleanup is pending. - Required plugins must make repeated calls safe and reject older revisions. A disabled or removed required plugin blocks completion until it is restored. - A new hire can run a second harness test after the existing UI test. Some adapters can charge for this test. - No live provider test was run. Provider implementation and resource backfill remain separate work. ## Model Used OpenAI GPT-6 through Codex, with reasoning, code tools, and local test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in this PR with the feature request field labels - [x] I have not referenced internal or instance-local issues or links - [x] My branch name describes the change - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented risks - [x] Targeted local tests pass - [x] All Paperclip CI gates are green on the new commit - [x] Greptile Apex is 5/5 with no open comments on the new commit - [x] I will address review comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandPaperclip
|
1cebdd4c4f |
Add plugin lifecycle delivery and an initial resource baseline (#15343)
## Thinking Path > - Paperclip manages agents and projects for work. > - Their lifecycle changes already commit to a durable journal. > - Plugins need to read these records and track completed work. > - Existing resources need a one-time baseline before delivery is enabled. > - Worker crashes must leave unfinished records available for retry. > - Each plugin needs its own progress and company access checks. > - This PR adds a pull inbox through the existing plugin SDK and job system. ## Linked Issues or Issue Description **Problem or motivation** Plugins cannot consume the durable resource lifecycle journal. In-process notifications can disappear during a restart and cannot record successful completion. Resource plugins need retries, company boundaries, and ordered transitions. **Proposed solution** Add `ctx.events.listLifecycle(companyId, limit?, afterId?)` and `ctx.events.acknowledgeLifecycle(companyId, eventId)` under `events.subscribe`. Seed a one-time baseline from current resource state. Deliver creation first, then the remaining transitions in ID order. Store acknowledgments for each plugin. Use existing plugin jobs to poll configured companies. **Alternatives considered** A global sequence cursor can skip lower IDs that commit later. Fire-and-forget subscriptions cannot record completion. A new dispatcher is unnecessary because plugin jobs support polling. Consumers serialize polling and use provider idempotency keys. **Roadmap alignment** This extends the existing plugin system and builds on #15280 and #15306. Searches found no duplicate resource inbox work. Related #13306 exposes decision events on the in-process bus. This PR includes the initial journal baseline. Provider provisioning remains separate work. ## What Changed - Add company-scoped lifecycle reads and acknowledgments to the SDK and worker RPC host. - Gate both methods by capability, invocation or proactive company scope, plugin readiness, and company enablement. - Seed hired agents and all projects once. Preserve paused/terminated agent state and archived project state. Keep pending hires behind approval. - Preserve existing history and deliver backfilled creation before partial transition histories. - Deliver project archive events from #15371 through the SDK. Seed archive intents for archived projects and update intents for active projects with a partial archive history. - Store acknowledgments per plugin and reject acknowledgments that skip earlier resource events. - Page past failed resources while retaining their pending records. Reset the page cursor each sweep to include late commits. - Add acknowledgment storage, an index for resource ordering, tests, and authoring guidance. - Preserve native identity definitions and sequence progress in JavaScript backups. Repair journal ID generators lost by older backups before seeding the baseline, without changing existing IDs. ## Verification - `pnpm -r typecheck` passed after the final origin/master rebase. - `pnpm build` passed before the final metadata rebase. The final CI build also passed. - All 46 focused lifecycle, SDK harness/RPC, migration snapshot, and legacy restore checks passed again with migration 0309. - Checks cover retries, per-plugin progress, resource order, capability/company boundaries, baseline idempotency, approval gates, archive delivery, restored projects with partial histories, and atomic migration rollback. - Full CI passed on final commit `0e674b87fc`. One unrelated Cursor fixture hit a 10-second timeout; it passed locally in under one second, and the failed server shard passed on retry. - Greptile scored the final commit 5/5 with no unresolved review threads. - The branch is rebased onto origin/master and is conflict-free. - Earlier full local runs were stopped as scope changed. CI ran the complete repository suite. - `git diff --check` and a local secret/PII scan passed. ## Risks - Apply migration `0309_loving_the_hood.sql` before starting the new server. It creates acknowledgment storage and an index, then seeds the baseline in the same transaction. Agent/project writes wait for the migration to commit. Keep these writes quiesced through the migration and activation of the new capture-capable server; do not resume an older runtime that lacks capture after the baseline. - The baseline records current desired state, not historical transitions. It includes archived projects and terminated-agent cleanup intents. It runs once with the delivery migration; no later or runtime journal backfill is planned. - Delivery is at least once. Concurrent reads can repeat an event. Consumers must serialize polling, use stable company/event idempotency keys, and acknowledge successful operations only. - Reset `afterId` at the start of every polling sweep. It is a page cursor, not a persisted high-water mark. - Deleting a plugin removes its acknowledgment records. A new installation may replay existing journal events. - Records contain identity and action. Consumers must load current authorized data before acting. Cleanup and retention policy belong to the provider plugin. - Provider calls, VM/volume provisioning, and journal retention are outside this PR. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, repository inspection, code execution, and tool use. The exact deployment model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandPaperclip
|
d9f600043f |
Capture project archive and restore lifecycle events (#15371)
## Thinking Path > - Paperclip manages projects and their repositories. > - Resource changes commit to a durable lifecycle journal. > - Project archive changes currently leave no journal record. > - Consumers need to observe archive and restore transitions. > - The record must commit with the project change. > - This PR adds project archive capture before plugin delivery in #15343. ## Linked Issues or Issue Description **Problem or motivation** Archiving a project does not record a lifecycle event. Restoring an archived project also emits no event when it is the only change. Consumers cannot observe these transitions through the journal. **Proposed solution** Record `archive` when an active project becomes archived. Record `update` when an archived project is restored. Keep both writes in the existing project transaction and row lock. Repeated status requests emit no new status record. **Alternatives considered** An in-process notification can disappear on restart. A separate archive service would duplicate the existing mutation path. Use the existing journal and project transaction. **Roadmap alignment** This extends lifecycle capture from #15280 and #15306. It should merge before the delivery PR #15343. The roadmap and related PR search showed no duplicate archive lifecycle capture. ## What Changed - Allow project `archive` records in the journal constraint and TypeScript type. - Capture archive and restore transitions under the existing project row lock. - Emit `update` then `archive` for an edit combined with archive. - Preserve repository records and roll back the project change if capture fails. - Add migration `0307_cool_naoko.sql`, tests, and database documentation. ## Verification - `pnpm -r typecheck` passed after the final master rebase. - `pnpm build` passed before the final master rebase; the final CI build also passed. - 20 targeted lifecycle, migration snapshot, and legacy restore tests passed on the current migration. - Tests cover concurrent archives, repeated archive/restore requests, repository retention, combined edits, atomic rollback, and upgrades from older JavaScript backups without the action constraint. - All 53 applicable CI checks passed on final commit `1021953035`; the two Storybook checks were skipped as expected. - Greptile scored the final commit 5/5 with no unresolved review threads. - `git diff --check` and a local secret/PII scan passed. - Prior CI found a missing-constraint restore failure; the migration now handles it. A separate runtime readiness timeout passed locally. The current full CI run passes both paths. ## Risks - Older JavaScript restores may omit the action check. The migration tolerates its absence and installs the complete check. - Apply the constraint migration before running the new capture code. It takes a short table lock and validates existing journal rows; it changes no existing records. - This PR captures future transitions only. The one-time baseline and plugin consumption remain in #15343, which must be rebased after this PR merges. - Archive records authorize no provider cleanup. Provider behavior and volume retention remain separate work. - An edit combined with archive emits two ordered records in one transaction. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, repository inspection, code execution, and tool use. The exact deployment model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandPaperclip
|
44e4979d23 |
Capture project and repository update lifecycle events (#15306)
## Thinking Path > - Existing lifecycle capture records agent transitions and project creation. > - Project and repository edits need matching project update records. > - The record must commit with the mutation so failed writes cannot lose a hook. > - Creation with repositories is one creation, and repository replacement is one update. > - Archive-only changes preserve project state without creating a hook. > - Plugin consumption and provider behavior are separate work. ## Linked Issues or Issue Description **Problem or motivation** Project edits and repository/workspace changes lack durable lifecycle records. A future resource plugin needs those changes captured alongside the existing project creation hook. Archive-only changes must produce no hook. **Proposed solution** Allow project `update` records in the existing lifecycle journal. Record project and workspace mutations in their database transaction while holding the project row lock. Suppress intermediate workspace hooks during project creation and aggregate repository replacement. **Alternatives considered** Route-only hooks miss shared service callers. Recording after commit can lose an event. Emitting a hook for each child mutation exposes intermediate repository state. **Roadmap alignment** This completes project lifecycle capture begun in #15280. Plugin delivery, VM/volume provisioning, and backfill remain separate. Searches found no duplicate project lifecycle work; related #13306 concerns decision events on the in-process plugin bus. ## What Changed - Record project edits and workspace additions, updates, and removals as project `update` events. - Commit each event atomically with its mutation under the project row lock. - Keep project creation with repositories to one creation event and repository replacement to one aggregate update. - Ignore archive-only changes and retain workspace records. - Extend the journal action constraint and document project update capture. ## Verification - `pnpm -r typecheck` passed on the narrowed scope. - 57 tests passed across six lifecycle, project, repository, and chat-project suites. - Seven managed-sandbox workspace route tests and the CLI lagging-worktree migration regression passed (65 targeted tests total). - Full GitHub CI passed on `8ba6f97f22`; all required gates are green. - Greptile scored the final project-only commit 5/5 with zero unresolved review threads. - The branch is current with `master` and has no merge conflicts. - `git diff --check` and a local secret/PII scan passed. ## Risks - Apply migration `0300_chunky_chamber.sql` before running the new server. It permits project update actions and tolerates older JavaScript worktree backups that omitted the prior CHECK constraint. - Event-write failure intentionally rolls back the project or repository mutation. - Records contain identity and action; future consumers must load current authorized project/workspace data. - Plugin consumption, provider calls, volume cleanup, and backfill are outside this PR. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, repository inspection, code execution, and tool use. The exact deployment model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandPaperclip
|
984f092ddf |
Add project and agent resource lifecycle hooks (#15280)
## Thinking Path > - Paperclip manages agents and projects for work. > - Plugins need reliable lifecycle hooks for these resources. > - In-process notifications can disappear during a restart. > - A lifecycle record must commit with the resource change. > - This pull request adds a generic lifecycle journal without provider calls. > - A later plugin delivery layer can use these records without losing lifecycle changes. ## Linked Issues or Issue Description **Problem or motivation** Resource provisioning needs durable hooks for agent hiring, pause, resume, termination, and project creation. Hooks must cover shared service callers, budget actions, and approval paths. Capture should work for all plugins and deployment modes. **Proposed solution** Record content-free events in the resource transaction. Deduplicate creation and unchanged status. Preserve each pause/resume cycle. Commit approval, activation, and creation together. Commit termination and API-key revocation together. **Alternatives considered** Event subscriptions alone cannot survive process failure. Cloud-only capture would exclude other plugins. Provider calls inside tenant transactions would couple resource creation to external services. **Roadmap alignment** This is lifecycle infrastructure for the existing plugin system. It adds no provider, adapter, UI, or plugin read API. Repository mutations and backfill remain separate work. Searches found no duplicate lifecycle-journal PR. ## What Changed - Add a company-scoped lifecycle journal and an additive migration. - Record hired-agent and project creation from shared services. - Record agent pause, resume, and termination, including generic updates and budget actions. - Lock agent state changes to suppress concurrent duplicate hooks. - Make hire approval, rejection, and termination transactions atomic with their lifecycle records. - Document capture, ordering, future per-plugin acknowledgments, and migration scope. ## Verification - Full workspace typecheck: `pnpm -r typecheck` passed. - Production build: `pnpm build` passed. - Focused agent, project, approval, budget, and built-in regressions: 102 tests passed across 9 suites. - Final lifecycle journal check: 14 tests passed. It covers repeated/concurrent transitions, rollback, key revocation, self-hosted capture, and company boundaries. - Review regression: 32 lifecycle and approval tests passed, including rejected-hire rollback and retry; server typecheck passed after the fix. - The initial local `pnpm test:run` overlapped the rejection fix and reported the new rollback regression against the earlier service code. A fresh lifecycle run passed all 14 tests. Fresh full local shards were stopped once the complete CI test matrix passed on the final commit. - `git diff --check` and a local secret/PII scan passed. - Greptile: 5/5 on `3ee3903f8e1a171ea3ba2bea9caa2b5766a3f807`, with the review thread resolved. - [Complete CI passed](https://github.com/paperclipai/paperclip/actions/runs/37374227222) on `3ee3903f8e1a171ea3ba2bea9caa2b5766a3f807`: general server/chat/workspace tests, serialized server suites, runner checks, build, typecheck, canary, and all end-to-end shards. The requester waived CI during the Actions outage, but the workflow subsequently completed successfully. ## Risks - The migration creates an empty table. It does not scan or backfill existing resources. - Apply the normal database migration before running this server version. Event-write failure intentionally rolls back the resource change. - Pending hires cannot bypass approval through pause or resume. - Capture works on all deployments. Plugin delivery, retention, retries, and provider actions remain separate work. No plugin can read this journal through a new API in this PR. - Future delivery must enforce company scope, track acknowledgments per plugin, and preserve resource order. A global sequence cursor can skip uncommitted transactions. - A termination hook does not authorize deleting persistent volumes. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, repository inspection, code execution, and tool use. The exact deployment model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |