mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-03 01:27:09 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its web server also owns controller admission and user commands. > - A rolling deployment starts a replacement before the old server exits. > - Shared database state alone does not preserve local agent files, browser chunks, or in-process commands. > - This pull request adds an opt-in ephemeral server contract and a local overlap test. > - The benefit is a web server that can relinquish traffic and local files without losing acknowledged work. ## Linked Issues or Issue Description Refs #18403, #18327, #18335, #18341. **Current behavior** The control-plane web server still has local agent-file authority and local command targets. A new container can miss saved bytes or send a command to the wrong process. An existing browser tab also needs chunks from its original build. **Proposed behavior** Eligible servers share only PostgreSQL, object storage and stable secrets. A replacement waits for readiness. The old server drains admitted requests, keeps owner commands available through handoff, and releases controller leases. Explicit maintenance adopts existing files before the volume is removed. **Reason and benefit** This is the application prerequisite for rolling app-only deployments. Schema migrations and permanent multi-replica operation are outside this change. ## What Changed - Store agent-file manifests in PostgreSQL and immutable bytes in S3. Treat local roots as disposable projections. Preserve existing permission checks and edit conflicts. - Add an explicit migration, eligibility and object-retention command to the production runtime bundle. - Add `/api/ready`, admitted-request drain, browser restart signaling and retained immutable UI chunks. - Carry transaction-aware work hints across controllers during handoff and active execution. Keep listeners closed when idle. - Deliver durable Stop and runtime responses to the execution owner. Recheck request lifecycle and access before response dispatch. - Reject local-only configuration in ephemeral mode. Keep the feature disabled for existing installations by default. - Add an isolated PostgreSQL/S3/proxy/browser deployment harness with independent server homes and retained failure evidence. ## Verification - All CI checks pass on `ce17b95019d3cf1cabd357d19b6d37f96782cc5c`: workspace typecheck, build, Runner checks, server/general tests and browser shards. Greptile reviewed the runtime changes at 5/5 with all findings resolved; the final commit updates maps and the plan. - Local targeted checks passed: coordinator 27; Stop/recovery 33; PostgreSQL command delivery 6; route/workspace regressions 8; bootstrap Stop 1; restart recovery 75. PostgreSQL command tests simulate the provider edge. - Local browser proof passed 91 reads and 19 saves; reconnect took 1,013ms. The tab opened Audit from S3 after deletion of A's whole home/build. This uses compiled workspace code and local-trusted auth, not released containers or live agents. - Local workspace typecheck/build passed; current server and runtime bundle passed. The earlier broad local suite was stopped after three hours and is not a passing receipt. Its retained failures and isolated reruns are documented in the plan. - Stacked #18526 adds stricter graceful-exit proof, an unready replacement control, live Runner qualification and a Stop handoff correction. Hosted enablement remains gated on those results and Cloud #1096. No hosted stack has been enabled. ## Risks - Apply migration 0422 before enabling the feature. Existing data must pass each owner's migration and eligibility checks. Do not discard a volume based on a single migration page. - Durable manifests remain authoritative after the opt-in is disabled. Rolling back to a binary that ignores them is unsupported. - Cold directory restoration and complete shutdown timing still need qualification before enabling a hosted stack. - Retained UI assets must cover the supported browser and rollback window. Cleanup is explicit; no new idle maintenance polling is introduced. - This change does not claim production zero downtime or authorize a fleet-wide enablement. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model ID and context-window limit are not exposed in this session. Used reasoning, code editing, shell execution and browser automation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
103 lines
5.5 KiB
TypeScript
103 lines
5.5 KiB
TypeScript
import { randomUUID } from "node:crypto";
|
|
import { setTimeout as delay } from "node:timers/promises";
|
|
import { sql } from "drizzle-orm";
|
|
import { afterAll, beforeAll, expect, it } from "vitest";
|
|
import { createDb } from "@paperclipai/db";
|
|
import { enableDatabaseWorkNotifications, publishDatabaseWorkHint, signalDatabaseWork, subscribeDatabaseWork } from "@paperclipai/db/work-signals";
|
|
import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
|
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
|
|
import { deploymentNotifications } from "../services/deployment-notifications.js";
|
|
import { setNativeExecutionActive } from "../services/native-runtime/native-active-runs.js";
|
|
|
|
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
|
let a: ReturnType<typeof createDb>, b: ReturnType<typeof createDb>;
|
|
beforeAll(async () => {
|
|
database = await startEmbeddedPostgresTestDatabase("deployment-notifications-");
|
|
a = createDb(database.connectionString); b = createDb(database.connectionString);
|
|
enableDatabaseWorkNotifications(b);
|
|
await a.execute(sql`create table deployment_notification_fixture (id text primary key)`);
|
|
}, 90_000);
|
|
afterAll(async () => { await a?.$client.end({ timeout: 1 }); await b?.$client.end({ timeout: 1 }); await database?.cleanup(); });
|
|
async function eventually(check: () => Promise<boolean>) {
|
|
const deadline = Date.now() + 5000;
|
|
while (!await check()) { if (Date.now() > deadline) throw new Error("Notification condition did not settle"); await delay(10); }
|
|
}
|
|
async function open(handoffMs = 120_000) {
|
|
const errors: unknown[] = [];
|
|
const coordinator = createDeliveryWorkCoordinator({ owner: a, canRun: () => true, onError: error => errors.push(error) });
|
|
const bridge = deploymentNotifications(a, coordinator, { directDatabaseUrl: database.connectionString, handoffMs });
|
|
await bridge.ready();
|
|
return { bridge, errors, async stop() { await bridge.stop(); await coordinator.stop(); } };
|
|
}
|
|
async function listenerCount() {
|
|
const [row] = await b.execute<{ count: number }>(sql`select count(*)::int as count from pg_stat_activity
|
|
where datname = current_database() and application_name = 'paperclip-deployment-notifications'`);
|
|
return row!.count;
|
|
}
|
|
|
|
it("delivers committed hints across independent pools, without exposing an outer rollback or nested savepoint", async () => {
|
|
const fixture = await open();
|
|
const observed: number[] = [];
|
|
const topic = `test-${randomUUID()}`;
|
|
const unsubscribe = subscribeDatabaseWork(a, topic, () => {
|
|
void a.execute(sql`select id from deployment_notification_fixture`).then(rows => observed.push(rows.length));
|
|
});
|
|
try {
|
|
await b.transaction(async outer => {
|
|
await outer.transaction(async nested => {
|
|
await signalDatabaseWork(nested, topic);
|
|
await nested.execute(sql`insert into deployment_notification_fixture values ('committed')`);
|
|
});
|
|
expect(observed).toEqual([]);
|
|
});
|
|
await eventually(async () => observed.length === 1);
|
|
expect(observed).toEqual([1]);
|
|
await expect(b.transaction(async tx => {
|
|
await signalDatabaseWork(tx, topic);
|
|
await tx.execute(sql`insert into deployment_notification_fixture values ('rollback')`);
|
|
throw new Error("rollback");
|
|
})).rejects.toThrow("rollback");
|
|
let barrier = false;
|
|
const stopBarrier = subscribeDatabaseWork(a, "test-barrier", () => { barrier = true; });
|
|
await publishDatabaseWorkHint(b, "test-barrier");
|
|
await eventually(async () => barrier); stopBarrier();
|
|
expect(observed).toEqual([1]);
|
|
expect(fixture.errors).toEqual([]);
|
|
} finally { unsubscribe(); await fixture.stop(); }
|
|
});
|
|
|
|
it("reconstructs on listener reconnect and scopes live refresh hints to the company", async () => {
|
|
const fixture = await open();
|
|
let refreshed = 0, other = 0, reconstructed = 0;
|
|
const companyId = randomUUID();
|
|
const stopCompany = fixture.bridge.subscribeCompany(companyId, () => { refreshed++; });
|
|
const stopOther = fixture.bridge.subscribeCompany(randomUUID(), () => { other++; });
|
|
const stopQueue = subscribeDatabaseWork(a, "test-reconstruction", () => { reconstructed++; });
|
|
try {
|
|
await publishDatabaseWorkHint(b, `live:${companyId}`);
|
|
await eventually(async () => refreshed === 1);
|
|
expect(other).toBe(0);
|
|
await b.execute(sql`select pg_terminate_backend(pid) from pg_stat_activity where datname = current_database()
|
|
and application_name = 'paperclip-deployment-notifications'`);
|
|
await eventually(async () => reconstructed > 0);
|
|
// Reconnect cannot know which hints were lost, so it refreshes both company
|
|
// subscriptions and existing durable queue consumers once.
|
|
expect(other).toBeGreaterThan(0);
|
|
} finally { stopCompany(); stopOther(); stopQueue(); await fixture.stop(); }
|
|
});
|
|
|
|
it("disconnects at the handoff deadline, reopens for active execution and becomes quiet again", async () => {
|
|
const fixture = await open(25), runId = randomUUID();
|
|
const stopCompany = fixture.bridge.subscribeCompany(randomUUID(), () => {});
|
|
try {
|
|
expect(await listenerCount()).toBe(1);
|
|
fixture.bridge.markReady();
|
|
await eventually(async () => await listenerCount() === 0);
|
|
setNativeExecutionActive(runId, true);
|
|
await eventually(async () => await listenerCount() === 1);
|
|
setNativeExecutionActive(runId, false);
|
|
await eventually(async () => await listenerCount() === 0);
|
|
expect(fixture.errors).toEqual([]);
|
|
} finally { setNativeExecutionActive(runId, false); stopCompany(); await fixture.stop(); }
|
|
});
|