mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-06 05:56:19 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A web server replacement must preserve remote execution and later task continuation. > - The active-run handoff now survives deletion of the departing server home. > - The next admission still compared disposable instruction and skill paths as configuration. > - A native receipt could also settle without the old adapter returning its terminal metadata. > - A replacement can also collide with a run row locked by the serving controller. > - This change preserves durable session identity and terminal evidence, and defers locked recovery work so overlap can reach readiness. ## Linked Issues or Issue Description Refs #18543, #18366, #9977, #18118. **What happened?** A real Codex/Daytona journey replaced both control-plane servers and deleted their homes. Reads, saves, saved files and active execution passed. A later useful answer used a new provider session. All four run records lacked the native terminal contract. The whole qualification therefore failed. A later attempt also exposed startup failure: the replacement waited for a row lock held by the serving controller and aborted on its one-second lock timeout. **Expected behavior** An unchanged managed instruction entry and immutable skill version retain the session across server cache relocation. Actual content and configuration changes still reset it. A committed native receipt retains its terminal contract even when the originating adapter never returns. A busy recovery row defers through the existing scheduler while the replacement reaches readiness. **Steps to reproduce** Run the `rolling-web-deployment.runner-codex.daytona.recover-controller` Product E2E cell with separate controller homes and shared PostgreSQL/S3. The focused regressions reproduce the failures without a model call. ## What Changed - Read the managed instruction entry digest from the company-scoped durable file head. Do not restore or read the cache for this check. - Omit disposable instruction paths only when that durable authority exists. Fingerprint selected immutable skill versions in both runtime skill inputs. - Preserve the existing local and external instruction path checks, and unversioned skill identity. - Publish the accepted native terminal contract in the same transaction as the canonical result receipt. - Skip locked recovery rows and schedule the same run for retry. Avoid an audit write that would block on the same row. No ownership is transferred while busy. - Keep replacement readiness independent of an in-progress skill selection; migration and cache eviction still require exact pins. Missing durable skill versions still refuse startup. - Project only the instruction digest in SQL instead of loading the entire agent-file manifest per admission. - Add real PostgreSQL lock contention tests for the issue, run and coordinator, including the heartbeat startup caller and later claim. - Limit native test teardown to its owned database on a shared test host. The old behavior also fails on clean master. - Add cache-loss, scope, changed-content, changed-version and finalization regressions. Update the deployment qualification record. ## Verification - Both focused regressions failed before the source changes. - All 249 tests pass across `heartbeat-workspace-session`, `agent-file-durability`, and `paperclip-control-plane-port`. - Server typecheck and exact-revision build pass. - Assembled admission ran 59 passing cases but failed during native-suite teardown. The teardown failure reproduced on clean master. All 16 native resumption cases and teardown pass after the database-scope fix. - All three row-lock regressions fail with the pre-fix lock timeout and pass with deferred recovery. The full recovery suite passes 74 cases, including the startup caller; eight Docker-specific cases were not enabled. - The real attempt at `d2165dec5d` failed when B aborted on the run lock. Its credential and bucket were removed; all three provider sandboxes were independently confirmed absent. The next real attempt at `aae8a53d01` failed a different readiness gate while a run was preparing skill pins. That race now has a failing-before regression; all 33 readiness, durable-file and managed-skill tests pass after the fix. Its owned resources are cleaned. The `a15a12651c` live attempt passed both web replacements: 765 reads, 78 saves and 78 verified saves with zero HTTP probe failures. The first three turns preserved the provider session and native terminal contract. The fourth continuation failed because a safely suspended remote Runner was treated as live/idle; that distinct recovery gap is being handled in a follow-up. All three owned sandboxes were independently confirmed absent. No claim of full Railway qualification yet. ## Risks - The new fingerprint shape may reset a session once when upgrading from older fingerprint metadata. It is stable across subsequent cache replacements. - A versioned skill is identified by its immutable selected version; materialization and admission still enforce availability. Unversioned skills retain their existing path boundary. - A native terminal describes the accepted execution result. Workspace and issue settlement remain separate authorities. - This does not yet prove the ten-deploy staging campaign, natural database autosuspension or the production canary. ## Model Used OpenAI GPT-6 Codex, with reasoning, repository editing, shell execution and live integration-test tools. The exact runtime model variant and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
68 lines
6.0 KiB
TypeScript
68 lines
6.0 KiB
TypeScript
import { randomUUID } from "node:crypto";
|
|
import { beforeAll, afterAll, afterEach, expect, it, vi } from "vitest";
|
|
import { eq } from "drizzle-orm";
|
|
import { agentFileRoots, agents, companies, companySkills, companySkillVersions, executionWorkspaces, heartbeatRuns, providerTraceRecords, createDb } from "@paperclipai/db";
|
|
import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
|
import { assertEphemeralServerData, readEphemeralServerDataEligibility } from "../services/ephemeral-server-contract.js";
|
|
import { initializeDurableAgentFiles } from "../services/agent-file-durability.js";
|
|
vi.mock("../storage/index.js", () => ({ getStorageService: () => ({ provider: "s3" }) }));
|
|
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
|
let db: ReturnType<typeof createDb>;
|
|
beforeAll(async () => { database = await startEmbeddedPostgresTestDatabase("ephemeral-data-"); db = createDb(database.connectionString); }, 90_000);
|
|
afterAll(async () => { await db?.$client.end({ timeout: 1 }); await database?.cleanup(); });
|
|
afterEach(() => vi.unstubAllEnvs());
|
|
it("refuses volume-dependent rows before readiness and accepts their durable replacement", async () => {
|
|
vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "true"); vi.stubEnv("PAPERCLIP_DURABLE_AGENT_FILES", "true");
|
|
await expect(assertEphemeralServerData(db)).resolves.toBeUndefined();
|
|
const companyId = randomUUID(), agentId = randomUUID();
|
|
await db.insert(companies).values({ id: companyId, name: "Deployment gate", issuePrefix: "GATE" });
|
|
await db.insert(agents).values({ id: agentId, companyId, name: "Legacy files" });
|
|
await expect(assertEphemeralServerData(db)).rejects.toThrow("agent_files_require_migration");
|
|
await db.transaction(tx => initializeDurableAgentFiles(tx, { id: agentId, companyId }));
|
|
await expect(assertEphemeralServerData(db)).resolves.toBeUndefined();
|
|
await db.update(agents).set({ adapterConfig: { instructionsBundleMode: "external", instructionsRootPath: "/lost-disk" } }).where(eq(agents.id, agentId));
|
|
await expect(assertEphemeralServerData(db)).rejects.toThrow("external_agent_instructions");
|
|
await db.update(agents).set({ adapterConfig: {} }).where(eq(agents.id, agentId));
|
|
const [workspace] = await db.insert(executionWorkspaces).values({ companyId, mode: "isolated", strategyType: "directory", name: "Legacy workspace", cwd: "/lost-disk/work" }).returning();
|
|
await expect(assertEphemeralServerData(db)).rejects.toThrow("workspaces_require_migration");
|
|
await db.delete(executionWorkspaces).where(eq(executionWorkspaces.id, workspace!.id));
|
|
const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId, invocationSource: "on_demand", status: "succeeded", logStore: "local_file", logRef: "old-log" }).returning();
|
|
await expect(assertEphemeralServerData(db)).rejects.toThrow("run_logs_require_migration");
|
|
await db.update(heartbeatRuns).set({ logStore: "s3_segments", logRef: "committed-object-ref" }).where(eq(heartbeatRuns.id, run!.id));
|
|
const [trace] = await db.insert(providerTraceRecords).values({ companyId, runId: run!.id, provider: "fixture", requestedBy: "fixture", traceRef: "old-trace", expiresAt: new Date(Date.now() + 60_000) }).returning();
|
|
await expect(assertEphemeralServerData(db)).rejects.toThrow("provider_traces_require_migration");
|
|
await db.update(providerTraceRecords).set({ deletedAt: new Date(), status: "deleted" }).where(eq(providerTraceRecords.id, trace!.id));
|
|
await expect(assertEphemeralServerData(db)).resolves.toBeUndefined();
|
|
await expect(db.transaction(async tx => {
|
|
const [created] = await tx.insert(agents).values({ companyId, name: "Rolled back hire" }).returning();
|
|
await initializeDurableAgentFiles(tx, created!);
|
|
throw new Error("abort hire");
|
|
})).rejects.toThrow("abort hire");
|
|
expect(await db.select().from(agentFileRoots)).toHaveLength(1);
|
|
});
|
|
|
|
it("keeps replacement readiness independent of a serving controller's unfinished skill selection", async () => {
|
|
vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "true"); vi.stubEnv("PAPERCLIP_DURABLE_AGENT_FILES", "true");
|
|
const companyId = randomUUID(), agentId = randomUUID(), skillId = randomUUID(), versionId = randomUUID();
|
|
await db.insert(companies).values({ id: companyId, name: "Overlapping skill preparation", issuePrefix: "PINS" });
|
|
await db.insert(agents).values({ id: agentId, companyId, name: "Preparing agent" });
|
|
await db.transaction(tx => initializeDurableAgentFiles(tx, { id: agentId, companyId }));
|
|
await db.insert(companySkills).values({ id: skillId, companyId, key: "overlap", slug: "overlap", name: "Overlap",
|
|
sourceType: "managed", markdown: "# Overlap", fileInventory: [{ path: "SKILL.md", kind: "entrypoint" }] });
|
|
await db.insert(companySkillVersions).values({ id: versionId, companyId, companySkillId: skillId, revisionNumber: 1,
|
|
fileInventory: [{ path: "SKILL.md", kind: "entrypoint", content: "# Overlap" }] });
|
|
await db.update(companySkills).set({ currentVersionId: versionId }).where(eq(companySkills.id, skillId));
|
|
await expect(assertEphemeralServerData(db)).resolves.toBeUndefined();
|
|
// Admission marks the run running before preparation publishes its skill pins.
|
|
// That prevents cache eviction/migration, but does not make durable bytes local.
|
|
const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId, invocationSource: "on_demand",
|
|
status: "running", contextSnapshot: {} }).returning();
|
|
await expect(assertEphemeralServerData(db)).resolves.toBeUndefined();
|
|
expect((await readEphemeralServerDataEligibility(db)).blockers).toContain("skills_require_migration");
|
|
await db.update(heartbeatRuns).set({ contextSnapshot: { skillVersions: { overlap: versionId } } }).where(eq(heartbeatRuns.id, run.id));
|
|
await expect(assertEphemeralServerData(db)).resolves.toBeUndefined();
|
|
expect((await readEphemeralServerDataEligibility(db)).eligible).toBe(true);
|
|
await db.update(companySkills).set({ currentVersionId: null }).where(eq(companySkills.id, skillId));
|
|
await expect(assertEphemeralServerData(db)).rejects.toThrow("skills_require_migration");
|
|
});
|