diff --git a/doc/DATABASE.md b/doc/DATABASE.md index 13b2aad680e..f69fb4eca8a 100644 --- a/doc/DATABASE.md +++ b/doc/DATABASE.md @@ -1023,6 +1023,13 @@ objects can be reclaimed. Object existence alone is not a committed save. remote allocation/path, writer run, and dirty flag. Paths in this record belong to the remote allocation. Save publication checks the current writer/epoch and atomically commits the ledger row, authority pointer, and operation receipt. +`execution_workspaces.metadata.remoteWorkspaceSnapshotRepositories` records the +committed snapshot ID and the repository descriptors covered by that archive. +Reservation retains this coverage when it adds requested repositories to the +current source. Successful save publication advances coverage and the snapshot +pointer together. Older saved workspaces capture their previous source inventory +at the next reservation. Newly attached repositories use normal acquisition; +they cannot claim coverage from an older archive. No schema migration is needed. `workspace_operations` journals preparation, save, restore, and local import; large file bodies stay outside the database. Ledger identities deliberately survive source-record deletion so object cleanup can finish. diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index fcd0683e9f3..d7608647f99 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -1775,8 +1775,15 @@ import. An interrupted apply retains its journal and admission hold; retry the same command to reconcile it before starting another writer. Supported sources include empty task folders, repository URLs, and managed -mutable repositories from the same project. Configured local folders use the -legacy path until imported. V1 rejects read-only cross-project referenced folders +mutable repositories. Authorized mentioned projects attach all their configured +Git repositories as writable copies under `.paperclip-repositories/` in the task +workspace. These copies use the existing remote save/restore lifecycle and do not +write to the source project's folder. Later turns retain their saved paths, refs, +pins and edits, and recheck each retained project's access and trust policy. +New repository attachments after a save are acquired separately from the saved +archive. Repository refs use `repoRef`, then `defaultRef`, then the Git remote's +default. A configured remote branch is resolved before its checkout is published. +Configured local folders use the legacy path until imported. V1 rejects explicitly read-only sources and custom cleanup/teardown commands on remote close. Automatic Core services, manual workspace jobs, services, and runtime commands are also rejected before any local directory access; run those commands through an agent task. One writer owns each logical @@ -1784,6 +1791,11 @@ workspace; separate workspaces can run concurrently. The Files panel identifies **Live workspace** and **Saved copy** with its save time. An unavailable dirty allocation is not silently replaced with an older saved copy. +Workspace validation failures show their redacted explanation in the task's +failure notice. They retain the normal Retry permission and admission gates. +Retained-folder access and trust denials use this explanation; low-trust project +boundary denials retain their dedicated project-access action. + A child or follow-up task can use a remote workspace inherited through its persisted task binding, including an isolated workspace or operator branch. Reservation checks current task/run access and all retained workspace sources. diff --git a/doc/SPEC-implementation.md b/doc/SPEC-implementation.md index f8a9ceaf7cb..c9455c9de14 100644 --- a/doc/SPEC-implementation.md +++ b/doc/SPEC-implementation.md @@ -2650,8 +2650,17 @@ have stopped. Cleanup retains copies whose upload completion is unknown. This includes operations from older builds without save-attempt records. A final save alone does not prove that an older upload has stopped. -V1 supports plain task folders, Git sources, and managed mutable repositories -from the same project. It rejects read-only cross-project referenced folders, +V1 supports plain task folders, Git sources, and managed mutable repositories. +An authorized mentioned project's configured Git repositories become writable +copies inside the task workspace, under `.paperclip-repositories/`. The copies +retain their saved files, paths, refs and pins on subsequent turns. Preparation +rechecks the access and trust policy of each retained source project. This does +not grant permission to change a shared source folder or push to a repository. +Repositories attached on a later turn are acquired before they become part of +the next committed archive. Snapshot coverage is separate from the requested +source recipe. New copies honor `repoRef`, then `defaultRef`, then the remote's +default branch; an invalid ref cannot publish a partial secondary checkout. +It rejects explicitly read-only sources, explicit concurrent shared writers, and custom cleanup/teardown commands on remote close. Manual workspace runtime commands, jobs, and services are also unsupported in V1 and must fail before local filesystem work. Agent tasks remain @@ -2661,11 +2670,16 @@ must fail admission rather than fall back to a controller checkout. A reference to an authorized project with no configured workspace source does not attach a folder. It adds a resolution warning and preserves the task's -workspace. A configured cross-project source remains subject to the read-only -folder restriction. `PAPERCLIP_MULTI_PROJECT_WORKSPACE_SYNC_REMOTE` gates configured +workspace. A configured cross-project local folder must be imported before remote +acquisition. `PAPERCLIP_MULTI_PROJECT_WORKSPACE_SYNC_REMOTE` gates configured sources; a source-less project reference needs no folder and stays metadata-only when that switch is off. +Task failure notices show the redacted workspace validation explanation when +preparation fails. Existing retry permissions and execution gates still apply. +This includes retained-folder access and trust denials. Low-trust project boundary +denials retain their separate project-access action. + An imported saved Git workspace can resume from its saved files after its old local repository is removed. Local repository paths remain invalid for new remote acquisition. Saves reject unfinished Git operations, including rebase, diff --git a/maps/agent-access-map/workspaces.md b/maps/agent-access-map/workspaces.md index 1de13f4f63f..10cc21d6026 100644 --- a/maps/agent-access-map/workspaces.md +++ b/maps/agent-access-map/workspaces.md @@ -71,9 +71,10 @@ until an explicit import. Stored `false` disables new enrollment unless managed configuration overrides it. Enrolled workspaces keep remote authority when new enrollment is disabled, and missing persistence prerequisites fail admission without a controller checkout fallback. Current authorization and responsible-user -checks still apply; -read-only cross-project referenced folders and explicitly concurrent shared -writers are rejected. Managed mutable same-project repositories are supported. +checks still apply. Authorized referenced Git repositories use writable task +copies without changing shared source folders or upstream push permissions. +Explicitly read-only sources and explicitly concurrent shared writers are rejected. +Managed mutable same-project repositories are supported. Portable capture streams validated working files while staging self-contained Git metadata. The smaller temporary-space requirement grants no new agent access or execution authority. Changed sources still fail before a save receipt. diff --git a/maps/feature-map/workspaces.md b/maps/feature-map/workspaces.md index 32f54e84ced..90923e81268 100644 --- a/maps/feature-map/workspaces.md +++ b/maps/feature-map/workspaces.md @@ -70,8 +70,9 @@ work. Finish the run and inspect its saved version. Replace the webserver and verify the next task turn uses the same remote folder without a controller checkout. Then release the saved allocation and verify a new one restores the files, Git state, and managed same-project repositories without the old sandbox -or original Git remote. Read-only cross-project referenced folders must fail -admission. Separate workspaces may run concurrently; shared concurrent writers +or original Git remote. Authorized referenced Git repositories must be editable +task copies; explicitly read-only sources must fail admission. Separate workspaces +may run concurrently; shared concurrent writers must be rejected. The Files panel must label **Live workspace** or **Saved copy** with its save diff --git a/packages/shared/src/types/execution-projection.ts b/packages/shared/src/types/execution-projection.ts index f3b9ef7d5c4..0dd0c3a8898 100644 --- a/packages/shared/src/types/execution-projection.ts +++ b/packages/shared/src/types/execution-projection.ts @@ -100,6 +100,7 @@ export interface ExecutionReconciliation { * and recovery records; ordinary product surfaces render this projection. */ export interface TaskInterruptionPresentation { providerFailure?: import("../provider-failure.generated.js").ProviderFailure; + workspaceFailure?: { message: string }; intentId: string | null; responsibility: "system" | "user"; state: "working" | "retry_scheduled" | "failed" | "waiting_for_user" | "resolved"; diff --git a/server/src/__tests__/attention-retention-work.test.ts b/server/src/__tests__/attention-retention-work.test.ts index 36e727dbc8a..1f840e18ac8 100644 --- a/server/src/__tests__/attention-retention-work.test.ts +++ b/server/src/__tests__/attention-retention-work.test.ts @@ -37,6 +37,7 @@ const delay = DEFAULT_DECISION_ARCHIVE_DAYS * 86_400_000; const errors: unknown[] = []; const statements: string[] = []; const parameterCounts: number[] = []; + const testBodies = new Set>(); let captureBlockerPlan = false; let blockerPlanQuery: { query: string; parameters: unknown[] } | undefined; beforeAll(async () => { @@ -64,6 +65,7 @@ const delay = DEFAULT_DECISION_ARCHIVE_DAYS * 86_400_000; vi.useFakeTimers({ now: Number(clock.now), toFake: ["Date", "setTimeout", "clearTimeout", "setInterval", "clearInterval"], shouldAdvanceTime: true }); }); afterEach(async () => { + await Promise.allSettled([...testBodies]); await coordinator?.stop(); coordinator = undefined; vi.useRealTimers(); vi.restoreAllMocks(); await db.transaction(async tx => { @@ -76,6 +78,24 @@ const delay = DEFAULT_DECISION_ARCHIVE_DAYS * 86_400_000; }); afterAll(async () => { await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); }); const old = () => new Date(Date.now() - delay - 600_000); + async function tracked(body: () => Promise) { + const running = body(); testBodies.add(running); + try { await running; } finally { testBodies.delete(running); } + } + async function insertSharedWorkspaceHistory(workspaceId: string, query: SQL) { + await db.transaction(async tx => { + await tx.execute(sql`alter table issues disable trigger issues_record_workspace_privacy_source`); + await tx.execute(query); + await tx.execute(sql`update execution_workspaces set metadata = metadata + where id = ${workspaceId}::uuid and company_id = ${companyId}::uuid`); + await tx.execute(sql`alter table issues enable trigger issues_record_workspace_privacy_source`); + const [provenance] = await tx.execute<{ complete: boolean }>(sql`select + w.metadata->'_issuePrivacySources' = (select jsonb_object_agg(i.id::text, true) from issues i + where i.execution_workspace_id = w.id and i.company_id = w.company_id) as complete + from execution_workspaces w where w.id = ${workspaceId}::uuid and w.company_id = ${companyId}::uuid`); + expect(provenance?.complete).toBe(true); + }); + } async function insertIssue(status = "todo", values: Partial = {}, database: Db = db) { const [row] = await database.insert(issues).values({ companyId, title: randomUUID(), status, createdByAgentId: agentId, createdAt: old(), updatedAt: old(), ...values }).returning(); @@ -115,6 +135,19 @@ const delay = DEFAULT_DECISION_ARCHIVE_DAYS * 86_400_000; authActor: { type: "board" as const, source: "local_implicit" as const, userId: "board", companyIds: [companyId], isInstanceAdmin: true }, actor: { actorType: "user" as const, actorId: "board", userId: "board", agentId: null, runId: null, agentApiKeyId: null, responsibleUserId: "board" } }); + it("restores ordinary workspace provenance recording after a failed bulk fixture", async () => { + const workspaceId = randomUUID(); + await db.execute(sql`insert into execution_workspaces (id, company_id, name, mode, strategy_type, metadata) + values (${workspaceId}::uuid, ${companyId}::uuid, 'Fixture rollback', 'shared', 'project_primary', '{"fixture":"retained"}'::jsonb)`); + await expect(insertSharedWorkspaceHistory(workspaceId, sql`select 1 / 0`)).rejects.toThrow(); + const [trigger] = await db.execute<{ tgenabled: string }>(sql`select tgenabled from pg_trigger + where tgrelid = 'issues'::regclass and tgname = 'issues_record_workspace_privacy_source'`); + expect(trigger.tgenabled).toBe("O"); + const issue = await insertIssue("done", { executionWorkspaceId: workspaceId }); + const [workspace] = await db.execute<{ metadata: unknown }>(sql`select metadata from execution_workspaces where id = ${workspaceId}::uuid`); + expect(workspace.metadata).toEqual({ fixture: "retained", _issuePrivacySources: { [issue.id]: true } }); + }); + it("records the aggregate baseline SQL across old ticks and eliminates it after settlement", async () => { await insertIssue("done"); const ticks: number[] = []; @@ -442,13 +475,13 @@ const delay = DEFAULT_DECISION_ARCHIVE_DAYS * 86_400_000; expect(await db.select().from(decisionArchiveNotificationOutbox)).toHaveLength(0); }); - it.each(["running", "run", "confirmation", "completed"] as const)("filters completed blocker history before bounded %s attention and archive checks", async kind => { + it.each(["running", "run", "confirmation", "completed"] as const)("filters completed blocker history before bounded %s attention and archive checks", kind => tracked(async () => { const root = await insertIssue("blocked"), workspaceId = randomUUID(); await db.execute(sql`insert into execution_workspaces (id, company_id, name, mode, strategy_type) values (${workspaceId}::uuid, ${companyId}::uuid, 'Finalized shared history', 'shared', 'project_primary')`); await db.execute(sql`insert into workspace_operations (company_id, execution_workspace_id, phase, status, started_at) values (${companyId}::uuid, ${workspaceId}::uuid, 'workspace_finalize', 'succeeded', ${old().toISOString()}::timestamptz)`); - await db.execute(sql`with history as ( + await insertSharedWorkspaceHistory(workspaceId, sql`with history as ( insert into issues (id, company_id, title, status, updated_at, execution_workspace_id) select ('00000000-0000-4000-8000-' || lpad(to_hex(n), 12, '0'))::uuid, ${companyId}::uuid, 'completed-blocker-' || n, 'done', ${old().toISOString()}::timestamptz, ${workspaceId}::uuid @@ -503,7 +536,7 @@ const delay = DEFAULT_DECISION_ARCHIVE_DAYS * 86_400_000; expect(batch.deferredRootIds.size).toBe(0); expect(Boolean(batch.attention.get(root.id)?.blockingTreeLive)).toBe(kind !== "completed"); expect(parameterCounts.length).toBeLessThanOrEqual(100); - }); + })); it("does not archive an unknown truncated blocker tree during its fresh check", async () => { const root = await insertIssue("blocked"); @@ -569,7 +602,7 @@ const delay = DEFAULT_DECISION_ARCHIVE_DAYS * 86_400_000; values (${workspaceId}::uuid, ${companyId}::uuid, 'Pending shared export', 'shared', 'project_primary')`); await workspaceOperationService(db).createRecorder({ companyId, executionWorkspaceId: workspaceId }) .recordOperation({ phase: "workspace_provision", run: async () => ({ status: "succeeded" }) }); - await db.execute(sql`with history as ( + await insertSharedWorkspaceHistory(workspaceId, sql`with history as ( insert into issues (company_id, title, status, updated_at, execution_workspace_id) select ${companyId}::uuid, 'waiting-for-export-' || n, 'done', ${old().toISOString()}::timestamptz, ${workspaceId}::uuid from generate_series(1, 2100) n returning id) @@ -1045,7 +1078,7 @@ const delay = DEFAULT_DECISION_ARCHIVE_DAYS * 86_400_000; expect(await state("agent_error_alert", agentId)).toBeNull(); }); - it("drains a paged backlog with source-scoped self notifications and leaves completed history quiet", async () => { + it("drains a paged backlog with source-scoped self notifications and leaves completed history quiet", () => tracked(async () => { await db.insert(issues).values(Array.from({ length: 225 }, () => ({ companyId, title: randomUUID(), status: "in_review", assigneeUserId: "board", updatedAt: old() }))); const service = attentionRetentionService(db); const load = vi.spyOn(service, "load"); @@ -1057,7 +1090,7 @@ const delay = DEFAULT_DECISION_ARCHIVE_DAYS * 86_400_000; expect(scopes.filter(scope => scope && !scope.sourceId)).toHaveLength(0); expect(scopes.filter(scope => scope?.sourceId).length).toBeLessThanOrEqual(225); expect(new Set((await db.select().from(decisionRetention)).map(row => row.sourceId)).size).toBe(225); - }); + }), 45_000); it("retries a failed atomic archive across drain and preserves notifications during a scan", async () => { const row = await insertIssue("in_review", { assigneeUserId: "board" }); diff --git a/server/src/__tests__/task-interruption-query.test.ts b/server/src/__tests__/task-interruption-query.test.ts index 3983ab50572..a179045cb39 100644 --- a/server/src/__tests__/task-interruption-query.test.ts +++ b/server/src/__tests__/task-interruption-query.test.ts @@ -24,6 +24,28 @@ describe.skipIf(!support.supported)("task interruption latest run query", () => }, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS); afterAll(async () => { await db?.$client.end(); await database?.cleanup(); }); + it("redacts workspace preparation errors on task detail and list reads", async () => { + vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "a".repeat(64)); + try { + const f = await seedInterruptionFixture(db, { selected: 1, history: 1, unrelated: 0, payloadBytes: 0 }); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId)); + const secret = `private-workspace-value-${randomUUID()}`; + const error = `Could not prepare the project folder: ${secret}.`; + await db.update(heartbeatRuns).set({ errorCode: "workspace_validation_failed", error }).where(eq(heartbeatRuns.id, run!.id)); + await createRunSecretRedactionRegistry(db).register(f.companyId, run!.id, secret); + const list = await taskInterruptionPresentations(db, f.companyId, f.input); + const detail = await taskInterruptionPresentation(db, f.input[0]!); + for (const presentation of [list.get(f.input[0]!.id), detail]) { + expect(presentation?.workspaceFailure?.message).toContain("Could not prepare the project folder"); + expect(presentation?.workspaceFailure?.message).toContain("REDACTED"); + expect(JSON.stringify(presentation)).not.toContain(secret); + } + vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "b".repeat(64)); + expect(await taskInterruptionPresentation(db, f.input[0]!)).not.toHaveProperty("workspaceFailure"); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run!.id)))[0]?.error).toBe(error); + } finally { vi.unstubAllEnvs(); } + }); + it.each(["valid", "numeric_user", "numeric_audit"] as const)("retains bounded typed Stop evidence in the real batched projection (%s)", async variant => { const companyId = randomUUID(), agentId = randomUUID(), runId = randomUUID(); await db.insert(companies).values({ id: companyId, name: "Stopped task", issuePrefix: `S${companyId.slice(0, 7)}` }); diff --git a/server/src/services/heartbeat/remote-workspace-source.test.ts b/server/src/services/heartbeat/remote-workspace-source.test.ts index a7747eda699..a096cf73a09 100644 --- a/server/src/services/heartbeat/remote-workspace-source.test.ts +++ b/server/src/services/heartbeat/remote-workspace-source.test.ts @@ -130,21 +130,58 @@ describe.skipIf(!support.supported)("remote referenced project sources", () => { } }); - it("still rejects an actual repository reference before any workspace is created", async () => { + it("attaches a referenced project's repositories as editable task copies", async () => { const f = await fixture(true); - await expect(resolveRemoteWorkspaceSource(db, { ...f, responsibleUserId: null, - projectId: null, projectWorkspaceId: null, taskDirectory: true })).rejects.toMatchObject({ - code: "workspace_validation_failed", resultJson: { reason: "remote_workspace_readonly_sources_unsupported" }, - }); + await db.insert(projectWorkspaces).values({ companyId: f.companyId, projectId: f.projectId, + name: "Second repository", sourceType: "git_repo", repoUrl: "https://example.test/team/second.git", repoRef: "release" }); + const result = await resolveRemoteWorkspaceSource(db, { ...f, responsibleUserId: null, + projectId: null, projectWorkspaceId: null, taskDirectory: true }); + expect(result).toMatchObject({ authority: "remote", source: { kind: "empty", repositories: [ + { projectId: f.projectId, repoUrl: "https://example.test/team/repository.git", ref: null }, + { projectId: f.projectId, repoUrl: "https://example.test/team/second.git", ref: "release" }, + ] }, referencedProjectFailures: [], warnings: [] }); + if (result.authority !== "remote") throw new Error("Expected remote task copies"); + expect(result.source.repositories).toHaveLength(2); + for (const repository of result.source.repositories!) { + expect(repository.readOnly).not.toBe(true); + expect(repository.relativePath).toMatch(/^\.paperclip-repositories\/[^/]+$/); + expect(repository.projectWorkspaceId).toBe(repository.id); + } + expect(new Set(result.source.repositories!.map(repository => repository.relativePath)).size).toBe(2); expect(await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.companyId, f.companyId))).toEqual([]); }); + it("keeps the selected anchor and adds the other project's writable repository", async () => { + const f = await assignedTask(true), referencedProjectId = randomUUID(); + await db.insert(projects).values({ id: referencedProjectId, companyId: f.companyId, name: "Skill sources" }); + const [repository] = await db.insert(projectWorkspaces).values({ companyId: f.companyId, projectId: referencedProjectId, + name: "Skills", sourceType: "git_repo", repoUrl: "https://example.test/team/skills.git" }).returning(); + await db.insert(issueComments).values({ companyId: f.companyId, issueId: f.issueId, + body: `Write a skill in [@Skill sources](${buildProjectMentionHref(referencedProjectId)}).` }); + const result = await resolveRemoteWorkspaceSource(db, { ...f, responsibleUserId: null, + projectId: f.projectId, projectWorkspaceId: null }); + expect(result).toMatchObject({ authority: "remote", projectId: f.projectId, + source: { kind: "git", repoUrl: "https://example.test/team/repository.git", repositories: [ + { id: repository.id, projectId: referencedProjectId, repoUrl: "https://example.test/team/skills.git" }, + ] } }); + expect(result.authority === "remote" && result.source.repositories).toHaveLength(1); + expect(result.authority === "remote" && result.source.repositories![0]!.readOnly).not.toBe(true); + }); + + it.each([null, "feature"])("uses a referenced repository's default ref unless overridden by %s", async repoRef => { + const f = await fixture(true); + await db.update(projectWorkspaces).set({ defaultRef: "release", repoRef }).where(eq(projectWorkspaces.projectId, f.projectId)); + const result = await resolveRemoteWorkspaceSource(db, { ...f, responsibleUserId: null, + projectId: null, projectWorkspaceId: null, taskDirectory: true }); + expect(result).toMatchObject({ authority: "remote", source: { repositories: [{ ref: repoRef ?? "release" }] } }); + }); + it("does not treat a configured source without a repository URL as metadata-only", async () => { const f = await fixture(false); await db.insert(projectWorkspaces).values({ companyId: f.companyId, projectId: f.projectId, name: "Configured source" }); await expect(resolveRemoteWorkspaceSource(db, { ...f, responsibleUserId: null, projectId: null, projectWorkspaceId: null, taskDirectory: true })).rejects.toMatchObject({ - resultJson: { reason: "remote_workspace_readonly_sources_unsupported" }, + resultJson: { reason: "remote_workspace_source_unavailable", projectId: f.projectId }, }); expect(await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.companyId, f.companyId))).toEqual([]); }); @@ -163,15 +200,17 @@ describe.skipIf(!support.supported)("remote referenced project sources", () => { } }); - it("keeps a real repository denied when a source-less reference is also mentioned", async () => { + it("attaches a repository when a source-less project is also mentioned", async () => { const f = await fixture(true), metadataProjectId = randomUUID(); await db.insert(projects).values({ id: metadataProjectId, companyId: f.companyId, name: "Metadata" }); await db.insert(issueComments).values({ companyId: f.companyId, issueId: f.issueId, body: `Also mention [@Metadata](${buildProjectMentionHref(metadataProjectId)}).` }); - await expect(resolveRemoteWorkspaceSource(db, { ...f, responsibleUserId: null, - projectId: null, projectWorkspaceId: null, taskDirectory: true })).rejects.toMatchObject({ - resultJson: { reason: "remote_workspace_readonly_sources_unsupported" }, - }); + const result = await resolveRemoteWorkspaceSource(db, { ...f, responsibleUserId: null, + projectId: null, projectWorkspaceId: null, taskDirectory: true }); + expect(result).toMatchObject({ authority: "remote", source: { repositories: [ + { projectId: f.projectId, repoUrl: "https://example.test/team/repository.git" }, + ] }, referencedProjectFailures: [{ projectId: metadataProjectId, reason: "resolution" }] }); + expect(result.authority === "remote" && result.source.repositories).toHaveLength(1); expect(await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.companyId, f.companyId))).toEqual([]); }); @@ -186,6 +225,31 @@ describe.skipIf(!support.supported)("remote referenced project sources", () => { expect(await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.companyId, f.companyId))).toEqual([]); }); + it("does not attach a project outside the run's low-trust boundary", async () => { + const f = await fixture(true); + const policy = { trustPreset: "low_trust_review", authorizationPolicy: { trustPreset: "low_trust_review", + trustBoundary: { mode: "low_trust_review", companyId: f.companyId, rootIssueId: f.issueId, projectIds: [] } } }; + await db.update(agents).set({ permissions: policy }).where(eq(agents.id, f.agentId)); + await db.update(issues).set({ executionPolicy: policy }).where(eq(issues.id, f.issueId)); + await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: f.issueId, executionPolicy: policy } }).where(eq(heartbeatRuns.id, f.runId)); + const result = await resolveRemoteWorkspaceSource(db, { ...f, responsibleUserId: null, + projectId: null, projectWorkspaceId: null, taskDirectory: true }); + expect(result).toMatchObject({ authority: "remote", source: { kind: "empty" }, + referencedProjectFailures: [{ projectId: f.projectId, reason: "authorization" }] }); + expect(result.authority === "remote" && result.source.repositories).toBeUndefined(); + }); + + it("does not apply a referenced project's restricted trust policy to an ordinary task", async () => { + const f = await fixture(true); + await db.update(projects).set({ executionWorkspacePolicy: { trustPreset: "low_trust_review", authorizationPolicy: { + trustPreset: "low_trust_review", trustBoundary: { mode: "low_trust_review", companyId: f.companyId, + rootIssueId: f.issueId, projectIds: [f.projectId] }, + } } }).where(eq(projects.id, f.projectId)); + await expect(resolveRemoteWorkspaceSource(db, { ...f, responsibleUserId: null, + projectId: null, projectWorkspaceId: null, taskDirectory: true })).rejects.toMatchObject({ status: 403 }); + expect(await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.companyId, f.companyId))).toEqual([]); + }); + it("preserves an enrolled workspace's saved source for a metadata-only project mention", async () => { const f = await fixture(false), snapshotId = randomUUID(); const source = { kind: "git" as const, repoUrl: "https://example.test/team/saved.git", ref: "saved-branch" }; @@ -205,4 +269,58 @@ describe.skipIf(!support.supported)("remote referenced project sources", () => { expect(await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.companyId, f.companyId))) .toEqual([workspace]); }); + + it.each(["saved", "read-only", "foreign-project", "restricted-trust", "low-trust-boundary"])("preserves %s repository authority on a later turn", async scenario => { + const f = await fixture(true); + const [repository] = await db.select().from(projectWorkspaces).where(eq(projectWorkspaces.projectId, f.projectId)); + const savedRepository = { id: repository.id, projectWorkspaceId: repository.id, projectId: f.projectId, + relativePath: ".paperclip-repositories/saved-skills", repoUrl: repository.repoUrl!, ref: "saved-branch", + pinnedCommit: "a".repeat(40), ...(scenario === "read-only" ? { readOnly: true } : {}) }; + const source = { kind: "empty" as const, repositories: [savedRepository] }; + const [workspace] = await db.insert(executionWorkspaces).values({ companyId: f.companyId, + sourceIssueId: f.issueId, mode: "isolated_workspace", strategyType: "project_primary", + name: "Saved task copies", providerType: "cloud_sandbox", cwd: "/workspaces/saved", + metadata: { remoteWorkspace: { schema: "paperclip.remote-workspace.v1", epoch: 2, snapshotId: randomUUID(), + source, location: null, writerRunId: null, dirty: false } }, + }).returning(); + await db.update(issues).set({ executionWorkspaceId: workspace.id }).where(eq(issues.id, f.issueId)); + if (scenario === "foreign-project") { + const companyId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Other company", issuePrefix: companyId.slice(0, 8) }); + await db.update(projects).set({ companyId }).where(eq(projects.id, f.projectId)); + } else { + await db.update(projectWorkspaces).set({ repoRef: "new-default" }).where(eq(projectWorkspaces.id, repository.id)); + await db.delete(issueComments).where(eq(issueComments.issueId, f.issueId)); + } + if (scenario === "restricted-trust") { + await db.update(projects).set({ executionWorkspacePolicy: { trustPreset: "low_trust_review", authorizationPolicy: { + trustPreset: "low_trust_review", trustBoundary: { mode: "low_trust_review", companyId: f.companyId, + rootIssueId: f.issueId, projectIds: [f.projectId] }, + } } }).where(eq(projects.id, f.projectId)); + } else if (scenario === "low-trust-boundary") { + const policy = { trustPreset: "low_trust_review", authorizationPolicy: { trustPreset: "low_trust_review", + trustBoundary: { mode: "low_trust_review", companyId: f.companyId, rootIssueId: f.issueId, projectIds: [] } } }; + await db.update(agents).set({ permissions: policy }).where(eq(agents.id, f.agentId)); + await db.update(issues).set({ executionPolicy: policy }).where(eq(issues.id, f.issueId)); + await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: f.issueId, executionPolicy: policy } }).where(eq(heartbeatRuns.id, f.runId)); + } + const resolve = () => resolveRemoteWorkspaceSource(db, { ...f, responsibleUserId: null, + projectId: null, projectWorkspaceId: null, existingWorkspace: toExecutionWorkspace(workspace) }); + if (scenario === "saved") { + const result = await resolve(); + expect(result).toMatchObject({ authority: "remote", source }); + expect(result.authority === "remote" && result.source.repositories).toHaveLength(1); + } else if (scenario === "read-only") { + await expect(resolve()).rejects.toMatchObject({ + resultJson: { reason: "remote_workspace_readonly_sources_unsupported" }, + }); + } else if (scenario === "low-trust-boundary") { + await expect(resolve()).rejects.toMatchObject({ status: 403, message: "Project is outside this low-trust boundary." }); + } else await expect(resolve()).rejects.toMatchObject({ + code: "workspace_validation_failed", + message: "A saved project folder is no longer accessible or its trust policy no longer permits this task.", + resultJson: { reason: "remote_workspace_source_access_denied", projectId: f.projectId }, + }); + expect((await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspace.id)))[0]).toEqual(workspace); + }); }); diff --git a/server/src/services/heartbeat/remote-workspace-source.ts b/server/src/services/heartbeat/remote-workspace-source.ts index 22aea2105e7..ae4232036bc 100644 --- a/server/src/services/heartbeat/remote-workspace-source.ts +++ b/server/src/services/heartbeat/remote-workspace-source.ts @@ -1,7 +1,10 @@ import { asc, and, eq, isNull, notExists } from "drizzle-orm"; import { projects, projectWorkspaces, type Db } from "@paperclipai/db"; import type { TaskWorkspaceIntent } from "@paperclipai/shared"; -import { assertTaskWorkspaceExecutionAccess } from "../task-workspace-access.js"; +import { HttpError } from "../../errors.js"; +import { ProjectAccessDeniedError } from "../project-access-denial.js"; +import { assertTaskWorkspaceExecutionAccess, assertTaskWorkspaceSourceProjectAccess } from "../task-workspace-access.js"; +import { assertSelectedWorkspaceTrust } from "../task-repository-acquisition.js"; import { managedProjectCheckoutName, projectRepositoryIdentity } from "../managed-project-checkout-name.js"; import { issueService } from "../issues.js"; import { projectService } from "../projects.js"; @@ -47,11 +50,12 @@ export async function resolveRemoteWorkspaceSource(db: Db, input: { branchName?: string | null; branchMode?: "create" | "existing"; }): Promise { - const selected = await assertTaskWorkspaceExecutionAccess(db, { + const actor = { type: "agent", source: "agent_jwt", companyId: input.companyId, agentId: input.agentId, runId: input.runId, onBehalfOfUserId: input.responsibleUserId === "local-board" ? null : input.responsibleUserId, - }, input.companyId, { issueId: input.issueId, + } as const; + const selected = await assertTaskWorkspaceExecutionAccess(db, actor, input.companyId, { issueId: input.issueId, ...(!input.selection && input.existingWorkspace ? { workspaceId: input.existingWorkspace.id } : {}), ...(input.selection ? { persistedSelection: input.selection.selection } : {}), }); @@ -62,8 +66,22 @@ export async function resolveRemoteWorkspaceSource(db: Db, input: { }; const withReferences = async (base: RemoteWorkspaceSource, anchorProjectId: string | null, sourceId: string | null, savedSource = false): Promise => { - const extra: NonNullable = - (base.repositories ?? []).filter(repository => !repository.readOnly); + if (base.repositories?.some(repository => repository.readOnly)) throw new WorkspaceValidationFailure( + "This remote runtime cannot safely provide read-only project folders.", + { reason: "remote_workspace_readonly_sources_unsupported" }); + const extra = [...(base.repositories ?? [])]; + for (const projectId of new Set(extra.flatMap(repository => repository.projectId ? [repository.projectId] : []))) { + try { + await assertTaskWorkspaceSourceProjectAccess(db, actor, input.companyId, projectId); + await assertSelectedWorkspaceTrust(db, { companyId: input.companyId, actor, issueId: input.issueId, + sourceProjectId: projectId }); + } catch (error) { + if (error instanceof ProjectAccessDeniedError || !(error instanceof HttpError) || error.status !== 403) throw error; + throw new WorkspaceValidationFailure( + "A saved project folder is no longer accessible or its trust policy no longer permits this task.", + { reason: "remote_workspace_source_access_denied", projectId }); + } + } // A verified saved archive owns these files after import. Its original Git // source remains an identity and need not be reachable from the new sandbox. if (!savedSource) { @@ -72,19 +90,35 @@ export async function resolveRemoteWorkspaceSource(db: Db, input: { } const referenced = isMultiProjectWorkspaceSyncEnabled() ? await resolveRunReferencedProjects(input.issueId, anchorProjectId, { companyId: input.companyId, - actor: { type: "agent", source: "agent_jwt", companyId: input.companyId, - agentId: input.agentId, runId: input.runId, - onBehalfOfUserId: input.responsibleUserId === "local-board" ? null : input.responsibleUserId }, + actor, issues: issueService(db), projects: projectService(db), access: authorizationService(db), }) : { additional: [], warnings: [], failures: [] }; const sourceLess = referenced.additional.filter(({ project }) => Array.isArray(project.workspaces) && project.workspaces.length === 0); if (referenced.additional.length > sourceLess.length) { if (!isMultiProjectWorkspaceSyncRemoteEnabled()) throw new WorkspaceValidationFailure( "Referenced remote workspace access is disabled.", { reason: "remote_referenced_workspace_disabled" }); - // Native sandbox providers currently run with external sandbox authority. - // They cannot enforce a separate read-only filesystem root for a project. - throw new WorkspaceValidationFailure("This remote runtime cannot safely provide read-only project folders.", - { reason: "remote_workspace_readonly_sources_unsupported" }); + for (const { projectId, project } of referenced.additional) { + if (!project.workspaces.length) continue; + await assertSelectedWorkspaceTrust(db, { companyId: input.companyId, actor, issueId: input.issueId, + sourceProjectId: projectId }); + for (const repository of project.workspaces) { + if (extra.some(saved => saved.id === repository.id)) continue; + if (repository.sourceType === "remote_managed" || (repository.cwd && repository.cwd !== "/__paperclip_repo_only__") + || (repository.repoUrl && !remoteRepositorySource(repository.repoUrl))) { + throw new WorkspaceValidationFailure( + `Import the local folder for project "${project.name}" before attaching it to a remote task.`, + { reason: "remote_workspace_local_source_requires_import", projectId, projectWorkspaceId: repository.id }); + } + if (!repository.repoUrl) throw new WorkspaceValidationFailure( + `Project "${project.name}" has a configured folder without a repository URL.`, + { reason: "remote_workspace_source_unavailable", projectId, projectWorkspaceId: repository.id }); + const ref = repository.repoRef ?? repository.defaultRef ?? null; + extra.push({ id: repository.id, projectId, projectWorkspaceId: repository.id, + relativePath: `.paperclip-repositories/${managedProjectCheckoutName({ + id: repository.id, repoUrl: repository.repoUrl, repoRef: ref, + })}`, repoUrl: repository.repoUrl, ref }); + } + } } return { authority: "remote", source: { ...base, ...(extra.length ? { repositories: extra } : { repositories: undefined }) }, projectId: anchorProjectId, projectWorkspaceId: sourceId, @@ -156,9 +190,10 @@ export async function resolveRemoteWorkspaceSource(db: Db, input: { if (repository.cwd && repository.cwd !== "/__paperclip_repo_only__") return hostSource(); if (!remoteRepositorySource(repository.repoUrl)) return hostSource(); identities.add(projectRepositoryIdentity(repository.repoUrl)); + const ref = repository.repoRef ?? repository.defaultRef ?? null; repositories.push({ id: repository.id, relativePath: `.paperclip-repositories/${managedProjectCheckoutName({ - id: repository.id, repoUrl: repository.repoUrl, repoRef: repository.repoRef, - })}`, repoUrl: repository.repoUrl, ref: repository.repoRef, projectId: source.projectId, + id: repository.id, repoUrl: repository.repoUrl, repoRef: ref, + })}`, repoUrl: repository.repoUrl, ref, projectId: source.projectId, projectWorkspaceId: repository.id }); } return withReferences({ kind: "git", repoUrl: source.repoUrl, diff --git a/server/src/services/heartbeat/workspace-preparation.test.ts b/server/src/services/heartbeat/workspace-preparation.test.ts index ccfc02f56cf..971ec4f4d0b 100644 --- a/server/src/services/heartbeat/workspace-preparation.test.ts +++ b/server/src/services/heartbeat/workspace-preparation.test.ts @@ -666,7 +666,7 @@ describe.skipIf(!support.supported && !externalDatabaseUrl)("heartbeat workspace expect(await readFile(join(source, "README.md"), "utf8")).toBe("source content\n"); }); - it("rejects read-only project references before reserving or acquiring a remote workspace", async () => { + it("reserves editable task copies for referenced projects before acquiring the remote environment", async () => { const { input, source, issue, envOrchestrator } = await fixture(); const previous = new Map(["PAPERCLIP_MULTI_PROJECT_WORKSPACE_SYNC", "PAPERCLIP_MULTI_PROJECT_WORKSPACE_SYNC_REMOTE"] .map(key => [key, process.env[key]])); @@ -680,16 +680,23 @@ describe.skipIf(!support.supported && !externalDatabaseUrl)("heartbeat workspace await db.update(projectWorkspaces).set({ cwd: null, sourceType: "git", repoUrl: "https://github.com/example/remote.git" }) .where(eq(projectWorkspaces.id, issue.projectWorkspaceId!)); const [reference] = await db.insert(projects).values({ companyId: issue.companyId, name: "Referenced project" }).returning(); - await db.insert(projectWorkspaces).values({ companyId: issue.companyId, projectId: reference.id, - name: "Referenced repository", sourceType: "git", repoUrl: "https://github.com/example/reference.git" }); + const [repository] = await db.insert(projectWorkspaces).values({ companyId: issue.companyId, projectId: reference.id, + name: "Referenced repository", sourceType: "git_repo", repoUrl: "https://github.com/example/reference.git" }).returning(); await db.update(issues).set({ description: `[project](${buildProjectMentionHref(reference.id)})` }).where(eq(issues.id, issue.id)); - const acquire = vi.spyOn(envOrchestrator, "acquireForRun"); + const acquire = vi.spyOn(envOrchestrator, "acquireForRun").mockImplementation(async args => { + expect(args.persistedExecutionWorkspace).toMatchObject({ companyId: issue.companyId, sourceIssueId: issue.id, + metadata: { remoteWorkspace: { source: { kind: "git", repoUrl: "https://github.com/example/remote.git", repositories: [ + { id: repository.id, projectId: reference.id, projectWorkspaceId: repository.id, + repoUrl: "https://github.com/example/reference.git" }, + ] } } } }); + const [stored] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, args.persistedExecutionWorkspace!.id)); + expect(readRemoteWorkspaceAuthority(stored.metadata)?.source.repositories?.[0]?.readOnly).not.toBe(true); + throw new Error("editable project copies reached acquisition"); + }); try { - await expect(prepareHeartbeatWorkspace(db, input)).rejects.toMatchObject({ - code: "workspace_validation_failed", resultJson: { reason: "remote_workspace_readonly_sources_unsupported" }, - }); - expect(acquire).not.toHaveBeenCalled(); - expect(await db.select().from(executionWorkspaces)).toHaveLength(0); + await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("editable project copies reached acquisition"); + expect(acquire).toHaveBeenCalledOnce(); + expect(await db.select().from(executionWorkspaces)).toHaveLength(1); expect(await readdir(source)).not.toContain(".paperclip-repositories"); } finally { for (const [key, value] of previous) { diff --git a/server/src/services/heartbeat/workspaces.ts b/server/src/services/heartbeat/workspaces.ts index 9e8c39c9cab..467c8317338 100644 --- a/server/src/services/heartbeat/workspaces.ts +++ b/server/src/services/heartbeat/workspaces.ts @@ -1617,7 +1617,7 @@ export interface ReferencedProjectFailure { export interface ResolvedRunReferencedProjects { /** The anchor (primary) project — retains the existing git-worktree run path; never re-authorized here. */ anchor: RunReferencedProject | null; - /** Additional read-only referenced projects that each passed per-project `project:read` authorization. */ + /** Additional referenced projects that each passed per-project `project:read` authorization. */ additional: RunReferencedProject[]; /** Human-readable warnings for every referenced project that was dropped (unavailable, unauthorized, or capped). */ warnings: string[]; diff --git a/server/src/services/remote-workspace-transport.test.ts b/server/src/services/remote-workspace-transport.test.ts index a78612d2832..4b1d2f79987 100644 --- a/server/src/services/remote-workspace-transport.test.ts +++ b/server/src/services/remote-workspace-transport.test.ts @@ -440,6 +440,29 @@ describe("remote workspace transport safety with real sandbox commands", () => { expect(await fs.readFile(path.join(root, "unsaved.txt"), "utf8")).toBe("preserve local work\n"); }); + it.each(["root", "secondary"] as const)("checks out a non-default remote ref in the %s repository", async kind => { + const origin = await repository(), { remote } = target(), root = path.join(workspace, `release-${kind}`); + await git(origin, "checkout", "-b", "release"); + await fs.writeFile(path.join(origin, "release.txt"), "release branch\n"); + await git(origin, "add", "."); await git(origin, "commit", "-m", "Release"); + const release = (await git(origin, "rev-parse", "HEAD")).stdout.trim(); + await git(origin, "checkout", "main"); + if (kind === "root") await prepareRemoteWorkspaceDirectory({ target: remote, remoteCwd: root, + preparationIdentity: "b".repeat(64), source: { kind: "git", repoUrl: origin, ref: "release" }, resolveGitAuth: async () => null }); + else await prepareRemoteRepository({ target: remote, remoteCwd: root, repoUrl: origin, ref: "release", resolveGitAuth: async () => null }); + expect((await git(root, "rev-parse", "HEAD")).stdout.trim()).toBe(release); + expect(await fs.readFile(path.join(root, "release.txt"), "utf8")).toBe("release branch\n"); + }); + + it("does not publish a secondary checkout after its configured ref fails", async () => { + const origin = await repository(), { remote } = target(), root = path.join(workspace, "missing-secondary-ref"); + await expect(prepareRemoteRepository({ target: remote, remoteCwd: root, repoUrl: origin, + ref: "missing-ref", resolveGitAuth: async () => null })).rejects.toThrow("remote_workspace_operation_failed"); + await expect(fs.access(root)).rejects.toMatchObject({ code: "ENOENT" }); + await expect(prepareRemoteRepository({ target: remote, remoteCwd: root, repoUrl: origin, + ref: "HEAD", resolveGitAuth: async () => null })).resolves.toMatchObject({ cwd: root }); + }); + it.each(["path", "file-url"] as const)("verifies an archived secondary %s repository after its local origin disappears", async sourceKind => { const origin = await repository(), { remote } = target(); const repoUrl = sourceKind === "path" ? origin : pathToFileURL(origin).href; diff --git a/server/src/services/remote-workspace-transport.ts b/server/src/services/remote-workspace-transport.ts index ab4b17dbbd0..8355f2e4111 100644 --- a/server/src/services/remote-workspace-transport.ts +++ b/server/src/services/remote-workspace-transport.ts @@ -10,6 +10,11 @@ import type { GitRemoteAuthProvider } from "./git-credentials.js"; export type RemoteWorkspaceTarget = Extract; +const checkoutConfiguredRefScript = String.raw`if(input.ref){const remoteRef=input.ref.replace(/^refs\/heads\//,''); + const selected=await git(['rev-parse','--verify','--end-of-options',input.ref+'^{commit}']) + .catch(()=>git(['rev-parse','--verify','--end-of-options','refs/remotes/origin/'+remoteRef+'^{commit}'])); + await git(['checkout','--detach',selected.stdout.trim()]);}`; + export function requireRemoteWorkspaceTarget(target: AdapterExecutionTarget | null): RemoteWorkspaceTarget { if (target?.kind !== "remote" || target.transport !== "sandbox" || !target.runner || target.runnerTask || !target.runner.syncIn || !target.runner.syncOut) throw new Error("remote_workspace_transport_unsupported"); @@ -69,7 +74,7 @@ const {promisify}=require('node:util'); const exec=promisify(cp.execFile); const local=await git(['show-ref','--verify','--quiet','refs/heads/'+input.branchName]).then(()=>true,()=>false); await git(local?['checkout',input.branchName]:['checkout','-b',input.branchName,remoteRef]); }else{ - if(input.ref)await git(['checkout','--detach',input.ref]); + ${checkoutConfiguredRefScript} if(input.branchName)await git(['checkout','-B',input.branchName]); } await fs.mkdir(path.join(staged,'.git','info'),{recursive:true,mode:0o700}); @@ -129,8 +134,13 @@ export async function prepareRemoteRepository(input: { if(!entry.isDirectory()||entry.isSymbolicLink()||await fs.realpath(parent)!==parent)throw Error('unsafe_repository_parent');} const existing=await fs.lstat(root).catch(e=>{if(e.code==='ENOENT')return null;throw e}); if(existing){if(!existing.isDirectory()||existing.isSymbolicLink())throw Error('unsafe_repository')} - else{if(input.verifyOnly||input.savedArchive)throw Error('repository_missing');if(input.probeOnly){process.stdout.write(JSON.stringify({cloneRequired:true}));return;}await exec('git',['-c','core.hooksPath=/dev/null','-c','init.templateDir=',...input.authArgs,'clone','--no-hardlinks','--',input.repoUrl,root],{env:cloneEnv}); - if(input.ref)await exec('git',['-c','core.hooksPath=/dev/null','-C',root,'checkout','--detach',input.ref],opts);} + else{if(input.verifyOnly||input.savedArchive)throw Error('repository_missing');if(input.probeOnly){process.stdout.write(JSON.stringify({cloneRequired:true}));return;} + const staged=await fs.mkdtemp(root+'.preparing-');try{ + await exec('git',['-c','core.hooksPath=/dev/null','-c','init.templateDir=',...input.authArgs,'clone','--no-hardlinks','--',input.repoUrl,staged],{env:cloneEnv}); + const git=async args=>exec('git',['-c','core.hooksPath=/dev/null','-C',staged,...args],opts); + ${checkoutConfiguredRefScript} + await fs.rename(staged,root); + }finally{await fs.rm(staged,{recursive:true,force:true});}} const git=async args=>(await exec('git',['-c','core.hooksPath=/dev/null','-C',root,...args],opts)).stdout.trim(); if(await fs.realpath(root)!==root)throw Error('unsafe_repository'); const hasOrigin=(await git(['remote'])).split('\n').includes('origin'); diff --git a/server/src/services/remote-workspaces.integration.test.ts b/server/src/services/remote-workspaces.integration.test.ts index 8c8fbf671c6..077cb768037 100644 --- a/server/src/services/remote-workspaces.integration.test.ts +++ b/server/src/services/remote-workspaces.integration.test.ts @@ -1,5 +1,6 @@ import { randomUUID } from "node:crypto"; -import { execFile } from "node:child_process"; +import { execFile, spawn } from "node:child_process"; +import net from "node:net"; import { promisify } from "node:util"; import fs from "node:fs/promises"; import os from "node:os"; @@ -11,7 +12,7 @@ import { activityLog, agents, agentWakeupRequests, applyPendingMigrations, close executionWorkspaces, executionWorkspaceSnapshots, heartbeatRunEvents, heartbeatRuns, issueRecoveryActions, issues, livenessDb, nativeRunFinalizations, projects, projectWorkspaces, workspaceOperations } from "@paperclipai/db"; import type { AdapterExecutionTarget } from "@paperclipai/adapter-utils/execution-target"; import type { CommandManagedRuntimeRunner } from "@paperclipai/adapter-utils/command-managed-runtime"; -import type { Environment, EnvironmentLease } from "@paperclipai/shared"; +import { buildProjectMentionHref, type Environment, type EnvironmentLease } from "@paperclipai/shared"; import { createStorageService } from "../storage/service.js"; import { createLocalDiskStorageProvider } from "../storage/local-disk-provider.js"; import { createS3StorageProvider } from "../storage/s3-provider.js"; @@ -35,6 +36,8 @@ import { registerRemoteWorkspaceSnapshotRetentionWork, remoteWorkspaceSnapshotRe import { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js"; import { issueService } from "./issues.js"; import { instanceSettingsService } from "./instance-settings.js"; +import { resolveRemoteWorkspaceSource } from "./heartbeat/remote-workspace-source.js"; +import { toExecutionWorkspace } from "./execution-workspaces.js"; const exec = promisify(execFile); const databaseUrl = process.env.PAPERCLIP_REMOTE_WORKSPACE_TEST_DATABASE_URL; @@ -123,6 +126,113 @@ suite("remote workspace lifecycle with real Postgres and independent filesystem companyId: ids.company, issueId: ids.issue, runId: ids.run, agentId: ids.agent, environmentId: ids.environment, projectId: null, projectWorkspaceId: null, existingWorkspaceId: null, source }); + it.each(["initial", "follow-up"] as const)("edits and cold-restores both an anchor and a referenced project's task copy (%s attachment)", async attachment => { + vi.stubEnv("PAPERCLIP_MULTI_PROJECT_WORKSPACE_SYNC", "true"); + vi.stubEnv("PAPERCLIP_MULTI_PROJECT_WORKSPACE_SYNC_REMOTE", "true"); + const origins = path.join(root, "origins"); await fs.mkdir(origins); + const git = (cwd: string, ...args: string[]) => exec("git", ["-C", cwd, + "-c", "user.name=Task copies", "-c", "user.email=copies@example.invalid", ...args]); + for (const name of ["main", "skills"]) { + const directory = path.join(origins, name); await fs.mkdir(directory); + await git(directory, "init", "--initial-branch=main"); + await fs.writeFile(path.join(directory, "README.md"), `${name} source\n`); + await git(directory, "add", "."); await git(directory, "commit", "-m", "Source"); + if (name === "skills") { + await git(directory, "checkout", "-b", "release"); + await fs.writeFile(path.join(directory, "README.md"), "skills release\n"); + await git(directory, "add", "."); await git(directory, "commit", "-m", "Release"); + await git(directory, "checkout", "main"); + } + } + const port = await new Promise((resolve, reject) => { + const listener = net.createServer(); listener.on("error", reject); + listener.listen(0, "127.0.0.1", () => { + const port = (listener.address() as net.AddressInfo).port; + listener.close(error => error ? reject(error) : resolve(port)); + }); + }); + const daemon = spawn("git", ["daemon", "--reuseaddr", "--export-all", `--base-path=${origins}`, + "--listen=127.0.0.1", `--port=${port}`, "--verbose", origins], { stdio: ["ignore", "ignore", "pipe"] }); + try { + await new Promise((resolve, reject) => { + daemon.once("error", reject); daemon.once("exit", code => reject(new Error(`Git source exited: ${code}`))); + daemon.stderr.on("data", bytes => { if (String(bytes).includes("Ready to rumble")) resolve(); }); + }); + const [anchor, reference] = await db.insert(projects).values(["Main", "Skills"].map(name => ({ companyId: ids.company, name }))).returning(); + const [anchorSource] = await db.insert(projectWorkspaces).values([ + { companyId: ids.company, projectId: anchor.id, name: "Main", repoUrl: `git://127.0.0.1:${port}/main`, sourceType: "git_repo" }, + { companyId: ids.company, projectId: reference.id, name: "Skills", repoUrl: `git://127.0.0.1:${port}/skills`, defaultRef: "release", sourceType: "git_repo" }, + ]).returning(); + await db.update(issues).set({ projectId: anchor.id, projectWorkspaceId: anchorSource.id, + description: attachment === "initial" ? `Edit both projects, including [@Skills](${buildProjectMentionHref(reference.id)}).` + : "Start in the main project." }).where(eq(issues.id, ids.issue)); + const input = { companyId: ids.company, issueId: ids.issue, agentId: ids.agent, runId: ids.run, + responsibleUserId: null, projectId: anchor.id, projectWorkspaceId: anchorSource.id }; + const resolved = await resolveRemoteWorkspaceSource(db, input); + if (resolved.authority !== "remote") throw new Error("Expected remote task copies"); + const reserved = await reserveRemoteWorkspace({ db, storage, ...input, environmentId: ids.environment, + existingWorkspaceId: null, source: resolved.source }); + const first = await target("project-copies-A"); + const prepared = await prepareRemoteWorkspace({ db, storage, workspace: reserved, runId: ids.run, lease: await lease(), target: first }); + const initialCopy = resolved.source.repositories?.[0]; + if (attachment === "initial") { + expect(initialCopy?.projectId).toBe(reference.id); expect(initialCopy?.readOnly).not.toBe(true); + expect(await fs.readFile(path.join(prepared.remoteRoot, initialCopy!.relativePath, "README.md"), "utf8")).toBe("skills release\n"); + } + for (const relative of ["", ...(initialCopy ? [initialCopy.relativePath] : [])]) { + await fs.writeFile(path.join(prepared.remoteRoot, relative, "result.md"), "saved edits\n"); + } + for (const name of ["main", "skills"]) { + await expect(fs.access(path.join(origins, name, "result.md"))).rejects.toMatchObject({ code: "ENOENT" }); + } + await saveRemoteWorkspace({ db, storage, workspace: prepared.workspace, runId: ids.run, + lease: prepared.lease, target: first, assertOwned: async () => undefined }); + await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }).where(eq(heartbeatRuns.id, ids.run)); + await fs.rm(path.dirname(prepared.remoteRoot), { recursive: true }); + await fs.rm(path.join(origins, "main"), { recursive: true }); + if (attachment === "initial") await fs.rm(path.join(origins, "skills"), { recursive: true }); + else await db.update(issues).set({ description: `Now edit [@Skills](${buildProjectMentionHref(reference.id)}).` }).where(eq(issues.id, ids.issue)); + const nextRun = randomUUID(), nextLeaseId = randomUUID(); + await db.insert(heartbeatRuns).values({ id: nextRun, companyId: ids.company, agentId: ids.agent, + nativeIssueId: ids.issue, runtimeMode: "native", status: "running" }); + const [nextLease] = await db.insert(environmentLeases).values({ id: nextLeaseId, companyId: ids.company, + environmentId: ids.environment, heartbeatRunId: nextRun, issueId: ids.issue, + provider: "daytona", providerLeaseId: `allocation-${nextLeaseId}`, status: "active" }).returning(); + const resumed = await resolveRemoteWorkspaceSource(db, { ...input, runId: nextRun, + existingWorkspace: toExecutionWorkspace(await workspace(reserved.id)) }); + if (resumed.authority !== "remote") throw new Error("Expected retained remote task copies"); + if (attachment === "initial") expect(resumed.source).toEqual(resolved.source); + const copy = resumed.source.repositories![0]!; + expect(copy.projectId).toBe(reference.id); + const reservedAgain = await reserveRemoteWorkspace({ db, storage, ...input, runId: nextRun, + environmentId: ids.environment, existingWorkspaceId: reserved.id, source: resumed.source }); + const second = await target("project-copies-B"); + const restored = await prepareRemoteWorkspace({ db, storage, workspace: reservedAgain, runId: nextRun, + lease: nextLease as EnvironmentLease, target: second }); + for (const relative of ["", copy.relativePath]) { + if (!relative || attachment === "initial") expect(await fs.readFile(path.join(restored.remoteRoot, relative, "result.md"), "utf8")).toBe("saved edits\n"); + await fs.writeFile(path.join(restored.remoteRoot, relative, "follow-up.md"), "continued edits\n"); + } + expect(await fs.readFile(path.join(restored.remoteRoot, copy.relativePath, "README.md"), "utf8")).toBe("skills release\n"); + await saveRemoteWorkspace({ db, storage, workspace: restored.workspace, runId: nextRun, + lease: restored.lease, target: second, assertOwned: async () => undefined }); + const final = await workspace(reserved.id), authority = readRemoteWorkspaceAuthority(final.metadata)!; + expect(authority).toMatchObject({ dirty: false, writerRunId: null }); + expect(final.metadata?.remoteWorkspaceSnapshotRepositories).toEqual({ snapshotId: authority.snapshotId, repositories: resumed.source.repositories }); + await fs.rm(origins, { recursive: true, force: true }); + const saved = await createSavedWorkspaceLoader(db, storage)({ companyId: ids.company, workspaceId: reserved.id, snapshotId: authority.snapshotId! }); + try { + for (const relative of ["", copy.relativePath]) expect(await fs.readFile(path.join(saved.root, relative, "follow-up.md"), "utf8")).toBe("continued edits\n"); + expect(await fs.readFile(path.join(saved.root, copy.relativePath, "README.md"), "utf8")).toBe("skills release\n"); + } finally { await saved.cleanup(); } + } finally { + if (daemon.exitCode === null) { + const stopped = new Promise(resolve => daemon.once("exit", () => resolve())); + daemon.kill("SIGTERM"); await stopped; + } + } + }, 60_000); + it("prepares, saves and restores task files without a controller checkout or original allocation", async () => { const reserved = await reserve({ kind: "empty" }), first = await target("sandbox-A"); const prepared = await prepareRemoteWorkspace({ db, storage, workspace: reserved, runId: ids.run, lease: await lease(), target: first }); diff --git a/server/src/services/remote-workspaces.ts b/server/src/services/remote-workspaces.ts index 857da6c2c54..bb5849d2b55 100644 --- a/server/src/services/remote-workspaces.ts +++ b/server/src/services/remote-workspaces.ts @@ -217,7 +217,10 @@ export async function reserveRemoteWorkspace(input: { } if (authority.writerRunId === input.runId) return existing; const next: RemoteWorkspaceAuthority = { ...authority, source, epoch: authority.epoch + 1, writerRunId: input.runId }; - const [saved] = await tx.update(executionWorkspaces).set({ metadata: { ...existing.metadata, remoteWorkspace: next }, updatedAt: new Date() }) + const [saved] = await tx.update(executionWorkspaces).set({ metadata: { ...existing.metadata, remoteWorkspace: next, + ...(authority.snapshotId && existing.metadata?.remoteWorkspaceSnapshotRepositories === undefined + ? { remoteWorkspaceSnapshotRepositories: { snapshotId: authority.snapshotId, repositories: authority.source.repositories ?? [] } } : {}), + }, updatedAt: new Date() }) .where(eq(executionWorkspaces.id, existing.id)).returning(); if (issue.executionWorkspaceId !== existing.id) await tx.update(issues).set({ executionWorkspaceId: existing.id, executionWorkspacePreference: "reuse_existing", workspaceBindingRevision: issue.workspaceBindingRevision + 1, @@ -297,9 +300,17 @@ async function cleanupSavedExports(input: { db: Db; companyId: string; workspace } /** The caller retains execution admission and provider ownership; this owns file authority. */ -function isSavedWorkspaceRepository(authority: RemoteWorkspaceAuthority, +function isSavedWorkspaceRepository(workspace: Workspace, authority: RemoteWorkspaceAuthority, repository: NonNullable[number]) { - return authority.snapshotId !== null && (authority.source.repositories ?? []).some(saved => + if (authority.snapshotId === null) return false; + const receipt = workspace.metadata?.remoteWorkspaceSnapshotRepositories; + let repositories = authority.source.repositories ?? []; + if (receipt !== undefined) { + const saved = object(receipt); + if (saved.snapshotId !== authority.snapshotId || !Array.isArray(saved.repositories)) throw new Error("remote_workspace_snapshot_source_mismatch"); + repositories = remoteWorkspaceSourceSchema.parse({ kind: "empty", repositories: saved.repositories }).repositories ?? []; + } + return repositories.some(saved => saved.id === repository.id && saved.relativePath === repository.relativePath && saved.repoUrl === repository.repoUrl && (saved.ref ?? "HEAD") === (repository.ref ?? "HEAD") && saved.pinnedCommit === repository.pinnedCommit && (saved.readOnly ?? false) === (repository.readOnly ?? false)); @@ -353,7 +364,7 @@ export async function prepareRemoteWorkspace(input: { // descriptors still follow normal acquisition and origin verification. await prepareRemoteRepository({ target, remoteCwd: remoteWorkspaceRepositoryPath(remoteRoot, repository), repoUrl: repository.repoUrl, ref: repository.ref, pinnedCommit: repository.pinnedCommit, - savedArchive: isSavedWorkspaceRepository(initial, repository), + savedArchive: isSavedWorkspaceRepository(input.workspace, initial, repository), resolveGitAuth: await createRemoteWorkspaceGitAuth(input.db, input.workspace.companyId, input.runId) }); } const recoveredFiles: Array<{ operationId: string; sourceRunId: string; path: string }> = []; @@ -582,7 +593,9 @@ export async function saveRemoteWorkspace(input: { if (!committed) throw new Error("remote_workspace_snapshot_publication_changed"); await notifyRemoteWorkspaceSnapshots(tx, `workspace:${workspace.id}`); if (boundary === "final") await notifyTerminalWorkspaceWork(tx, `workspace:${workspace.id}`); - await tx.update(executionWorkspaces).set({ metadata: { ...workspace.metadata, remoteWorkspace: authority }, updatedAt: at }).where(eq(executionWorkspaces.id, workspace.id)); + await tx.update(executionWorkspaces).set({ metadata: { ...workspace.metadata, remoteWorkspace: authority, + remoteWorkspaceSnapshotRepositories: { snapshotId: saved.id, repositories: expected.source.repositories ?? [] }, + }, updatedAt: at }).where(eq(executionWorkspaces.id, workspace.id)); await tx.update(workspaceOperations).set({ status: "succeeded", finishedAt: at, updatedAt: at, metadata: sql`coalesce(${workspaceOperations.metadata}, '{}'::jsonb) || ${JSON.stringify({ snapshotId: saved.id, sha256: saved.sha256 })}::jsonb` }) .where(eq(workspaceOperations.id, operation.id)); @@ -621,7 +634,7 @@ export async function prepareRemoteWorkspaceRepository(input: { await input.assertOwned?.(); const target = requireRemoteWorkspaceTarget(input.target); const result = await prepareRemoteRepository({ target, remoteCwd: remoteWorkspaceRepositoryPath(authority.location.remoteCwd, input.repository), - ...input.repository, verifyOnly: input.verifyOnly, savedArchive: isSavedWorkspaceRepository(authority, input.repository), + ...input.repository, verifyOnly: input.verifyOnly, savedArchive: isSavedWorkspaceRepository(input.workspace, authority, input.repository), resolveGitAuth: input.resolveGitAuth ?? await createRemoteWorkspaceGitAuth(input.db, input.workspace.companyId, input.runId) }); await input.assertOwned?.(); await saveRemoteWorkspace({ ...input, target, boundary: `repository:${input.repository.id}:${result.pinnedCommit}`, diff --git a/server/src/services/task-interruption-presentation.test.ts b/server/src/services/task-interruption-presentation.test.ts index bf534b439d8..f9d67e6d7e9 100644 --- a/server/src/services/task-interruption-presentation.test.ts +++ b/server/src/services/task-interruption-presentation.test.ts @@ -7,6 +7,17 @@ const run = { id: "run", agentId: "agent", status: "failed", scheduledRetryAt: n resultJson: { projectAccessDenied: { projectId: "22222222-2222-4222-8222-222222222222" } } }; const agent = { status: "idle", pauseReason: null }; +it("preserves the actual workspace preparation error without changing retry authority", () => { + const failedRun = { ...run, errorCode: "workspace_validation_failed", + error: "The selected project has no repository source for a remote workspace.", resultJson: null }; + expect(projectTaskInterruption(issue, { run: failedRun, agent, paused: false, approval: false })) + .toMatchObject({ state: "failed", canRetry: true, workspaceFailure: { message: failedRun.error } }); + for (const status of ["done", "cancelled"]) expect(projectTaskInterruption({ ...issue, status }, + { run: failedRun, agent, paused: false, approval: false })).not.toHaveProperty("workspaceFailure"); + expect(projectTaskInterruption(issue, { run: { ...failedRun, errorCode: "unrelated_failure" }, agent, paused: false, approval: false })) + .not.toHaveProperty("workspaceFailure"); +}); + function operatorStoppedRun() { return { ...run, status: "cancelled", runtimeMode: "native" as const, companyId: issue.companyId, nativeIssueId: issue.id, errorCode: "cancelled", resultJson: { diff --git a/server/src/services/task-interruption-presentation.ts b/server/src/services/task-interruption-presentation.ts index eabc90cb86d..d84e5c5f25d 100644 --- a/server/src/services/task-interruption-presentation.ts +++ b/server/src/services/task-interruption-presentation.ts @@ -60,6 +60,8 @@ export function projectTaskInterruption(issue: ProjectionIssue, input: Projectio const providerFailure = run.status === "failed" || run.status === "scheduled_retry" ? readProviderFailure(run.resultJson?.providerFailure) : null; if (providerFailure) base.providerFailure = providerFailure; + if (["failed", "scheduled_retry"].includes(run.status) && run.errorCode === "workspace_validation_failed" + && typeof run.error === "string" && run.error.trim()) base.workspaceFailure = { message: run.error }; const userAction = !issue.assigneeAgentId ? "assign_agent" : input.approval || run.errorCode === "native_provider_approval_required" ? "approval" : isAiAuthenticationBlocked(run) ? "reconnect" : run.errorCode === "native_provider_model_rejected" ? "choose_model" @@ -124,7 +126,8 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i nativeIssueId: heartbeatRuns.nativeIssueId, runtimeMode: heartbeatRuns.runtimeMode, agentAdapter: agents.adapterType, error: sql`case when ${heartbeatRuns.errorCode} = 'setup_failed' - and ${heartbeatRuns.error} = 'Project is outside this low-trust boundary.' then ${heartbeatRuns.error} else null end`, + and ${heartbeatRuns.error} = 'Project is outside this low-trust boundary.' + or ${heartbeatRuns.errorCode} = 'workspace_validation_failed' then ${heartbeatRuns.error} else null end`, errorCode: heartbeatRuns.errorCode, livenessState: heartbeatRuns.livenessState, scheduledRetryAt: heartbeatRuns.scheduledRetryAt, resultJson: sql>`jsonb_build_object('configurationIncomplete', ${heartbeatRuns.resultJson}->'configurationIncomplete', 'projectAccessDenied', ${heartbeatRuns.resultJson}->'projectAccessDenied', @@ -218,25 +221,37 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i agentAdapter: run.agentAdapter ?? "unknown" }); } } - const disclosures = [...result.values()].flatMap(presentation => presentation?.providerFailure && presentation.observedRunId - ? [{ id: presentation.observedRunId, providerFailure: presentation.providerFailure, presentation }] : []); + const disclosures = [...result.values()].flatMap(presentation => (presentation?.providerFailure || presentation?.workspaceFailure) && presentation.observedRunId + ? [{ id: presentation.observedRunId, providerFailure: presentation.providerFailure, + workspaceFailure: presentation.workspaceFailure, presentation }] : []); if (disclosures.length) { try { const [safe, settings] = await Promise.all([ - createRunSecretRedactionRegistry(db).redactForRuns(companyId, disclosures.map(({ id, providerFailure }) => ({ id, providerFailure }))), + createRunSecretRedactionRegistry(db).redactForRuns(companyId, + disclosures.map(({ id, providerFailure, workspaceFailure }) => ({ id, providerFailure, workspaceFailure }))), instanceSettingsService(db).getGeneral(), ]); + const safeMessage = (value: unknown) => typeof value === "string" + ? [...redactCurrentUserText(value, { enabled: settings.censorUsernameInLogs })].slice(0, 4000).join("") : ""; disclosures.forEach(({ presentation }, index) => { - const value = safe[index]?.providerFailure; - const message = typeof value?.message === "string" - ? [...redactCurrentUserText(value.message, { enabled: settings.censorUsernameInLogs })].slice(0, 4000).join("") : ""; - const failure = readProviderFailure({ ...value, message }); - if (failure) presentation.providerFailure = failure; - else delete presentation.providerFailure; + if (presentation.providerFailure) { + const value = safe[index]?.providerFailure; + const failure = readProviderFailure({ ...value, message: safeMessage(value?.message) }); + if (failure) presentation.providerFailure = failure; + else delete presentation.providerFailure; + } + if (presentation.workspaceFailure) { + const message = safeMessage(safe[index]?.workspaceFailure?.message); + if (message.trim()) presentation.workspaceFailure = { message }; + else delete presentation.workspaceFailure; + } }); } catch { - for (const { presentation } of disclosures) delete presentation.providerFailure; - logger.warn({ companyId, runCount: disclosures.length }, "Could not redact provider failure display; explanations omitted"); + for (const { presentation } of disclosures) { + delete presentation.providerFailure; + delete presentation.workspaceFailure; + } + logger.warn({ companyId, runCount: disclosures.length }, "Could not redact run failure display; explanations omitted"); } } return result; diff --git a/ui/src/components/ExecutionBlockerNotice.test.tsx b/ui/src/components/ExecutionBlockerNotice.test.tsx index bfe07d25284..bf16c89ff5c 100644 --- a/ui/src/components/ExecutionBlockerNotice.test.tsx +++ b/ui/src/components/ExecutionBlockerNotice.test.tsx @@ -58,6 +58,18 @@ describe("ordinary task interruption", () => { expect(container.querySelector('[role="alert"]')?.textContent).toContain("Your task access changed."); expect(container.querySelectorAll("button")).toHaveLength(1); }); + it("shows workspace preparation failures as plain text while keeping retry permissions", async () => { + const workspaceFailure = { message: 'Could not prepare the project folder .' }; + await render({ ...failed, workspaceFailure }, { canRetry: false }); + expect(container.textContent).toContain("Couldn’t prepare the task folders."); + expect(container.textContent).toContain(workspaceFailure.message); + expect(container.querySelector("img, button")).toBeNull(); + await render({ ...failed, workspaceFailure }); + await act(async () => container.querySelector("button")!.click()); + expect(retry).toHaveBeenCalledOnce(); + await render({ ...failed, workspaceFailure, state: "working" }); + expect(container.textContent).toBe(""); + }); it("shows an unknown provider's original explanation as text and preserves read-only access", async () => { const providerFailure = { schema: "paperclip.provider-failure.v1" as const, provider: "new-provider", phase: "startup" as const, category: "unknown" as const, retryability: "unknown" as const, httpStatus: 429, diff --git a/ui/src/components/ExecutionBlockerNotice.tsx b/ui/src/components/ExecutionBlockerNotice.tsx index 721dfda3a0f..48b0ac8444b 100644 --- a/ui/src/components/ExecutionBlockerNotice.tsx +++ b/ui/src/components/ExecutionBlockerNotice.tsx @@ -43,7 +43,9 @@ export function ExecutionBlockerNotice({ presentation, canRetry, onRetry, optimi {presentation.state === "failed" &&