From f7ad1dc7cacedf7153a45e854ddfdd544aa87abd Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 3 Nov 2026 09:33:44 -0600 Subject: [PATCH] fix(workspaces): make attached project repositories writable (#18770) ## Thinking Path > - Paperclip lets people assign work to AI agents and inspect the result. > - A task can use repositories from more than one project. > - Remote workspace preparation treated every additional project as a read-only source. > - This rejected valid Git repositories before the provider started, even when no source was read-only. > - The task page then hid the preparation reason behind a generic failed-attempt notice. > - This pull request prepares authorized repositories as writable task copies and shows redacted workspace validation errors. > - People can work across both projects and see the reason if folder preparation fails. ## Linked Issues or Issue Description **What happened?** A remote task with a selected Git-backed project and another mentioned Git-backed project failed before provider execution. It reported `remote_workspace_readonly_sources_unsupported`. Neither project had a read-only setting. The task notice showed only that the attempt failed. **Expected behavior** Each attached repository should be writable inside the task workspace. This should preserve source access checks and saved work. A workspace validation failure should explain its cause on the task page. **Steps to reproduce** 1. Configure remote workspace storage and a native sandbox environment. 2. Select a Git-backed project for a task. 3. Mention another authorized Git-backed project in the task. 4. Start the task. Preparation rejects the additional source before it starts the provider. **Paperclip version or commit** Reproduced against `dc9f387edc`, before this change. **Deployment mode** Managed remote workspace storage. Local regression checks use disposable PostgreSQL and independent filesystem processes. Related merged work: Refs #18403 and #18504. Those changes established remote storage and metadata-only references. They still reject configured cross-project repositories. No duplicate writable-repository repair was found. Refs #18774 for the test-only retention-fixture repair reused to unblock CI. ## What Changed - Add every configured Git repository of an authorized referenced project to `.paperclip-repositories/` as an editable task copy. - Retain saved paths, refs, pins and files. Recheck retained project access and source trust. Keep explicit read-only, local-source import and remote feature gates. - Keep committed snapshot repository coverage separate from later requests. Honor configured default refs, resolve remote branch names, and publish secondary checkouts only after checkout succeeds. Share configured-ref resolution between root and attached-repository scripts. - Include redacted workspace validation messages in the task interruption projection and existing failure notice. Preserve Retry permissions and render messages as plain text. Explain retained-folder access and trust denials while preserving the dedicated low-trust project-access path. - Add source, authorization, redaction, UI and save/restore regressions. Update the implementation contract and workspace maps. - Reuse the reviewed test-only repair from #18774. Batch the shared-workspace history fixture, verify its complete privacy-source map, and drain owned test bodies before cleanup. Production retention behavior is unchanged. ## Verification - Red before repair: four source-resolution cases failed with the reported read-only error. The task-error projection test also failed because it omitted the cause. - Green: 49 source and transport tests, 16 task projection and redaction tests, and 10 UI tests. The assembled preparation check passes. All 48 remote workspace lifecycle tests pass. The full preparation suite passed all 126 assertions but its cleanup hook exceeded 30 seconds. The unchanged base reproduces the same cleanup timeout with all 126 assertions passing. The focused preparation check finishes cleanly. - The real PostgreSQL/Git lifecycle checks cover both first-turn and later project attachments. They verify the configured release branch, saved edits, replacement allocations, unchanged source folders, committed archive contents, and released writers. - Review regressions were red before repair: the default ref became null, later attachment failed during restore, non-default remote branch checkout failed, and invalid secondary refs left partial checkouts. They now pass. - The local browser shows the workspace validation explanation and Retry action. The fixture uses a saved failed run; this proves the error display, not a successful live sandbox run. - `pnpm -r typecheck` and `pnpm build` pass. Map, module-boundary and token checks pass. - A local two-turn acceptance probe repeated at `17ed67eaa7` used real Codex CLI 0.165.0, `gpt-6-astra`, and `xhigh` effort. Codex wrote files in both task repositories. The services saved them, removed the first allocation and its Git sources, restored another allocation, and Codex read the saved files and wrote a follow-up in both folders. This uses the real source resolver and persistence services with a local command-runner stand-in for sandbox transport. It does not qualify Daytona, managed subscription forwarding, or the original task. The first probe waited for stdin; closing the test harness pipe fixed that setup error before provider work. - `pnpm test:run` passed the first chat shard, then failed one chat response concurrency assertion in the second shard. That case passes in isolation on this head. The same isolated chat case passes on the unchanged base. Both corresponding unchanged-base chat shards also pass (351 and 324 tests). The cause of the earlier wider-suite chat failure is not established. The preparation cleanup timeout and chat failure remain part of this record. - CI at `17ed67eaa7`: all other verification jobs passed. Browser shard 6 initially reported a request callback from the preceding test after teardown; its single unchanged-code retry passed. Server shard 10 timed out the 2,100-row retention confirmation fixture at 15 seconds on both attempts. That exact case passes locally, but the repeated CI failure required a repair. Its setup repeatedly rebuilt a shared-workspace privacy map. The logs and failures remain part of this record. - Reused the test-only repair from #18774 without changing its source file. At `ab774e9418`, all 54 retention tests and all 25 ordinary production privacy tests pass locally (79 total). Full CI [41650626903](https://github.com/paperclipai/paperclip/actions/runs/41650626903) passed on that commit, and its exact-head Greptile review was 5/5. The fixture preserves its 2,100 rows, query-bound assertions, privacy-source map, and completed-history test deadlines. - The additional review identified retained-source access/trust denials that still bypassed the workspace explanation. Real-resolver regressions reproduced both plain-403 failures before the correction. Both now yield a safe workspace validation explanation, while the dedicated low-trust denial and saved workspace remain unchanged. The optional duplicate ref-resolution block is also consolidated. - Final review corrections: 65 source, transport and error-projection tests passed, followed by all 48 lifecycle cases against an owned disposable PostgreSQL database (113 total). The initial combined command skipped the explicitly gated lifecycle suite; the subsequent dedicated invocation enabled and passed all 48. Full workspace typecheck and build pass. The first standalone lifecycle launcher lacked the repository TypeScript loader; rerunning with that loader resolved its import setup error. At previous head `7be02ea09c4ac76df560e8b517050cdf398f33f0`, full CI [41654422329, attempt 2](https://github.com/paperclipai/paperclip/actions/runs/41654422329/attempts/2) passed, Greptile is 5/5 on that exact head, and all review threads are resolved. The first attempt timed out the unchanged voice-session report/history case at 15 seconds; 3,027 cases in its shard passed. That exact case passed locally, then its single unchanged-code shard retry passed. The wider-suite timeout cause remains unproven. The completed Paperclip advisory check is neutral; its earlier findings are corrected. No staging deployment or original-task recovery is claimed. - Master advanced while checks ran, producing conflicts in two adjacent test additions. Merged `27d35314be` into this branch as `d5c8d552939c86056abf8fe10833881b2ade41a4`, preserving both workspace-error tests and the audited Stop/receipt tests. Kept workspace maps to minimal corrections under the updated map rule. All 82 source, transport and interruption cases, all 48 real-database save/restore lifecycle cases, full typecheck, build, and map checks pass locally on the combined code. Current-head [CI 41660719214](https://github.com/paperclipai/paperclip/actions/runs/41660719214) and Greptile are pending; the earlier verified head has not been merged. ## Risks - Additional remote project repositories are now editable task copies. Shared source folders and upstream push permissions do not change. - Snapshot coverage uses existing workspace metadata. Older workspaces capture their previous inventory at reservation; successful save publication advances coverage and snapshot identity in one transaction. No SQL migration is needed. - A retained copy can contain files from a project that is no longer mentioned. Preparation rechecks that project's current access and trust policy before reuse. - Explicit read-only descriptors still fail. Source-less projects remain metadata-only. Local folders still require import. No migration or provider-pack update is needed. - Error display reads only workspace validation errors. The existing run-secret registry and username censor run before display truncation. A redaction failure omits the explanation. - The retained fixture repair pauses its row trigger only within the isolated seed transaction, rebuilds the exact source map, and restores the trigger before commit. A rollback regression and 25 ordinary privacy cases pass. Its paged-case test budget honors the existing bounded inner waits; production deadlines are unchanged. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use and code execution. The exact serving model ID and context-window size are not exposed by this session. The test-only fixture changes are reused from #18774, whose model record identifies OpenAI GPT-6.1 Sol. The optional live acceptance probe uses `gpt-6-astra` with `xhigh` effort through Codex CLI 0.165.0. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/DATABASE.md | 7 + doc/DEVELOPING.md | 16 +- doc/SPEC-implementation.md | 22 ++- maps/agent-access-map/workspaces.md | 7 +- maps/feature-map/workspaces.md | 5 +- .../shared/src/types/execution-projection.ts | 1 + .../attention-retention-work.test.ts | 45 +++++- .../__tests__/task-interruption-query.test.ts | 22 +++ .../heartbeat/remote-workspace-source.test.ts | 140 ++++++++++++++++-- .../heartbeat/remote-workspace-source.ts | 63 ++++++-- .../heartbeat/workspace-preparation.test.ts | 25 ++-- server/src/services/heartbeat/workspaces.ts | 2 +- .../remote-workspace-transport.test.ts | 23 +++ .../services/remote-workspace-transport.ts | 16 +- .../remote-workspaces.integration.test.ts | 114 +++++++++++++- server/src/services/remote-workspaces.ts | 25 +++- .../task-interruption-presentation.test.ts | 11 ++ .../task-interruption-presentation.ts | 39 +++-- .../ExecutionBlockerNotice.test.tsx | 12 ++ ui/src/components/ExecutionBlockerNotice.tsx | 4 +- 20 files changed, 523 insertions(+), 76 deletions(-) diff --git a/doc/DATABASE.md b/doc/DATABASE.md index 13b2aad680e..f69fb4eca8a 100644 --- a/doc/DATABASE.md +++ b/doc/DATABASE.md @@ -1023,6 +1023,13 @@ objects can be reclaimed. Object existence alone is not a committed save. remote allocation/path, writer run, and dirty flag. Paths in this record belong to the remote allocation. Save publication checks the current writer/epoch and atomically commits the ledger row, authority pointer, and operation receipt. +`execution_workspaces.metadata.remoteWorkspaceSnapshotRepositories` records the +committed snapshot ID and the repository descriptors covered by that archive. +Reservation retains this coverage when it adds requested repositories to the +current source. Successful save publication advances coverage and the snapshot +pointer together. Older saved workspaces capture their previous source inventory +at the next reservation. Newly attached repositories use normal acquisition; +they cannot claim coverage from an older archive. No schema migration is needed. `workspace_operations` journals preparation, save, restore, and local import; large file bodies stay outside the database. Ledger identities deliberately survive source-record deletion so object cleanup can finish. diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index fcd0683e9f3..d7608647f99 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -1775,8 +1775,15 @@ import. An interrupted apply retains its journal and admission hold; retry the same command to reconcile it before starting another writer. Supported sources include empty task folders, repository URLs, and managed -mutable repositories from the same project. Configured local folders use the -legacy path until imported. V1 rejects read-only cross-project referenced folders +mutable repositories. Authorized mentioned projects attach all their configured +Git repositories as writable copies under `.paperclip-repositories/` in the task +workspace. These copies use the existing remote save/restore lifecycle and do not +write to the source project's folder. Later turns retain their saved paths, refs, +pins and edits, and recheck each retained project's access and trust policy. +New repository attachments after a save are acquired separately from the saved +archive. Repository refs use `repoRef`, then `defaultRef`, then the Git remote's +default. A configured remote branch is resolved before its checkout is published. +Configured local folders use the legacy path until imported. V1 rejects explicitly read-only sources and custom cleanup/teardown commands on remote close. Automatic Core services, manual workspace jobs, services, and runtime commands are also rejected before any local directory access; run those commands through an agent task. One writer owns each logical @@ -1784,6 +1791,11 @@ workspace; separate workspaces can run concurrently. The Files panel identifies **Live workspace** and **Saved copy** with its save time. An unavailable dirty allocation is not silently replaced with an older saved copy. +Workspace validation failures show their redacted explanation in the task's +failure notice. They retain the normal Retry permission and admission gates. +Retained-folder access and trust denials use this explanation; low-trust project +boundary denials retain their dedicated project-access action. + A child or follow-up task can use a remote workspace inherited through its persisted task binding, including an isolated workspace or operator branch. Reservation checks current task/run access and all retained workspace sources. diff --git a/doc/SPEC-implementation.md b/doc/SPEC-implementation.md index f8a9ceaf7cb..c9455c9de14 100644 --- a/doc/SPEC-implementation.md +++ b/doc/SPEC-implementation.md @@ -2650,8 +2650,17 @@ have stopped. Cleanup retains copies whose upload completion is unknown. This includes operations from older builds without save-attempt records. A final save alone does not prove that an older upload has stopped. -V1 supports plain task folders, Git sources, and managed mutable repositories -from the same project. It rejects read-only cross-project referenced folders, +V1 supports plain task folders, Git sources, and managed mutable repositories. +An authorized mentioned project's configured Git repositories become writable +copies inside the task workspace, under `.paperclip-repositories/`. The copies +retain their saved files, paths, refs and pins on subsequent turns. Preparation +rechecks the access and trust policy of each retained source project. This does +not grant permission to change a shared source folder or push to a repository. +Repositories attached on a later turn are acquired before they become part of +the next committed archive. Snapshot coverage is separate from the requested +source recipe. New copies honor `repoRef`, then `defaultRef`, then the remote's +default branch; an invalid ref cannot publish a partial secondary checkout. +It rejects explicitly read-only sources, explicit concurrent shared writers, and custom cleanup/teardown commands on remote close. Manual workspace runtime commands, jobs, and services are also unsupported in V1 and must fail before local filesystem work. Agent tasks remain @@ -2661,11 +2670,16 @@ must fail admission rather than fall back to a controller checkout. A reference to an authorized project with no configured workspace source does not attach a folder. It adds a resolution warning and preserves the task's -workspace. A configured cross-project source remains subject to the read-only -folder restriction. `PAPERCLIP_MULTI_PROJECT_WORKSPACE_SYNC_REMOTE` gates configured +workspace. A configured cross-project local folder must be imported before remote +acquisition. `PAPERCLIP_MULTI_PROJECT_WORKSPACE_SYNC_REMOTE` gates configured sources; a source-less project reference needs no folder and stays metadata-only when that switch is off. +Task failure notices show the redacted workspace validation explanation when +preparation fails. Existing retry permissions and execution gates still apply. +This includes retained-folder access and trust denials. Low-trust project boundary +denials retain their separate project-access action. + An imported saved Git workspace can resume from its saved files after its old local repository is removed. Local repository paths remain invalid for new remote acquisition. Saves reject unfinished Git operations, including rebase, diff --git a/maps/agent-access-map/workspaces.md b/maps/agent-access-map/workspaces.md index 1de13f4f63f..10cc21d6026 100644 --- a/maps/agent-access-map/workspaces.md +++ b/maps/agent-access-map/workspaces.md @@ -71,9 +71,10 @@ until an explicit import. Stored `false` disables new enrollment unless managed configuration overrides it. Enrolled workspaces keep remote authority when new enrollment is disabled, and missing persistence prerequisites fail admission without a controller checkout fallback. Current authorization and responsible-user -checks still apply; -read-only cross-project referenced folders and explicitly concurrent shared -writers are rejected. Managed mutable same-project repositories are supported. +checks still apply. Authorized referenced Git repositories use writable task +copies without changing shared source folders or upstream push permissions. +Explicitly read-only sources and explicitly concurrent shared writers are rejected. +Managed mutable same-project repositories are supported. Portable capture streams validated working files while staging self-contained Git metadata. The smaller temporary-space requirement grants no new agent access or execution authority. Changed sources still fail before a save receipt. diff --git a/maps/feature-map/workspaces.md b/maps/feature-map/workspaces.md index 32f54e84ced..90923e81268 100644 --- a/maps/feature-map/workspaces.md +++ b/maps/feature-map/workspaces.md @@ -70,8 +70,9 @@ work. Finish the run and inspect its saved version. Replace the webserver and verify the next task turn uses the same remote folder without a controller checkout. Then release the saved allocation and verify a new one restores the files, Git state, and managed same-project repositories without the old sandbox -or original Git remote. Read-only cross-project referenced folders must fail -admission. Separate workspaces may run concurrently; shared concurrent writers +or original Git remote. Authorized referenced Git repositories must be editable +task copies; explicitly read-only sources must fail admission. Separate workspaces +may run concurrently; shared concurrent writers must be rejected. The Files panel must label **Live workspace** or **Saved copy** with its save diff --git a/packages/shared/src/types/execution-projection.ts b/packages/shared/src/types/execution-projection.ts index f3b9ef7d5c4..0dd0c3a8898 100644 --- a/packages/shared/src/types/execution-projection.ts +++ b/packages/shared/src/types/execution-projection.ts @@ -100,6 +100,7 @@ export interface ExecutionReconciliation { * and recovery records; ordinary product surfaces render this projection. */ export interface TaskInterruptionPresentation { providerFailure?: import("../provider-failure.generated.js").ProviderFailure; + workspaceFailure?: { message: string }; intentId: string | null; responsibility: "system" | "user"; state: "working" | "retry_scheduled" | "failed" | "waiting_for_user" | "resolved"; diff --git a/server/src/__tests__/attention-retention-work.test.ts b/server/src/__tests__/attention-retention-work.test.ts index 36e727dbc8a..1f840e18ac8 100644 --- a/server/src/__tests__/attention-retention-work.test.ts +++ b/server/src/__tests__/attention-retention-work.test.ts @@ -37,6 +37,7 @@ const delay = DEFAULT_DECISION_ARCHIVE_DAYS * 86_400_000; const errors: unknown[] = []; const statements: string[] = []; const parameterCounts: number[] = []; + const testBodies = new Set>(); let captureBlockerPlan = false; let blockerPlanQuery: { query: string; parameters: unknown[] } | undefined; beforeAll(async () => { @@ -64,6 +65,7 @@ const delay = DEFAULT_DECISION_ARCHIVE_DAYS * 86_400_000; vi.useFakeTimers({ now: Number(clock.now), toFake: ["Date", "setTimeout", "clearTimeout", "setInterval", "clearInterval"], shouldAdvanceTime: true }); }); afterEach(async () => { + await Promise.allSettled([...testBodies]); await coordinator?.stop(); coordinator = undefined; vi.useRealTimers(); vi.restoreAllMocks(); await db.transaction(async tx => { @@ -76,6 +78,24 @@ const delay = DEFAULT_DECISION_ARCHIVE_DAYS * 86_400_000; }); afterAll(async () => { await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); }); const old = () => new Date(Date.now() - delay - 600_000); + async function tracked(body: () => Promise) { + const running = body(); testBodies.add(running); + try { await running; } finally { testBodies.delete(running); } + } + async function insertSharedWorkspaceHistory(workspaceId: string, query: SQL) { + await db.transaction(async tx => { + await tx.execute(sql`alter table issues disable trigger issues_record_workspace_privacy_source`); + await tx.execute(query); + await tx.execute(sql`update execution_workspaces set metadata = metadata + where id = ${workspaceId}::uuid and company_id = ${companyId}::uuid`); + await tx.execute(sql`alter table issues enable trigger issues_record_workspace_privacy_source`); + const [provenance] = await tx.execute<{ complete: boolean }>(sql`select + w.metadata->'_issuePrivacySources' = (select jsonb_object_agg(i.id::text, true) from issues i + where i.execution_workspace_id = w.id and i.company_id = w.company_id) as complete + from execution_workspaces w where w.id = ${workspaceId}::uuid and w.company_id = ${companyId}::uuid`); + expect(provenance?.complete).toBe(true); + }); + } async function insertIssue(status = "todo", values: Partial = {}, database: Db = db) { const [row] = await database.insert(issues).values({ companyId, title: randomUUID(), status, createdByAgentId: agentId, createdAt: old(), updatedAt: old(), ...values }).returning(); @@ -115,6 +135,19 @@ const delay = DEFAULT_DECISION_ARCHIVE_DAYS * 86_400_000; authActor: { type: "board" as const, source: "local_implicit" as const, userId: "board", companyIds: [companyId], isInstanceAdmin: true }, actor: { actorType: "user" as const, actorId: "board", userId: "board", agentId: null, runId: null, agentApiKeyId: null, responsibleUserId: "board" } }); + it("restores ordinary workspace provenance recording after a failed bulk fixture", async () => { + const workspaceId = randomUUID(); + await db.execute(sql`insert into execution_workspaces (id, company_id, name, mode, strategy_type, metadata) + values (${workspaceId}::uuid, ${companyId}::uuid, 'Fixture rollback', 'shared', 'project_primary', '{"fixture":"retained"}'::jsonb)`); + await expect(insertSharedWorkspaceHistory(workspaceId, sql`select 1 / 0`)).rejects.toThrow(); + const [trigger] = await db.execute<{ tgenabled: string }>(sql`select tgenabled from pg_trigger + where tgrelid = 'issues'::regclass and tgname = 'issues_record_workspace_privacy_source'`); + expect(trigger.tgenabled).toBe("O"); + const issue = await insertIssue("done", { executionWorkspaceId: workspaceId }); + const [workspace] = await db.execute<{ metadata: unknown }>(sql`select metadata from execution_workspaces where id = ${workspaceId}::uuid`); + expect(workspace.metadata).toEqual({ fixture: "retained", _issuePrivacySources: { [issue.id]: true } }); + }); + it("records the aggregate baseline SQL across old ticks and eliminates it after settlement", async () => { await insertIssue("done"); const ticks: number[] = []; @@ -442,13 +475,13 @@ const delay = DEFAULT_DECISION_ARCHIVE_DAYS * 86_400_000; expect(await db.select().from(decisionArchiveNotificationOutbox)).toHaveLength(0); }); - it.each(["running", "run", "confirmation", "completed"] as const)("filters completed blocker history before bounded %s attention and archive checks", async kind => { + it.each(["running", "run", "confirmation", "completed"] as const)("filters completed blocker history before bounded %s attention and archive checks", kind => tracked(async () => { const root = await insertIssue("blocked"), workspaceId = randomUUID(); await db.execute(sql`insert into execution_workspaces (id, company_id, name, mode, strategy_type) values (${workspaceId}::uuid, ${companyId}::uuid, 'Finalized shared history', 'shared', 'project_primary')`); await db.execute(sql`insert into workspace_operations (company_id, execution_workspace_id, phase, status, started_at) values (${companyId}::uuid, ${workspaceId}::uuid, 'workspace_finalize', 'succeeded', ${old().toISOString()}::timestamptz)`); - await db.execute(sql`with history as ( + await insertSharedWorkspaceHistory(workspaceId, sql`with history as ( insert into issues (id, company_id, title, status, updated_at, execution_workspace_id) select ('00000000-0000-4000-8000-' || lpad(to_hex(n), 12, '0'))::uuid, ${companyId}::uuid, 'completed-blocker-' || n, 'done', ${old().toISOString()}::timestamptz, ${workspaceId}::uuid @@ -503,7 +536,7 @@ const delay = DEFAULT_DECISION_ARCHIVE_DAYS * 86_400_000; expect(batch.deferredRootIds.size).toBe(0); expect(Boolean(batch.attention.get(root.id)?.blockingTreeLive)).toBe(kind !== "completed"); expect(parameterCounts.length).toBeLessThanOrEqual(100); - }); + })); it("does not archive an unknown truncated blocker tree during its fresh check", async () => { const root = await insertIssue("blocked"); @@ -569,7 +602,7 @@ const delay = DEFAULT_DECISION_ARCHIVE_DAYS * 86_400_000; values (${workspaceId}::uuid, ${companyId}::uuid, 'Pending shared export', 'shared', 'project_primary')`); await workspaceOperationService(db).createRecorder({ companyId, executionWorkspaceId: workspaceId }) .recordOperation({ phase: "workspace_provision", run: async () => ({ status: "succeeded" }) }); - await db.execute(sql`with history as ( + await insertSharedWorkspaceHistory(workspaceId, sql`with history as ( insert into issues (company_id, title, status, updated_at, execution_workspace_id) select ${companyId}::uuid, 'waiting-for-export-' || n, 'done', ${old().toISOString()}::timestamptz, ${workspaceId}::uuid from generate_series(1, 2100) n returning id) @@ -1045,7 +1078,7 @@ const delay = DEFAULT_DECISION_ARCHIVE_DAYS * 86_400_000; expect(await state("agent_error_alert", agentId)).toBeNull(); }); - it("drains a paged backlog with source-scoped self notifications and leaves completed history quiet", async () => { + it("drains a paged backlog with source-scoped self notifications and leaves completed history quiet", () => tracked(async () => { await db.insert(issues).values(Array.from({ length: 225 }, () => ({ companyId, title: randomUUID(), status: "in_review", assigneeUserId: "board", updatedAt: old() }))); const service = attentionRetentionService(db); const load = vi.spyOn(service, "load"); @@ -1057,7 +1090,7 @@ const delay = DEFAULT_DECISION_ARCHIVE_DAYS * 86_400_000; expect(scopes.filter(scope => scope && !scope.sourceId)).toHaveLength(0); expect(scopes.filter(scope => scope?.sourceId).length).toBeLessThanOrEqual(225); expect(new Set((await db.select().from(decisionRetention)).map(row => row.sourceId)).size).toBe(225); - }); + }), 45_000); it("retries a failed atomic archive across drain and preserves notifications during a scan", async () => { const row = await insertIssue("in_review", { assigneeUserId: "board" }); diff --git a/server/src/__tests__/task-interruption-query.test.ts b/server/src/__tests__/task-interruption-query.test.ts index 3983ab50572..a179045cb39 100644 --- a/server/src/__tests__/task-interruption-query.test.ts +++ b/server/src/__tests__/task-interruption-query.test.ts @@ -24,6 +24,28 @@ describe.skipIf(!support.supported)("task interruption latest run query", () => }, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS); afterAll(async () => { await db?.$client.end(); await database?.cleanup(); }); + it("redacts workspace preparation errors on task detail and list reads", async () => { + vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "a".repeat(64)); + try { + const f = await seedInterruptionFixture(db, { selected: 1, history: 1, unrelated: 0, payloadBytes: 0 }); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId)); + const secret = `private-workspace-value-${randomUUID()}`; + const error = `Could not prepare the project folder: ${secret}.`; + await db.update(heartbeatRuns).set({ errorCode: "workspace_validation_failed", error }).where(eq(heartbeatRuns.id, run!.id)); + await createRunSecretRedactionRegistry(db).register(f.companyId, run!.id, secret); + const list = await taskInterruptionPresentations(db, f.companyId, f.input); + const detail = await taskInterruptionPresentation(db, f.input[0]!); + for (const presentation of [list.get(f.input[0]!.id), detail]) { + expect(presentation?.workspaceFailure?.message).toContain("Could not prepare the project folder"); + expect(presentation?.workspaceFailure?.message).toContain("REDACTED"); + expect(JSON.stringify(presentation)).not.toContain(secret); + } + vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "b".repeat(64)); + expect(await taskInterruptionPresentation(db, f.input[0]!)).not.toHaveProperty("workspaceFailure"); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run!.id)))[0]?.error).toBe(error); + } finally { vi.unstubAllEnvs(); } + }); + it.each(["valid", "numeric_user", "numeric_audit"] as const)("retains bounded typed Stop evidence in the real batched projection (%s)", async variant => { const companyId = randomUUID(), agentId = randomUUID(), runId = randomUUID(); await db.insert(companies).values({ id: companyId, name: "Stopped task", issuePrefix: `S${companyId.slice(0, 7)}` }); diff --git a/server/src/services/heartbeat/remote-workspace-source.test.ts b/server/src/services/heartbeat/remote-workspace-source.test.ts index a7747eda699..a096cf73a09 100644 --- a/server/src/services/heartbeat/remote-workspace-source.test.ts +++ b/server/src/services/heartbeat/remote-workspace-source.test.ts @@ -130,21 +130,58 @@ describe.skipIf(!support.supported)("remote referenced project sources", () => { } }); - it("still rejects an actual repository reference before any workspace is created", async () => { + it("attaches a referenced project's repositories as editable task copies", async () => { const f = await fixture(true); - await expect(resolveRemoteWorkspaceSource(db, { ...f, responsibleUserId: null, - projectId: null, projectWorkspaceId: null, taskDirectory: true })).rejects.toMatchObject({ - code: "workspace_validation_failed", resultJson: { reason: "remote_workspace_readonly_sources_unsupported" }, - }); + await db.insert(projectWorkspaces).values({ companyId: f.companyId, projectId: f.projectId, + name: "Second repository", sourceType: "git_repo", repoUrl: "https://example.test/team/second.git", repoRef: "release" }); + const result = await resolveRemoteWorkspaceSource(db, { ...f, responsibleUserId: null, + projectId: null, projectWorkspaceId: null, taskDirectory: true }); + expect(result).toMatchObject({ authority: "remote", source: { kind: "empty", repositories: [ + { projectId: f.projectId, repoUrl: "https://example.test/team/repository.git", ref: null }, + { projectId: f.projectId, repoUrl: "https://example.test/team/second.git", ref: "release" }, + ] }, referencedProjectFailures: [], warnings: [] }); + if (result.authority !== "remote") throw new Error("Expected remote task copies"); + expect(result.source.repositories).toHaveLength(2); + for (const repository of result.source.repositories!) { + expect(repository.readOnly).not.toBe(true); + expect(repository.relativePath).toMatch(/^\.paperclip-repositories\/[^/]+$/); + expect(repository.projectWorkspaceId).toBe(repository.id); + } + expect(new Set(result.source.repositories!.map(repository => repository.relativePath)).size).toBe(2); expect(await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.companyId, f.companyId))).toEqual([]); }); + it("keeps the selected anchor and adds the other project's writable repository", async () => { + const f = await assignedTask(true), referencedProjectId = randomUUID(); + await db.insert(projects).values({ id: referencedProjectId, companyId: f.companyId, name: "Skill sources" }); + const [repository] = await db.insert(projectWorkspaces).values({ companyId: f.companyId, projectId: referencedProjectId, + name: "Skills", sourceType: "git_repo", repoUrl: "https://example.test/team/skills.git" }).returning(); + await db.insert(issueComments).values({ companyId: f.companyId, issueId: f.issueId, + body: `Write a skill in [@Skill sources](${buildProjectMentionHref(referencedProjectId)}).` }); + const result = await resolveRemoteWorkspaceSource(db, { ...f, responsibleUserId: null, + projectId: f.projectId, projectWorkspaceId: null }); + expect(result).toMatchObject({ authority: "remote", projectId: f.projectId, + source: { kind: "git", repoUrl: "https://example.test/team/repository.git", repositories: [ + { id: repository.id, projectId: referencedProjectId, repoUrl: "https://example.test/team/skills.git" }, + ] } }); + expect(result.authority === "remote" && result.source.repositories).toHaveLength(1); + expect(result.authority === "remote" && result.source.repositories![0]!.readOnly).not.toBe(true); + }); + + it.each([null, "feature"])("uses a referenced repository's default ref unless overridden by %s", async repoRef => { + const f = await fixture(true); + await db.update(projectWorkspaces).set({ defaultRef: "release", repoRef }).where(eq(projectWorkspaces.projectId, f.projectId)); + const result = await resolveRemoteWorkspaceSource(db, { ...f, responsibleUserId: null, + projectId: null, projectWorkspaceId: null, taskDirectory: true }); + expect(result).toMatchObject({ authority: "remote", source: { repositories: [{ ref: repoRef ?? "release" }] } }); + }); + it("does not treat a configured source without a repository URL as metadata-only", async () => { const f = await fixture(false); await db.insert(projectWorkspaces).values({ companyId: f.companyId, projectId: f.projectId, name: "Configured source" }); await expect(resolveRemoteWorkspaceSource(db, { ...f, responsibleUserId: null, projectId: null, projectWorkspaceId: null, taskDirectory: true })).rejects.toMatchObject({ - resultJson: { reason: "remote_workspace_readonly_sources_unsupported" }, + resultJson: { reason: "remote_workspace_source_unavailable", projectId: f.projectId }, }); expect(await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.companyId, f.companyId))).toEqual([]); }); @@ -163,15 +200,17 @@ describe.skipIf(!support.supported)("remote referenced project sources", () => { } }); - it("keeps a real repository denied when a source-less reference is also mentioned", async () => { + it("attaches a repository when a source-less project is also mentioned", async () => { const f = await fixture(true), metadataProjectId = randomUUID(); await db.insert(projects).values({ id: metadataProjectId, companyId: f.companyId, name: "Metadata" }); await db.insert(issueComments).values({ companyId: f.companyId, issueId: f.issueId, body: `Also mention [@Metadata](${buildProjectMentionHref(metadataProjectId)}).` }); - await expect(resolveRemoteWorkspaceSource(db, { ...f, responsibleUserId: null, - projectId: null, projectWorkspaceId: null, taskDirectory: true })).rejects.toMatchObject({ - resultJson: { reason: "remote_workspace_readonly_sources_unsupported" }, - }); + const result = await resolveRemoteWorkspaceSource(db, { ...f, responsibleUserId: null, + projectId: null, projectWorkspaceId: null, taskDirectory: true }); + expect(result).toMatchObject({ authority: "remote", source: { repositories: [ + { projectId: f.projectId, repoUrl: "https://example.test/team/repository.git" }, + ] }, referencedProjectFailures: [{ projectId: metadataProjectId, reason: "resolution" }] }); + expect(result.authority === "remote" && result.source.repositories).toHaveLength(1); expect(await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.companyId, f.companyId))).toEqual([]); }); @@ -186,6 +225,31 @@ describe.skipIf(!support.supported)("remote referenced project sources", () => { expect(await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.companyId, f.companyId))).toEqual([]); }); + it("does not attach a project outside the run's low-trust boundary", async () => { + const f = await fixture(true); + const policy = { trustPreset: "low_trust_review", authorizationPolicy: { trustPreset: "low_trust_review", + trustBoundary: { mode: "low_trust_review", companyId: f.companyId, rootIssueId: f.issueId, projectIds: [] } } }; + await db.update(agents).set({ permissions: policy }).where(eq(agents.id, f.agentId)); + await db.update(issues).set({ executionPolicy: policy }).where(eq(issues.id, f.issueId)); + await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: f.issueId, executionPolicy: policy } }).where(eq(heartbeatRuns.id, f.runId)); + const result = await resolveRemoteWorkspaceSource(db, { ...f, responsibleUserId: null, + projectId: null, projectWorkspaceId: null, taskDirectory: true }); + expect(result).toMatchObject({ authority: "remote", source: { kind: "empty" }, + referencedProjectFailures: [{ projectId: f.projectId, reason: "authorization" }] }); + expect(result.authority === "remote" && result.source.repositories).toBeUndefined(); + }); + + it("does not apply a referenced project's restricted trust policy to an ordinary task", async () => { + const f = await fixture(true); + await db.update(projects).set({ executionWorkspacePolicy: { trustPreset: "low_trust_review", authorizationPolicy: { + trustPreset: "low_trust_review", trustBoundary: { mode: "low_trust_review", companyId: f.companyId, + rootIssueId: f.issueId, projectIds: [f.projectId] }, + } } }).where(eq(projects.id, f.projectId)); + await expect(resolveRemoteWorkspaceSource(db, { ...f, responsibleUserId: null, + projectId: null, projectWorkspaceId: null, taskDirectory: true })).rejects.toMatchObject({ status: 403 }); + expect(await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.companyId, f.companyId))).toEqual([]); + }); + it("preserves an enrolled workspace's saved source for a metadata-only project mention", async () => { const f = await fixture(false), snapshotId = randomUUID(); const source = { kind: "git" as const, repoUrl: "https://example.test/team/saved.git", ref: "saved-branch" }; @@ -205,4 +269,58 @@ describe.skipIf(!support.supported)("remote referenced project sources", () => { expect(await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.companyId, f.companyId))) .toEqual([workspace]); }); + + it.each(["saved", "read-only", "foreign-project", "restricted-trust", "low-trust-boundary"])("preserves %s repository authority on a later turn", async scenario => { + const f = await fixture(true); + const [repository] = await db.select().from(projectWorkspaces).where(eq(projectWorkspaces.projectId, f.projectId)); + const savedRepository = { id: repository.id, projectWorkspaceId: repository.id, projectId: f.projectId, + relativePath: ".paperclip-repositories/saved-skills", repoUrl: repository.repoUrl!, ref: "saved-branch", + pinnedCommit: "a".repeat(40), ...(scenario === "read-only" ? { readOnly: true } : {}) }; + const source = { kind: "empty" as const, repositories: [savedRepository] }; + const [workspace] = await db.insert(executionWorkspaces).values({ companyId: f.companyId, + sourceIssueId: f.issueId, mode: "isolated_workspace", strategyType: "project_primary", + name: "Saved task copies", providerType: "cloud_sandbox", cwd: "/workspaces/saved", + metadata: { remoteWorkspace: { schema: "paperclip.remote-workspace.v1", epoch: 2, snapshotId: randomUUID(), + source, location: null, writerRunId: null, dirty: false } }, + }).returning(); + await db.update(issues).set({ executionWorkspaceId: workspace.id }).where(eq(issues.id, f.issueId)); + if (scenario === "foreign-project") { + const companyId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Other company", issuePrefix: companyId.slice(0, 8) }); + await db.update(projects).set({ companyId }).where(eq(projects.id, f.projectId)); + } else { + await db.update(projectWorkspaces).set({ repoRef: "new-default" }).where(eq(projectWorkspaces.id, repository.id)); + await db.delete(issueComments).where(eq(issueComments.issueId, f.issueId)); + } + if (scenario === "restricted-trust") { + await db.update(projects).set({ executionWorkspacePolicy: { trustPreset: "low_trust_review", authorizationPolicy: { + trustPreset: "low_trust_review", trustBoundary: { mode: "low_trust_review", companyId: f.companyId, + rootIssueId: f.issueId, projectIds: [f.projectId] }, + } } }).where(eq(projects.id, f.projectId)); + } else if (scenario === "low-trust-boundary") { + const policy = { trustPreset: "low_trust_review", authorizationPolicy: { trustPreset: "low_trust_review", + trustBoundary: { mode: "low_trust_review", companyId: f.companyId, rootIssueId: f.issueId, projectIds: [] } } }; + await db.update(agents).set({ permissions: policy }).where(eq(agents.id, f.agentId)); + await db.update(issues).set({ executionPolicy: policy }).where(eq(issues.id, f.issueId)); + await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: f.issueId, executionPolicy: policy } }).where(eq(heartbeatRuns.id, f.runId)); + } + const resolve = () => resolveRemoteWorkspaceSource(db, { ...f, responsibleUserId: null, + projectId: null, projectWorkspaceId: null, existingWorkspace: toExecutionWorkspace(workspace) }); + if (scenario === "saved") { + const result = await resolve(); + expect(result).toMatchObject({ authority: "remote", source }); + expect(result.authority === "remote" && result.source.repositories).toHaveLength(1); + } else if (scenario === "read-only") { + await expect(resolve()).rejects.toMatchObject({ + resultJson: { reason: "remote_workspace_readonly_sources_unsupported" }, + }); + } else if (scenario === "low-trust-boundary") { + await expect(resolve()).rejects.toMatchObject({ status: 403, message: "Project is outside this low-trust boundary." }); + } else await expect(resolve()).rejects.toMatchObject({ + code: "workspace_validation_failed", + message: "A saved project folder is no longer accessible or its trust policy no longer permits this task.", + resultJson: { reason: "remote_workspace_source_access_denied", projectId: f.projectId }, + }); + expect((await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspace.id)))[0]).toEqual(workspace); + }); }); diff --git a/server/src/services/heartbeat/remote-workspace-source.ts b/server/src/services/heartbeat/remote-workspace-source.ts index 22aea2105e7..ae4232036bc 100644 --- a/server/src/services/heartbeat/remote-workspace-source.ts +++ b/server/src/services/heartbeat/remote-workspace-source.ts @@ -1,7 +1,10 @@ import { asc, and, eq, isNull, notExists } from "drizzle-orm"; import { projects, projectWorkspaces, type Db } from "@paperclipai/db"; import type { TaskWorkspaceIntent } from "@paperclipai/shared"; -import { assertTaskWorkspaceExecutionAccess } from "../task-workspace-access.js"; +import { HttpError } from "../../errors.js"; +import { ProjectAccessDeniedError } from "../project-access-denial.js"; +import { assertTaskWorkspaceExecutionAccess, assertTaskWorkspaceSourceProjectAccess } from "../task-workspace-access.js"; +import { assertSelectedWorkspaceTrust } from "../task-repository-acquisition.js"; import { managedProjectCheckoutName, projectRepositoryIdentity } from "../managed-project-checkout-name.js"; import { issueService } from "../issues.js"; import { projectService } from "../projects.js"; @@ -47,11 +50,12 @@ export async function resolveRemoteWorkspaceSource(db: Db, input: { branchName?: string | null; branchMode?: "create" | "existing"; }): Promise { - const selected = await assertTaskWorkspaceExecutionAccess(db, { + const actor = { type: "agent", source: "agent_jwt", companyId: input.companyId, agentId: input.agentId, runId: input.runId, onBehalfOfUserId: input.responsibleUserId === "local-board" ? null : input.responsibleUserId, - }, input.companyId, { issueId: input.issueId, + } as const; + const selected = await assertTaskWorkspaceExecutionAccess(db, actor, input.companyId, { issueId: input.issueId, ...(!input.selection && input.existingWorkspace ? { workspaceId: input.existingWorkspace.id } : {}), ...(input.selection ? { persistedSelection: input.selection.selection } : {}), }); @@ -62,8 +66,22 @@ export async function resolveRemoteWorkspaceSource(db: Db, input: { }; const withReferences = async (base: RemoteWorkspaceSource, anchorProjectId: string | null, sourceId: string | null, savedSource = false): Promise => { - const extra: NonNullable = - (base.repositories ?? []).filter(repository => !repository.readOnly); + if (base.repositories?.some(repository => repository.readOnly)) throw new WorkspaceValidationFailure( + "This remote runtime cannot safely provide read-only project folders.", + { reason: "remote_workspace_readonly_sources_unsupported" }); + const extra = [...(base.repositories ?? [])]; + for (const projectId of new Set(extra.flatMap(repository => repository.projectId ? [repository.projectId] : []))) { + try { + await assertTaskWorkspaceSourceProjectAccess(db, actor, input.companyId, projectId); + await assertSelectedWorkspaceTrust(db, { companyId: input.companyId, actor, issueId: input.issueId, + sourceProjectId: projectId }); + } catch (error) { + if (error instanceof ProjectAccessDeniedError || !(error instanceof HttpError) || error.status !== 403) throw error; + throw new WorkspaceValidationFailure( + "A saved project folder is no longer accessible or its trust policy no longer permits this task.", + { reason: "remote_workspace_source_access_denied", projectId }); + } + } // A verified saved archive owns these files after import. Its original Git // source remains an identity and need not be reachable from the new sandbox. if (!savedSource) { @@ -72,19 +90,35 @@ export async function resolveRemoteWorkspaceSource(db: Db, input: { } const referenced = isMultiProjectWorkspaceSyncEnabled() ? await resolveRunReferencedProjects(input.issueId, anchorProjectId, { companyId: input.companyId, - actor: { type: "agent", source: "agent_jwt", companyId: input.companyId, - agentId: input.agentId, runId: input.runId, - onBehalfOfUserId: input.responsibleUserId === "local-board" ? null : input.responsibleUserId }, + actor, issues: issueService(db), projects: projectService(db), access: authorizationService(db), }) : { additional: [], warnings: [], failures: [] }; const sourceLess = referenced.additional.filter(({ project }) => Array.isArray(project.workspaces) && project.workspaces.length === 0); if (referenced.additional.length > sourceLess.length) { if (!isMultiProjectWorkspaceSyncRemoteEnabled()) throw new WorkspaceValidationFailure( "Referenced remote workspace access is disabled.", { reason: "remote_referenced_workspace_disabled" }); - // Native sandbox providers currently run with external sandbox authority. - // They cannot enforce a separate read-only filesystem root for a project. - throw new WorkspaceValidationFailure("This remote runtime cannot safely provide read-only project folders.", - { reason: "remote_workspace_readonly_sources_unsupported" }); + for (const { projectId, project } of referenced.additional) { + if (!project.workspaces.length) continue; + await assertSelectedWorkspaceTrust(db, { companyId: input.companyId, actor, issueId: input.issueId, + sourceProjectId: projectId }); + for (const repository of project.workspaces) { + if (extra.some(saved => saved.id === repository.id)) continue; + if (repository.sourceType === "remote_managed" || (repository.cwd && repository.cwd !== "/__paperclip_repo_only__") + || (repository.repoUrl && !remoteRepositorySource(repository.repoUrl))) { + throw new WorkspaceValidationFailure( + `Import the local folder for project "${project.name}" before attaching it to a remote task.`, + { reason: "remote_workspace_local_source_requires_import", projectId, projectWorkspaceId: repository.id }); + } + if (!repository.repoUrl) throw new WorkspaceValidationFailure( + `Project "${project.name}" has a configured folder without a repository URL.`, + { reason: "remote_workspace_source_unavailable", projectId, projectWorkspaceId: repository.id }); + const ref = repository.repoRef ?? repository.defaultRef ?? null; + extra.push({ id: repository.id, projectId, projectWorkspaceId: repository.id, + relativePath: `.paperclip-repositories/${managedProjectCheckoutName({ + id: repository.id, repoUrl: repository.repoUrl, repoRef: ref, + })}`, repoUrl: repository.repoUrl, ref }); + } + } } return { authority: "remote", source: { ...base, ...(extra.length ? { repositories: extra } : { repositories: undefined }) }, projectId: anchorProjectId, projectWorkspaceId: sourceId, @@ -156,9 +190,10 @@ export async function resolveRemoteWorkspaceSource(db: Db, input: { if (repository.cwd && repository.cwd !== "/__paperclip_repo_only__") return hostSource(); if (!remoteRepositorySource(repository.repoUrl)) return hostSource(); identities.add(projectRepositoryIdentity(repository.repoUrl)); + const ref = repository.repoRef ?? repository.defaultRef ?? null; repositories.push({ id: repository.id, relativePath: `.paperclip-repositories/${managedProjectCheckoutName({ - id: repository.id, repoUrl: repository.repoUrl, repoRef: repository.repoRef, - })}`, repoUrl: repository.repoUrl, ref: repository.repoRef, projectId: source.projectId, + id: repository.id, repoUrl: repository.repoUrl, repoRef: ref, + })}`, repoUrl: repository.repoUrl, ref, projectId: source.projectId, projectWorkspaceId: repository.id }); } return withReferences({ kind: "git", repoUrl: source.repoUrl, diff --git a/server/src/services/heartbeat/workspace-preparation.test.ts b/server/src/services/heartbeat/workspace-preparation.test.ts index ccfc02f56cf..971ec4f4d0b 100644 --- a/server/src/services/heartbeat/workspace-preparation.test.ts +++ b/server/src/services/heartbeat/workspace-preparation.test.ts @@ -666,7 +666,7 @@ describe.skipIf(!support.supported && !externalDatabaseUrl)("heartbeat workspace expect(await readFile(join(source, "README.md"), "utf8")).toBe("source content\n"); }); - it("rejects read-only project references before reserving or acquiring a remote workspace", async () => { + it("reserves editable task copies for referenced projects before acquiring the remote environment", async () => { const { input, source, issue, envOrchestrator } = await fixture(); const previous = new Map(["PAPERCLIP_MULTI_PROJECT_WORKSPACE_SYNC", "PAPERCLIP_MULTI_PROJECT_WORKSPACE_SYNC_REMOTE"] .map(key => [key, process.env[key]])); @@ -680,16 +680,23 @@ describe.skipIf(!support.supported && !externalDatabaseUrl)("heartbeat workspace await db.update(projectWorkspaces).set({ cwd: null, sourceType: "git", repoUrl: "https://github.com/example/remote.git" }) .where(eq(projectWorkspaces.id, issue.projectWorkspaceId!)); const [reference] = await db.insert(projects).values({ companyId: issue.companyId, name: "Referenced project" }).returning(); - await db.insert(projectWorkspaces).values({ companyId: issue.companyId, projectId: reference.id, - name: "Referenced repository", sourceType: "git", repoUrl: "https://github.com/example/reference.git" }); + const [repository] = await db.insert(projectWorkspaces).values({ companyId: issue.companyId, projectId: reference.id, + name: "Referenced repository", sourceType: "git_repo", repoUrl: "https://github.com/example/reference.git" }).returning(); await db.update(issues).set({ description: `[project](${buildProjectMentionHref(reference.id)})` }).where(eq(issues.id, issue.id)); - const acquire = vi.spyOn(envOrchestrator, "acquireForRun"); + const acquire = vi.spyOn(envOrchestrator, "acquireForRun").mockImplementation(async args => { + expect(args.persistedExecutionWorkspace).toMatchObject({ companyId: issue.companyId, sourceIssueId: issue.id, + metadata: { remoteWorkspace: { source: { kind: "git", repoUrl: "https://github.com/example/remote.git", repositories: [ + { id: repository.id, projectId: reference.id, projectWorkspaceId: repository.id, + repoUrl: "https://github.com/example/reference.git" }, + ] } } } }); + const [stored] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, args.persistedExecutionWorkspace!.id)); + expect(readRemoteWorkspaceAuthority(stored.metadata)?.source.repositories?.[0]?.readOnly).not.toBe(true); + throw new Error("editable project copies reached acquisition"); + }); try { - await expect(prepareHeartbeatWorkspace(db, input)).rejects.toMatchObject({ - code: "workspace_validation_failed", resultJson: { reason: "remote_workspace_readonly_sources_unsupported" }, - }); - expect(acquire).not.toHaveBeenCalled(); - expect(await db.select().from(executionWorkspaces)).toHaveLength(0); + await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow("editable project copies reached acquisition"); + expect(acquire).toHaveBeenCalledOnce(); + expect(await db.select().from(executionWorkspaces)).toHaveLength(1); expect(await readdir(source)).not.toContain(".paperclip-repositories"); } finally { for (const [key, value] of previous) { diff --git a/server/src/services/heartbeat/workspaces.ts b/server/src/services/heartbeat/workspaces.ts index 9e8c39c9cab..467c8317338 100644 --- a/server/src/services/heartbeat/workspaces.ts +++ b/server/src/services/heartbeat/workspaces.ts @@ -1617,7 +1617,7 @@ export interface ReferencedProjectFailure { export interface ResolvedRunReferencedProjects { /** The anchor (primary) project — retains the existing git-worktree run path; never re-authorized here. */ anchor: RunReferencedProject | null; - /** Additional read-only referenced projects that each passed per-project `project:read` authorization. */ + /** Additional referenced projects that each passed per-project `project:read` authorization. */ additional: RunReferencedProject[]; /** Human-readable warnings for every referenced project that was dropped (unavailable, unauthorized, or capped). */ warnings: string[]; diff --git a/server/src/services/remote-workspace-transport.test.ts b/server/src/services/remote-workspace-transport.test.ts index a78612d2832..4b1d2f79987 100644 --- a/server/src/services/remote-workspace-transport.test.ts +++ b/server/src/services/remote-workspace-transport.test.ts @@ -440,6 +440,29 @@ describe("remote workspace transport safety with real sandbox commands", () => { expect(await fs.readFile(path.join(root, "unsaved.txt"), "utf8")).toBe("preserve local work\n"); }); + it.each(["root", "secondary"] as const)("checks out a non-default remote ref in the %s repository", async kind => { + const origin = await repository(), { remote } = target(), root = path.join(workspace, `release-${kind}`); + await git(origin, "checkout", "-b", "release"); + await fs.writeFile(path.join(origin, "release.txt"), "release branch\n"); + await git(origin, "add", "."); await git(origin, "commit", "-m", "Release"); + const release = (await git(origin, "rev-parse", "HEAD")).stdout.trim(); + await git(origin, "checkout", "main"); + if (kind === "root") await prepareRemoteWorkspaceDirectory({ target: remote, remoteCwd: root, + preparationIdentity: "b".repeat(64), source: { kind: "git", repoUrl: origin, ref: "release" }, resolveGitAuth: async () => null }); + else await prepareRemoteRepository({ target: remote, remoteCwd: root, repoUrl: origin, ref: "release", resolveGitAuth: async () => null }); + expect((await git(root, "rev-parse", "HEAD")).stdout.trim()).toBe(release); + expect(await fs.readFile(path.join(root, "release.txt"), "utf8")).toBe("release branch\n"); + }); + + it("does not publish a secondary checkout after its configured ref fails", async () => { + const origin = await repository(), { remote } = target(), root = path.join(workspace, "missing-secondary-ref"); + await expect(prepareRemoteRepository({ target: remote, remoteCwd: root, repoUrl: origin, + ref: "missing-ref", resolveGitAuth: async () => null })).rejects.toThrow("remote_workspace_operation_failed"); + await expect(fs.access(root)).rejects.toMatchObject({ code: "ENOENT" }); + await expect(prepareRemoteRepository({ target: remote, remoteCwd: root, repoUrl: origin, + ref: "HEAD", resolveGitAuth: async () => null })).resolves.toMatchObject({ cwd: root }); + }); + it.each(["path", "file-url"] as const)("verifies an archived secondary %s repository after its local origin disappears", async sourceKind => { const origin = await repository(), { remote } = target(); const repoUrl = sourceKind === "path" ? origin : pathToFileURL(origin).href; diff --git a/server/src/services/remote-workspace-transport.ts b/server/src/services/remote-workspace-transport.ts index ab4b17dbbd0..8355f2e4111 100644 --- a/server/src/services/remote-workspace-transport.ts +++ b/server/src/services/remote-workspace-transport.ts @@ -10,6 +10,11 @@ import type { GitRemoteAuthProvider } from "./git-credentials.js"; export type RemoteWorkspaceTarget = Extract; +const checkoutConfiguredRefScript = String.raw`if(input.ref){const remoteRef=input.ref.replace(/^refs\/heads\//,''); + const selected=await git(['rev-parse','--verify','--end-of-options',input.ref+'^{commit}']) + .catch(()=>git(['rev-parse','--verify','--end-of-options','refs/remotes/origin/'+remoteRef+'^{commit}'])); + await git(['checkout','--detach',selected.stdout.trim()]);}`; + export function requireRemoteWorkspaceTarget(target: AdapterExecutionTarget | null): RemoteWorkspaceTarget { if (target?.kind !== "remote" || target.transport !== "sandbox" || !target.runner || target.runnerTask || !target.runner.syncIn || !target.runner.syncOut) throw new Error("remote_workspace_transport_unsupported"); @@ -69,7 +74,7 @@ const {promisify}=require('node:util'); const exec=promisify(cp.execFile); const local=await git(['show-ref','--verify','--quiet','refs/heads/'+input.branchName]).then(()=>true,()=>false); await git(local?['checkout',input.branchName]:['checkout','-b',input.branchName,remoteRef]); }else{ - if(input.ref)await git(['checkout','--detach',input.ref]); + ${checkoutConfiguredRefScript} if(input.branchName)await git(['checkout','-B',input.branchName]); } await fs.mkdir(path.join(staged,'.git','info'),{recursive:true,mode:0o700}); @@ -129,8 +134,13 @@ export async function prepareRemoteRepository(input: { if(!entry.isDirectory()||entry.isSymbolicLink()||await fs.realpath(parent)!==parent)throw Error('unsafe_repository_parent');} const existing=await fs.lstat(root).catch(e=>{if(e.code==='ENOENT')return null;throw e}); if(existing){if(!existing.isDirectory()||existing.isSymbolicLink())throw Error('unsafe_repository')} - else{if(input.verifyOnly||input.savedArchive)throw Error('repository_missing');if(input.probeOnly){process.stdout.write(JSON.stringify({cloneRequired:true}));return;}await exec('git',['-c','core.hooksPath=/dev/null','-c','init.templateDir=',...input.authArgs,'clone','--no-hardlinks','--',input.repoUrl,root],{env:cloneEnv}); - if(input.ref)await exec('git',['-c','core.hooksPath=/dev/null','-C',root,'checkout','--detach',input.ref],opts);} + else{if(input.verifyOnly||input.savedArchive)throw Error('repository_missing');if(input.probeOnly){process.stdout.write(JSON.stringify({cloneRequired:true}));return;} + const staged=await fs.mkdtemp(root+'.preparing-');try{ + await exec('git',['-c','core.hooksPath=/dev/null','-c','init.templateDir=',...input.authArgs,'clone','--no-hardlinks','--',input.repoUrl,staged],{env:cloneEnv}); + const git=async args=>exec('git',['-c','core.hooksPath=/dev/null','-C',staged,...args],opts); + ${checkoutConfiguredRefScript} + await fs.rename(staged,root); + }finally{await fs.rm(staged,{recursive:true,force:true});}} const git=async args=>(await exec('git',['-c','core.hooksPath=/dev/null','-C',root,...args],opts)).stdout.trim(); if(await fs.realpath(root)!==root)throw Error('unsafe_repository'); const hasOrigin=(await git(['remote'])).split('\n').includes('origin'); diff --git a/server/src/services/remote-workspaces.integration.test.ts b/server/src/services/remote-workspaces.integration.test.ts index 8c8fbf671c6..077cb768037 100644 --- a/server/src/services/remote-workspaces.integration.test.ts +++ b/server/src/services/remote-workspaces.integration.test.ts @@ -1,5 +1,6 @@ import { randomUUID } from "node:crypto"; -import { execFile } from "node:child_process"; +import { execFile, spawn } from "node:child_process"; +import net from "node:net"; import { promisify } from "node:util"; import fs from "node:fs/promises"; import os from "node:os"; @@ -11,7 +12,7 @@ import { activityLog, agents, agentWakeupRequests, applyPendingMigrations, close executionWorkspaces, executionWorkspaceSnapshots, heartbeatRunEvents, heartbeatRuns, issueRecoveryActions, issues, livenessDb, nativeRunFinalizations, projects, projectWorkspaces, workspaceOperations } from "@paperclipai/db"; import type { AdapterExecutionTarget } from "@paperclipai/adapter-utils/execution-target"; import type { CommandManagedRuntimeRunner } from "@paperclipai/adapter-utils/command-managed-runtime"; -import type { Environment, EnvironmentLease } from "@paperclipai/shared"; +import { buildProjectMentionHref, type Environment, type EnvironmentLease } from "@paperclipai/shared"; import { createStorageService } from "../storage/service.js"; import { createLocalDiskStorageProvider } from "../storage/local-disk-provider.js"; import { createS3StorageProvider } from "../storage/s3-provider.js"; @@ -35,6 +36,8 @@ import { registerRemoteWorkspaceSnapshotRetentionWork, remoteWorkspaceSnapshotRe import { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js"; import { issueService } from "./issues.js"; import { instanceSettingsService } from "./instance-settings.js"; +import { resolveRemoteWorkspaceSource } from "./heartbeat/remote-workspace-source.js"; +import { toExecutionWorkspace } from "./execution-workspaces.js"; const exec = promisify(execFile); const databaseUrl = process.env.PAPERCLIP_REMOTE_WORKSPACE_TEST_DATABASE_URL; @@ -123,6 +126,113 @@ suite("remote workspace lifecycle with real Postgres and independent filesystem companyId: ids.company, issueId: ids.issue, runId: ids.run, agentId: ids.agent, environmentId: ids.environment, projectId: null, projectWorkspaceId: null, existingWorkspaceId: null, source }); + it.each(["initial", "follow-up"] as const)("edits and cold-restores both an anchor and a referenced project's task copy (%s attachment)", async attachment => { + vi.stubEnv("PAPERCLIP_MULTI_PROJECT_WORKSPACE_SYNC", "true"); + vi.stubEnv("PAPERCLIP_MULTI_PROJECT_WORKSPACE_SYNC_REMOTE", "true"); + const origins = path.join(root, "origins"); await fs.mkdir(origins); + const git = (cwd: string, ...args: string[]) => exec("git", ["-C", cwd, + "-c", "user.name=Task copies", "-c", "user.email=copies@example.invalid", ...args]); + for (const name of ["main", "skills"]) { + const directory = path.join(origins, name); await fs.mkdir(directory); + await git(directory, "init", "--initial-branch=main"); + await fs.writeFile(path.join(directory, "README.md"), `${name} source\n`); + await git(directory, "add", "."); await git(directory, "commit", "-m", "Source"); + if (name === "skills") { + await git(directory, "checkout", "-b", "release"); + await fs.writeFile(path.join(directory, "README.md"), "skills release\n"); + await git(directory, "add", "."); await git(directory, "commit", "-m", "Release"); + await git(directory, "checkout", "main"); + } + } + const port = await new Promise((resolve, reject) => { + const listener = net.createServer(); listener.on("error", reject); + listener.listen(0, "127.0.0.1", () => { + const port = (listener.address() as net.AddressInfo).port; + listener.close(error => error ? reject(error) : resolve(port)); + }); + }); + const daemon = spawn("git", ["daemon", "--reuseaddr", "--export-all", `--base-path=${origins}`, + "--listen=127.0.0.1", `--port=${port}`, "--verbose", origins], { stdio: ["ignore", "ignore", "pipe"] }); + try { + await new Promise((resolve, reject) => { + daemon.once("error", reject); daemon.once("exit", code => reject(new Error(`Git source exited: ${code}`))); + daemon.stderr.on("data", bytes => { if (String(bytes).includes("Ready to rumble")) resolve(); }); + }); + const [anchor, reference] = await db.insert(projects).values(["Main", "Skills"].map(name => ({ companyId: ids.company, name }))).returning(); + const [anchorSource] = await db.insert(projectWorkspaces).values([ + { companyId: ids.company, projectId: anchor.id, name: "Main", repoUrl: `git://127.0.0.1:${port}/main`, sourceType: "git_repo" }, + { companyId: ids.company, projectId: reference.id, name: "Skills", repoUrl: `git://127.0.0.1:${port}/skills`, defaultRef: "release", sourceType: "git_repo" }, + ]).returning(); + await db.update(issues).set({ projectId: anchor.id, projectWorkspaceId: anchorSource.id, + description: attachment === "initial" ? `Edit both projects, including [@Skills](${buildProjectMentionHref(reference.id)}).` + : "Start in the main project." }).where(eq(issues.id, ids.issue)); + const input = { companyId: ids.company, issueId: ids.issue, agentId: ids.agent, runId: ids.run, + responsibleUserId: null, projectId: anchor.id, projectWorkspaceId: anchorSource.id }; + const resolved = await resolveRemoteWorkspaceSource(db, input); + if (resolved.authority !== "remote") throw new Error("Expected remote task copies"); + const reserved = await reserveRemoteWorkspace({ db, storage, ...input, environmentId: ids.environment, + existingWorkspaceId: null, source: resolved.source }); + const first = await target("project-copies-A"); + const prepared = await prepareRemoteWorkspace({ db, storage, workspace: reserved, runId: ids.run, lease: await lease(), target: first }); + const initialCopy = resolved.source.repositories?.[0]; + if (attachment === "initial") { + expect(initialCopy?.projectId).toBe(reference.id); expect(initialCopy?.readOnly).not.toBe(true); + expect(await fs.readFile(path.join(prepared.remoteRoot, initialCopy!.relativePath, "README.md"), "utf8")).toBe("skills release\n"); + } + for (const relative of ["", ...(initialCopy ? [initialCopy.relativePath] : [])]) { + await fs.writeFile(path.join(prepared.remoteRoot, relative, "result.md"), "saved edits\n"); + } + for (const name of ["main", "skills"]) { + await expect(fs.access(path.join(origins, name, "result.md"))).rejects.toMatchObject({ code: "ENOENT" }); + } + await saveRemoteWorkspace({ db, storage, workspace: prepared.workspace, runId: ids.run, + lease: prepared.lease, target: first, assertOwned: async () => undefined }); + await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }).where(eq(heartbeatRuns.id, ids.run)); + await fs.rm(path.dirname(prepared.remoteRoot), { recursive: true }); + await fs.rm(path.join(origins, "main"), { recursive: true }); + if (attachment === "initial") await fs.rm(path.join(origins, "skills"), { recursive: true }); + else await db.update(issues).set({ description: `Now edit [@Skills](${buildProjectMentionHref(reference.id)}).` }).where(eq(issues.id, ids.issue)); + const nextRun = randomUUID(), nextLeaseId = randomUUID(); + await db.insert(heartbeatRuns).values({ id: nextRun, companyId: ids.company, agentId: ids.agent, + nativeIssueId: ids.issue, runtimeMode: "native", status: "running" }); + const [nextLease] = await db.insert(environmentLeases).values({ id: nextLeaseId, companyId: ids.company, + environmentId: ids.environment, heartbeatRunId: nextRun, issueId: ids.issue, + provider: "daytona", providerLeaseId: `allocation-${nextLeaseId}`, status: "active" }).returning(); + const resumed = await resolveRemoteWorkspaceSource(db, { ...input, runId: nextRun, + existingWorkspace: toExecutionWorkspace(await workspace(reserved.id)) }); + if (resumed.authority !== "remote") throw new Error("Expected retained remote task copies"); + if (attachment === "initial") expect(resumed.source).toEqual(resolved.source); + const copy = resumed.source.repositories![0]!; + expect(copy.projectId).toBe(reference.id); + const reservedAgain = await reserveRemoteWorkspace({ db, storage, ...input, runId: nextRun, + environmentId: ids.environment, existingWorkspaceId: reserved.id, source: resumed.source }); + const second = await target("project-copies-B"); + const restored = await prepareRemoteWorkspace({ db, storage, workspace: reservedAgain, runId: nextRun, + lease: nextLease as EnvironmentLease, target: second }); + for (const relative of ["", copy.relativePath]) { + if (!relative || attachment === "initial") expect(await fs.readFile(path.join(restored.remoteRoot, relative, "result.md"), "utf8")).toBe("saved edits\n"); + await fs.writeFile(path.join(restored.remoteRoot, relative, "follow-up.md"), "continued edits\n"); + } + expect(await fs.readFile(path.join(restored.remoteRoot, copy.relativePath, "README.md"), "utf8")).toBe("skills release\n"); + await saveRemoteWorkspace({ db, storage, workspace: restored.workspace, runId: nextRun, + lease: restored.lease, target: second, assertOwned: async () => undefined }); + const final = await workspace(reserved.id), authority = readRemoteWorkspaceAuthority(final.metadata)!; + expect(authority).toMatchObject({ dirty: false, writerRunId: null }); + expect(final.metadata?.remoteWorkspaceSnapshotRepositories).toEqual({ snapshotId: authority.snapshotId, repositories: resumed.source.repositories }); + await fs.rm(origins, { recursive: true, force: true }); + const saved = await createSavedWorkspaceLoader(db, storage)({ companyId: ids.company, workspaceId: reserved.id, snapshotId: authority.snapshotId! }); + try { + for (const relative of ["", copy.relativePath]) expect(await fs.readFile(path.join(saved.root, relative, "follow-up.md"), "utf8")).toBe("continued edits\n"); + expect(await fs.readFile(path.join(saved.root, copy.relativePath, "README.md"), "utf8")).toBe("skills release\n"); + } finally { await saved.cleanup(); } + } finally { + if (daemon.exitCode === null) { + const stopped = new Promise(resolve => daemon.once("exit", () => resolve())); + daemon.kill("SIGTERM"); await stopped; + } + } + }, 60_000); + it("prepares, saves and restores task files without a controller checkout or original allocation", async () => { const reserved = await reserve({ kind: "empty" }), first = await target("sandbox-A"); const prepared = await prepareRemoteWorkspace({ db, storage, workspace: reserved, runId: ids.run, lease: await lease(), target: first }); diff --git a/server/src/services/remote-workspaces.ts b/server/src/services/remote-workspaces.ts index 857da6c2c54..bb5849d2b55 100644 --- a/server/src/services/remote-workspaces.ts +++ b/server/src/services/remote-workspaces.ts @@ -217,7 +217,10 @@ export async function reserveRemoteWorkspace(input: { } if (authority.writerRunId === input.runId) return existing; const next: RemoteWorkspaceAuthority = { ...authority, source, epoch: authority.epoch + 1, writerRunId: input.runId }; - const [saved] = await tx.update(executionWorkspaces).set({ metadata: { ...existing.metadata, remoteWorkspace: next }, updatedAt: new Date() }) + const [saved] = await tx.update(executionWorkspaces).set({ metadata: { ...existing.metadata, remoteWorkspace: next, + ...(authority.snapshotId && existing.metadata?.remoteWorkspaceSnapshotRepositories === undefined + ? { remoteWorkspaceSnapshotRepositories: { snapshotId: authority.snapshotId, repositories: authority.source.repositories ?? [] } } : {}), + }, updatedAt: new Date() }) .where(eq(executionWorkspaces.id, existing.id)).returning(); if (issue.executionWorkspaceId !== existing.id) await tx.update(issues).set({ executionWorkspaceId: existing.id, executionWorkspacePreference: "reuse_existing", workspaceBindingRevision: issue.workspaceBindingRevision + 1, @@ -297,9 +300,17 @@ async function cleanupSavedExports(input: { db: Db; companyId: string; workspace } /** The caller retains execution admission and provider ownership; this owns file authority. */ -function isSavedWorkspaceRepository(authority: RemoteWorkspaceAuthority, +function isSavedWorkspaceRepository(workspace: Workspace, authority: RemoteWorkspaceAuthority, repository: NonNullable[number]) { - return authority.snapshotId !== null && (authority.source.repositories ?? []).some(saved => + if (authority.snapshotId === null) return false; + const receipt = workspace.metadata?.remoteWorkspaceSnapshotRepositories; + let repositories = authority.source.repositories ?? []; + if (receipt !== undefined) { + const saved = object(receipt); + if (saved.snapshotId !== authority.snapshotId || !Array.isArray(saved.repositories)) throw new Error("remote_workspace_snapshot_source_mismatch"); + repositories = remoteWorkspaceSourceSchema.parse({ kind: "empty", repositories: saved.repositories }).repositories ?? []; + } + return repositories.some(saved => saved.id === repository.id && saved.relativePath === repository.relativePath && saved.repoUrl === repository.repoUrl && (saved.ref ?? "HEAD") === (repository.ref ?? "HEAD") && saved.pinnedCommit === repository.pinnedCommit && (saved.readOnly ?? false) === (repository.readOnly ?? false)); @@ -353,7 +364,7 @@ export async function prepareRemoteWorkspace(input: { // descriptors still follow normal acquisition and origin verification. await prepareRemoteRepository({ target, remoteCwd: remoteWorkspaceRepositoryPath(remoteRoot, repository), repoUrl: repository.repoUrl, ref: repository.ref, pinnedCommit: repository.pinnedCommit, - savedArchive: isSavedWorkspaceRepository(initial, repository), + savedArchive: isSavedWorkspaceRepository(input.workspace, initial, repository), resolveGitAuth: await createRemoteWorkspaceGitAuth(input.db, input.workspace.companyId, input.runId) }); } const recoveredFiles: Array<{ operationId: string; sourceRunId: string; path: string }> = []; @@ -582,7 +593,9 @@ export async function saveRemoteWorkspace(input: { if (!committed) throw new Error("remote_workspace_snapshot_publication_changed"); await notifyRemoteWorkspaceSnapshots(tx, `workspace:${workspace.id}`); if (boundary === "final") await notifyTerminalWorkspaceWork(tx, `workspace:${workspace.id}`); - await tx.update(executionWorkspaces).set({ metadata: { ...workspace.metadata, remoteWorkspace: authority }, updatedAt: at }).where(eq(executionWorkspaces.id, workspace.id)); + await tx.update(executionWorkspaces).set({ metadata: { ...workspace.metadata, remoteWorkspace: authority, + remoteWorkspaceSnapshotRepositories: { snapshotId: saved.id, repositories: expected.source.repositories ?? [] }, + }, updatedAt: at }).where(eq(executionWorkspaces.id, workspace.id)); await tx.update(workspaceOperations).set({ status: "succeeded", finishedAt: at, updatedAt: at, metadata: sql`coalesce(${workspaceOperations.metadata}, '{}'::jsonb) || ${JSON.stringify({ snapshotId: saved.id, sha256: saved.sha256 })}::jsonb` }) .where(eq(workspaceOperations.id, operation.id)); @@ -621,7 +634,7 @@ export async function prepareRemoteWorkspaceRepository(input: { await input.assertOwned?.(); const target = requireRemoteWorkspaceTarget(input.target); const result = await prepareRemoteRepository({ target, remoteCwd: remoteWorkspaceRepositoryPath(authority.location.remoteCwd, input.repository), - ...input.repository, verifyOnly: input.verifyOnly, savedArchive: isSavedWorkspaceRepository(authority, input.repository), + ...input.repository, verifyOnly: input.verifyOnly, savedArchive: isSavedWorkspaceRepository(input.workspace, authority, input.repository), resolveGitAuth: input.resolveGitAuth ?? await createRemoteWorkspaceGitAuth(input.db, input.workspace.companyId, input.runId) }); await input.assertOwned?.(); await saveRemoteWorkspace({ ...input, target, boundary: `repository:${input.repository.id}:${result.pinnedCommit}`, diff --git a/server/src/services/task-interruption-presentation.test.ts b/server/src/services/task-interruption-presentation.test.ts index bf534b439d8..f9d67e6d7e9 100644 --- a/server/src/services/task-interruption-presentation.test.ts +++ b/server/src/services/task-interruption-presentation.test.ts @@ -7,6 +7,17 @@ const run = { id: "run", agentId: "agent", status: "failed", scheduledRetryAt: n resultJson: { projectAccessDenied: { projectId: "22222222-2222-4222-8222-222222222222" } } }; const agent = { status: "idle", pauseReason: null }; +it("preserves the actual workspace preparation error without changing retry authority", () => { + const failedRun = { ...run, errorCode: "workspace_validation_failed", + error: "The selected project has no repository source for a remote workspace.", resultJson: null }; + expect(projectTaskInterruption(issue, { run: failedRun, agent, paused: false, approval: false })) + .toMatchObject({ state: "failed", canRetry: true, workspaceFailure: { message: failedRun.error } }); + for (const status of ["done", "cancelled"]) expect(projectTaskInterruption({ ...issue, status }, + { run: failedRun, agent, paused: false, approval: false })).not.toHaveProperty("workspaceFailure"); + expect(projectTaskInterruption(issue, { run: { ...failedRun, errorCode: "unrelated_failure" }, agent, paused: false, approval: false })) + .not.toHaveProperty("workspaceFailure"); +}); + function operatorStoppedRun() { return { ...run, status: "cancelled", runtimeMode: "native" as const, companyId: issue.companyId, nativeIssueId: issue.id, errorCode: "cancelled", resultJson: { diff --git a/server/src/services/task-interruption-presentation.ts b/server/src/services/task-interruption-presentation.ts index eabc90cb86d..d84e5c5f25d 100644 --- a/server/src/services/task-interruption-presentation.ts +++ b/server/src/services/task-interruption-presentation.ts @@ -60,6 +60,8 @@ export function projectTaskInterruption(issue: ProjectionIssue, input: Projectio const providerFailure = run.status === "failed" || run.status === "scheduled_retry" ? readProviderFailure(run.resultJson?.providerFailure) : null; if (providerFailure) base.providerFailure = providerFailure; + if (["failed", "scheduled_retry"].includes(run.status) && run.errorCode === "workspace_validation_failed" + && typeof run.error === "string" && run.error.trim()) base.workspaceFailure = { message: run.error }; const userAction = !issue.assigneeAgentId ? "assign_agent" : input.approval || run.errorCode === "native_provider_approval_required" ? "approval" : isAiAuthenticationBlocked(run) ? "reconnect" : run.errorCode === "native_provider_model_rejected" ? "choose_model" @@ -124,7 +126,8 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i nativeIssueId: heartbeatRuns.nativeIssueId, runtimeMode: heartbeatRuns.runtimeMode, agentAdapter: agents.adapterType, error: sql`case when ${heartbeatRuns.errorCode} = 'setup_failed' - and ${heartbeatRuns.error} = 'Project is outside this low-trust boundary.' then ${heartbeatRuns.error} else null end`, + and ${heartbeatRuns.error} = 'Project is outside this low-trust boundary.' + or ${heartbeatRuns.errorCode} = 'workspace_validation_failed' then ${heartbeatRuns.error} else null end`, errorCode: heartbeatRuns.errorCode, livenessState: heartbeatRuns.livenessState, scheduledRetryAt: heartbeatRuns.scheduledRetryAt, resultJson: sql>`jsonb_build_object('configurationIncomplete', ${heartbeatRuns.resultJson}->'configurationIncomplete', 'projectAccessDenied', ${heartbeatRuns.resultJson}->'projectAccessDenied', @@ -218,25 +221,37 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i agentAdapter: run.agentAdapter ?? "unknown" }); } } - const disclosures = [...result.values()].flatMap(presentation => presentation?.providerFailure && presentation.observedRunId - ? [{ id: presentation.observedRunId, providerFailure: presentation.providerFailure, presentation }] : []); + const disclosures = [...result.values()].flatMap(presentation => (presentation?.providerFailure || presentation?.workspaceFailure) && presentation.observedRunId + ? [{ id: presentation.observedRunId, providerFailure: presentation.providerFailure, + workspaceFailure: presentation.workspaceFailure, presentation }] : []); if (disclosures.length) { try { const [safe, settings] = await Promise.all([ - createRunSecretRedactionRegistry(db).redactForRuns(companyId, disclosures.map(({ id, providerFailure }) => ({ id, providerFailure }))), + createRunSecretRedactionRegistry(db).redactForRuns(companyId, + disclosures.map(({ id, providerFailure, workspaceFailure }) => ({ id, providerFailure, workspaceFailure }))), instanceSettingsService(db).getGeneral(), ]); + const safeMessage = (value: unknown) => typeof value === "string" + ? [...redactCurrentUserText(value, { enabled: settings.censorUsernameInLogs })].slice(0, 4000).join("") : ""; disclosures.forEach(({ presentation }, index) => { - const value = safe[index]?.providerFailure; - const message = typeof value?.message === "string" - ? [...redactCurrentUserText(value.message, { enabled: settings.censorUsernameInLogs })].slice(0, 4000).join("") : ""; - const failure = readProviderFailure({ ...value, message }); - if (failure) presentation.providerFailure = failure; - else delete presentation.providerFailure; + if (presentation.providerFailure) { + const value = safe[index]?.providerFailure; + const failure = readProviderFailure({ ...value, message: safeMessage(value?.message) }); + if (failure) presentation.providerFailure = failure; + else delete presentation.providerFailure; + } + if (presentation.workspaceFailure) { + const message = safeMessage(safe[index]?.workspaceFailure?.message); + if (message.trim()) presentation.workspaceFailure = { message }; + else delete presentation.workspaceFailure; + } }); } catch { - for (const { presentation } of disclosures) delete presentation.providerFailure; - logger.warn({ companyId, runCount: disclosures.length }, "Could not redact provider failure display; explanations omitted"); + for (const { presentation } of disclosures) { + delete presentation.providerFailure; + delete presentation.workspaceFailure; + } + logger.warn({ companyId, runCount: disclosures.length }, "Could not redact run failure display; explanations omitted"); } } return result; diff --git a/ui/src/components/ExecutionBlockerNotice.test.tsx b/ui/src/components/ExecutionBlockerNotice.test.tsx index bfe07d25284..bf16c89ff5c 100644 --- a/ui/src/components/ExecutionBlockerNotice.test.tsx +++ b/ui/src/components/ExecutionBlockerNotice.test.tsx @@ -58,6 +58,18 @@ describe("ordinary task interruption", () => { expect(container.querySelector('[role="alert"]')?.textContent).toContain("Your task access changed."); expect(container.querySelectorAll("button")).toHaveLength(1); }); + it("shows workspace preparation failures as plain text while keeping retry permissions", async () => { + const workspaceFailure = { message: 'Could not prepare the project folder .' }; + await render({ ...failed, workspaceFailure }, { canRetry: false }); + expect(container.textContent).toContain("Couldn’t prepare the task folders."); + expect(container.textContent).toContain(workspaceFailure.message); + expect(container.querySelector("img, button")).toBeNull(); + await render({ ...failed, workspaceFailure }); + await act(async () => container.querySelector("button")!.click()); + expect(retry).toHaveBeenCalledOnce(); + await render({ ...failed, workspaceFailure, state: "working" }); + expect(container.textContent).toBe(""); + }); it("shows an unknown provider's original explanation as text and preserves read-only access", async () => { const providerFailure = { schema: "paperclip.provider-failure.v1" as const, provider: "new-provider", phase: "startup" as const, category: "unknown" as const, retryability: "unknown" as const, httpStatus: 429, diff --git a/ui/src/components/ExecutionBlockerNotice.tsx b/ui/src/components/ExecutionBlockerNotice.tsx index 721dfda3a0f..48b0ac8444b 100644 --- a/ui/src/components/ExecutionBlockerNotice.tsx +++ b/ui/src/components/ExecutionBlockerNotice.tsx @@ -43,7 +43,9 @@ export function ExecutionBlockerNotice({ presentation, canRetry, onRetry, optimi {presentation.state === "failed" &&