Files
PaperClipAI/doc/execution-github-identity.md
T
DottaandPaperclip 27fbdb2a74 perf(workspaces): reuse verified seeds across task branches (#18159)
## Thinking Path

> - Paperclip manages AI agents and their workspaces.
> - Remote runs stage a recoverable copy of the selected host workspace.
> - New tasks use separate workspace IDs and Git branches.
> - The verified seed cache currently treats those identities as
different content.
> - This causes repeated clones and seed verification when task files
and Git commits are identical.
> - This pull request shares immutable seed content within a company and
applies each task's branch to a private archive.
> - It also removes credential broker calls for two local Git snapshot
commands.

## Linked Issues or Issue Description

**What happened?**

New remote tasks rebuild the same Git seed under different workspace IDs
and branches. In a measured staging run, preparation took 29.7 seconds.
Git clone took 6.1 seconds and optional seed publication took 7.8
seconds. Publication overlapped the 4.4-second transfer, so those values
cannot be added. Local `diff-files` and `write-tree` commands also
request GitHub credentials.

**What did you expect?**

Reuse a verified seed when filesystem contents, Git HEAD, origin, and
nested repository layout match. Preserve each task's branch, recovery
archive, ignored files, and copyback behavior. Keep local snapshot
plumbing independent of GitHub credential capture.

**Steps to reproduce**

1. Configure a remote native environment and a Git project workspace.
2. Create two tasks from the same source commit with different task
branches and workspace IDs.
3. Inspect the workspace preparation spans. Both tasks currently clone
and publish the same seed content.

**Paperclip version and deployment mode**

Reproduced on Core `25f1070fbdabd96b26cdf199e624a77905f3ca37`, with a
managed remote sandbox and native Runner. The local comparison used a
public repository at a fixed commit with roughly 13,000 entries.

Refs #17886, #16398, #16399, #17357. The Git launcher extension builds
on Jamie Buchanan's work in #17886. The other PRs establish the seed
verification, recovery, and eviction contracts preserved here. A search
found no open PR for sharing these verified seeds across task branches.

## What Changed

- Use a company-scoped cache bucket for native host-current preparation.
Keep the existing bucket and company storage limits and eviction rules.
- Remove branch names from the shared generation identity. Keep file
contents, Git HEAD, origin, and nested layout in that identity. Keep the
default workspace-scoped identity unchanged for other callers.
- Store a detached Git archive in the shared cache. Copy it and append
the selected branch metadata before upload and private per-run
persistence. Existing durable-seed replay can read the private archive
without the new cache option.
- Forward the scope through the execution target and both
command-runtime paths.
- Treat `diff-files` and `write-tree` as local Git commands. Network
commands, unknown commands, and managed commit identity keep their
existing credential path.
- Test new workspace IDs, named and detached branches, nested
repositories, dirty and ignored files, symlinks, recovery, corrupt
caches, refused publication, and copyback.

## Verification

- Current head `6087bac274d50804d1b332646e43f3b8d41d798f`: 53 checks
passed, two optional checks skipped, Greptile 5/5, no unresolved review
threads. Linux CI passed.
- Adapter-utils TypeScript 7.0.2 typecheck passed.
- Server package typecheck passed, including its Runner and plugin build
prerequisites.
- 140 seed-cache, seed-reuse, and sandbox-runtime tests passed after
rebasing.
- 44 Git launcher and command-runtime tests passed after rebasing.
- 30 native workspace preparation, warm history, and
retention-configuration tests passed. All 56 retention tests also passed
(5.72 seconds) after repairing missing library aliases inside the
installed macOS Postgres package. That local dependency repair changed
no repository source. The earlier startup timeouts occurred before test
assertions.
- The new seed reuse regression failed before the change because the
second branch cloned again. It passes through the execution target and
both command-runtime paths after the change.
- Clean local preparation comparison: two alternating measured pairs,
with an existing verified company seed and new task branches. Baseline:
19.63 and 9.85 seconds. Shared cache: 5.18 and 4.80 seconds. Median:
14.74 to 4.99 seconds. Each baseline cloned and published; each
shared-cache run skipped both. Branch and file checks passed. The
complete comparison loop took 58 seconds.
- The comparison used real Git, archives, snapshot workers, and local
file transport. It used no model calls. It does not establish a complete
staging response time. The first generation still pays the cold
preparation cost.

## Risks

- A shared cache hit requires an identical generation that is still
present. The company bucket retains the existing 16-generation/1 GiB
bucket limit; busy companies may miss more often.
- The branch binding appends entries to a private tar archive. Tests
cover the host platform; Linux CI must also pass before merge.
- A cache miss, corruption, or capacity refusal must retain the private
recovery path. Existing tests exercise those cases.
- Git plumbing can still encounter repository-specific filters or
missing objects. This extends the existing local-command policy by two
commands; it does not grant network credentials to them.
- The latest measured real staging reply took 77.8 seconds. This change
targets a bounded part of that path and does not claim the
under-15-second response goal is met.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning and repository tools. The
session does not expose the exact model revision or context window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-29 09:19:23 -05:00

224 lines
20 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# GitHub identity during agent execution
Shared agents use the GitHub connection of the person whose accepted instructions they are executing. Task ownership remains unchanged. GitHub is optional: ordinary work can start without a connection. Managed launchers provide a default author for local commits; private checkouts and authenticated API calls still require eligible credentials.
## Accepted instructions and continuations
`run_identity_contexts` records ordered revisions, stored message authors, originating causes, parent contexts, acceptance state, and redacted GitHub outcomes. `heartbeat_runs.active_identity_context_id` selects the current revision. Existing historical runs are not backfilled with inferred authorship.
Human messages use their stored authenticated author. Queued messages retain their delivery order. Accepting steering reserves a pending revision before delivery, then activates it after the provider acknowledgement. Rejected delivery leaves the prior revision active. An uncertain acknowledgement holds new credential acquisition; a later acknowledgement or its authenticated native event receipt reconciles the reservation. Replays cannot reactivate an older revision. Activation locks the task before the run, matching task and queue mutations so concurrent status changes cannot deadlock identity initialization.
Delegated work and interactions persist their originating context. Retries retain the originating run's active context. Background continuations carry their source run; dependency wakes use the task's continuation context, independently of its owner. Scheduled and webhook routines use the routine's responsible person; manual invocations use the caller, and edits preserve the routine's responsible person.
## Managed GitHub operations
Executions with managed GitHub configured receive token-free `git` and `gh` launchers. Each launcher invocation resolves one eligible credential from the current run's accepted identity at operation start. A `gh` command's child Git processes inherit that command's captured identity; later steering does not change already-started operations.
A `git` command that never uses the network does not ask the broker. The launcher reads past the global options, such as `-C` and `-c`, and checks the subcommand against a short list (for example `status`, `diff`, `diff-files`, `write-tree`, `log`, `rev-parse`, `add`, `config`, `branch`, `checkout`). This includes the index comparison and tree writing used by workspace snapshots. An unknown subcommand or option still asks. `commit` and `var` still ask, because the managed author comes from the broker answer. The launcher keeps no broker answer between invocations and never writes a credential to a file.
GitHub credential capture reads the current task, project, and retained trust
policy bindings without transferring saved prompts, runner profiles, results,
or log excerpts. It keeps the existing task/run lock order and accepted-identity
reconciliation. Other identity callers retain their complete execution input,
including the native input required by project creation authorization.
Native runners retain a session-owned broker and launcher path across warm turns. The provider keeps an opaque transport token, not a GitHub credential or the previous run's signed capability. After acquiring exclusive session ownership, the controller binds the broker to the current company, agent, task, and run. Requests cannot choose another run or responsible person. The broker rejects requests while idle and discards credential responses if their run binding changed during acquisition. Each operation still rechecks the live run, accepted identity, grants, and trust policy through the shared credential resolver. Changing run IDs alone no longer replaces the provider process; changes to authentication mode, provider credentials, permissions, or other session configuration retain their existing retirement rules.
The session broker listens only on controller loopback and accepts only its authenticated GitHub credential operation. Remote executions reach it through the existing authenticated callback bridge. Its transport and launchers are retired with the provider session. If remote bridge startup fails, anonymous launchers keep ordinary work available; the next run retries setup with a fresh session. Controller restart/cold recovery still uses the existing checkpoint and process-recovery rules; the broker's in-memory authority is not persisted for adoption.
The native GitHub callback bridge uses HTTP/2 only when the server-selected
`enableSandboxDuplexBridge` flag and the sandbox's verified duplex capability
permit it. Missing settings, unsupported providers and failed channel startup
keep the authenticated file bridge. Changing the server flag retires a retained
warm bridge at the next normal run boundary, before a retained instruction copy
is claimed. A lost HTTP/2 channel also makes the broker unavailable and retires
that warm session at its next admission. An unchanged flag and a healthy broker
permit reuse. Normal session cleanup closes the channel as an orderly completion.
This selection changes transport only; operation-time credential checks remain
the same. Local and SSH execution retain their existing transport.
Other adapters continue using the run-scoped signed capability and public broker endpoint. That endpoint rejects browser origins and session cookies, validates a distinct signed runtime scope, and rechecks the company, agent, and live run. GitHub credentials are returned only to the managed command process, never persisted in identity history or injected into the long-lived provider process.
Low-trust executions cannot receive raw GitHub credentials, including dedicated
agent tokens. The broker rechecks current agent, project, task, and retained run
policies before credential resolution. An external guest's internal sponsor is
accountable for the task, but does not authorize using the sponsor's account.
Read-only access must use separately authorized tools that enforce that boundary.
Server-side Git operations and GitHub gateway calls follow the same selection rules. Approved gateway operations retain their signed originating identity. Connection audience and tool policies continue to apply to the selected person's connection. Native catalogs remain stable across identity changes, but each invocation resolves the selected grant again. Personal OAuth secret declarations survive connection pauses and metadata edits.
Managed commands disable ambient Git credential helpers, Git global/system configuration, host GitHub CLI configuration, and host SSH identity access. Per-operation GitHub CLI configuration is isolated in a writable configuration directory beneath the managed launcher directory. Missing credentials clear previous author and token values; no teammate, standing delegation, host token, or company-default user's account is substituted. Anonymous/local operations remain available where supported.
OpenCode retains the controller-projected GitHub broker environment through
runner startup and provider startup. Shell operations acquire credentials for
the active run through that broker. They do not require a persistent GitHub CLI
login or inherit an ambient host broker capability.
Without a managed identity, local commits default to
`Paperclip <noreply@paperclip.ing>`. The launcher writes this token-free default
to a private per-operation global Git configuration file, removed on exit.
Repository identity and `git -c user.name=... -c user.email=...` override it.
The launcher leaves author/committer environment variables unset instead of
guessing the host user's details. `git config --global` changes only this
temporary file; use repository configuration for persistent overrides.
If the private configuration directory or default author file cannot be prepared,
Git ignores the incomplete file. Local commits still require an explicit
repository or command identity when no managed identity is available. A failed
default author write does not remove valid managed credentials. Managed shell profiles
remove empty identity overrides after environment merging, so agents do not
need to unset them per command. A captured managed identity still takes
precedence over repository configuration.
Remote launchers prepend their directory to the execution target's effective
`PATH`. An explicit remote `PATH` override is preserved; otherwise Paperclip
reads the provider's environment before staging the launcher shell files.
This keeps legacy NVM and user-local agent installations available alongside
newer images with system-wide CLIs. The generated shell files retain that
combined path with managed `git` and `gh` first. The launcher directory has
its own CommonJS package scope, so the extensionless Node launchers work
inside repositories that declare `"type": "module"` without changing the
project's package configuration. Sandbox command checks use
the same sanitized environment as execution, so a CLI visible only in the
provider's default environment cannot pass the launch check. Failed path
discovery stops startup instead of silently falling back to a minimal path.
Providers that declare concurrent sync support stage at most three independent
launcher files at once. Other providers keep serial staging. Each file retains
its content hash, upload lock, integrity check and atomic rename. A failed batch
settles all started uploads before startup rejects; later files and the final
permission step do not start. This changes startup scheduling, not credentials
or callback transport selection.
Scripts that previously read a persistent `GH_TOKEN` must use managed `git`, `gh`, or GitHub gateway tools. Managed execution skips legacy GitHub token bindings in agent, environment, project, and routine configuration before secret preflight. Configure personal or dedicated access through the GitHub connection instead. Directly invoking an unmanaged executable or retaining a token obtained during an earlier invocation is outside the managed invocation contract.
## Legacy hosts and networking
When no managed GitHub connection is installed for an agent, standard-trust
local and SSH executions retain that execution host's existing Git and GitHub
CLI credentials, configuration, credential helpers, and SSH agent. Paperclip
does not import controller credentials into an SSH target. Sandbox, plugin,
and low-trust executions do not receive this compatibility fallback. Once a
managed connection is configured, unavailable or revoked access never falls
back to host authentication. Switching modes replaces the provider process
while preserving the settled conversation.
Runner network access is independent of GitHub credentials. The controller
enables networking for standard-trust execution. Low-trust runs and runners
without a controller network decision retain a restricted default. An operator
can set `PAPERCLIP_RUNNER_NETWORK_ACCESS=disabled` to restrict normal execution;
user environment bindings cannot override that decision. Outer execution-
environment network restrictions still apply. The controller projects the assigned worktree's Git metadata paths so
Git can operate without exposing unrelated workspace or provider state. The
sandbox also receives read access to validated provider executable resources
and the target host's DNS and CA files, including resolver symlink targets
outside `/etc`. Provider credential directories remain isolated.
A managed broker outage does not prevent local Git operations. Launchers clear
credentials and run the command without authentication, with a redacted error
category identifying configuration setup, transport, or capability rejection.
They do not retain a previous operation's token or replay a GitHub operation.
Healthy eligible grants for the same stable GitHub account ID take precedence
over duplicates with failed health checks. Credential acquisition can retry
once against another grant for that same principal and account, before any
GitHub operation begins. Run identity diagnostics include the selected
connection and grant IDs, without credential values. Access-refresh conflicts
retry once against current state and never turn a concurrency conflict into
a reconnect requirement.
## Dedicated accounts and diagnostics
An explicit dedicated-agent grant overrides personal selection. Revoked, disabled, unavailable, or ambiguous dedicated grants do not fall back to a person's account. Removing the dedicated configuration restores personal selection.
Connection setup and permissions display: “This agent uses this GitHub account for everyone's work, instead of the person giving instructions.”
The GitHub permissions page shows repositories across all connected accounts in one scrollable list. It has no account filter or repository search. Repository icons, private-repository indicators, refresh, and GitHub configuration links remain available. The “Add More Repos on GitHub” button opens GitHub’s app installation and repository-access setup.
Multiple eligible connections for the same GitHub account are treated as one
identity, using GitHub's stable account ID rather than its login. The resolver
selects an available grant, preferring the newest authorization with a stable
ID tie-breaker. Duplicate eligibility includes an active credential record with
the correct owner, the OAuth access-token reference, and repository access
metadata. It keeps that grant's credential and connection policy together;
it does not combine repository access or bypass connection audiences. Distinct
accounts or unidentifiable duplicate grants remain ambiguous. Managed commands
print the redacted reason when GitHub access is unavailable, while unrelated
local operations can still proceed without credentials.
Run details show identity revisions and redacted GitHub results: responsible person, selected login when available, personal/dedicated source, and an unavailable reason. Tasks do not receive an additional identity indicator or takeover action.
## Deployment and verification
Deploy the schema, server broker, launcher staging, and runtime environment contract together. Already-running processes retain their original environment; only newly dispatched processes receive the broker contract. Run-scoped capabilities remain valid only while their bound run is active.
Focused coverage lives in `run-identity.test.ts`, `github-operation-credentials.test.ts`, and `github-launcher.test.ts`, alongside the native steering, gateway, routine, and callback-bridge suites. Live acceptance additionally requires two authenticated Paperclip users, two authorized GitHub accounts, and a designated disposable repository for push verification. Local commit metadata and mocked API results do not replace that live push test.
### Release procedure
1. Back up the instance database using the normal deployment procedure.
2. Build and deploy one revision containing migrations 0240–0245, the server broker, managed launchers, and the runner artifacts. Run the standard pending-migration check before admitting new runs. These additive migrations are safe to replay and do not infer authorship for historical runs.
3. Let pre-rollout executions finish with their original runtime contract. New executions must have an active identity context and the managed launcher capability before provider startup.
4. Check one ordinary run without a GitHub connection, then an authenticated GitHub operation. Inspect the run details for the responsible person and credential outcome. Verify a queued continuation on the same conversation.
5. If rollback is needed, finish or explicitly stop executions using the new broker before removing its endpoint. Keep the additive schema and identity history. Do not drop identity columns or tables to roll back application code.
Remote acceptance uses the existing paid runner workflow with a narrow selection. Run it against the same immutable revision as the release; a successful local test does not qualify a different remote runner artifact.
Identity history survives deletion of the originating agent or run, so surviving
subtasks and approvals retain their responsible person. The company foreign key and company-deletion service remove
these company-scoped records when their company is deleted. Run-scoped adapters remove their managed launcher files at the terminal boundary.
Native warm sessions retain their token-free launchers and inactive broker until
session retirement; environment deletion and orderly controller shutdown close
idle owners first. Cleanup failures are logged and do not change the run result.
## Project authorization for background repository work
Background provisioning and workspace maintenance have no responsible task user.
They must not retain a task's identity. A signed-in project workspace writer can
select their own active, company-installed GitHub Connector grant in project
settings and explicitly authorize the project. The binding stores the exact grant
ID and consenting user, never a credential. Revocation records a project lifecycle
event so subscribed providers can reconcile the change.
`GET /api/projects/:id/git-authorization` returns the current binding and only the
viewer's eligible personal grants. `PUT` accepts `{ grantId }` or `{ grantId: null }`.
Agents cannot create this standing consent. Project workspace writers can remove it.
Core rechecks membership, Connector installation/enablement, grant status, and
repository metadata on each background use. Public repositories never require
a connection or credential. Private repositories wait for authorization before
provisioning starts, so missing consent cannot leave a failed anonymous clone.
Repository selections retain the same visibility metadata used by the UI lock.
Legacy workspaces use existing Connector catalog metadata, with anonymous
GitHub discovery only when visibility is unknown. Unknown visibility defers
provisioning; it is never assumed public. It never substitutes another grant
when the selected one fails. Replacing a grant requires explicit reauthorization.
Project consent follows the selected connection's current GitHub repository access,
including repositories added to the project later. It does not create a second
repository approval list. Project repository editors may add sources within that
connection's access; manage permitted repositories in the GitHub connection.
Background export also rechecks the consenting user's project read access and the
project's existing trust policy. Low-trust or unverified projects cannot export a
raw credential through standing consent.
Cloud owner access is rechecked through current Cloud authority and the managed
owner-elevation setting on every private background export; saved local admin
roles cannot replace that authority. An unavailable authority check cannot confer
owner access, though explicit project membership still applies.
The trusted plugin capability `projects.git-credentials` enables
`ctx.projects.resolveGitCredential(projectId, workspaceId, companyId)`. The host
binds requests to the active project lifecycle or workspace operation; data, UI,
tool, and proactive calls do not carry this authority. Repository URLs come from
stored project workspaces. Foreground preparation uses the live run's accepted
identity and standard trust checks, even when background project consent exists.
Maintenance ignores originating run/agent context and uses project consent only.
After archival, existing consent remains usable only for host-issued workspace
inspection and removal. New provisioning and new consent remain blocked.
The provider receives a transient GitHub token with its existing provider
permissions. This is not a repository-scoped or read-only token minting API.
Only grant-listed repositories can trigger background export. Plugins must never
save tokens in entities, state, configuration, operation receipts, or logs, and
must send them only to trusted execution infrastructure over HTTPS. Revocation
prevents subsequent credential acquisition; it cannot recall a token already
sent to an in-flight operation. Existing tasks and Git credential fallback rules
outside this plugin API are unchanged.