mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-30 13:20:54 +01:00
## Thinking Path > - Paperclip manages AI agents and their workspaces. > - Remote runs stage a recoverable copy of the selected host workspace. > - New tasks use separate workspace IDs and Git branches. > - The verified seed cache currently treats those identities as different content. > - This causes repeated clones and seed verification when task files and Git commits are identical. > - This pull request shares immutable seed content within a company and applies each task's branch to a private archive. > - It also removes credential broker calls for two local Git snapshot commands. ## Linked Issues or Issue Description **What happened?** New remote tasks rebuild the same Git seed under different workspace IDs and branches. In a measured staging run, preparation took 29.7 seconds. Git clone took 6.1 seconds and optional seed publication took 7.8 seconds. Publication overlapped the 4.4-second transfer, so those values cannot be added. Local `diff-files` and `write-tree` commands also request GitHub credentials. **What did you expect?** Reuse a verified seed when filesystem contents, Git HEAD, origin, and nested repository layout match. Preserve each task's branch, recovery archive, ignored files, and copyback behavior. Keep local snapshot plumbing independent of GitHub credential capture. **Steps to reproduce** 1. Configure a remote native environment and a Git project workspace. 2. Create two tasks from the same source commit with different task branches and workspace IDs. 3. Inspect the workspace preparation spans. Both tasks currently clone and publish the same seed content. **Paperclip version and deployment mode** Reproduced on Core `25f1070fbdabd96b26cdf199e624a77905f3ca37`, with a managed remote sandbox and native Runner. The local comparison used a public repository at a fixed commit with roughly 13,000 entries. Refs #17886, #16398, #16399, #17357. The Git launcher extension builds on Jamie Buchanan's work in #17886. The other PRs establish the seed verification, recovery, and eviction contracts preserved here. A search found no open PR for sharing these verified seeds across task branches. ## What Changed - Use a company-scoped cache bucket for native host-current preparation. Keep the existing bucket and company storage limits and eviction rules. - Remove branch names from the shared generation identity. Keep file contents, Git HEAD, origin, and nested layout in that identity. Keep the default workspace-scoped identity unchanged for other callers. - Store a detached Git archive in the shared cache. Copy it and append the selected branch metadata before upload and private per-run persistence. Existing durable-seed replay can read the private archive without the new cache option. - Forward the scope through the execution target and both command-runtime paths. - Treat `diff-files` and `write-tree` as local Git commands. Network commands, unknown commands, and managed commit identity keep their existing credential path. - Test new workspace IDs, named and detached branches, nested repositories, dirty and ignored files, symlinks, recovery, corrupt caches, refused publication, and copyback. ## Verification - Current head `6087bac274d50804d1b332646e43f3b8d41d798f`: 53 checks passed, two optional checks skipped, Greptile 5/5, no unresolved review threads. Linux CI passed. - Adapter-utils TypeScript 7.0.2 typecheck passed. - Server package typecheck passed, including its Runner and plugin build prerequisites. - 140 seed-cache, seed-reuse, and sandbox-runtime tests passed after rebasing. - 44 Git launcher and command-runtime tests passed after rebasing. - 30 native workspace preparation, warm history, and retention-configuration tests passed. All 56 retention tests also passed (5.72 seconds) after repairing missing library aliases inside the installed macOS Postgres package. That local dependency repair changed no repository source. The earlier startup timeouts occurred before test assertions. - The new seed reuse regression failed before the change because the second branch cloned again. It passes through the execution target and both command-runtime paths after the change. - Clean local preparation comparison: two alternating measured pairs, with an existing verified company seed and new task branches. Baseline: 19.63 and 9.85 seconds. Shared cache: 5.18 and 4.80 seconds. Median: 14.74 to 4.99 seconds. Each baseline cloned and published; each shared-cache run skipped both. Branch and file checks passed. The complete comparison loop took 58 seconds. - The comparison used real Git, archives, snapshot workers, and local file transport. It used no model calls. It does not establish a complete staging response time. The first generation still pays the cold preparation cost. ## Risks - A shared cache hit requires an identical generation that is still present. The company bucket retains the existing 16-generation/1 GiB bucket limit; busy companies may miss more often. - The branch binding appends entries to a private tar archive. Tests cover the host platform; Linux CI must also pass before merge. - A cache miss, corruption, or capacity refusal must retain the private recovery path. Existing tests exercise those cases. - Git plumbing can still encounter repository-specific filters or missing objects. This extends the existing local-command policy by two commands; it does not grant network credentials to them. - The latest measured real staging reply took 77.8 seconds. This change targets a bounded part of that path and does not claim the under-15-second response goal is met. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and repository tools. The session does not expose the exact model revision or context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
224 lines
20 KiB
Markdown
224 lines
20 KiB
Markdown
# GitHub identity during agent execution
|
||
|
||
Shared agents use the GitHub connection of the person whose accepted instructions they are executing. Task ownership remains unchanged. GitHub is optional: ordinary work can start without a connection. Managed launchers provide a default author for local commits; private checkouts and authenticated API calls still require eligible credentials.
|
||
|
||
## Accepted instructions and continuations
|
||
|
||
`run_identity_contexts` records ordered revisions, stored message authors, originating causes, parent contexts, acceptance state, and redacted GitHub outcomes. `heartbeat_runs.active_identity_context_id` selects the current revision. Existing historical runs are not backfilled with inferred authorship.
|
||
|
||
Human messages use their stored authenticated author. Queued messages retain their delivery order. Accepting steering reserves a pending revision before delivery, then activates it after the provider acknowledgement. Rejected delivery leaves the prior revision active. An uncertain acknowledgement holds new credential acquisition; a later acknowledgement or its authenticated native event receipt reconciles the reservation. Replays cannot reactivate an older revision. Activation locks the task before the run, matching task and queue mutations so concurrent status changes cannot deadlock identity initialization.
|
||
|
||
Delegated work and interactions persist their originating context. Retries retain the originating run's active context. Background continuations carry their source run; dependency wakes use the task's continuation context, independently of its owner. Scheduled and webhook routines use the routine's responsible person; manual invocations use the caller, and edits preserve the routine's responsible person.
|
||
|
||
## Managed GitHub operations
|
||
|
||
Executions with managed GitHub configured receive token-free `git` and `gh` launchers. Each launcher invocation resolves one eligible credential from the current run's accepted identity at operation start. A `gh` command's child Git processes inherit that command's captured identity; later steering does not change already-started operations.
|
||
|
||
A `git` command that never uses the network does not ask the broker. The launcher reads past the global options, such as `-C` and `-c`, and checks the subcommand against a short list (for example `status`, `diff`, `diff-files`, `write-tree`, `log`, `rev-parse`, `add`, `config`, `branch`, `checkout`). This includes the index comparison and tree writing used by workspace snapshots. An unknown subcommand or option still asks. `commit` and `var` still ask, because the managed author comes from the broker answer. The launcher keeps no broker answer between invocations and never writes a credential to a file.
|
||
|
||
GitHub credential capture reads the current task, project, and retained trust
|
||
policy bindings without transferring saved prompts, runner profiles, results,
|
||
or log excerpts. It keeps the existing task/run lock order and accepted-identity
|
||
reconciliation. Other identity callers retain their complete execution input,
|
||
including the native input required by project creation authorization.
|
||
|
||
Native runners retain a session-owned broker and launcher path across warm turns. The provider keeps an opaque transport token, not a GitHub credential or the previous run's signed capability. After acquiring exclusive session ownership, the controller binds the broker to the current company, agent, task, and run. Requests cannot choose another run or responsible person. The broker rejects requests while idle and discards credential responses if their run binding changed during acquisition. Each operation still rechecks the live run, accepted identity, grants, and trust policy through the shared credential resolver. Changing run IDs alone no longer replaces the provider process; changes to authentication mode, provider credentials, permissions, or other session configuration retain their existing retirement rules.
|
||
|
||
The session broker listens only on controller loopback and accepts only its authenticated GitHub credential operation. Remote executions reach it through the existing authenticated callback bridge. Its transport and launchers are retired with the provider session. If remote bridge startup fails, anonymous launchers keep ordinary work available; the next run retries setup with a fresh session. Controller restart/cold recovery still uses the existing checkpoint and process-recovery rules; the broker's in-memory authority is not persisted for adoption.
|
||
|
||
The native GitHub callback bridge uses HTTP/2 only when the server-selected
|
||
`enableSandboxDuplexBridge` flag and the sandbox's verified duplex capability
|
||
permit it. Missing settings, unsupported providers and failed channel startup
|
||
keep the authenticated file bridge. Changing the server flag retires a retained
|
||
warm bridge at the next normal run boundary, before a retained instruction copy
|
||
is claimed. A lost HTTP/2 channel also makes the broker unavailable and retires
|
||
that warm session at its next admission. An unchanged flag and a healthy broker
|
||
permit reuse. Normal session cleanup closes the channel as an orderly completion.
|
||
This selection changes transport only; operation-time credential checks remain
|
||
the same. Local and SSH execution retain their existing transport.
|
||
|
||
Other adapters continue using the run-scoped signed capability and public broker endpoint. That endpoint rejects browser origins and session cookies, validates a distinct signed runtime scope, and rechecks the company, agent, and live run. GitHub credentials are returned only to the managed command process, never persisted in identity history or injected into the long-lived provider process.
|
||
|
||
Low-trust executions cannot receive raw GitHub credentials, including dedicated
|
||
agent tokens. The broker rechecks current agent, project, task, and retained run
|
||
policies before credential resolution. An external guest's internal sponsor is
|
||
accountable for the task, but does not authorize using the sponsor's account.
|
||
Read-only access must use separately authorized tools that enforce that boundary.
|
||
|
||
Server-side Git operations and GitHub gateway calls follow the same selection rules. Approved gateway operations retain their signed originating identity. Connection audience and tool policies continue to apply to the selected person's connection. Native catalogs remain stable across identity changes, but each invocation resolves the selected grant again. Personal OAuth secret declarations survive connection pauses and metadata edits.
|
||
|
||
Managed commands disable ambient Git credential helpers, Git global/system configuration, host GitHub CLI configuration, and host SSH identity access. Per-operation GitHub CLI configuration is isolated in a writable configuration directory beneath the managed launcher directory. Missing credentials clear previous author and token values; no teammate, standing delegation, host token, or company-default user's account is substituted. Anonymous/local operations remain available where supported.
|
||
|
||
OpenCode retains the controller-projected GitHub broker environment through
|
||
runner startup and provider startup. Shell operations acquire credentials for
|
||
the active run through that broker. They do not require a persistent GitHub CLI
|
||
login or inherit an ambient host broker capability.
|
||
|
||
Without a managed identity, local commits default to
|
||
`Paperclip <noreply@paperclip.ing>`. The launcher writes this token-free default
|
||
to a private per-operation global Git configuration file, removed on exit.
|
||
Repository identity and `git -c user.name=... -c user.email=...` override it.
|
||
The launcher leaves author/committer environment variables unset instead of
|
||
guessing the host user's details. `git config --global` changes only this
|
||
temporary file; use repository configuration for persistent overrides.
|
||
If the private configuration directory or default author file cannot be prepared,
|
||
Git ignores the incomplete file. Local commits still require an explicit
|
||
repository or command identity when no managed identity is available. A failed
|
||
default author write does not remove valid managed credentials. Managed shell profiles
|
||
remove empty identity overrides after environment merging, so agents do not
|
||
need to unset them per command. A captured managed identity still takes
|
||
precedence over repository configuration.
|
||
|
||
Remote launchers prepend their directory to the execution target's effective
|
||
`PATH`. An explicit remote `PATH` override is preserved; otherwise Paperclip
|
||
reads the provider's environment before staging the launcher shell files.
|
||
This keeps legacy NVM and user-local agent installations available alongside
|
||
newer images with system-wide CLIs. The generated shell files retain that
|
||
combined path with managed `git` and `gh` first. The launcher directory has
|
||
its own CommonJS package scope, so the extensionless Node launchers work
|
||
inside repositories that declare `"type": "module"` without changing the
|
||
project's package configuration. Sandbox command checks use
|
||
the same sanitized environment as execution, so a CLI visible only in the
|
||
provider's default environment cannot pass the launch check. Failed path
|
||
discovery stops startup instead of silently falling back to a minimal path.
|
||
|
||
Providers that declare concurrent sync support stage at most three independent
|
||
launcher files at once. Other providers keep serial staging. Each file retains
|
||
its content hash, upload lock, integrity check and atomic rename. A failed batch
|
||
settles all started uploads before startup rejects; later files and the final
|
||
permission step do not start. This changes startup scheduling, not credentials
|
||
or callback transport selection.
|
||
|
||
Scripts that previously read a persistent `GH_TOKEN` must use managed `git`, `gh`, or GitHub gateway tools. Managed execution skips legacy GitHub token bindings in agent, environment, project, and routine configuration before secret preflight. Configure personal or dedicated access through the GitHub connection instead. Directly invoking an unmanaged executable or retaining a token obtained during an earlier invocation is outside the managed invocation contract.
|
||
|
||
## Legacy hosts and networking
|
||
|
||
When no managed GitHub connection is installed for an agent, standard-trust
|
||
local and SSH executions retain that execution host's existing Git and GitHub
|
||
CLI credentials, configuration, credential helpers, and SSH agent. Paperclip
|
||
does not import controller credentials into an SSH target. Sandbox, plugin,
|
||
and low-trust executions do not receive this compatibility fallback. Once a
|
||
managed connection is configured, unavailable or revoked access never falls
|
||
back to host authentication. Switching modes replaces the provider process
|
||
while preserving the settled conversation.
|
||
|
||
Runner network access is independent of GitHub credentials. The controller
|
||
enables networking for standard-trust execution. Low-trust runs and runners
|
||
without a controller network decision retain a restricted default. An operator
|
||
can set `PAPERCLIP_RUNNER_NETWORK_ACCESS=disabled` to restrict normal execution;
|
||
user environment bindings cannot override that decision. Outer execution-
|
||
environment network restrictions still apply. The controller projects the assigned worktree's Git metadata paths so
|
||
Git can operate without exposing unrelated workspace or provider state. The
|
||
sandbox also receives read access to validated provider executable resources
|
||
and the target host's DNS and CA files, including resolver symlink targets
|
||
outside `/etc`. Provider credential directories remain isolated.
|
||
|
||
A managed broker outage does not prevent local Git operations. Launchers clear
|
||
credentials and run the command without authentication, with a redacted error
|
||
category identifying configuration setup, transport, or capability rejection.
|
||
They do not retain a previous operation's token or replay a GitHub operation.
|
||
|
||
Healthy eligible grants for the same stable GitHub account ID take precedence
|
||
over duplicates with failed health checks. Credential acquisition can retry
|
||
once against another grant for that same principal and account, before any
|
||
GitHub operation begins. Run identity diagnostics include the selected
|
||
connection and grant IDs, without credential values. Access-refresh conflicts
|
||
retry once against current state and never turn a concurrency conflict into
|
||
a reconnect requirement.
|
||
|
||
## Dedicated accounts and diagnostics
|
||
|
||
An explicit dedicated-agent grant overrides personal selection. Revoked, disabled, unavailable, or ambiguous dedicated grants do not fall back to a person's account. Removing the dedicated configuration restores personal selection.
|
||
|
||
Connection setup and permissions display: “This agent uses this GitHub account for everyone's work, instead of the person giving instructions.”
|
||
|
||
The GitHub permissions page shows repositories across all connected accounts in one scrollable list. It has no account filter or repository search. Repository icons, private-repository indicators, refresh, and GitHub configuration links remain available. The “Add More Repos on GitHub” button opens GitHub’s app installation and repository-access setup.
|
||
|
||
Multiple eligible connections for the same GitHub account are treated as one
|
||
identity, using GitHub's stable account ID rather than its login. The resolver
|
||
selects an available grant, preferring the newest authorization with a stable
|
||
ID tie-breaker. Duplicate eligibility includes an active credential record with
|
||
the correct owner, the OAuth access-token reference, and repository access
|
||
metadata. It keeps that grant's credential and connection policy together;
|
||
it does not combine repository access or bypass connection audiences. Distinct
|
||
accounts or unidentifiable duplicate grants remain ambiguous. Managed commands
|
||
print the redacted reason when GitHub access is unavailable, while unrelated
|
||
local operations can still proceed without credentials.
|
||
|
||
Run details show identity revisions and redacted GitHub results: responsible person, selected login when available, personal/dedicated source, and an unavailable reason. Tasks do not receive an additional identity indicator or takeover action.
|
||
|
||
## Deployment and verification
|
||
|
||
Deploy the schema, server broker, launcher staging, and runtime environment contract together. Already-running processes retain their original environment; only newly dispatched processes receive the broker contract. Run-scoped capabilities remain valid only while their bound run is active.
|
||
|
||
Focused coverage lives in `run-identity.test.ts`, `github-operation-credentials.test.ts`, and `github-launcher.test.ts`, alongside the native steering, gateway, routine, and callback-bridge suites. Live acceptance additionally requires two authenticated Paperclip users, two authorized GitHub accounts, and a designated disposable repository for push verification. Local commit metadata and mocked API results do not replace that live push test.
|
||
|
||
### Release procedure
|
||
|
||
1. Back up the instance database using the normal deployment procedure.
|
||
2. Build and deploy one revision containing migrations 0240–0245, the server broker, managed launchers, and the runner artifacts. Run the standard pending-migration check before admitting new runs. These additive migrations are safe to replay and do not infer authorship for historical runs.
|
||
3. Let pre-rollout executions finish with their original runtime contract. New executions must have an active identity context and the managed launcher capability before provider startup.
|
||
4. Check one ordinary run without a GitHub connection, then an authenticated GitHub operation. Inspect the run details for the responsible person and credential outcome. Verify a queued continuation on the same conversation.
|
||
5. If rollback is needed, finish or explicitly stop executions using the new broker before removing its endpoint. Keep the additive schema and identity history. Do not drop identity columns or tables to roll back application code.
|
||
|
||
Remote acceptance uses the existing paid runner workflow with a narrow selection. Run it against the same immutable revision as the release; a successful local test does not qualify a different remote runner artifact.
|
||
|
||
Identity history survives deletion of the originating agent or run, so surviving
|
||
subtasks and approvals retain their responsible person. The company foreign key and company-deletion service remove
|
||
these company-scoped records when their company is deleted. Run-scoped adapters remove their managed launcher files at the terminal boundary.
|
||
Native warm sessions retain their token-free launchers and inactive broker until
|
||
session retirement; environment deletion and orderly controller shutdown close
|
||
idle owners first. Cleanup failures are logged and do not change the run result.
|
||
|
||
## Project authorization for background repository work
|
||
|
||
Background provisioning and workspace maintenance have no responsible task user.
|
||
They must not retain a task's identity. A signed-in project workspace writer can
|
||
select their own active, company-installed GitHub Connector grant in project
|
||
settings and explicitly authorize the project. The binding stores the exact grant
|
||
ID and consenting user, never a credential. Revocation records a project lifecycle
|
||
event so subscribed providers can reconcile the change.
|
||
|
||
`GET /api/projects/:id/git-authorization` returns the current binding and only the
|
||
viewer's eligible personal grants. `PUT` accepts `{ grantId }` or `{ grantId: null }`.
|
||
Agents cannot create this standing consent. Project workspace writers can remove it.
|
||
Core rechecks membership, Connector installation/enablement, grant status, and
|
||
repository metadata on each background use. Public repositories never require
|
||
a connection or credential. Private repositories wait for authorization before
|
||
provisioning starts, so missing consent cannot leave a failed anonymous clone.
|
||
Repository selections retain the same visibility metadata used by the UI lock.
|
||
Legacy workspaces use existing Connector catalog metadata, with anonymous
|
||
GitHub discovery only when visibility is unknown. Unknown visibility defers
|
||
provisioning; it is never assumed public. It never substitutes another grant
|
||
when the selected one fails. Replacing a grant requires explicit reauthorization.
|
||
Project consent follows the selected connection's current GitHub repository access,
|
||
including repositories added to the project later. It does not create a second
|
||
repository approval list. Project repository editors may add sources within that
|
||
connection's access; manage permitted repositories in the GitHub connection.
|
||
Background export also rechecks the consenting user's project read access and the
|
||
project's existing trust policy. Low-trust or unverified projects cannot export a
|
||
raw credential through standing consent.
|
||
Cloud owner access is rechecked through current Cloud authority and the managed
|
||
owner-elevation setting on every private background export; saved local admin
|
||
roles cannot replace that authority. An unavailable authority check cannot confer
|
||
owner access, though explicit project membership still applies.
|
||
|
||
The trusted plugin capability `projects.git-credentials` enables
|
||
`ctx.projects.resolveGitCredential(projectId, workspaceId, companyId)`. The host
|
||
binds requests to the active project lifecycle or workspace operation; data, UI,
|
||
tool, and proactive calls do not carry this authority. Repository URLs come from
|
||
stored project workspaces. Foreground preparation uses the live run's accepted
|
||
identity and standard trust checks, even when background project consent exists.
|
||
Maintenance ignores originating run/agent context and uses project consent only.
|
||
After archival, existing consent remains usable only for host-issued workspace
|
||
inspection and removal. New provisioning and new consent remain blocked.
|
||
|
||
The provider receives a transient GitHub token with its existing provider
|
||
permissions. This is not a repository-scoped or read-only token minting API.
|
||
Only grant-listed repositories can trigger background export. Plugins must never
|
||
save tokens in entities, state, configuration, operation receipts, or logs, and
|
||
must send them only to trusted execution infrastructure over HTTPS. Revocation
|
||
prevents subsequent credential acquisition; it cannot recall a token already
|
||
sent to an in-flight operation. Existing tasks and Git credential fallback rules
|
||
outside this plugin API are unchanged.
|