pioduino/uv
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

Support Git LFS with opt-in (#16143)

Browse Source 
## Summary

Follow up to https://github.com/astral-sh/uv/pull/15563
Closes https://github.com/astral-sh/uv/issues/13485

This is a first-pass at adding support for conditional support for Git
LFS between git sources, initial feedback welcome.

e.g.
```
[tool.uv.sources]
test-lfs-repo = { git = "https://github.com/zanieb/test-lfs-repo.git", lfs = true }
```

For context previously a user had to set `UV_GIT_LFS` to have uv fetch
lfs objects on git sources. This env var was all or nothing, meaning you
must always have it set to get consistent behavior and it applied to all
git sources. If you fetched lfs objects at a revision and then turned
off lfs (or vice versa), the git db, corresponding checkout lfs
artifacts would not be updated properly. Similarly, when git source
distributions were built, there would be no distinction between sources
with lfs and without lfs. Hence, it could corrupt the git, sdist, and
archive caches.

In order to support some sources being LFS enabled and other not, this
PR adds a stateful layer roughly similar to how `subdirectory` works but
for `lfs` since the git database, the checkouts and the corresponding
caching layers needed to be LFS aware (requested vs installed). The
caches also had to isolated and treated entirely separate when handling
LFS sources.

Summary
* Adds `lfs = true` or `lfs = false` to git sources in pyproject.toml
* Added `lfs=true` query param / fragments to most relevant url structs
(not parsed as user input)
  * In the case of uv add / uv tool, `--lfs` is supported instead
* `UV_GIT_LFS` environment variable support is still functional for
non-project entrypoints (e.g. uv pip)
* `direct-url.json` now has an custom `git_lfs` entry under VcsInfo
(note, this is not in the spec currently -- see caveats).
* git database and checkouts have an different cache key as the sources
should be treated effectively different for the same rev.
* sdists cache also differ in the cache key of a built distribution if
it was built using LFS enabled revisions to distinguish between non-LFS
same revisions. This ensures the strong assumption for archive-v0 that
an unpacked revision "doesn't change sources" stays valid.

Caveats
* `pylock.toml` import support has not been added via git_lfs=true,
going through the spec it wasn't clear to me it's something we'd support
outside of the env var (for now).
* direct-url struct was modified by adding a non-standard `git_lfs`
field under VcsInfo which may be undersirable although the PEP 610 does
say `Additional fields that would be necessary to support such VCS
SHOULD be prefixed with the VCS command name` which could be interpret
this change as ok.
* There will be a slight lockfile and cache churn for users that use
`UV_GIT_LFS` as all git lockfile entries will get a `lfs=true` fragment.
The cache version does not need an update, but LFS sources will get
their own namespace under git-v0 and sdist-v9/git hence a cache-miss
will occur once but this can be sufficient to label this as breaking for
workflows always setting `UV_GIT_LFS`.

## Test Plan

Some initial tests were added. More tests likely to follow as we reach
consensus on a final approach.

For IT test, we may want to move to use a repo under astral namespace in
order to test lfs functionality.

Manual testing was done for common pathological cases like killing LFS
fetch mid-way, uninstalling LFS after installing an sdist with it and
reinstalling, fetching LFS artifacts in different commits, etc.

PSA: Please ignore the docker build failures as its related to depot
OIDC issues.

---------

Co-authored-by: Zanie Blue <contact@zanie.dev>
Co-authored-by: konstin <konstin@mailbox.org>
This commit is contained in:
samypr100
2025-12-02 07:23:51 -05:00
committed by GitHub
parent 5947fb0c83
commit fee7f9d093
45 changed files with 1978 additions and 85 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/uv-git/src/source.rs
+42 -10
View File
@@ -63,6 +63,8 @@ impl GitSource {
/// Fetch the underlying Git repository at the given revision.
#[instrument(skip(self), fields(repository = %self.git.repository(), rev = ?self.git.precise()))]
pub fn fetch(self) -> Result<Fetch> {
let lfs_requested = self.git.lfs().enabled();
// Compute the canonical URL for the repository.
let canonical = RepositoryUrl::new(self.git.repository());
@@ -85,24 +87,37 @@ impl GitSource {
// If we have a locked revision, and we have a pre-existing database which has that
// revision, then no update needs to happen.
// When requested, we also check if LFS artifacts have been fetched and validated.
if let (Some(rev), Some(db)) = (self.git.precise(), &maybe_db) {
if db.contains(rev) {
if db.contains(rev) && (!lfs_requested || db.contains_lfs_artifacts(rev)) {
debug!("Using existing Git source `{}`", self.git.repository());
return Ok((maybe_db.unwrap(), rev, None));
return Ok((
maybe_db
.unwrap()
.with_lfs_ready(lfs_requested.then_some(true)),
rev,
None,
));
}
}
// If the revision isn't locked, but it looks like it might be an exact commit hash,
// and we do have a pre-existing database, then check whether it is, in fact, a commit
// hash. If so, treat it like it's locked.
// When requested, we also check if LFS artifacts have been fetched and validated.
if let Some(db) = &maybe_db {
if let GitReference::BranchOrTagOrCommit(maybe_commit) = self.git.reference() {
if let Ok(oid) = maybe_commit.parse::<GitOid>() {
if db.contains(oid) {
// This reference is an exact commit. Treat it like it's
// locked.
if db.contains(oid) && (!lfs_requested || db.contains_lfs_artifacts(oid)) {
// This reference is an exact commit. Treat it like it's locked.
debug!("Using existing Git source `{}`", self.git.repository());
return Ok((maybe_db.unwrap(), oid, None));
return Ok((
maybe_db
.unwrap()
.with_lfs_ready(lfs_requested.then_some(true)),
oid,
None,
));
}
}
}
@@ -125,6 +140,7 @@ impl GitSource {
self.git.precise(),
self.disable_ssl,
self.offline,
lfs_requested,
)?;
Ok((db, actual_rev, task))
@@ -134,16 +150,25 @@ impl GitSource {
// path length limit on Windows.
let short_id = db.to_short_id(actual_rev)?;
// Check out `actual_rev` from the database to a scoped location on the
// filesystem. This will use hard links and such to ideally make the
// checkout operation here pretty fast.
// Compute the canonical URL for the repository checkout.
let canonical = canonical.with_lfs(Some(lfs_requested));
// Recompute the checkout hash when Git LFS is enabled as we want
// to distinctly differentiate between LFS vs non-LFS source trees.
let ident = if lfs_requested {
cache_digest(&canonical)
} else {
ident
};
let checkout_path = self
.cache
.join("checkouts")
.join(&ident)
.join(short_id.as_str());
db.copy_to(actual_rev, &checkout_path)?;
// Check out `actual_rev` from the database to a scoped location on the
// filesystem. This will use hard links and such to ideally make the
// checkout operation here pretty fast.
let checkout = db.copy_to(actual_rev, &checkout_path)?;
// Report the checkout operation to the reporter.
if let Some(task) = maybe_task {
@@ -155,6 +180,7 @@ impl GitSource {
Ok(Fetch {
git: self.git.with_precise(actual_rev),
path: checkout_path,
lfs_ready: checkout.lfs_ready().unwrap_or(false),
})
}
}
@@ -164,6 +190,8 @@ pub struct Fetch {
git: GitUrl,
/// The path to the checked out repository.
path: PathBuf,
/// Git LFS artifacts have been initialized (if requested).
lfs_ready: bool,
}
impl Fetch {
@@ -175,6 +203,10 @@ impl Fetch {
&self.path
}
pub fn lfs_ready(&self) -> &bool {
&self.lfs_ready
}
pub fn into_git(self) -> GitUrl {
self.git
}