Commit Graph

21 Commits

Author SHA1 Message Date
justin 1ff7265e8a respect --offline flag for git cli operations (#12619)
## Summary
closes #12234


[fetch_with_cli](https://github.com/thejchap/uv/blob/e0f81f0d4a904d8c743e776d9f8c9ef5b96f769c/crates/uv-git/src/git.rs#L573)
doesn't respect the registry client's [connectivity
setting](https://github.com/thejchap/uv/blob/e0f81f0d4a904d8c743e776d9f8c9ef5b96f769c/crates/uv-client/src/registry_client.rs#L1009)
- this pr updates `fetch_with_cli` to set `GIT_ALLOW_PROTOCOL=file` when
the client's connectivity setting is `Connectivity::Offline`

## Test Plan
E2E

```sh
cargo run add "pycurl @ git+https://github.com/pycurl/pycurl.git" --directory ~/src/offline-test/ --offline
```

```sh
   Compiling uv-cli v0.0.1 (/Users/justinchapman/src/uv/crates/uv-cli)
   Compiling uv v0.6.11 (/Users/justinchapman/src/uv/crates/uv)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 4.47s
     Running `target/debug/uv add 'pycurl @ git+https://github.com/pycurl/pycurl.git' --directory /Users/justinchapman/src/offline-test/ --offline`
   Updating https://github.com/pycurl/pycurl.git (HEAD)                                                                                                                                   × Failed to download and build `pycurl @ git+https://github.com/pycurl/pycurl.git`
  ├─▶ Git operation failed
  ├─▶ failed to fetch into: /Users/justinchapman/.cache/uv/git-v0/db/9a596e5213c3162d
  ╰─▶ process didn't exit successfully: `/usr/bin/git fetch --force --update-head-ok 'https://github.com/pycurl/pycurl.git' '+HEAD:refs/remotes/origin/HEAD'` (exit status: 128)
      --- stderr
      fatal: transport 'https' not allowed

  help: If you want to add the package regardless of the failed resolution, provide the `--frozen` flag to skip locking and syncing.
```

---------

Co-authored-by: Zanie Blue <contact@zanie.dev>
2025-04-04 11:02:54 -05:00
samypr100 878497a014 Upgrade Rust toolchain to 1.85 (#11720)
## Summary

* Upgrade the rust toolchain to 1.85.0. This does not increase the MSRV.
* Update windows trampoline to 1.86 nightly beta (previously in 1.85
nightly beta).

## Test Plan

Existing tests
2025-02-23 16:52:34 +01:00
konsti 248da23f6d Split uv-git and uv-git-types (#11448)
We want to build `uv-build` without depending on the network crates. In
preparation for that, we split uv-git into uv-git and uv-git-types,
where only uv-git depends on reqwest, so that uv-build can use
uv-git-types.
2025-02-17 10:37:55 +01:00
konsti 59b46bc216 Show messages for builds and large downloads in non-interactive mode (#11165)
When stderr is not a tty, we currently don't show any messages for build
or large downloads, since indicatif is hidden. We can improve this by
showing a message for:

* Starting and finishing a large download (>1MB)
* Starting and finishing a build

Downloads are limited to 1MB or unknown size to keep the logs concise
and not scroll the entire terminal away for a download that finishes
almost immediately.

These messages are not captured in the tests since their order is
non-deterministic (downloads and builds race to finish).

There are no "tick" messages for large downloads yet, we could e.g. show
an update on runnning downloads every n seconds.

Part of #11121

**Test Plan**

```
$ uv venv && FORCE_COLOR=1 cargo run -q pip install numpy --no-binary :all: --no-cache 2>&1 | tee a.txt
  Using CPython 3.13.0
  Creating virtual environment at: .venv
  Activate with: source .venv/bin/activate
  Resolved 1 package in 221ms
     Building numpy==2.2.2
        Built numpy==2.2.2
  Prepared 1 package in 2m 34s
  Installed 1 package in 6ms
   + numpy==2.2.2
```


![image](https://github.com/user-attachments/assets/f4b64313-afa7-449f-9e5b-2b1b7026bef3)


```
$ uv venv && FORCE_COLOR=1 cargo run -q pip install torch --no-cache 2>&1 | tee b.txt
  Using CPython 3.13.0
  Creating virtual environment at: .venv
  Activate with: source .venv/bin/activate
  Resolved 24 packages in 648ms
  Downloading setuptools (1.2MiB)
  Downloading nvidia-cuda-cupti-cu12 (13.2MiB)
  Downloading torch (731.1MiB)
  Downloading nvidia-nvjitlink-cu12 (20.1MiB)
  Downloading nvidia-cufft-cu12 (201.7MiB)
  Downloading nvidia-cuda-nvrtc-cu12 (23.5MiB)
  Downloading nvidia-curand-cu12 (53.7MiB)
  Downloading nvidia-nccl-cu12 (179.9MiB)
  Downloading nvidia-cudnn-cu12 (634.0MiB)
  Downloading nvidia-cublas-cu12 (346.6MiB)
  Downloading sympy (5.9MiB)
  Downloading nvidia-cusparse-cu12 (197.8MiB)
  Downloading nvidia-cusparselt-cu12 (143.1MiB)
  Downloading networkx (1.6MiB)
  Downloading nvidia-cusolver-cu12 (122.0MiB)
  Downloading triton (241.4MiB)
   Downloaded setuptools
   Downloaded networkx
   Downloaded sympy
   Downloaded nvidia-cuda-cupti-cu12
   Downloaded nvidia-nvjitlink-cu12
   Downloaded nvidia-cuda-nvrtc-cu12
   Downloaded nvidia-curand-cu12
[...]
```


![image](https://github.com/user-attachments/assets/71918d94-c5c0-44ce-bea8-aaba6cd80ef7)
2025-02-05 17:38:02 -06:00
Charlie Marsh 748582ee6f Disable SSL in Git commands for --allow-insecure-host (#11210)
## Summary

Closes https://github.com/astral-sh/uv/issues/11176.

## Test Plan

- Created a self-signed certificate.
- Ran `openssl s_server -cert cert.pem -key key.pem -WWW -port 8443`.
- Verified that `cargo run pip install
git+https://localhost:8443/repo.git` failed with:

```
error: Git operation failed
  Caused by: failed to fetch into: /Users/crmarsh/.cache/uv/git-v0/db/0773914b3ec4a56e
  Caused by: process didn't exit successfully: `/usr/bin/git fetch --force --update-head-ok 'https://localhost:8443/repo.git' '+HEAD:refs/remotes/origin/HEAD'` (exit status: 128)
--- stderr
fatal: unable to access 'https://localhost:8443/repo.git/': SSL certificate problem: self signed certificate
```

- Verified that `cargo run pip install
git+https://localhost:8443/repo.git --allow-insecure-host
https://localhost:8443` continued further.
2025-02-04 10:57:57 -05:00
Charlie Marsh 5c7fba86e1 Deduplicate GitSha and GitOid types (#10802)
## Summary

I think this split is leftover from using `libgit2`. I kept `Oid` since
that seems to be the official terminology.
2025-01-21 09:15:11 -05:00
Charlie Marsh 154fd6bd23 Make GitOid a crate-public type (#10799) 2025-01-21 01:41:35 +00:00
Charlie Marsh 5e5cb8797d Remove unused Git resolver methods (#10754) 2025-01-19 17:02:53 +00:00
Charlie Marsh 66a603b6c4 Use dynamic dispatch to simplify reporters (#10086)
## Summary

Sort of undecided on this. These are already stored as `dyn Reporter` in
each struct, so we're already using dynamic dispatch in that sense. But
all the methods take `impl Reporter`. This is sometimes nice (the
callsites are simpler?), but it also means that in practice, you often
_can't_ pass `None` to these methods that accept `Option<impl
Reporter>`, because Rust can't infer the generic type.

Anyway, this adds more consistency and simplifies the setup by using
`Arc<dyn Reporter>` everywhere.
2025-01-06 12:04:00 -05:00
Charlie Marsh cc734ea2b6 Allow dependency metadata entries for direct URL requirements (#7846)
## Summary

This is part of making
https://github.com/astral-sh/uv/issues/7299#issuecomment-2385286341
better. You can now use `tool.uv.dependency-metadata` for direct URL
requirements. Unfortunately, you _must_ include a version, since we need
one to perform resolution.
2024-10-22 22:01:23 -04:00
Charlie Marsh 14507a1793 Add uv- prefix to all internal crates (#7853)
## Summary

Brings more consistency to the repo and ensures that all crates
automatically show up in `--verbose` logging.
2024-10-01 20:15:32 -04:00
Zanie Blue 4a98e1eceb Avoid using debug representation for git source urls (#6779)
e.g.

> DEBUG Using existing Git source
`https://github.com/StarfishStorage/python-swiftclient.git`

instead of

> DEBUG Using existing git source `Url { scheme: "https",
cannot_be_a_base: false, username: "", password: None, host:
Some(Domain("github.com")), port: None, path:
"/StarfishStorage/gunicorn.git", query: None, fragment: None }`
2024-08-28 22:02:24 +00:00
Charlie Marsh 8fac63d4ce Redact Git credentials from pyproject.toml (#6074)
## Summary

We retain them if you use `--raw-sources`, but otherwise they're
removed. We still respect them in the subsequent `uv.lock` via an
in-process store.

Closes #6056.
2024-08-14 01:30:02 +00:00
Charlie Marsh 561625ed8c Use hasher to compute resolution hash (#5495)
## Summary

Addressing one TODO. This should be more efficient.
2024-07-26 23:24:09 +00:00
Charlie Marsh 9905521957 Use shared client in Git fetch implementation (#4487)
## Summary

It turns out that the Git fetch implementation is initializing its own
client, which can be really expensive on macOS (due to loading native
certificates) _and_ bypasses any of our middleware. This PR modifies the
Git implementation to accept a shared client.
2024-06-24 17:09:29 -04:00
Ibraheem Ahmed 261aa2c70a Port all git functionality to use git CLI (#3833)
## Summary

We currently rely on libgit2 for most git-related functionality.
However, libgit2 has long-standing performance issues, as well as lags
significantly behind git in terms of new features. For these reasons we
now use the git CLI by default for fetching repositories
(https://github.com/astral-sh/uv/pull/1781). This PR completely drops
libgit2 in favor of the git CLI for all git-related functionality, which
should allow us to use features such as partial clones and sparse
checkouts in the future for performance.

There is also a lot of technical debt in the current git code as it's
mostly taken from Cargo. Switching to the git CLI *vastly* simplifies
the `uv-git` codebase.

Eventually we might want to look into switching to
[`gitoxide`](https://github.com/Byron/gitoxide), but it's currently too
immature for our use case.
2024-05-30 15:28:48 -04:00
konsti c85c52d4ce Unify packse find links urls (#2969)
The sync scenarios script is broken, so i did the updates manually

```
$ ./scripts/sync_scenarios.sh
Setting up a temporary environment...
Using Python 3.12.1 interpreter at: /home/konsti/projects/uv/.venv/bin/python3
Creating virtualenv at: .venv
Activate with: source .venv/bin/activate
  × No solution found when resolving dependencies:
  ╰─▶ Because docutils==0.21.post1 is unusable because the package metadata was inconsistent and you require docutils==0.21.post1, we can conclude that the requirements are unsatisfiable.

      hint: Metadata for docutils==0.21.post1 was inconsistent:
        Package metadata version `0.21` does not match given version `0.21.post1`
```

---------

Co-authored-by: Zanie Blue <contact@zanie.dev>
2024-04-11 08:35:22 +00:00
Charlie Marsh 684f790d5d Preserve .git suffixes and casing in Git dependencies (#2789)
## Summary

I noticed in #2769 that I was now stripping `.git` suffixes from Git
URLs after resolving to a precise commit. This PR cleans up the internal
caching to use a better canonical representation: a `RepositoryUrl`
along with a `GitReference`, instead of a `GitUrl` which can contain
non-canonical data. This gives us both better fidelity (preserving the
`.git`, along with any casing that the user provided when defining the
URL) and is overall cleaner and more robust.
2024-04-03 00:24:29 +00:00
konsti 70dad51cd9 Remove spawn_blocking from version map (#1966)
I previously add `spawn_blocking` to the version map construction as it
had become a bottleneck
(https://github.com/astral-sh/uv/pull/1163/files#diff-704ceeaedada99f90369eac535713ec82e19550bff166cd44745d7277ecae527R116).
With the zero copy deserialization, this has become so fast we don't
need to move it to the thread pool anymore. I've also checked
`DataWithCachePolicy` but it seems to still take a significant amount of
time. Span visualization:

Resolving jupyter warm:

![image](https://github.com/astral-sh/uv/assets/6826232/692b03da-61c5-4f96-b413-199c14aa47c4)

Resolving jupyter cold:

![image](https://github.com/astral-sh/uv/assets/6826232/a6893155-d327-40c9-a83a-7c537b7c99c4)

![image](https://github.com/astral-sh/uv/assets/6826232/213556a3-a331-42db-aaf5-bdef5e0205dd)

I've also updated the instrumentation a little.

We don't seem cpu bound for the cold cache (top) and refresh case
(bottom) from jupyter:

![image](https://github.com/astral-sh/uv/assets/6826232/cb976add-3d30-465a-a470-8490b7b6caea)

![image](https://github.com/astral-sh/uv/assets/6826232/d7ecb745-dd2d-4f91-939c-2e46b7c812dd)
2024-02-26 09:44:24 +00:00
Zanie Blue 71ec568d0f Use git command to fetch repositories instead of libgit2 for robust SSH support (#1781)
Closes https://github.com/astral-sh/uv/issues/1775
Closes https://github.com/astral-sh/uv/issues/1452
Closes https://github.com/astral-sh/uv/issues/1514
Follows https://github.com/astral-sh/uv/pull/1717

libgit2 does not support host names with extra identifiers during SSH
lookup (e.g. [`github.com-some_identifier`](

https://docs.github.com/en/authentication/connecting-to-github-with-ssh/managing-deploy-keys#using-multiple-repositories-on-one-server))
so we use the `git` command instead for fetching. This is required for
`pip` parity.

See the [Cargo
documentation](https://doc.rust-lang.org/nightly/cargo/reference/config.html#netgit-fetch-with-cli)
for more details on using the `git` CLI instead of libgit2. We may want
to try to use libgit2 first in the future, as it is more performant
(#1786).

We now support authentication with:

```
git+ssh://git@<hostname>/...
git+ssh://git@<hostname>-<identifier>/...
```

Tested with a deploy key e.g.

```
cargo run -- \
    pip install uv-private-pypackage@git+ssh://git@github.com-test-uv-private-pypackage/astral-test/uv-private-pypackage.git \
    --reinstall --no-cache -v
```

and

```
cargo run -- \
    pip install uv-private-pypackage@git+ssh://git@github.com/astral-test/uv-private-pypackage.git \
    --reinstall --no-cache -v     
```

with a ssh config like

```
Host github.com
        Hostname github.com
        IdentityFile=/Users/mz/.ssh/id_ed25519

Host github.com-test-uv-private-pypackage
        Hostname github.com
        IdentityFile=/Users/mz/.ssh/id_ed25519
```

It seems quite hard to add test coverage for this to the test suite, as
we'd need to add the SSH key and I don't know how to isolate that from
affecting other developer's machines.
2024-02-21 12:44:32 -06:00
Zanie Blue 2586f655bb Rename to uv (#1302)
First, replace all usages in files in-place. I used my editor for this.
If someone wants to add a one-liner that'd be fun.

Then, update directory and file names:

```
# Run twice for nested directories
find . -type d -print0 | xargs -0 rename s/puffin/uv/g
find . -type d -print0 | xargs -0 rename s/puffin/uv/g

# Update files
find . -type f -print0 | xargs -0 rename s/puffin/uv/g
```

Then add all the files again

```
# Add all the files again
git add crates
git add python/uv

# This one needs a force-add
git add -f crates/uv-trampoline
```
2024-02-15 11:19:46 -06:00