Closes https://github.com/astral-sh/uv/issues/11288
I tested the reproduction there manually.
I'm a little uncertain about this behavior, it's not true to the spirit
of `--python <dir>` selecting a target environment but this method is
only used to see if an existing environment matches for the purpose of
invalidation in projects and tools where I think we always force a
separate environment anyway?
Closes https://github.com/astral-sh/uv/issues/11214
Special-cases the first Python executable we find on the `PATH`,
allowing it to be considered during searches for virtual environments.
For some context, there are two stages to Python interpreter discovery
1. We find possible Python executables in various sources
2. We query the executables to determine canonical metadata about the
interpreter
We can't really be "sure" if an executable is a complaint virtual
environment during (1), we need to query the interpreter first. This
means that if you're only allowed to installed into virtual
environments, we'll query every interpreter on your PATH. This is not
performant, and causes confusion for users. Notably, I recently improved
error messaging when we can't find any valid interpreters, by showing
the error message we encounter while querying an interpreter (if any).
However, this is problematic when there's an error for an interpreter
that is not relevant to your search. In
https://github.com/astral-sh/uv/pull/11143, I added filtering to avoid
querying additional interpreters, but that regressed some user
experiences where they were relying on us finding implicitly active
virtual environments via the PATH.
## Summary
In preview mode on windows, register und un-register the managed python build standalone installations in the Windows registry following PEP 514.
We write the values defined in the PEP plus the download URL and hash. We add an entry when installing a version, remove an entry when uninstalling and removing all values when uninstalling with `--all`. We update entries only by overwriting existing values, there is no "syncing" involved.
Since they are not official builds, pbs gets a prefix. `py -V:Astral/CPython3.13.1` works, `py -3.13` doesn't.
```
$ py --list-paths
-V:3.12 * C:\Users\Konsti\AppData\Local\Programs\Python\Python312\python.exe
-V:3.11.9 C:\Users\Konsti\.pyenv\pyenv-win\versions\3.11.9\python.exe
-V:3.11 C:\Users\micro\AppData\Local\Programs\Python\Python311\python.exe
-V:3.8 C:\Users\micro\AppData\Local\Programs\Python\Python38\python.exe
-V:Astral/CPython3.13.1 C:\Users\Konsti\AppData\Roaming\uv\data\python\cpython-3.13.1-windows-x86_64-none\python.exe
```
Registry errors are reported but not fatal, except for operations on the company key since it's not bound to any specific python interpreter.
On uninstallation, we prune registry entries that have no matching Python installation (i.e. broken entries).
The code uses the official `windows_registry` crate of the `winreg` crate.
Best reviewed commit-by-commit.
## Test Plan
We're reusing an existing system check to test different (un)installation scenarios.
Previously, these errors would only be visible in the debug logs as
"Skipping bad interpreter ..." which can lead us to making some
ridiculous claims like "There is no virtual environment" or "Python is
not installed" when really we just failed to query the interpreter for
some reason.
We show the first error, sort of arbitrarily — but I think it matches
user expectation, i.e., this would be the first Python on your PATH.
Related to #10713
## Summary
This PR reimplements
[`sysconfigpatcher`](https://github.com/bluss/sysconfigpatcher) in Rust
and applies it to our Python installations at install-time, ensuring
that the `sysconfig` data is more likely to be correct.
For now, we only rewrite prefixes (i.e., any path that starts with
`/install` gets rewritten to the correct absolute path for the current
machine).
Unlike `sysconfigpatcher`, this PR does not yet do any of the following:
- Patch `pkginfo` files.
- Change `clang` references to `cc`.
A few things that we should do as follow-ups, in my opinion:
1. Rewrite
[`AR`](https://github.com/bluss/sysconfigpatcher/blob/c1ebf8ab9274dcde255484d93ce0f1fd1f76a248/src/sysconfigpatcher.py#L61).
2. Remove `-isysroot`, which we already do for newer builds.
## Summary
Fix#8075.
Invalid discovered environments in the working directory should be
filtered out.
## Test Plan
- Test python_find
---------
Co-authored-by: Zanie Blue <contact@zanie.dev>
## Summary
This PR adds `--install-dir` argument for the following commands:
- `uv python install`
- `uv python uninstall`
The `UV_PYTHON_INSTALL_DIR` env variable can be used to set it
(previously it was also used internally).
Any more commands we would want to add this to?
## Test Plan
For now just manual test (works on my machine hehe)
```
❯ ./target/debug/uv python install --install-dir /tmp/pythons 3.8.12
Searching for Python versions matching: Python 3.8.12
Installed Python 3.8.12 in 4.31s
+ cpython-3.8.12-linux-x86_64-gnu
❯ /tmp/pythons/cpython-3.8.12-linux-x86_64-gnu/bin/python --help
usage: /tmp/pythons/cpython-3.8.12-linux-x86_64-gnu/bin/python [option] ... [-c cmd | -m mod | file | -] [arg] ...
```
Open to add some tests after the initial feedback.
---------
Co-authored-by: Zanie Blue <contact@zanie.dev>
## Summary
I'm not sure why this hasn't come up before... But it looks like this
method is only looking at `python.exe` and `python3.exe`? From the user
screenshots, the `python3.12.exe` and `python3.13.exe` are also present,
though.
Closes https://github.com/astral-sh/uv/issues/9667.
The `SysVersion` registry entry may or may not include the patch
version, so if we encounter a registry entry without a patch version, we
must not assume that the patch version is 0.
```
Name Property
---- --------
3.9 DisplayName : Python 3.9 (64-bit)
SupportUrl : https://www.python.org/
Version : 3.9.13
SysVersion : 3.9
SysArchitecture : 64bit
Hive: HKEY_CURRENT_USER\Software\Python\PythonCore\3.9
```
Confirmed the fix manually.
Fixes#9668
## Summary
A lot of good new lints, and most importantly, error stabilizations. I
tried to find a few usages of the new stabilizations, but I'm sure there
are more.
IIUC, this _does_ require bumping our MSRV.
## Summary
These were moved as part of a broader refactor to create a single
integration test module. That "single integration test module" did
indeed have a big impact on compile times, which is great! But we aren't
seeing any benefit from moving these tests into their own files (despite
the claim in [this blog
post](https://matklad.github.io/2021/02/27/delete-cargo-integration-tests.html),
I see the same compilation pattern regardless of where the tests are
located). Plus, we don't have many of these, and same-file tests is such
a strong Rust convention.
This restores behavior previously removed in
https://github.com/astral-sh/uv/pull/7649.
I thought it'd be clearer (and simpler) to have a consistent Python
executable name ordering. However, we've seen some cases where this can
be surprising and, in combination with #8481, can result in incorrect
behavior. For example, see https://github.com/astral-sh/uv/issues/9046
where we prefer `python3` over `python3.12` in the same directory even
though `python3.12` was requested. While `python3` and `python3.12` both
point to valid Python 3.12 environments there, the expectation is that
when `python3.12` is requested that the `python3.12` executable is
preferred. This expectation may be less obvious if the user requests
`python@3.12`, but uv does not distinguish between these request forms.
Similarly, this may be surprising as by default uv prefers `python` over
`python3` but when requesting `python3.12` the preference will be
swapped.
Implement a full working version of local version semantics. The (AFAIA)
major move towards this was implemented in #2430. This added support
such that the version specifier `torch==2.1.0+cpu` would install
`torch@2.1.0+cpu` and consider `torch@2.1.0+cpu` a valid way to satisfy
the requirement `torch==2.1.0` in further dependency resolution.
In this feature, we more fully support local version semantics. Namely,
we now allow `torch==2.1.0` to install `torch@2.1.0+cpu` regardless of
whether `torch@2.1.0` (no local tag) actually exists.
We do this by adding an internal-only `Max` value to local versions that
compare greater to all other local versions. Then we can translate
`torch==2.1.0` into bounds: greater than 2.1.0 with no local tag and
less than 2.1.0 with the `Max` local tag.
Depends on https://github.com/astral-sh/packse/pull/227.
e.g.
```
❯ echo "anyio" | cargo run -q -- pip compile - -v
DEBUG uv 0.4.30 (107ab3d71 2024-11-07)
DEBUG Starting Python discovery for a default Python
DEBUG Looking for exact match for request a default Python
DEBUG Searching for default Python interpreter in virtual environments, managed installations, or search path
DEBUG Found `cpython-3.12.7-macos-aarch64-none` at `/Users/zb/workspace/uv/.venv/bin/python3` (virtual environment)
```
```
❯ cargo run -q -- pip install anyio -v
DEBUG uv 0.4.30 (107ab3d71 2024-11-07)
DEBUG Searching for default Python interpreter in virtual environments
DEBUG Found `cpython-3.12.7-macos-aarch64-none` at `/Users/zb/workspace/uv/.venv/bin/python3` (virtual environment)
```
vs
```
❯ uv pip install anyio -v
DEBUG uv 0.4.30 (61ed2a236 2024-11-04)
DEBUG Searching for default Python interpreter in system path
DEBUG Found `cpython-3.12.7-macos-aarch64-none` at `/Users/zb/workspace/uv/.venv/bin/python3` (virtual environment)
```
```
❯ echo "anyio" | uv pip compile - -v
DEBUG uv 0.4.30 (61ed2a236 2024-11-04)
DEBUG Starting Python discovery for a default Python
DEBUG Looking for exact match for request a default Python
DEBUG Searching for default Python interpreter in managed installations or system path
DEBUG Found `cpython-3.12.7-macos-aarch64-none` at `/Users/zb/workspace/uv/.venv/bin/python3` (virtual environment)
```
Updates `uv python install` to link `python3.x` in the executable
directory (i.e., `~/.local/bin`) to the the managed interpreter path.
Includes
- #8569
- #8571
Remaining work
- #8663
- #8650
- Add an opt-out setting and flag
- Update documentation
Closes https://github.com/astral-sh/uv/issues/8228
e.g., on this branch
```
❯ uv python install 3.13t 3.13
❯ cargo build
❯ cargo run -q --bin uvx -- --from build python -c "import sys; print(sys.base_prefix)"
/Users/zb/.local/share/uv/python/cpython-3.13.0-macos-aarch64-none
❯ cargo run -q --bin uvx -- -p 3.13 --from build python -c "import sys; print(sys.base_prefix)"
/Users/zb/.local/share/uv/python/cpython-3.13.0-macos-aarch64-none
❯ cargo run -q --bin uvx -- -p 3.13t --from build python -c "import sys; print(sys.base_prefix)"
/Users/zb/.local/share/uv/python/cpython-3.13.0+freethreaded-macos-aarch64-none
```
and on main
```
❯ cargo build
❯ cargo run -q --bin uvx -- --from build python -c "import sys; print(sys.base_prefix)"
Installed 3 packages in 12ms
/Users/zb/.local/share/uv/python/cpython-3.13.0+freethreaded-macos-aarch64-none
```
I want to add more test coverage around this, but I've noticed the
free-threaded discovery tests are a bit off as-is and it'll be a bigger
task. I think the recent bugs around discovery indicate we should invest
more into that test framework.
As mentioned in https://github.com/astral-sh/uv/issues/8189
We only checked if an interpreter was free-threaded _when_ free-threaded
variants were requested. But we should not use free-threaded
interpreters unless explicitly requested.
## Summary
This PR declares and documents all environment variables that are used
in one way or another in `uv`, either internally, or externally, or
transitively under a common struct.
I think over time as uv has grown there's been many environment
variables introduced. Its harder to know which ones exists, which ones
are missing, what they're used for, or where are they used across the
code. The docs only documents a handful of them, for others you'd have
to dive into the code and inspect across crates to know which crates
they're used on or where they're relevant.
This PR is a starting attempt to unify them, make it easier to discover
which ones we have, and maybe unlock future posibilities in automating
generating documentation for them.
I think we can split out into multiple structs later to better organize,
but given the high influx of PR's and possibly new environment variables
introduced/re-used, it would be hard to try to organize them all now
into their proper namespaced struct while this is all happening given
merge conflicts and/or keeping up to date.
I don't think this has any impact on performance as they all should
still be inlined, although it may affect local build times on changes to
the environment vars as more crates would likely need a rebuild. Lastly,
some of them are declared but not used in the code, for example those in
`build.rs`. I left them declared because I still think it's useful to at
least have a reference.
Did I miss any? Are their initial docs cohesive?
Note, `uv-static` is a terrible name for a new crate, thoughts? Others
considered `uv-vars`, `uv-consts`.
## Test Plan
Existing tests
As per
https://matklad.github.io/2021/02/27/delete-cargo-integration-tests.html
Before that, there were 91 separate integration tests binary.
(As discussed on Discord — I've done the `uv` crate, there's still a few
more commits coming before this is mergeable, and I want to see how it
performs in CI and locally).
## Summary
Closes#7977. Makes `PythonDownloadRequest` account for the prerelease
part if allowed. Also stores the prerelease in `PythonInstallationKey`
directly as a `Prerelease` rather than a string.
## Test Plan
Correctly picks the relevant prerelease (rather than picking the most
recent one):
```
λ cargo run python install 3.13.0rc2
Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.17s
Running `target/debug/uv python install 3.13.0rc2`
Searching for Python versions matching: Python 3.13rc2
cpython-3.13.0rc2-macos-aarch64-none ------------------------------ 457.81 KiB/14.73 MiB ^C
λ cargo run python install 3.13.0rc3
Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.17s
Running `target/debug/uv python install 3.13.0rc3`
Searching for Python versions matching: Python 3.13rc3
Found existing installation for Python 3.13rc3: cpython-3.13.0rc3-macos-aarch64-none
```
Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>
## Summary
This PR adds the ability to list available scripts in the environment
when `uv run` is invoked without any arguments.
It somewhat mimics the behavior of `rye run` command
(See https://rye.astral.sh/guide/commands/run).
This is an attempt to fix#4024.
## Test Plan
I added test cases. The CI pipeline should pass.
### Manuel Tests
```shell
❯ uv run
Provide a command or script to invoke with `uv run <command>` or `uv run script.py`.
The following scripts are available:
normalizer
python
python3
python3.12
See `uv run --help` for more information.
```
---------
Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>
Co-authored-by: Zanie Blue <contact@zanie.dev>
Closes#7118
This only really affects managed interpreters, as we exclude alternative
Python implementations from the search path during the
`VersionRequest::executable_names` part of discovery.
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
closes#4828
First iteration for an implementation. I need to add more tests but
wanted your opinion on the implementation first.
<!-- What's the purpose of the change? What does it do, and why? -->
## Test Plan
Currently tested using the following command but will add tests shortly:
```console
D:\repo\uv> cargo run venv -p 3.13t && .venv\Scripts\python.exe
Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.52s
Running `target\debug\uv.exe venv -p 3.13t`
Using Python 3.13.0rc1 interpreter at: C:\Users\bschoen\AppData\Local\Programs\Python\Python313\python3.13t.exe
Creating virtualenv at: .venv
Activate with: .venv\Scripts\activate
Python 3.13.0rc1 experimental free-threading build (tags/v3.13.0rc1:e4a3e78, Jul 31 2024, 21:06:58) [MSC v.1940 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>
```
---------
Co-authored-by: Zanie Blue <contact@zanie.dev>
There are two parts to this.
The first is a restructuring and refactoring. We had some debt around
expected executable name generation, which we address here by
consolidating into a single function that generates a combination of
names. This includes a bit of extra code around free-threaded variants
because this was written on top of #7431 — I'll rebase that on top of
this.
The second addresses some bugs around alternative implementations.
Notably, `uv python list` does not discovery executables with
alternative implementation names. Now, we properly generate all of the
executable names for `VersionRequest::Any` (originally implemented in
https://github.com/astral-sh/uv/pull/7508) to properly show all the
implementations we can find:
```
❯ cargo run -q -- python list --no-python-downloads
cpython-3.12.6-macos-aarch64-none /opt/homebrew/opt/python@3.12/bin/python3.12 -> ../Frameworks/Python.framework/Versions/3.12/bin/python3.12
cpython-3.11.10-macos-aarch64-none /opt/homebrew/opt/python@3.11/bin/python3.11 -> ../Frameworks/Python.framework/Versions/3.11/bin/python3.11
cpython-3.9.6-macos-aarch64-none /Library/Developer/CommandLineTools/usr/bin/python3 -> ../../Library/Frameworks/Python3.framework/Versions/3.9/bin/python3
pypy-3.10.14-macos-aarch64-none /opt/homebrew/bin/pypy3 -> ../Cellar/pypy3.10/7.3.17/bin/pypy3
```
While doing both of these changes, I ended up changing the priority of
interpreter discovery slightly. For example, given that the executables
are in the same directory, do we query `python` or `python3.10` first
when you request `--python 3.10`? Previously, we'd check `python3.10`
but I think that was an incorrect optimization. I think we should always
prefer the bare name (i.e. `python`) first. Similarly, this applies to
`python` and an executable for an alternative implementation like
`pypy`. If it's not compatible with the request, we'll skip it anyway.
We might have to query more interpreters with this approach but it seems
rare.
Closes https://github.com/astral-sh/uv/issues/7286 superseding
https://github.com/astral-sh/uv/pull/7508
Zanie Blue