## Summary
This adds warnings to both our steam and sync ZIP handling on ZIP
entries that aren't "well-known." For now, "well-known" means stored
(i.e. no compression), DEFLATE, or zstd.
In practice we have duplicated codepaths for this check: one for the
"sync" (pre-downloaded) path, and one for the streaming path.
See #16911 and #17467 for context.
## Test Plan
Will update snapshots if/when they change. I'll also add a ZIP test for
this.
(Upd: added some "futzed" wheels for the ZIP tests.)
---------
Signed-off-by: William Woodruff <william@astral.sh>
We were using slightly different retry code in multiple places, this PR
unifies it.
Also fixes retry undercounting in publish if the retry middleware was
involved.
---------
Co-authored-by: Tomasz Kramkowski <tom@astral.sh>
Resolves https://github.com/astral-sh/uv/issues/16616
This PR detects managed prerelease interpreters during discovery and
warns when a matching stable build is available, wiring the new helper
into `PythonInstallation::find`, `find_best`, and `find_or_download`.
The `uv-python` crate reads the checked-in download metadata to write
out a minified version to be statically included into the binary.
Sometimes, the resulting `download-metadata-minified.json` can go
missing, such as when `uv` is installed via `cargo` as a git source, and
there is an update; `cargo` will do a hard reset on the git repo it
stores in the cache, causing it to be deleted. In these instances, the
`uv-python` crate does not currently re-run its build script, because as
it happens, there was no change to the `download-metadata.json` since it
was last compiled. This causes a build error like:
```
Updating git repository `https://github.com/astral-sh/uv`
Installing uv v0.8.24 (https://github.com/astral-sh/uv?tag=0.8.24#252f8873)
Updating crates.io index
Compiling uv-version v0.8.24 (/home/skhawtho/.cargo/git/checkouts/uv-c9e40703e19509a8/252f887/crates/uv-version)
Compiling uv-git v0.0.1 (/home/skhawtho/.cargo/git/checkouts/uv-c9e40703e19509a8/252f887/crates/uv-git)
Compiling uv-build-backend v0.1.0 (/home/skhawtho/.cargo/git/checkouts/uv-c9e40703e19509a8/252f887/crates/uv-build-backend)
Compiling uv-configuration v0.0.1 (/home/skhawtho/.cargo/git/checkouts/uv-c9e40703e19509a8/252f887/crates/uv-configuration)
Compiling uv-client v0.0.1 (/home/skhawtho/.cargo/git/checkouts/uv-c9e40703e19509a8/252f887/crates/uv-client)
Compiling uv-extract v0.0.1 (/home/skhawtho/.cargo/git/checkouts/uv-c9e40703e19509a8/252f887/crates/uv-extract)
Compiling uv-workspace v0.0.1 (/home/skhawtho/.cargo/git/checkouts/uv-c9e40703e19509a8/252f887/crates/uv-workspace)
Compiling uv-python v0.0.1 (/home/skhawtho/.cargo/git/checkouts/uv-c9e40703e19509a8/252f887/crates/uv-python)
Compiling uv-requirements-txt v0.0.1 (/home/skhawtho/.cargo/git/checkouts/uv-c9e40703e19509a8/252f887/crates/uv-requirements-txt)
Compiling uv-bin-install v0.0.1 (/home/skhawtho/.cargo/git/checkouts/uv-c9e40703e19509a8/252f887/crates/uv-bin-install)
Compiling uv-publish v0.1.0 (/home/skhawtho/.cargo/git/checkouts/uv-c9e40703e19509a8/252f887/crates/uv-publish)
error: couldn't read `crates/uv-python/src/download-metadata-minified.json`: No such file or directory (os error 2)
--> crates/uv-python/src/downloads.rs:795:45
|
795 | const BUILTIN_PYTHON_DOWNLOADS_JSON: &str = include_str!("download-metadata-minified.json");
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
error: could not compile `uv-python` (lib) due to 1 previous error
error: failed to compile `uv v0.8.24 (https://github.com/astral-sh/uv?tag=0.8.24#252f8873)`, intermediate artifacts can be found at `/home/skhawtho/.cache/cargo`.
To reuse those artifacts with a future compilation, set the environment variable `CARGO_TARGET_DIR` to that path.
```
This fixes the issue by also adding a `rerun-if-changed` on the
resulting minified JSON file.
Separately, this revision also changes the output directory of the
minified file to the build script's output directory via the `OUT_DIR`
environment variable, so as not to pollute the source code.
## Summary
I spent time trying to figure out how to support this but came up empty.
It _seems_ like maybe the `DefaultFields` implementation in
`tracing-subscriber` uses debug formatting for fields...? So if you have
a string with ANSI codes, they end up printing as unformatted values? I
even reverted all our custom formatting in `logging.rs` and saw the same
thing.
Closes https://github.com/astral-sh/uv/issues/15840.
When there is an error during the streaming download and unpack for
Python interpreter and bin installs, we would previously fail, causing a
lot of CI flakes on GitHub Actions.
The problem was that the error is not one of the extended IO errors we
were previously handling, but a regular reqwest error, nested below
layers of errors of other crates processing the stream, including some
IO errors. We now handle nested reqwest errors, too.
This surfaced another problem: Our manual retry loop couldn't inform the
retry middleware that it already performed the limit of retries, and
that the middleware should not retry anymore. While too many retries are
more a problem for debugging than for the user, this causes confusing
error output. To work around this, we disable the retries in the client
and handle all retry errors in our loop.
Fixes https://github.com/astral-sh/uv/issues/14171
Co-authored-by: Charlie Marsh <charlie.r.marsh@gmail.com>
Allows pinning the Python build version via environment variables, e.g.,
`UV_PYTHON_CPYTHON_BUILD=...`. Each variable is implementation specific,
because they use different versioning schemes.
Updates the Python download metadata to include a `build` string, so we
can filter downloads by the pin. Writes the build version to a file in
the managed install, e.g., `cpython-3.10.18-macos-aarch64-none/BUILD`,
so we can filter installed versions by the pin.
Some important follow-up here:
- Include the build version in not found errors (when pinned)
- Automatically use a remote list of Python downloads to satisfy build
versions not present in the latest embedded download metadata
Some less important follow-ups to consider:
- Allow using ranges for build version pins
We should not unnecessarily leak memory. Instead, we follow the general
patterns and use `Cow` for strings that can be from either a static or a
dynamic source.
This allows `PythonDownloadRequest` which is used for parsing general
install key requests to have missing segments, which unblocks requests
like `windows-aarch64` or `cpython-linux` (whereas before those would
require `any-any-windows-aarch64` and `cpython-any-linux` respectively).
We still require strict ordering of segments.
Previously, we only allowed missing segments at the end of the key.
This uses a state machine for parsing, which is quite a bit more
complicated.
I'm a little hesitant about the possibility that this regresses error
messages and the complexity of the implementation, but `uv run -p
aarch64` seems valuable following #13724. The alternative to this would
probably be to make these explicit in various places? e.g., expose
`--python-arch`, `--python-libc`, and `--python-os`? Or make
`--python-platform` (which already exists) accept a subset of the keys?
There is a possibility of regressions here, e.g., if something matches
this parser it will not fallback to the `PythonRequest::ExecutableName`
case and we've made this parser more permissive, but I think that should
be quite rare?
## Summary
Make the use of `Self` consistent. Mostly done by running `cargo clippy
--fix -- -A clippy::all -W clippy::use_self`.
## Test Plan
<!-- How was it tested? -->
No need.
If/when we see https://github.com/astral-sh/uv/issues/14171 again, this
should clarify whether our retry logic was skipped (i.e. a transient
error wasn't correctly identified as transient), or whether we exhausted
our retries. Previously, if you ran a local example fileserver as in
https://github.com/astral-sh/uv/issues/14171#issuecomment-3014580701 and
then you tried to install Python from it, you'd get:
```
$ export UV_TEST_NO_CLI_PROGRESS=1
$ uv python install 3.8.20 --mirror http://localhost:8000 2>&1 | cat
error: Failed to install cpython-3.8.20-linux-x86_64-gnu
Caused by: Failed to extract archive: cpython-3.8.20-20241002-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz
Caused by: failed to unpack `/home/jacko/.local/share/uv/python/.temp/.tmpS4sHHZ/python/lib/libpython3.8.so.1.0`
Caused by: failed to unpack `python/lib/libpython3.8.so.1.0` into `/home/jacko/.local/share/uv/python/.temp/.tmpS4sHHZ/python/lib/libpython3.8.so.1.0`
Caused by: error decoding response body
Caused by: request or response body error
Caused by: error reading a body from connection
Caused by: Connection reset by peer (os error 104)
```
With this change you get:
```
error: Failed to install cpython-3.8.20-linux-x86_64-gnu
Caused by: Request failed after 3 retries
Caused by: Failed to extract archive: cpython-3.8.20-20241002-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz
Caused by: failed to unpack `/home/jacko/.local/share/uv/python/.temp/.tmp4Ia24w/python/lib/libpython3.8.so.1.0`
Caused by: failed to unpack `python/lib/libpython3.8.so.1.0` into `/home/jacko/.local/share/uv/python/.temp/.tmp4Ia24w/python/lib/libpython3.8.so.1.0`
Caused by: error decoding response body
Caused by: request or response body error
Caused by: error reading a body from connection
Caused by: Connection reset by peer (os error 104)
```
At the same time, I'm updating the way we handle the retry count to
avoid nested retry loops exceeding the intended number of attempts, as I
mentioned at
https://github.com/astral-sh/uv/issues/14069#issuecomment-3020634281.
It's not clear to me whether we actually want this part of the change,
and I need feedback here.
> NOTE: The PRs that were merged into this feature branch have all been
independently reviewed. But it's also useful to see all of the changes
in their final form. I've added comments to significant changes
throughout the PR to aid discussion.
This PR introduces transparent Python version upgrades to uv, allowing
for a smoother experience when upgrading to new patch versions.
Previously, upgrading Python patch versions required manual updates to
each virtual environment. Now, virtual environments can transparently
upgrade to newer patch versions.
Due to significant changes in how uv installs and executes managed
Python executables, this functionality is initially available behind a
`--preview` flag. Once an installation has been made upgradeable through
`--preview`, subsequent operations (like `uv venv -p 3.10` or patch
upgrades) will work without requiring the flag again. This is
accomplished by checking for the existence of a minor version symlink
directory (or junction on Windows).
### Features
* New `uv python upgrade` command to upgrade installed Python versions
to the latest available patch release:
```
# Upgrade specific minor version
uv python upgrade 3.12 --preview
# Upgrade all installed minor versions
uv python upgrade --preview
```
* Transparent upgrades also occur when installing newer patch versions:
```
uv python install 3.10.8 --preview
# Automatically upgrades existing 3.10 environments
uv python install 3.10.18
```
* Support for transparently upgradeable Python `bin` installations via
`--preview` flag
```
uv python install 3.13 --preview
# Automatically upgrades the `bin` installation if there is a newer patch version available
uv python upgrade 3.13 --preview
```
* Virtual environments can still be tied to a patch version if desired
(ignoring patch upgrades):
```
uv venv -p 3.10.8
```
### Implementation
Transparent upgrades are implemented using:
* Minor version symlink directories (Unix) or junctions (Windows)
* On Windows, trampolines simulate paths with junctions
* Symlink directory naming follows Python build standalone format: e.g.,
`cpython-3.10-macos-aarch64-none`
* Upgrades are scoped to the minor version key (as represented in the
naming format: implementation-minor version+variant-os-arch-libc)
* If the context does not provide a patch version request and the
interpreter is from a managed CPython installation, the `Interpreter`
used by `uv python run` will use the full symlink directory executable
path when available, enabling transparently upgradeable environments
created with the `venv` module (`uv run python -m venv`)
New types:
* `PythonMinorVersionLink`: in a sense, the core type for this PR, this
is a representation of a minor version symlink directory (or junction on
Windows) that points to the highest installed managed CPython patch
version for a minor version key.
* `PythonInstallationMinorVersionKey`: provides a view into a
`PythonInstallationKey` that excludes the patch and prerelease. This is
used for grouping installations by minor version key (e.g., to find the
highest available patch installation for that minor version key) and for
minor version directory naming.
### Compatibility
* Supports virtual environments created with:
* `uv venv`
* `uv run python -m venv` (using managed Python that was installed or
upgraded with `--preview`)
* Virtual environments created within these environments
* Existing virtual environments from before these changes continue to
work but aren't transparently upgradeable without being recreated
* Supports both standard Python (`python3.10`) and freethreaded Python
(`python3.10t`)
* Support for transparently upgrades is currently only available for
managed CPython installations
Closes#7287Closes#7325Closes#7892Closes#9031Closes#12977
---------
Co-authored-by: Zanie Blue <contact@zanie.dev>
Using a companion change in the middleware
(https://github.com/TrueLayer/reqwest-middleware/pull/235, forked&tagged
pending review), we can check and show retries for HTTP status core
errors, to consistently report retries again.
We fix two cases:
* Show retries for status code errors for cache client requests
* Show retries for status code errors for Python download requests
Not handled:
* Show previous retries when a distribution download fails mid-streaming
* Perform retries when a distribution download fails mid-streaming
* Show previous retries when a Python download fails mid-streaming
* Perform retries when a Python download fails mid-streaming
This PR contains the following updates:
| Package | Type | Update | Change |
|---|---|---|---|
| [tempfile](https://stebalien.com/projects/tempfile-rs/)
([source](https://redirect.github.com/Stebalien/tempfile)) |
workspace.dependencies | minor | `3.19.1` -> `3.20.0` |
---
> [!WARNING]
> Some dependencies could not be looked up. Check the Dependency
Dashboard for more information.
---
### Release Notes
<details>
<summary>Stebalien/tempfile (tempfile)</summary>
###
[`v3.20.0`](https://redirect.github.com/Stebalien/tempfile/blob/HEAD/CHANGELOG.md#3200)
[Compare
Source](https://redirect.github.com/Stebalien/tempfile/compare/v3.19.1...v3.20.0)
This release mostly unifies the behavior/capabilities around "keeping"
temporary files:
- Rename `Builder::keep(bool)` (via deprecation) to
`Builder::disable_cleanup(bool)` to make it clear that behaves
differently from `NamedTempFile::keep()`. The former disables automatic
cleanup while the latter *consumes* the `NamedTempFile` object entirely
and unsets the "temporary file" attribute (on Windows).
- Rename `TempDir::into_path` (via deprecation) to `TempDir::keep` to
mirror `NamedTempFile::keep`.
- Add `TempDir::disable_cleanup`, `NamedTempFile::disable_cleanup`, and
`TempPath::disable_cleanup` making it possible to disable automatic
cleanup in-place *after* creating a temporary file/directory (equivalent
to calling `Builder::disable_cleanup` before creating the
file/directory).
Additionally, it adds a few spooled temporary file features:
- Add `SpooledTempFile::into_file` for turning a `SpooledTempFile` into
a regular unnamed temporary file, writing it to the backing storage
("rolling" it) if it was still stored in-memory.
- Add `spooled_tempfile_in` and `SpooledTempFile::new_in` methods for
creating spooled temporary files in a specific directory. This makes it
possible to choose the backing device for your spooled temporary file
which is rather important on Linux where the default temporary directory
is likely backed by memory (defeating the entire point of having a
spooled temporary file).
Finally, this release improves documentation, especially the top-level
documentation explaining which temporary file type to use.
**BREAKING** for those with `deny(warnings)`:
- `Builder::keep` deprecated in favor of `Builder::disable_cleanup`.
- `TempDir::into_path` is deprecated in favor of `TempDir::keep`.
**BREAKING**:
</details>
---
### Configuration
📅 **Schedule**: Branch creation - "before 4am on Monday" (UTC),
Automerge - At any time (no schedule defined).
🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.
♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.
🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.
---
- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box
---
This PR was generated by [Mend Renovate](https://mend.io/renovate/).
View the [repository job
log](https://developer.mend.io/github/astral-sh/uv).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0MC4zMy42IiwidXBkYXRlZEluVmVyIjoiNDAuMzMuNiIsInRhcmdldEJyYW5jaCI6Im1haW4iLCJsYWJlbHMiOlsiaW50ZXJuYWwiXX0=-->
---------
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: konstin <konstin@mailbox.org>
In https://github.com/astral-sh/uv/pull/13721#issuecomment-2920530601 I
presumed that all the installation problems in
https://github.com/astral-sh/uv/pull/13722 were solved by
https://github.com/astral-sh/uv/pull/13709 but they were not because we
don't differentiate between implicit and explicit architecture requests
so a request for `aarch64` is considered satisfied by an existing
`x86-64` installation even if the user explicitly requested that
architecture.
Now, we track if it was explicit or implicit, requiring an exact match
in the former case, and a `supports` in the latter.
We considered doing this for other items in the request, like the
operating system but we don't have a `supports()` concept there. It
might make sense for libc in the future.
Previously you could get a confusing error message like this:
```
$ docker run ghcr.io/astral-sh/uv run python
error: Could not read ELF interpreter from any of the following paths: /bin/sh, /usr/bin/env, /bin/dash, /bin/ls
```
Now the error message is better:
```
error: Failed to discover managed Python installations
Caused by: Failed to determine the libc used on the current platform
Caused by: Failed to find any common binaries to determine libc from: /bin/sh, /usr/bin/env, /bin/dash, /bin/ls
```
See https://github.com/astral-sh/uv/issues/8635.
---------
Co-authored-by: konsti <konstin@mailbox.org>
Co-authored-by: Zanie Blue <contact@zanie.dev>
Currently, it is not possible to set a custom Python downloads JSON on
Windows, as Windows paths can be valid URLs.
```rust
use url::Url;
fn main() {
dbg!(Url::parse(r"C:\Users\Ferris\download.json"));
}
```
Tested by https://github.com/astral-sh/uv/pull/13585 (where it is
currently failing CI).
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
This change allows for `uv python install --reinstall` to include
pre-releases when reinstalling. It does this by explicitly allowing
pre-releases to be included within `PythonRequest::Any` if the user does
not specify a version to reinstall.
Fixes#13582
## Test Plan
<!-- How was it tested? -->
```bash
uv python install 3.14 3.13 3.10
uv python install --no-config --reinstall
```
Prior to this PR, there were numerous places where uv would leak
credentials in logs. We had a way to mask credentials by calling methods
or a recently-added `redact_url` function, but this was not secure by
default. There were a number of other types (like `GitUrl`) that would
leak credentials on display.
This PR adds a `DisplaySafeUrl` newtype to prevent leaking credentials
when logging by default. It takes a maximalist approach, replacing the
use of `Url` almost everywhere. This includes when first parsing config
files, when storing URLs in types like `GitUrl`, and also when storing
URLs in types that in practice will never contain credentials (like
`DirectorySourceUrl`). The idea is to make it easy for developers to do
the right thing and for the compiler to support this (and to minimize
ever having to manually convert back and forth). Displaying credentials
now requires an active step. Note that despite this maximalist approach,
the use of the newtype should be zero cost.
One conspicuous place this PR does not use `DisplaySafeUrl` is in the
`uv-auth` crate. That would require new clones since there are calls to
`request.url()` that return a `&Url`. One option would have been to make
`DisplaySafeUrl` wrap a `Cow`, but this would lead to lifetime
annotations all over the codebase. I've created a separate PR based on
this one (#13576) that updates `uv-auth` to use `DisplaySafeUrl` with
one new clone. We can discuss the tradeoffs there.
Most of this PR just replaces `Url` with `DisplaySafeUrl`. The core is
`uv_redacted/lib.rs`, where the newtype is implemented. To make it
easier to review the rest, here are some points of note:
* `DisplaySafeUrl` has a `Display` implementation that masks
credentials. Currently, it will still display the username when there is
both a username and password. If we think is the wrong choice, it can
now be changed in one place.
* `DisplaySafeUrl` has a `remove_credentials()` method and also a
`.to_string_with_credentials()` method. This allows us to use it in a
variety of scenarios.
* `IndexUrl::redacted()` was renamed to
`IndexUrl::removed_credentials()` to make it clearer that we are not
masking.
* We convert from a `DisplaySafeUrl` to a `Url` when calling `reqwest`
methods like `.get()` and `.head()`.
* We convert from a `DisplaySafeUrl` to a `Url` when creating a
`uv_auth::Index`. That is because, as mentioned above, I will be
updating the `uv_auth` crate to use this newtype in a separate PR.
* A number of tests (e.g., in `pip_install.rs`) that formerly used
filters to mask tokens in the test output no longer need those filters
since tokens in URLs are now masked automatically.
* The one place we are still knowingly writing credentials to
`pyproject.toml` is when a URL with credentials is passed to `uv add`
with `--raw`. Since displaying credentials is no longer automatic, I
have added a `to_string_with_credentials()` method to the `Pep508Url`
trait. This is used when `--raw` is passed. Adding it to that trait is a
bit weird, but it's the simplest way to achieve the goal. I'm open to
suggestions on how to improve this, but note that because of the way
we're using generic bounds, it's not as simple as just creating a
separate trait for that method.
Rustfmt introduces a lot of formatting changes in the 2024 edition. To
not break everything all at once, we split out the set of formatting
changes compatible with both the 2021 and 2024 edition by first
formatting with the 2024 style, and then again with the currently used
2021 style.
Notable changes are the formatting of derive macro attributes and lines
with overly long strings and adding trailing semicolons after statements
consistently.
Zanie Blue