konsti
d9dbb8a4af
Downstack PR: #4481 ## Introduction We support forking the dependency resolution to support conflicting registry requirements for different platforms, say on package range is required for an older python version while a newer is required for newer python versions, or dependencies that are different per platform. We need to extend this support to direct URL requirements. ```toml dependencies = [ "iniconfig @ https://files.pythonhosted.org/packages/ef/a6/62565a6e1cf69e10f5727360368e451d4b7f58beeac6173dc9db836a5b46/iniconfig-2.0.0-py3-none-any.whl ; python_version >= '3.12'", "iniconfig @ https://files.pythonhosted.org/packages/9b/dd/b3c12c6d707058fa947864b67f0c4e0c39ef8610988d7baea9578f3c48f3/iniconfig-1.1.1-py2.py3-none-any.whl ; python_version < '3.12'" ] ``` This did not work because `Urls` was built on the assumption that there is a single allowed URL per package. We collect all allowed URL ahead of resolution by following direct URL dependencies (including path dependencies) transitively, i.e. a registry distribution can't require a URL. ## The same package can have Registry and URL requirements Consider the following two cases: requirements.in: ```text werkzeug==2.0.0 werkzeug @ https://files.pythonhosted.org/packages/ff/1d/960bb4017c68674a1cb099534840f18d3def3ce44aed12b5ed8b78e0153e/Werkzeug-2.0.0-py3-none-any.whl ``` pyproject.toml: ```toml dependencies = [ "iniconfig == 1.1.1 ; python_version < '3.12'", "iniconfig @ git+https://github.com/pytest-dev/iniconfig@93f5930e668c0d1ddf4597e38dd0dea4e2665e7a ; python_version >= '3.12'", ] ``` In the first case, we want the URL to override the registry dependency, in the second case we want to fork and have one branch use the registry and the other the URL. We have to know about this in `PubGrubRequirement::from_registry_requirement`, but we only fork after the current method. Consider the following case too: a: ``` c==1.0.0 b @ https://b.zip ``` b: ``` c @ https://c_new.zip ; python_version >= '3.12'", c @ https://c_old.zip ; python_version < '3.12'", ``` When we convert the requirements of `a`, we can't know the url of `c` yet. The solution is to remove the `Url` from `PubGrubPackage`: The `Url` is redundant with `PackageName`, there can be only one url per package name per fork. We now do the following: We track the urls from requirements in `PubGrubDependency`. After forking, we call `add_package_version_dependencies` where we apply override URLs, check if the URL is allowed and check if the url is unique in this fork. When we request a distribution, we ask the fork urls for the real URL. Since we prioritize url dependencies over registry dependencies and skip packages with `Urls` entries in pre-visiting, we know that when fetching a package, we know if it has a url or not. ## URL conflicts pyproject.toml (invalid): ```toml dependencies = [ "iniconfig @ https://files.pythonhosted.org/packages/44/39/e96292c7f7068e58877f476908c5974dc76c37c623f1fa332fe4ed6dfbec/iniconfig-1.1.0.tar.gz", "iniconfig @ https://files.pythonhosted.org/packages/9b/dd/b3c12c6d707058fa947864b67f0c4e0c39ef8610988d7baea9578f3c48f3/iniconfig-1.1.1-py2.py3-none-any.whl ; python_version < '3.12'", "iniconfig @ https://files.pythonhosted.org/packages/ef/a6/62565a6e1cf69e10f5727360368e451d4b7f58beeac6173dc9db836a5b46/iniconfig-2.0.0-py3-none-any.whl ; python_version >= '3.12'", ] ``` On the fork state, we keep `ForkUrls` that check for conflicts after forking, rejecting the third case because we added two packages of the same name with different URLs. We need to flatten out the requirements before transformation into pubgrub requirements to get the full list of other requirements which may contain a URL, which was changed in a previous PR: #4430. ## Complex Example a: ```toml dependencies = [ # Force a split "anyio==4.3.0 ; python_version >= '3.12'", "anyio==4.2.0 ; python_version < '3.12'", # Include URLs transitively "b" ] ``` b: ```toml dependencies = [ # Only one is used in each split. "b1 ; python_version < '3.12'", "b2 ; python_version >= '3.12'", "b3 ; python_version >= '3.12'", ] ``` b1: ```toml dependencies = [ "iniconfig @ https://files.pythonhosted.org/packages/9b/dd/b3c12c6d707058fa947864b67f0c4e0c39ef8610988d7baea9578f3c48f3/iniconfig-1.1.1-py2.py3-none-any.whl", ] ``` b2: ```toml dependencies = [ "iniconfig @ https://files.pythonhosted.org/packages/ef/a6/62565a6e1cf69e10f5727360368e451d4b7f58beeac6173dc9db836a5b46/iniconfig-2.0.0-py3-none-any.whl", ] ``` b3: ```toml dependencies = [ "iniconfig @ https://files.pythonhosted.org/packages/44/39/e96292c7f7068e58877f476908c5974dc76c37c623f1fa332fe4ed6dfbec/iniconfig-1.1.0.tar.gz", ] ``` In this example, all packages are url requirements (directory requirements) and the root package is `a`. We first split on `a`, `b` being in each split. In the first fork, we reach `b1`, the fork URLs are empty, we insert the iniconfig 1.1.1 URL, and then we skip over `b2` and `b3` since the mark is disjoint with the fork markers. In the second fork, we skip over `b1`, visit `b2`, insert the iniconfig 2.0.0 URL into the again empty fork URLs, then visit `b3` and try to insert the iniconfig 1.1.0 URL. At this point we find a conflict for the iniconfig URL and error. ## Closing The git tests are slow, but they make the best example for different URL types i could find. Part of #3927. This PR does not handle `Locals` or pre-releases yet.
Crates
bench
Functionality for benchmarking uv.
cache-key
Generic functionality for caching paths, URLs, and other resources across platforms.
distribution-filename
Parse built distribution (wheel) and source distribution (sdist) filenames to extract structured metadata.
distribution-types
Abstractions for representing built distributions (wheels) and source distributions (sdists), and the sources from which they can be downloaded.
install-wheel-rs
Install built distributions (wheels) into a virtual environment.]
once-map
A waitmap-like concurrent hash map for executing tasks
exactly once.
pep440-rs
Utilities for interacting with Python version numbers and specifiers.
pep508-rs
Utilities for interacting with PEP 508 dependency specifiers.
platform-host
Functionality for detecting the current platform (operating system, architecture, etc.).
platform-tags
Functionality for parsing and inferring Python platform tags as per PEP 425.
uv
Command-line interface for the uv package manager.
uv-build
A PEP 517-compatible build frontend for uv.
uv-cache
Functionality for caching Python packages and associated metadata.
uv-client
Client for interacting with PyPI-compatible HTTP APIs.
uv-dev
Development utilities for uv.
uv-dispatch
A centralized struct for resolving and building source distributions in isolated environments.
Implements the traits defined in uv-types.
uv-distribution
Client for interacting with built distributions (wheels) and source distributions (sdists). Capable of fetching metadata, distribution contents, etc.
uv-extract
Utilities for extracting files from archives.
uv-fs
Utilities for interacting with the filesystem.
uv-git
Functionality for interacting with Git repositories.
uv-installer
Functionality for installing Python packages into a virtual environment.
uv-toolchain
Functionality for detecting and leveraging the current Python interpreter.
uv-normalize
Normalize package and extra names as per Python specifications.
uv-package
Types and functionality for working with Python packages, e.g., parsing wheel files.
uv-requirements
Utilities for reading package requirements from pyproject.toml and requirements.txt files.
uv-resolver
Functionality for resolving Python packages and their dependencies.
uv-types
Shared traits for uv, to avoid circular dependencies.
pypi-types
General-purpose type definitions for types used in PyPI-compatible APIs.
uv-virtualenv
A venv replacement to create virtual environments in Rust.
uv-warnings
User-facing warnings for uv.
requirements-txt
Functionality for parsing requirements.txt files.