fee7f9d093
## Summary Follow up to https://github.com/astral-sh/uv/pull/15563 Closes https://github.com/astral-sh/uv/issues/13485 This is a first-pass at adding support for conditional support for Git LFS between git sources, initial feedback welcome. e.g. ``` [tool.uv.sources] test-lfs-repo = { git = "https://github.com/zanieb/test-lfs-repo.git", lfs = true } ``` For context previously a user had to set `UV_GIT_LFS` to have uv fetch lfs objects on git sources. This env var was all or nothing, meaning you must always have it set to get consistent behavior and it applied to all git sources. If you fetched lfs objects at a revision and then turned off lfs (or vice versa), the git db, corresponding checkout lfs artifacts would not be updated properly. Similarly, when git source distributions were built, there would be no distinction between sources with lfs and without lfs. Hence, it could corrupt the git, sdist, and archive caches. In order to support some sources being LFS enabled and other not, this PR adds a stateful layer roughly similar to how `subdirectory` works but for `lfs` since the git database, the checkouts and the corresponding caching layers needed to be LFS aware (requested vs installed). The caches also had to isolated and treated entirely separate when handling LFS sources. Summary * Adds `lfs = true` or `lfs = false` to git sources in pyproject.toml * Added `lfs=true` query param / fragments to most relevant url structs (not parsed as user input) * In the case of uv add / uv tool, `--lfs` is supported instead * `UV_GIT_LFS` environment variable support is still functional for non-project entrypoints (e.g. uv pip) * `direct-url.json` now has an custom `git_lfs` entry under VcsInfo (note, this is not in the spec currently -- see caveats). * git database and checkouts have an different cache key as the sources should be treated effectively different for the same rev. * sdists cache also differ in the cache key of a built distribution if it was built using LFS enabled revisions to distinguish between non-LFS same revisions. This ensures the strong assumption for archive-v0 that an unpacked revision "doesn't change sources" stays valid. Caveats * `pylock.toml` import support has not been added via git_lfs=true, going through the spec it wasn't clear to me it's something we'd support outside of the env var (for now). * direct-url struct was modified by adding a non-standard `git_lfs` field under VcsInfo which may be undersirable although the PEP 610 does say `Additional fields that would be necessary to support such VCS SHOULD be prefixed with the VCS command name` which could be interpret this change as ok. * There will be a slight lockfile and cache churn for users that use `UV_GIT_LFS` as all git lockfile entries will get a `lfs=true` fragment. The cache version does not need an update, but LFS sources will get their own namespace under git-v0 and sdist-v9/git hence a cache-miss will occur once but this can be sufficient to label this as breaking for workflows always setting `UV_GIT_LFS`. ## Test Plan Some initial tests were added. More tests likely to follow as we reach consensus on a final approach. For IT test, we may want to move to use a repo under astral namespace in order to test lfs functionality. Manual testing was done for common pathological cases like killing LFS fetch mid-way, uninstalling LFS after installing an sdist with it and reinstalling, fetching LFS artifacts in different commits, etc. PSA: Please ignore the docker build failures as its related to depot OIDC issues. --------- Co-authored-by: Zanie Blue <contact@zanie.dev> Co-authored-by: konstin <konstin@mailbox.org>
232 lines
8.2 KiB
Markdown
232 lines
8.2 KiB
Markdown
# Contributing
|
|
|
|
## Finding ways to help
|
|
|
|
We label issues that would be good for a first time contributor as
|
|
[`good first issue`](https://github.com/astral-sh/uv/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22).
|
|
These usually do not require significant experience with Rust or the uv code base.
|
|
|
|
We label issues that we think are a good opportunity for subsequent contributions as
|
|
[`help wanted`](https://github.com/astral-sh/uv/issues?q=is%3Aopen+is%3Aissue+label%3A%22help+wanted%22).
|
|
These require varying levels of experience with Rust and uv. Often, we want to accomplish these
|
|
tasks but do not have the resources to do so ourselves.
|
|
|
|
You don't need our permission to start on an issue we have labeled as appropriate for community
|
|
contribution as described above. However, it's a good idea to indicate that you are going to work on
|
|
an issue to avoid concurrent attempts to solve the same problem.
|
|
|
|
Please check in with us before starting work on an issue that has not been labeled as appropriate
|
|
for community contribution. We're happy to receive contributions for other issues, but it's
|
|
important to make sure we have consensus on the solution to the problem first.
|
|
|
|
Outside of issues with the labels above, issues labeled as
|
|
[`bug`](https://github.com/astral-sh/uv/issues?q=is%3Aopen+is%3Aissue+label%3A%22bug%22) are the
|
|
best candidates for contribution. In contrast, issues labeled with `needs-decision` or
|
|
`needs-design` are _not_ good candidates for contribution. Please do not open pull requests for
|
|
issues with these labels.
|
|
|
|
Please do not open pull requests for new features without prior discussion. While we appreciate
|
|
exploration of new features, we will almost always close these pull requests immediately. Adding a
|
|
new feature to uv creates a long-term maintenance burden and requires strong consensus from the uv
|
|
team before it is appropriate to begin work on an implementation.
|
|
|
|
## Setup
|
|
|
|
[Rust](https://rustup.rs/) (and a C compiler) are required to build uv.
|
|
|
|
On Ubuntu and other Debian-based distributions, you can install a C compiler with:
|
|
|
|
```shell
|
|
sudo apt install build-essential
|
|
```
|
|
|
|
On Fedora-based distributions, you can install a C compiler with:
|
|
|
|
```shell
|
|
sudo dnf install gcc
|
|
```
|
|
|
|
## Testing
|
|
|
|
For running tests, we recommend [nextest](https://nexte.st/).
|
|
|
|
If tests fail due to a mismatch in the JSON Schema, run: `cargo dev generate-json-schema`.
|
|
|
|
### Python
|
|
|
|
Testing uv requires multiple specific Python versions; they can be installed with:
|
|
|
|
```shell
|
|
cargo run python install
|
|
```
|
|
|
|
The storage directory can be configured with `UV_PYTHON_INSTALL_DIR`. (It must be an absolute path.)
|
|
|
|
### Snapshot testing
|
|
|
|
uv uses [insta](https://insta.rs/) for snapshot testing. It's recommended (but not necessary) to use
|
|
`cargo-insta` for a better snapshot review experience. See the
|
|
[installation guide](https://insta.rs/docs/cli/) for more information.
|
|
|
|
In tests, you can use `uv_snapshot!` macro to simplify creating snapshots for uv commands. For
|
|
example:
|
|
|
|
```rust
|
|
#[test]
|
|
fn test_add() {
|
|
let context = TestContext::new("3.12");
|
|
uv_snapshot!(context.filters(), context.add().arg("requests"), @"");
|
|
}
|
|
```
|
|
|
|
To run and review a specific snapshot test:
|
|
|
|
```shell
|
|
cargo test --package <package> --test <test> -- <test_name> -- --exact
|
|
cargo insta review
|
|
```
|
|
|
|
### Git and Git LFS
|
|
|
|
A subset of uv tests require both [Git](https://git-scm.com) and [Git LFS](https://git-lfs.com/) to
|
|
execute properly.
|
|
|
|
These tests can be disabled by turning off either `git` or `git-lfs` uv features.
|
|
|
|
### Local testing
|
|
|
|
You can invoke your development version of uv with `cargo run -- <args>`. For example:
|
|
|
|
```shell
|
|
cargo run -- venv
|
|
cargo run -- pip install requests
|
|
```
|
|
|
|
## Running inside a Docker container
|
|
|
|
Source distributions can run arbitrary code on build and can make unwanted modifications to your
|
|
system
|
|
(["Someone's Been Messing With My Subnormals!" on Blogspot](https://moyix.blogspot.com/2022/09/someones-been-messing-with-my-subnormals.html),
|
|
["nvidia-pyindex" on PyPI](https://pypi.org/project/nvidia-pyindex/)), which can even occur when
|
|
just resolving requirements. To prevent this, there's a Docker container you can run commands in:
|
|
|
|
```console
|
|
$ docker build -t uv-builder -f crates/uv-dev/builder.dockerfile --load .
|
|
# Build for musl to avoid glibc errors, might not be required with your OS version
|
|
cargo build --target x86_64-unknown-linux-musl --profile profiling
|
|
docker run --rm -it -v $(pwd):/app uv-builder /app/target/x86_64-unknown-linux-musl/profiling/uv-dev resolve-many --cache-dir /app/cache-docker /app/scripts/popular_packages/pypi_10k_most_dependents.txt
|
|
```
|
|
|
|
We recommend using this container if you don't trust the dependency tree of the package(s) you are
|
|
trying to resolve or install.
|
|
|
|
## Profiling and Benchmarking
|
|
|
|
Please refer to Ruff's
|
|
[Profiling Guide](https://github.com/astral-sh/ruff/blob/main/CONTRIBUTING.md#profiling-projects),
|
|
it applies to uv, too.
|
|
|
|
We provide diverse sets of requirements for testing and benchmarking the resolver in
|
|
`scripts/requirements` and for the installer in `scripts/requirements/compiled`.
|
|
|
|
You can use `scripts/benchmark` to benchmark predefined workloads between uv versions and with other
|
|
tools, e.g., from the `scripts/benchmark` directory:
|
|
|
|
```shell
|
|
uv run resolver \
|
|
--uv-pip \
|
|
--poetry \
|
|
--benchmark \
|
|
resolve-cold \
|
|
../scripts/requirements/trio.in
|
|
```
|
|
|
|
### Analyzing concurrency
|
|
|
|
You can use [tracing-durations-export](https://github.com/konstin/tracing-durations-export) to
|
|
visualize parallel requests and find any spots where uv is CPU-bound. Example usage, with `uv` and
|
|
`uv-dev` respectively:
|
|
|
|
```shell
|
|
RUST_LOG=uv=info TRACING_DURATIONS_FILE=target/traces/jupyter.ndjson cargo run --features tracing-durations-export --profile profiling -- pip compile scripts/requirements/jupyter.in
|
|
```
|
|
|
|
```shell
|
|
RUST_LOG=uv=info TRACING_DURATIONS_FILE=target/traces/jupyter.ndjson cargo run --features tracing-durations-export --bin uv-dev --profile profiling -- resolve jupyter
|
|
```
|
|
|
|
### Trace-level logging
|
|
|
|
You can enable `trace` level logging using the `RUST_LOG` environment variable, i.e.
|
|
|
|
```shell
|
|
RUST_LOG=trace uv
|
|
```
|
|
|
|
## Documentation
|
|
|
|
To preview any changes to the documentation locally:
|
|
|
|
1. Install the [Rust toolchain](https://www.rust-lang.org/tools/install).
|
|
|
|
2. Run `cargo dev generate-all`, to update any auto-generated documentation.
|
|
|
|
3. Run the development server with:
|
|
|
|
```shell
|
|
# For contributors.
|
|
uvx --with-requirements docs/requirements.txt -- mkdocs serve -f mkdocs.public.yml
|
|
|
|
# For members of the Astral org, which has access to MkDocs Insiders via sponsorship.
|
|
uvx --with-requirements docs/requirements-insiders.txt -- mkdocs serve -f mkdocs.insiders.yml
|
|
```
|
|
|
|
The documentation should then be available locally at
|
|
[http://127.0.0.1:8000/uv/](http://127.0.0.1:8000/uv/).
|
|
|
|
To update the documentation dependencies, edit `docs/requirements.in` and
|
|
`docs/requirements-insiders.in`, then run:
|
|
|
|
```shell
|
|
uv pip compile docs/requirements.in -o docs/requirements.txt --universal -p 3.12
|
|
uv pip compile docs/requirements-insiders.in -o docs/requirements-insiders.txt --universal -p 3.12
|
|
```
|
|
|
|
Documentation is deployed automatically on release by publishing to the
|
|
[Astral documentation](https://github.com/astral-sh/docs) repository, which itself deploys via
|
|
Cloudflare Pages.
|
|
|
|
After making changes to the documentation, format the markdown files with:
|
|
|
|
```shell
|
|
npx prettier --prose-wrap always --write "**/*.md"
|
|
```
|
|
|
|
Note that the command above requires Node.js and npm to be installed on your system. As an
|
|
alternative, you can run this command using Docker:
|
|
|
|
```console
|
|
$ docker run --rm -v .:/src/ -w /src/ node:alpine npx prettier --prose-wrap always --write "**/*.md"
|
|
```
|
|
|
|
## Releases
|
|
|
|
Releases can only be performed by Astral team members.
|
|
|
|
Changelog entries and version bumps are automated. First, run:
|
|
|
|
```shell
|
|
./scripts/release.sh
|
|
```
|
|
|
|
Then, editorialize the `CHANGELOG.md` file to ensure entries are consistently styled.
|
|
|
|
Then, open a pull request, e.g., `Bump version to ...`.
|
|
|
|
Binary builds will automatically be tested for the release.
|
|
|
|
After merging the pull request, run the
|
|
[release workflow](https://github.com/astral-sh/uv/actions/workflows/release.yml) with the version
|
|
tag. **Do not include a leading `v`**. The release will automatically be created on GitHub after
|
|
everything else publishes.
|