Use codspeed runner for wall time benchmarking (#13586)

We couldn't use the CodSpeed "walltime" runner because it required
administrative permissions on our repositories, but following some
feedback they've adjusted the required permissions so we can give it a
try now.

As a brief background, CodSpeed uses Valgrind for instrumented
benchmarking, emulating the execution for improved stability on GitHub's
runners. This is nice, but means things like allocs and io are not
measured. Now, they support standard wall time benchmarking, using their
own managed runners for stable measurements. Here, we add support for
those while retaining the old workflow — you can toggle between views in
their UI.
This commit is contained in:
Zanie Blue
2025-05-29 12:52:17 -05:00
committed by GitHub
parent 08f2fa48ff
commit 59070b5b3f
+39 -1
View File
@@ -2254,7 +2254,45 @@ jobs:
- name: "Validate embedded Python install"
run: python ./scripts/check_embedded_python.py --uv ./uv.exe
benchmarks:
benchmarks-walltime:
name: "benchmarks | walltime aarch64 linux"
runs-on: codspeed-macro
needs: determine_changes
if: ${{ github.repository == 'astral-sh/uv' && !contains(github.event.pull_request.labels.*.name, 'no-test') && (needs.determine_changes.outputs.code == 'true' || github.ref == 'refs/heads/main') }}
timeout-minutes: 20
steps:
- name: "Checkout Branch"
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
- uses: Swatinem/rust-cache@9d47c6ad4b02e050fd481d890b2ea34778fd09d6 # v2.7.8
- name: "Install Rust toolchain"
run: rustup show
- name: "Install codspeed"
uses: taiki-e/install-action@ab3728c7ba6948b9b429627f4d55a68842b27f18 # v2.50.3
with:
tool: cargo-codspeed
- name: "Install requirements and prime cache"
run: |
sudo apt-get update
sudo apt-get install -y libsasl2-dev libldap2-dev libkrb5-dev
cargo run --bin uv -- venv --cache-dir .cache
cargo run --bin uv -- pip compile scripts/requirements/jupyter.in --universal --exclude-newer 2024-08-08 --cache-dir .cache
cargo run --bin uv -- pip compile scripts/requirements/airflow.in --universal --exclude-newer 2024-08-08 --cache-dir .cache
- name: "Build benchmarks"
run: cargo codspeed build --profile profiling --features codspeed -p uv-bench
- name: "Run benchmarks"
uses: CodSpeedHQ/action@0010eb0ca6e89b80c88e8edaaa07cfe5f3e6664d # v3.5.0
with:
run: cargo codspeed run
token: ${{ secrets.CODSPEED_TOKEN }}
benchmarks-instrumented:
name: "benchmarks | instrumented"
runs-on: ubuntu-latest
needs: determine_changes
if: ${{ github.repository == 'astral-sh/uv' && !contains(github.event.pull_request.labels.*.name, 'no-test') && (needs.determine_changes.outputs.code == 'true' || github.ref == 'refs/heads/main') }}