Use codspeed runner for wall time benchmarking (#13586)
We couldn't use the CodSpeed "walltime" runner because it required administrative permissions on our repositories, but following some feedback they've adjusted the required permissions so we can give it a try now. As a brief background, CodSpeed uses Valgrind for instrumented benchmarking, emulating the execution for improved stability on GitHub's runners. This is nice, but means things like allocs and io are not measured. Now, they support standard wall time benchmarking, using their own managed runners for stable measurements. Here, we add support for those while retaining the old workflow — you can toggle between views in their UI.
This commit is contained in:
@@ -2254,7 +2254,45 @@ jobs:
|
||||
- name: "Validate embedded Python install"
|
||||
run: python ./scripts/check_embedded_python.py --uv ./uv.exe
|
||||
|
||||
benchmarks:
|
||||
benchmarks-walltime:
|
||||
name: "benchmarks | walltime aarch64 linux"
|
||||
runs-on: codspeed-macro
|
||||
needs: determine_changes
|
||||
if: ${{ github.repository == 'astral-sh/uv' && !contains(github.event.pull_request.labels.*.name, 'no-test') && (needs.determine_changes.outputs.code == 'true' || github.ref == 'refs/heads/main') }}
|
||||
timeout-minutes: 20
|
||||
steps:
|
||||
- name: "Checkout Branch"
|
||||
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
|
||||
|
||||
- uses: Swatinem/rust-cache@9d47c6ad4b02e050fd481d890b2ea34778fd09d6 # v2.7.8
|
||||
|
||||
- name: "Install Rust toolchain"
|
||||
run: rustup show
|
||||
|
||||
- name: "Install codspeed"
|
||||
uses: taiki-e/install-action@ab3728c7ba6948b9b429627f4d55a68842b27f18 # v2.50.3
|
||||
with:
|
||||
tool: cargo-codspeed
|
||||
|
||||
- name: "Install requirements and prime cache"
|
||||
run: |
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y libsasl2-dev libldap2-dev libkrb5-dev
|
||||
cargo run --bin uv -- venv --cache-dir .cache
|
||||
cargo run --bin uv -- pip compile scripts/requirements/jupyter.in --universal --exclude-newer 2024-08-08 --cache-dir .cache
|
||||
cargo run --bin uv -- pip compile scripts/requirements/airflow.in --universal --exclude-newer 2024-08-08 --cache-dir .cache
|
||||
|
||||
- name: "Build benchmarks"
|
||||
run: cargo codspeed build --profile profiling --features codspeed -p uv-bench
|
||||
|
||||
- name: "Run benchmarks"
|
||||
uses: CodSpeedHQ/action@0010eb0ca6e89b80c88e8edaaa07cfe5f3e6664d # v3.5.0
|
||||
with:
|
||||
run: cargo codspeed run
|
||||
token: ${{ secrets.CODSPEED_TOKEN }}
|
||||
|
||||
benchmarks-instrumented:
|
||||
name: "benchmarks | instrumented"
|
||||
runs-on: ubuntu-latest
|
||||
needs: determine_changes
|
||||
if: ${{ github.repository == 'astral-sh/uv' && !contains(github.event.pull_request.labels.*.name, 'no-test') && (needs.determine_changes.outputs.code == 'true' || github.ref == 'refs/heads/main') }}
|
||||
|
||||
Reference in New Issue
Block a user