konsti
4ac78f673b
When doing a directory traversal for source dist inclusion, we want to offer the user include and exclude options, and we want to avoid traversing irrelevant directories. The latter is important for performance, especially on network file systems, but also with large data directories, or (not-included) directories with other permissions. To support this, we introduce `GlobDirFilter`, which uses a DFA from regex_automata to determine whether any children of a directory can be included and skips the directory if not. The globs are based on PEP 639. The syntax is more restricted than glob or globset, but it's standardized. I chose it over glob or globset because we're already using this syntax for `project.license-files` a required by PEP 639, so it makes sense to use the same globs for all includes (see e.g. https://github.com/google-deepmind/alphafold3/blob/4f52a3bb624fd7245fb1ad0bf62d752ffa46f0ce/pyproject.toml#L36-L48 for example with same semantics for include and exclude) ### Semantics Glob semantics are complex due to mixing directories and files, expectations around simplicity and our need to exclude most of the tree in the project from traversal. The current draft uses a syntax that optimizes for simple default use cases for the start. #### includes Glob expressions which files and directories to include in the source distribution. Includes are anchored, which means that `pyproject.toml` includes only `<project root>/pyproject.toml`. Use for example `assets/**/sample.csv` to include for all `sample.csv` files in `<project root>/assets` or any child directory. To recursively include all files under a directory, use a `/**` suffix, e.g. `src/**`. For performance and reproducibility, avoid unanchored matches such as `**/sample.csv`. The glob syntax is the reduced portable glob from [PEP 639](https://peps.python.org/pep-0639/#add-license-FILES-key). #### excludes Glob expressions which files and directories to exclude from the previous source distribution includes. Excludes are not, which means that `__pycache__` excludes all directories named `__pycache__` and it's children anywhere. To anchor a directory, use a `/` prefix, e.g., `/dist` will exclude only `<project root>/dist`. The glob syntax is the reduced portable glob from [PEP 639](https://peps.python.org/pep-0639/#add-license-FILES-key).
1.5 KiB
1.5 KiB
globfilter
Portable directory walking with includes and excludes.
Motivating example: You want to allow the user to select paths within a project.
include = ["src", "License.txt", "resources/icons/*.svg"]
exclude = ["target", "/dist", ".cache", "*.tmp"]
When traversing the directory, you can use
GlobDirFilter::from_globs(...)?.match_directory(&relative) skip directories that never match in
WalkDirs filter_entry.
Syntax
This crate supports the cross-language, restricted glob syntax from PEP 639:
- Alphanumeric characters, underscores (
_), hyphens (-) and dots (.) are matched verbatim. - The special glob characters are:
*: Matches any number of characters except path separators?: Matches a single character except the path separator**: Matches any number of characters including path separators[], containing only the verbatim matched characters: Matches a single of the characters contained. Within[...], the hyphen indicates a locale-agnostic range (e.g.a-z, order based on Unicode code points). Hyphens at the start or end are matched literally.
- The path separator is the forward slash character (
/). Patterns are relative to the given directory, a leading slash character for absolute paths is not supported. - Parent directory indicators (
..) are not allowed.
These rules mean that matching the backslash (\) is forbidden, which avoid collisions with the
windows path separator.