Figuring out what is missing from dedicated packages for supporting downloading and installing a wheel from PyPI

As per Building distributions and drawing the Platypus, I figured I would kick off a discussion to figure out where any gaps exist for downloading and installing an appropriate wheel from PyPI. I’m thinking of the case where you know the name of the project and version you want already, so this is for selecting the correct wheel from those available on PyPI, downloading it, and then installing it.

Here is the outline of steps that I’m aware of to accomplish this feature/story and whether something exists as a spec and/or package:

  1. Check if package is already installed (spec / importlib-metadata & importlib.metadata)
  2. Check local wheel cache (? / ?; how pip does it, XDG Base Directory spec)
  3. Choose appropriate file from PyPI / index
    1. Process the list of files (Simple repository spec / ?; PyPI JSON API / ?)
    2. Calculate best-fitting wheel (spec / packaging.tags)
  4. Download the wheel
  5. Cache the wheel locally (? / ?; see local cache check for potential details)
  6. Install the wheel
    1. Choose which directory to install the files to (e.g. user versus system/virtual environment; ? / ?)
    2. Install the files (spec / distlib.wheel although works on file paths versus bytes and so can’t go straight from reading off the socket to installation)
    3. Record the installation (spec / ?)

I have made this post a wiki so others can add or edit details as appropriate.

Currently in the weeds that is the pip codebase. :upside_down_face:

Cleanup work to make this easier – Adopting "working" scheme for every run · Issue #4575 · pypa/pip · GitHub

@cjerdonek and @uranusjr will have a better idea of state of affairs than me.

I’m not sure if we want this to be one of the interoperable-units. @brettcannon why did you include it?

Yes, I have a PR that I’ll be updating shortly with more detailed documentation on how this works in pip currently. (It was requested that I add this to an “architecture” document.)

1 Like

Because I would rather not see five copies of pip or request’s wheels installed on my machine due to caching by 5 different tools. :smile: Basically why waste the disk and networking if we could all agree on where wheels should get cached?

Great! In terms of coding this it shouldn’t be hard:

# Keeping it simple by assuming PyPI.
def find_release_file(project, version):
    files = pypi_release(project, version)
    wheel_mapping = map_wheels_to_files(files)  # Because of compressed wheel tags.
    for tag in packaging.tags.sys_tags():
        if file := wheel_mapping.get(tag):
            return file
    else:
        raise NoCompatibleFileError

I imagine that’s the doc I started with Pradyun at the minisummit in May and have been polishing. @cjerdonek Please feel free to add commits.