Pip download doesn't download all dependencies (setuptools and wheel are missing)

christian · June 27, 2024, 12:22pm

Hi, I need to offline install a python package via pip into a venv, but without internet connection. The package I actually want to install is here: GitHub - giggls/osml10n: Localization functions for Openstreetmap and I cloned it to ~/osml10n.

Then I make a new venv: python3 -mvenv ~/venv
and then i do ~/venv/bin/pip download ~/osml10n/
and i get a bunch of packages downloaded in my cwd.

After that, I do ~/venv/bin/pip install --no-index --find-links . ~/osml10n/, but i get an error:

      ERROR: Could not find a version that satisfies the requirement setuptools (from versions: none)
      ERROR: No matching distribution found for setuptools

I could already figure out, that it helps to ~/venv/bin/pip download setuptools
Then a setuptools package gets downloaded and the previous pip install finds it.
Then the same goes for wheel, after separate pip download wheel it’s fine.

Now I’m wondering (TBH, I don’t have much experience with the details of pip or python packaging), why doesn’t pip download all packages, the following pip install needs? Is there a mistake somewhere, maybe in osml10n, maybe theses packages aren’t direct dependencies, but recursive from some other package and pip download requires a flag for recursive dependencies?

If it matters, i’m on ubuntu 24.04

pradyunsg · June 27, 2024, 12:25pm

Please see

github.com/pypa/pip

Build dependencies are not downloaded by `pip download`

opened 07:44AM - 15 Mar 20 UTC

qwhelan

type: bug state: needs discussion PEP implementation C: build logic

**Environment** * pip version: 20.0.2 * Python version: 3.6 * OS: Ubuntu 18….04 **Description** An issue was reported in my repo https://github.com/pydata/bottleneck/issues/333 by a user utilizing `pip download` to create a local cache prior to installation, but was seeing the following error despite having an up-to-date copy of `setuptools`: ``` ERROR: Could not find a version that satisfies the requirement setuptools (from versions: none) ERROR: No matching distribution found for setuptools ``` I was able to reproduce the behavior via a `Dockerfile` and identify that the root issue is that `pip download` is not fetching the PEP 517 dependencies. This error message appears to be reproducible for other packages (such as `numpy`) provided that `pip download` fetches a source package that requires a PEP 517 build step. My findings can be found here: https://github.com/pydata/bottleneck/issues/333#issuecomment-599177688 **Expected behavior** As far as expected behavior, I would expect `pip download` to fetch all packages needed to successfully install the target package(s), including build-only dependencies. **How to Reproduce** `Dockerfile`: ``` FROM nvidia/cuda:10.0-cudnn7-runtime-ubuntu18.04 RUN apt-get update RUN apt-get install -y python3 python3-pip RUN pip3 install -U pip RUN cd /tmp && python3 -m pip download numpy --no-binary numpy -d ./ RUN ls -alh /tmp RUN python3 -m pip install --no-binary numpy numpy --find-links /tmp --no-index RUN pip3 list ``` Output: ``` Step 7/8 : RUN python3 -m pip install --no-binary numpy numpy --find-links /tmp --no-index ---> Running in b34d3e92b764 Looking in links: /tmp Processing /tmp/numpy-1.18.1.zip Installing build dependencies: started Installing build dependencies: finished with status 'error' ERROR: Command errored out with exit status 1: command: /usr/bin/python3 /usr/local/lib/python3.6/dist-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-6p0zdywu/overlay --no-warn-script-location --no-binary numpy --only-binary :none: --no-index --find-links /tmp -- setuptools wheel 'Cython>=0.29.14' cwd: None Complete output (3 lines): Looking in links: /tmp ERROR: Could not find a version that satisfies the requirement setuptools (from versions: none) ERROR: No matching distribution found for setuptools ---------------------------------------- ERROR: Command errored out with exit status 1: /usr/bin/python3 /usr/local/lib/python3.6/dist-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-6p0zdywu/overlay --no-warn-script-location --no-binary numpy --only-binary :none: --no-index --find-links /tmp -- setuptools wheel 'Cython>=0.29.14' Check the logs for full command output. The command '/bin/sh -c python3 -m pip install --no-binary numpy numpy --find-links /tmp --no-index' returned a non-zero code: 1 ```

christian · June 27, 2024, 12:56pm

Ok, thx, than I will stick to manual download, until this is fixed.

sinoroc · June 27, 2024, 4:47pm

Would it help to build a wheelhouse?

I am not sure, I have not tested this exactly in a while, but for example instead of ~/venv/bin/pip download ~/osml10n/ you could try ~/venv/bin/pip wheel ~/osml10n/. If I am not mistaken this should get you wheels (it will build them if necessary) and then you can move all those wheels to the host machine that does not have internet access.

https://pip.pypa.io/en/stable/topics/repeatable-installs/#using-a-wheelhouse-aka-installation-bundles

christian · June 28, 2024, 8:13am

Thx, didn’t now about pip wheel. Sounds like it fits my needs, and will give it a try.

dhellmann · June 28, 2024, 6:26pm

You may also be interested in GitHub - python-wheel-build/fromager: Build your own wheels as a tool for building that wheelhouse. It does include build dependencies. It’s a new project, so the documentation is still a bit sparse. Let me know if you try it and run into issues.

christian · July 1, 2024, 9:55am

Ok, i checked this, but what is the difference to pip wheel?
As far as I understand, they both do the same thing, aren’t they?

christian · July 2, 2024, 10:19am

After trying, and reading about what wheels actually are, this is the right solution for my case, thx.

dhellmann · July 2, 2024, 1:14pm

pip wheel builds a wheel of something, but it may use pre-built wheels from pypi for the build dependencies (if they exist).

fromager allows you to alway starts from source for every package, even build dependencies.

christian · July 2, 2024, 2:27pm

Ok, I understand. so, pip wheel is sufficient for me, but thx for pointing me there.