Pip install package from GitLab Package Registry and dependencies from PyPI mirror index

Python Help
,
1

Hey everyone,

I would like to install my-package and only my-package from the package registry of a specific GitLab project, but install its dependencies from an alternative index, which is a PyPI mirror (we’ll use https://some-pypi-mirror.org/pypi/simple here).

The closest I’ve been able to come up with is:

python3 -m pip install -v --no-cache-dir \
    "my-package @ git+https://__token__:<GITLAB_TOKEN>@gitlab.com/some-group/my-package" \
    --index-url https://some-pypi-mirror.org/pypi/simple

This command works. However, I do not want to install from source (i.e. the project URL) but rather the built wheel available in the package registry. In addition, I would like to be able to optionally specify a package version, which this approach obviously doesn’t support either:

python3 -m pip install -v --no-cache-dir \
    "my-package==0.8 @ git+https://__token__:<GITLAB_TOKEN>@gitlab.com/some-group/my-package" \
    --index-url https://some-pypi-mirror.org/pypi/simple

ERROR: Invalid requirement 'my-package==0.8 git+https://[...]'

To summarize, is there a way:

  • to install my-package (and only that) via the built wheel available in a GitLab package registry,
  • which supports optionally defining a version specifier (e.g. my-package==0.8),
  • and to then resolve the dependencies of my-package via the PyPI mirror?

Any help is greatly appreciated. Thank you very much in advance!

Best regards,

Ryan

2

Maybe use something like simpleindex, or any other more complex proxy/mirror solution.

3

A workaround, if you know what the dependencies should be: install them first (to the same environment), and then when installing the main package, Pip will recognize that the dependencies are already installed and skip trying to grab them from the main index.

4

I’ve been able to do something similar using a pip configuration file within a Python virtual environment (the only difference between what I had done and what you’re asking is that I had my wheel’s dependencies pulled from official PyPI instead of a mirror, but I think this method should still work for you).

Essentially, what you’ll need to do is configure pip’s index URL to point to your GitLab project repository, then configure pip’s extra index URL to point to your PyPI mirror. Configuring the index URL to point to your project’s repository rather than using the extra index URL will make pip prioritize searching there first (so just in case there’s a package with the same name in your mirror, pip will download the one from your project’s repository). Then pip will install the dependencies from the extra index URL.

With this configuration, you should be able to do pip install my-package==0.8 from within your virtual environment to install from your GitLab repository (and the package’s dependencies from your PyPI mirror).

Also, just as a disclosure, this is all from memory as I don’t currently have access to my configuration to refer to, so I may have gotten something slightly incorrect (in particular, the exact pip configuration settings), but this is the gist of it. I’ll try to update this response tomorrow once I’ve double-checked the exact configuration settings.

UPDATE:

# <venv>/pip.conf
[global]
index-url = https://__token__:<access token>@gitlab.com/api/v4/groups/<group>/-/packages/pypi/simple
extra-index-url = <PyPI mirror>
5

Assuming you are using Gitlab 14.2 or later, Gitlab will automatically forward requests for any packages of doesn’t have to PyPI

Not sure if there’s an easy way to configure that to point to a PyPI mirror instead of PyPI itself though

6

@ sinoroc

Thanks for the hint. I’ve looked into that project, but I’m looking for a solution as native as possible. In other words, the use case I’ve described (i.e. have pip resolve dependencies over a specific index) doesn’t appear too special to me, and I think that it should be possible to solve with pip itself.

@ kknechtel

Thanks for your reply. That’s actually a good hint. However, in my case, dependencies vary and there’s no reasonable way to find them out in advance, I’m afraid.

@ bytemarx

Thank you for your reply. I’ve tried what you suggested, but, unfortunately, that doesn’t work in my case. I’ve set up pip.conf to use the GitLab Package Registry URL as index-url and to use the PyPI mirror URL as extra-index-url. However, I get the following error:

python3 -m pip --debug install -v --no-cache-dir --ignore-installed --dry-run my-package

Using pip 23.3.1 from [...]\Lib\site-packages\pip (python 3.11)
Looking in indexes: https://[...]@gitlab.com/api/v4/projects/[...]/packages/pypi/simple, https://some-pypi-mirror.org/pypi/simple
Collecting my-package
  Downloading https://gitlab.com/api/v4/projects/[...]/packages/pypi/files/[...]/my_package-0.1-py3-none-any.whl (7.4 kB)
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionE
rror('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x0000019C6251E350>: Failed to establish a new connection: [Errno
11001] getaddrinfo failed')': /simple/requests/

What I suspect is happening here is that GitLab is forwarding me to pypi.org in order to resolve the dependencies, as suggested here. And since my work machine is behind a firewall that prevents me from accessing pypi.org (which is the origin of the entire problem), it cannot connect to it. I know that the error shown above indicates a DNS resolution problem, but I think that this is related to the firewall configuration, too. If I run nslookup pypi.org in cmd (Windows), it fails to resolve the address, too.

In fact, I had already tried out what you’ve suggested, with the only difference that I hadn’t configured these settings permanently in pip.conf but just passed them as command-line arguments (--index-url <GitLab> --extra-index-url <PyPI mirror>), which I would expect to have the same effect.

@ jamesdow21

Thanks for your reply. What you’re suggesting seems to be true. We’ve got our own internal instance of GitLab running in the company VPN, which is the GitLab instance I’m targeting in terms of accessing its Package Registry. And that instance of GitLab seems to be forwarding me to pypi.org. However, as pointed out above, the company firewall prevents me from accessing PyPI, so the GitLab’s forwarding fails. Unfortunately, I doubt that our GitLab admins will configure GitLab to globally point to the PyPI mirror.