Avoiding compromised certificates in pip

Packaging
1

Hi, another pip maintainer here :waving_hand:

I’ve been investigating alternative remedies for those unable to upgrade pip immediately. As far as I can tell, setting PIP_CERT to a valid CA bundle that has been patched (edit: AND setting PIP_USE_DEPRECATED=legacy-certs if on Python 3.10+, see below) is sufficient to stop pip from using the vendored certifi bundle.

Some caveats:

This remedy can be confirmed by passing a patched, empty, or near empty CA bundle via PIP_CERT and observing that the TLS verification fails despite the vendored certifi for a command like PIP_CERT=bundle... pip install -r https://dv-g3b-emudhra.we-have-a-problem.com/. (If it is vulnerable, then you should get a parse error that HTML is not a valid requirement file.)

I would investigate further to determine whether PIP_USE_DEPRECATED=legacy-certs is needed for Python 3.10 and higher, but I am very busy in my personal life so I won’t have the time to do so until much later, so I’m sharing what I have so far for now.

I have also not investigated REQUESTS_CA_BUNDLE at all.

2 Likes
Information for Python users regarding eMudhra / emSign CA certificate security incident
2

The PIP_CERT trick does not work for me with pip 26.2.1.

$ pip install -U pip==26.2.1
$ PIP_CERT=/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem pip install -v emudhra --index-url https://dv-g3b-emudhra.we-have-a-problem.com/
Using pip 26.2.1 from /tmp/testenv/lib64/python3.12/site-packages/pip (python 3.12)
Looking in indexes: https://dv-g3b-emudhra.we-have-a-problem.com/
ERROR: Could not find a version that satisfies the requirement emudhra (from versions: none)
ERROR: No matching distribution found for emudhra

legacy-certs + PIP_CERT work:

$ PIP_USE_DEPRECATED=legacy-certs PIP_CERT=/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem pip install -v emudhra --index-url https://dv-g3b-emudhra.we-have-a-problem.com/
Using pip 26.2.1 from /tmp/testenv/lib64/python3.12/site-packages/pip (python 3.12)
Looking in indexes: https://dv-g3b-emudhra.we-have-a-problem.com/
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:998)'))': /emudhra/

curl and uv refuse a connection:

$ curl --cacert /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem https://dv-g3b-emudhra.we-have-a-problem.com/
curl: (60) SSL certificate problem: unable to get local issuer certificate
...
$ UV_SYSTEM_CERTS=1 uv pip install emudhra --index-url https://dv-g3b-emudhra.we-have-a-problem.com/
Using Python 3.12.14 environment at: /tmp/testenv
⠙ Resolving dependencies...                                                                                                                                                                                    
error: Failed to fetch: `https://dv-g3b-emudhra.we-have-a-problem.com/emudhra/`
  Caused by: error sending request for url (https://dv-g3b-emudhra.we-have-a-problem.com/emudhra/)
  Caused by: client error (Connect)
  Caused by: invalid peer certificate: UnknownIssuer
1 Like
3

I really wish I knew why PIP_CERT isn’t sufficient on Python 3.10+ for you, but I honestly do not know. Are you sure that pip isn’t caching the index pages? Recent versions of pip respect the cache control header for simple API index page responses. Try setting PIP_NO_CACHE=1 or pip cache purge.

I’m glad the legacy-certs and PIP_CERT combo does work, though.

Edit: the cache shouldn’t… matter? or at least I’d expect the other commands to work if it were cached, but TBH, I am exhausted and not of the mental clarity needed to look further. I’ll update my comment to recommend PIP_USE_DEPRECATED for now.

4

@ichard26 Your recommendation does not work because pip always loads certifi certificates in addition to system certs from truststore. PIP_CERT or --cert do not change the code path. Only legacy-certs trigger a different path that does not load certifi. I explained the issue in my initial comment Information for Python users regarding eMudhra / emSign CA certificate security incident - #3 by tiran

You can easily reproduce the problem by either strace-ing the process or by making the cacert.pem unreadable:

$ chmod 0 /tmp/testenv/lib/python3.14/site-packages/pip/_vendor/certifi/cacert.pem
$ PIP_CERT=/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem pip install emudhra --index-url https://dv-g3b-emudhra.we-have-a-problem.com/
WARNING: There was an error checking the latest version of pip.
ERROR: Exception:
Traceback (most recent call last):
  File "/tmp/testenv/lib64/python3.14/site-packages/pip/_internal/cli/base_command.py", line 109, in _run_wrapper
    status = _inner_run()
  File "/tmp/testenv/lib64/python3.14/site-packages/pip/_internal/cli/base_command.py", line 102, in _inner_run
    return self.run(options, args)
           ~~~~~~~~^^^^^^^^^^^^^^^
  File "/tmp/testenv/lib64/python3.14/site-packages/pip/_internal/cli/req_command.py", line 106, in wrapper
    return func(self, options, args)
  File "/tmp/testenv/lib64/python3.14/site-packages/pip/_internal/commands/install.py", line 431, in run
    session = self.get_default_session(options)
  File "/tmp/testenv/lib64/python3.14/site-packages/pip/_internal/cli/index_command.py", line 80, in get_default_session
    self._session = self.enter_context(self._build_session(options))
                                       ~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/tmp/testenv/lib64/python3.14/site-packages/pip/_internal/cli/index_command.py", line 99, in _build_session
    ssl_context = _create_truststore_ssl_context()
  File "/tmp/testenv/lib64/python3.14/site-packages/pip/_internal/cli/index_command.py", line 50, in _create_truststore_ssl_context
    ctx.load_verify_locations(certifi.where())
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/tmp/testenv/lib64/python3.14/site-packages/pip/_vendor/truststore/_api.py", line 160, in load_verify_locations
    return self._ctx.load_verify_locations(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        cafile=cafile, capath=capath, cadata=cadata
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
PermissionError: [Errno 13] Permission denied
3 Likes
5

Yes, over at pip we need to reassess our defaults, what options we provide to users, fix some documentation, and fix some bugs. We’re starting on Switch to using truststore only by default (currently mixed certifi + truststore) · Issue #14374 · pypa/pip · GitHub, but I’m sure there will be several follow up issues and PRs.

For now, as you say, if you want to enforce your cert bundle only you can run:

--use-deprecated=legacy-certs --cert bundle.pem

To set this legacy-certs mode in an env you can run:

# Unix-like:
export PIP_USE_DEPRECATED=legacy-certs 

# Powershell:
$env:PIP_USE_DEPRECATED = "legacy-certs"

# CMD:
set "PIP_USE_DEPRECATED=legacy-certs"

To set it via configuration you can run:

pip config --user set global.use-deprecated legacy-certs

To give a little bit of a defense as to how we got here, a lot of the choices were made to encourage people to start using SSL verification without breaking their existing setups. It was not that long ago that every guide I read inside a large organization included something like --trusted-host pypi.org --trusted-host files.pythonhosted.org for running any pip command, completely disabling verification.

4 Likes
6

(For Linux systems)

Correct me if I’m wrong, but can’t this be overridden with PIP_CERT env vars, the --cert option to pip, or in the configuration file for pip by defining this? (at least on Linux systems)

[global]
cert = /etc/ssl/certs/ca-certificates.crt

From what I can tell this would override the certificate bundle as well.

(For what it’s worth, I fully agree with switching to using truststore by default - modified versions of pip and certifi do this in Ubuntu and Debian distributions when installed via apt to enforce the use of the system CA certificates store - though that’s not as relevant to Python and pip itself, it makes more sense to at least note it)

7

No, it does not disable the certificate bundle. The default code path always injects the CA bundle from vendored certifi. You also have to enable the legacy-cert code path to disable ctx.load_verify_locations(certifi.where()).

3 Likes
8

@tiran Hmm, that explains some things there. Assuming this is done via either --use-deprecated=legacy-cert or similar config files in the pip config file, then the system CA cert bundle (inherited by default I presume from base SSL contexts) would be used, in addition to --cert defined certificates?

9

@teward It is … complicated. ssl.SSLContext() does not load any certificates from the system trust store. Many years ago I introduced ssl.create_default_context(). That function returns an SSLContext with trust anchors from the system trust store.

But there is an additional catch! OpenSSL has hard-coded path to cert PEM bundle and directory to hashed CA cert files. Linux distributions do not agree on the location of the CA certs. Alpine, Debian/Ubuntu, Fedora/CentOS/RHEL, Gentoo, SuSE … they all disagree and compile OpenSSL with different paths. Here is where truststore package comes into play. It checks multiple locations.

Most recent Linux distros use /etc/ssl/cert.pem. For example I added a cross-distro compatibility feature to Fedora, CentOS, and RHEL in 2020, 1895619 – Symlink cert.pem and openssl.cnf to /etc/ssl for cross-distro compatibility . Fedora 44 removed the link for a brief moment during its development cycle.

1 Like
10

If I understand correctly, is ssl.create_default_context() just enough if I’m not building my own OpenSSL from source? Since the stdlib ssl links against the system OpenSSL library, I would expect the certificate paths to be correct. Is that right?

11

Yes – assuming your OpenSSL is correctly configured, you are not on macOS, and you don’t copy your binaries to a machine with a different Linux distribution.

1 Like