Background:
Currently the bundled pip version in 3.8 - 3.10 did not receive the latest version bumps for the bundled pip wheel file as already in security maintenance mode.
Currently the version bump in e.g. 3.10 from 23.0.1 would address the following bundled issues according to my research:
CVE-2023-5752 (in pip)
CVE-2023-43804 (in bundled urllib)
CVE-2022-40896 (in bundled pygments)
CVE-2023-37920 (in bundled certifi)
I did not do an in depth review if this issues are actually exploitable from the way the libraries are used within pip. Are there some rules in place when a potential security issue in the bundled version triggers a backport?
FWIW, the pip maintainers’ policy is currently to backport new pip releases just to maintained CPython releases, not to security-only ones. We could easily enough change that policy to backport to all maintained releases, including security-only ones. However, most of the changes in any given pip release are not security fixes, so I don’t know if that would be acceptable.
IIRC, some pip releases have gotten backports on older Python versions and each time it’s been on a case-by-case basis; due to a PR being maybe filed by a pip maintainer/CPython release manager and the release manager landing them on the security-only branches.
Thanks for the ping @tjreedy, my understanding of the situation is that ensurepip can be used to bootstrap to a newer pip version which has fixes so the existence of any vulnerability isn’t necessary cause to create a new CPython release. As long as the vulnerability doesn’t impact a python -m pip install --upgrade pip we’re good.
The only vulnerability on that list that has a chance of impacting the bootstrapping process is CVE-2023-37920, which would only matter if e-Tugra was subverted into issuing an invalid certificate for pythonhosted.org (obviously something we don’t want to be possible, but if this happens then we have many more issues beyond ensurepip).
Certifi bundling its certificates is exactly why I’ve been working on getting Truststore support into pip, but it’s only available in Python 3.10+ and not enabled by default.
I think it’d be a good change of policy to always backport new pip releases to older CPython branches where there isn’t a compatibility issue, if only so we don’t miss a security fix and end up having to do another CPython release to ship a new pip version. This seems like something that could be automated with GitHub Actions creating PRs, maybe even Dependabot if it allows custom pip parameters?
I had a look at the last couple of pip update PRs for ensurepip and the current flow is
pip maintainers create an update PR for main
pip maintainer adding the backport label for bug fix releases
backports are handled automatically or manually depending on merge conflicts
From my naïve point of view I do not see much automation possibilities apart from the initial PR creation (step 1). I had a brief look at dependabot and from my understanding this does not support what would be required (see GitHub doc). So I see two topics here:
Should step 2 include also security-only releases? To me it sounds like @sethmlarson is in favor and pip maintainers are open to change current policy. IMHO it would be good to also get input from release managers (@ambv / @pablogsal)
Will pip maintainers benefit from a more automated PR creations for pip updates? Different topic only loosely coupled to this one. If there is some benefit seen I could volunteer to look into it.
Personally (as a pip maintainer) I’d say no. The process is pretty straightforward, and short of some sort of full automation of the whole pip release process, I don’t think we’d gain from automating just this part of it.
Automation that allows us checking the logs of the entire artifact creation would be better to RMs in the sense that it wouldn’t be a random binary blob sent over by a pip contributor. I mean, we trust the people behind the accounts, but as shown recently we can never be 100% sure who is controlling the account.
As for “should we upgrade pip in security-only releases as well”, it seems like we could at least when it comes to pip releases which themselves contain security fixes. However, this might not be entirely easy. As ensurepip in 3.8 - 3.10 also ships (outdated) setuptools, we were faced with a decision whether to upgrade our bundled setuptools to fix a ReDoS CVE. We decided against doing major bumps for setuptools due to risk of breakage as there were breaking changes between what we ship and what is in the current versions.
It follows that we would do the same for pip, i.e. as long as it’s backwards compatible, it should be fine. But, again, it would need to cohabitate with those old setuptools versions.
It should be noted that pip does not have any compatibility guarantees, it uses a roughly {year}.{release}[.{bugfix}] version number and old {year}.{release} versions are not given new bugfixes.
For example the upcoming 24.1 is likely to be a big breaking change for a subset of users compared to 24.0 due to dropping support for legacy versions and specifiers.
Anyone upgrading would therefore need to manually assess the changelog and have a good understanding of the impact on the Python packaging ecosystem.
It should also be noted that ensurepip simply delivers the baseline pip you get in new (venv-based) virtual environments and in your base Python install. All of these can be updated with python -m pip install --upgrade pip. So there’s a very simple fix[1] if ensurepip has an out of date pip.
if it’s not simple, we probably wouldn’t be able to backport anyway ↩︎
Know what? I would assume it’s obvious that you can upgrade pip that way (and if not, it’s documented in the pip docs), and pip’s self-check will warn you that you need to.
FWIW, this process doesn’t require trusting binary blobs unconditionally – we have CI checks in CPython to ensure that the ensurepip binary blobs match what’s on PyPI, and the PRs would be failing CI if they don’t match the (IIRC) sha256 checksum.
It’s also feasible to do such a check manually, and we can include a blurb to execute in pip/devguide docs (basically somewhere trusted), and link from the PR description if that’d be useful for the RMs.
All that said, I don’t think we backported the CI checks to 3.8/3.9 (and am on mobile, so it’s non trivial to check/verify for me right now).
Edit: Also, yea, I’d like to automate the whole release process for pip but it’s a thing that’s non trivial, and I ran out of time/motivation the last few times I looked into this.
Yea, that works fine for pip.
I agree overall. Although pip does make intentional backwards incompatible changes less often than setuptools, my feeling is that Hyrum’s law applies for pip still – basically every change is a breaking change for someone. That said, I think this is a judgement call at the end of the day and it’s fine to handle this on a case by case basis (as we already do?).
