SELECT
details.system.name,
details.distro.name,
details.distro.version,
details.implementation.version,
details.installer.name,
COUNT(*) AS count
FROM
[the-psf:pypi.downloads20190612]
WHERE
file.project = "awscli"
GROUP BY
details.system.name,
details.distro.name,
details.distro.version,
details.implementation.version,
details.installer.name,
ORDER BY
count DESC
The result is here:
Oh, “null” is the most major distro! I don’t know what environment are included in “null”.
Ubuntu 16.04 is the second most major distro. But I believe Ubuntu 18.04 will overtake it at some point.
Not that I don’t share this goal, but IIRC Donald has 20% of his time allocated to OSS, so Amazon is one of the few companies paying people to support packaging. Of course, the burden spreads further than Donald, but I like to take every opportunity I can to recognize companies that are actually doing The Right Thing™ and supporting OSS.
Oh yeah, for sure. I think it’s more than 20% even. I’m not saying we should get out the pitchforks :-). It’s much appreciated. But, just to overstate the obvious, it doesn’t mean they can stay on py2 forever while volunteers like Paul and Pradyun support them. So they still need some kind of plan.
You’re forgiven I forgot that there is at least one good reason why ensurepip isn’t as well known in Python 2.7. Besides the fact that is was added late in 2.7’s life, we also elected to make its installation opt-in so, unlike with Python 3, you need to explicitly add --with-ensurepip to ./configure.
On Windows and Mac OS X, the CPython installers now default to installing pip along with CPython itself (users may opt out of installing it during the installation process).
I’m pretty sure it’s possible to craft a query that shows what proportion of Py2 downloads come from what packages.
If we see a substantial amount of downloads for a certain package, that’ll be an indicator here if there’s other places to look. (It’s basically the same analysis as @methane and @njs applied to aws-cli but more generally applied)
I ran a query (group by project-python_version-week-year, top 16000 by count, for past 6 months) on the downloads table in BigQuery (context), and dumped the data into a GitHub Gist. The simple API records, are too many to query from it without hitting my puny 1TB free tier limit.
Be warned, opening the Gist will result in a huge HTML page being served to you (JSON is ~2.5 MB and GitHub serves all of it highlighted – the loaded DOM is ~500 MB).
If someone knows of a cool way to generate insights from this, that’d be nice. In the interest of maintaining everyone’s sanity, here’s the raw (plain text) dump of the Gist files:
BTW, awscli-dwlogs is downloaded about 330k/day, and awslogs-agent-setup.py downloads 17 packages (pip, virtualenv, awscli-cwlogs, and it’s dependencies). 330k * 17 is about 5.6M DL/day.
If awslogs-agent-setup.py is updated to download from S3 instead of PyPI, 5.6M DL/day will be
disappeared from PyPI download stats!
I’d be curious as to whether the projects listed in the Python 3 statement have seen declines in the proportions of Python 2-related downloads from PyPI since we published the FAQ in September.
@jaraco has merged and released a PR dropping Python 2 support for setuptoolshere. We still have the compatibility shims in place, so it shouldn’t stop working if people use it even in Python 2, but we are still discussing whether or not to revert this PR.
If other PyPA projects have strong feelings one way or the other, now is a good time to comment.
I’m on record as wanting pip to drop Python 2 support sooner rather than later, so please take this comment with that in mind.
As pip vendors pkg_resources (which is part of setuptools), we have to pick a version to vendor (I suppose we could in theory vendor two versions, and pick one at runtime, or something like that, but that’s a lot messier). Am I correct in that assumption, or is there some clever way of keeping up to date while still supporting Python 2 that I’m missing?
If we do have to (in effect) pin pkg_resources, what will be the impact on interoperability standards development? Do such standards typically affect pkg_resources? I’m not so worried about build processes, as we install an independent copy of setuptools in the build environment.
My feeling is that pip will need to pin pkg_resources, but this won’t be as significant a problem as it would be if we relied on the broader setuptools package. I do think that we should have a discussion, and projects should be careful about unilaterally dropping support for Python 2, so thanks for raising this, @pganssle.
I think the key thing we need to consider here is moving the ecosystem forward. If all that mattered was support and bug fixes, I think that telling Python 2 users to just use the last version of the packaging tools with Python 2 support would be fine. But the real question is how do we drive adoption of new standards like pyproject.toml and later versions of manylinux if Python 2 projects are required to pin their packaging tools. I don’t think it’s impossible (for example, pure Python projects can just build universal wheels using Python 3 - they might need to change their workflow, but I’m OK with that) but if we could offer transition advice then that would likely smooth the process.
This is definitely something we should discuss at the packaging summit, IMO.
Edit: Apparently (see this pip issue) some people build their own wheel mirrors, and don’t necessarily remember to extract the Requires-Python metadata and expose it via the index. Unfortunately, this seems like it would be a common mistake to make, which somewhat reduces the effectiveness of the data-requires-python tag
We have an open issue in pip to transition from pkg_resources to importlib_metadata. It doesn’t address the larger issue, but would make setuptools dropping support for Python 2 less of a risk to pip specifically.
As much as I would love to drop Python 2 on pip, it seems that a significant number of our users aren’t there yet.
Concerning setuptools, I think a deprecation notice of a few months would have made sense and feel that the drop was quite sudden.
All the libraries using setuptools and still supporting Python 2 are now likely in an uncomfortable place of choosing between pinning setuptools or having a different version (and thus different options) depending on the interpreter version…
I wouldn’t be surprised if importlib_metadata were to also drop support for Python 2 in a near future
Of course, the burden spreads further than Donald, but I like to take every opportunity I can to recognize companies that are actually doing The Right Thing™ and supporting OSS. 
I forgot that there is at least one good reason why ensurepip isn’t as well known in Python 2.7. Besides the fact that is was added late in 2.7’s life, we also elected to make its installation opt-in so, unlike with Python 3, you need to explicitly add 