Packaging and Python 2

I tried this query:

SELECT
  details.system.name,
  details.distro.name,
  details.distro.version,
  details.implementation.version,
  details.installer.name,
  COUNT(*) AS count
FROM
  [the-psf:pypi.downloads20190612]
WHERE
  file.project = "awscli"
GROUP BY
  details.system.name,
  details.distro.name,
  details.distro.version,
  details.implementation.version,
  details.installer.name,
ORDER BY
  count DESC

The result is here:

Oh, “null” is the most major distro! I don’t know what environment are included in “null”.
Ubuntu 16.04 is the second most major distro. But I believe Ubuntu 18.04 will overtake it at some point.

I found very interesting comment.

For CLI v2 we’re planning on distributing standalone installers for all platforms to avoid these and other issues.
https://github.com/aws/aws-cli/issues/4175#issuecomment-497881116

Not that I don’t share this goal, but IIRC Donald has 20% of his time allocated to OSS, so Amazon is one of the few companies paying people to support packaging. :wink: Of course, the burden spreads further than Donald, but I like to take every opportunity I can to recognize companies that are actually doing The Right Thing™ and supporting OSS. :smile:

1 Like

(Especially since py2 doesn’t ship with pip.)

( ?? -> https://docs.python.org/2/library/ensurepip.html)

Oh yeah, for sure. I think it’s more than 20% even. I’m not saying we should get out the pitchforks :-). It’s much appreciated. But, just to overstate the obvious, it doesn’t mean they can stay on py2 forever while volunteers like Paul and Pradyun support them. So they still need some kind of plan.

Huh, whoops. Today I learned!

Huh, whoops. Today I learned!

You’re forgiven :slight_smile: I forgot that there is at least one good reason why ensurepip isn’t as well known in Python 2.7. Besides the fact that is was added late in 2.7’s life, we also elected to make its installation opt-in so, unlike with Python 3, you need to explicitly add --with-ensurepip to ./configure.

.

Is it included by default on the python.org installer?


Edit: Yes. https://docs.python.org/3/whatsnew/2.7.html#bootstrapping-pip-by-default says:

On Windows and Mac OS X, the CPython installers now default to installing pip along with CPython itself (users may opt out of installing it during the installation process).

1 Like

I’m pretty sure it’s possible to craft a query that shows what proportion of Py2 downloads come from what packages.

If we see a substantial amount of downloads for a certain package, that’ll be an indicator here if there’s other places to look. (It’s basically the same analysis as @methane and @njs applied to aws-cli but more generally applied)

I ran a query (group by project-python_version-week-year, top 16000 by count, for past 6 months) on the downloads table in BigQuery (context), and dumped the data into a GitHub Gist. The simple API records, are too many to query from it without hitting my puny 1TB free tier limit.

Be warned, opening the Gist will result in a huge HTML page being served to you (JSON is ~2.5 MB and GitHub serves all of it highlighted – the loaded DOM is ~500 MB).

The GitHub Gist: https://gist.github.com/pradyunsg/f6ebfd39a5b21a097179f8b5f8b5fcb9


If someone knows of a cool way to generate insights from this, that’d be nice. In the interest of maintaining everyone’s sanity, here’s the raw (plain text) dump of the Gist files:

The raw query: https://gist.githubusercontent.com/pradyunsg/f6ebfd39a5b21a097179f8b5f8b5fcb9/raw/abb15158674b576c896814e69a8ffe1b9f9dbf22/query.sql
The raw resulting JSON: https://gist.githubusercontent.com/pradyunsg/f6ebfd39a5b21a097179f8b5f8b5fcb9/raw/abb15158674b576c896814e69a8ffe1b9f9dbf22/results-20190621-195401.json


And, with that I hit the “3 consecutive posts only” limit of Discourse.

I wrote an article to advocate using Python 3 for awscli.

https://dev.to/methane/use-pip3-to-install-awscli-44pk

And I made a pull request to Travis document.

Let’s advocate Python 3 more!

1 Like

I think about 14 (awscli and it’s dependencies) * 200k DL/day are from not normal users. I suspect AWS downloads them.
I filed an issue about it.

1 Like

Thanks @methane for the investigation, the blog post and for filing the issue w/ AWS. Much appreciated. =D

1 Like

I found awslogs-agent-setup.py installs awscli-cwlogs after installing pip<7.0.0.

I reported it to GitHub already.

BTW, awscli-dwlogs is downloaded about 330k/day, and awslogs-agent-setup.py downloads 17 packages (pip, virtualenv, awscli-cwlogs, and it’s dependencies). 330k * 17 is about 5.6M DL/day.

If awslogs-agent-setup.py is updated to download from S3 instead of PyPI, 5.6M DL/day will be
disappeared from PyPI download stats!

2 Likes

Update regarding the general Python 2 sunset:

Here’s the simple FAQ to circulate widely, originally published in early September (focusing on the fact that updates to Python 2 will stop on Jan. 1, 2020), and here’s the press release that just went out (the last minor release of Python 2.7 will be released in April 2020).

I’d be curious as to whether the projects listed in the Python 3 statement have seen declines in the proportions of Python 2-related downloads from PyPI since we published the FAQ in September.

I made charts for some projects, between January 2016 and November 2019 (pip, six, coverage.py, Django, Flake8, pylint haven’t signed it):

(Jan 2016 - Oct 2019: https://dev.to/hugovk/python-version-share-over-time-5-7ik)

1 Like

@jaraco has merged and released a PR dropping Python 2 support for setuptools here. We still have the compatibility shims in place, so it shouldn’t stop working if people use it even in Python 2, but we are still discussing whether or not to revert this PR.

If other PyPA projects have strong feelings one way or the other, now is a good time to comment.

I’m on record as wanting pip to drop Python 2 support sooner rather than later, so please take this comment with that in mind.

As pip vendors pkg_resources (which is part of setuptools), we have to pick a version to vendor (I suppose we could in theory vendor two versions, and pick one at runtime, or something like that, but that’s a lot messier). Am I correct in that assumption, or is there some clever way of keeping up to date while still supporting Python 2 that I’m missing?

If we do have to (in effect) pin pkg_resources, what will be the impact on interoperability standards development? Do such standards typically affect pkg_resources? I’m not so worried about build processes, as we install an independent copy of setuptools in the build environment.

My feeling is that pip will need to pin pkg_resources, but this won’t be as significant a problem as it would be if we relied on the broader setuptools package. I do think that we should have a discussion, and projects should be careful about unilaterally dropping support for Python 2, so thanks for raising this, @pganssle.

I think the key thing we need to consider here is moving the ecosystem forward. If all that mattered was support and bug fixes, I think that telling Python 2 users to just use the last version of the packaging tools with Python 2 support would be fine. But the real question is how do we drive adoption of new standards like pyproject.toml and later versions of manylinux if Python 2 projects are required to pin their packaging tools. I don’t think it’s impossible (for example, pure Python projects can just build universal wheels using Python 3 - they might need to change their workflow, but I’m OK with that) but if we could offer transition advice then that would likely smooth the process.

This is definitely something we should discuss at the packaging summit, IMO.

2 Likes

https://github.com/pypa/virtualenv/issues/1493 appears relevant as well.

Edit: Apparently (see this pip issue) some people build their own wheel mirrors, and don’t necessarily remember to extract the Requires-Python metadata and expose it via the index. Unfortunately, this seems like it would be a common mistake to make, which somewhat reduces the effectiveness of the data-requires-python tag :slightly_frowning_face:

We have an open issue in pip to transition from pkg_resources to importlib_metadata. It doesn’t address the larger issue, but would make setuptools dropping support for Python 2 less of a risk to pip specifically.

1 Like

As much as I would love to drop Python 2 on pip, it seems that a significant number of our users aren’t there yet.
Concerning setuptools, I think a deprecation notice of a few months would have made sense and feel that the drop was quite sudden.
All the libraries using setuptools and still supporting Python 2 are now likely in an uncomfortable place of choosing between pinning setuptools or having a different version (and thus different options) depending on the interpreter version…

I wouldn’t be surprised if importlib_metadata were to also drop support for Python 2 in a near future :wink:

1 Like