EDIT: I added Miro Hrončok as co-author.
Hi,
While I was reading the great summary of PEP 602 and 605 on LWN, and after my first failed attempt to handle incompatible changes (PEP 606: Python Compatibility Version), I decided to write down an idea that I was thinking about for almost one year. I wrote the PEP that you can find below.
Red Hat has a team working on Fedora to ensure that all Python packages are working well with the next Python (we’re almost done with 3.8, the next will be Python 3.9). We do bug triage for each broken package and attempt to report issues to upstream (broken projects and Python upstream) as much as possible. We are also actively working on fixing these issues (with our limited resources). My colleague Miro Hrončok is doing an amazing job in this area!
The idea here is to do something similar, but directly on Python upstream with a smaller set of Python projects and make it part of the Python release process (PEP 101).
Inexact summary: I propose to include 27 projects to the Python CI
HTML version:
Full text bellow to reply inline.
Victor
PEP: 608
Title: Coordinated Python release
Author: Miro Hrončok miro@hroncok.cz, Victor Stinner vstinner@python.org
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 25-Oct-2019
Python-Version: 3.9
Abstract
Block a Python release until a compatible version of selected projects
is available.
The Python release manager can decide to release Python even if a
project is not compatible, if they decide that the project is going to
be fixed soon enough, or if the issue severity is low enough.
Rationale
The PEP involves maintainers of the selected projects in the Python
release cycle. There are multiple benefit:
- Detect more bugs before a Python final release
- Discuss and maybe revert incompatible changes before a Python final
release - Increase the number of compatible projects when the new Python final
version is released
Too few projects are involved in the Python beta phase
Currently, Python beta versions are available four months before the
final 3.x.0 release.
Bugs reported during the beta phase can be easily fixed and can block a
release if they are serious enough.
Incompatible changes are discussed during the beta phase: enhance
documentation explaining how to update code, or consider to revert these
changes.
Even if more and more projects are tested on the master branch of Python
in their CI, too many projects of the top 50 PyPI projects are only
compatible with the new Python a few weeks, or even months, after the
final Python release.
DeprecatedWarning is being ignored
Python has well defined process to deprecate features. A
DeprecatedWarning must be emitted during at least one Python release,
before a feature can be removed.
In practice, DeprecatedWarning warnings are ignored for years in major
Python projects. Usually, maintainers explain that there are too many
warnings and so they simply ignore warnings. Moreover, DeprecatedWarning
is silent by default (except in the __main__
module: PEP 565 <https://www.python.org/dev/peps/pep-0565/>
_).
Even if more and more projects are running their test suite with
warnings treated as errors (-Werror
), Python core developers still
have no idea how many projects are broken when a feature is removed.
Need to coordinate
When issues and incompatible changes are discovered and discussed after
the final Python release, it becomes way more complicated and expensive
to fix Python. Once an API is part of an official final release, Python
should provide backward compatibility for the whole 3.x release
lifetime. Some operating systems can be shipped with the buggy final
release and can take several months before being updated.
Too many projects are only updated to the new Python after the final
Python release, which makes this new Python version barely usable to run
large applications when Python is released.
It is proposed to block a Python release until a compatible version of
all selected projects is available.
Shorter Python release schedule
The PEP 602: Annual Release Cycle for Python <https://www.python.org/dev/peps/pep-0602/>
_ and the PEP 605: A rolling feature release stream for CPython <https://www.python.org/dev/peps/pep-0605/>
_ would like to release
Python more often to ship new features more quickly.
The problem is that each Python 3.x
release breaks many projects.
Coordinated Python releases reduces the number of broken projects and
makes new Python release more usable.
Specification
By default, a Python release is blocked until a compatible version of
all selected projects is available.
Before releasing the final Python version, the Python release manager is
responsible to send a report of the compatibility status of each project
of the selected projects. It is recommended to send such report at
each beta release to see the evolution and detects issues as soon as
possible.
The Python release manager can decide to release Python even if a
project is not compatible, if they decide that the project is going to
be fixed soon enough, or if the issue severity is low enough.
After each Python release, the project list can be updated to remove
projects and add new ones. For example, to remove old unused
dependencies and add new ones. The list can grow if the whole process
doesn’t block Python releases for too long.
Limit the delay
When a build or test issue with the next Python version is reported to a
project, maintainers have one month to answer. With no answer, the
project can be excluded from the list of projects blocking the Python
release.
Multiple projects are already tested on the master branch of Python in a
CI. Problems can be detected very early in a Python release which should
provide enough time to handle them. More CI can be added for projects
which are not tested on the next Python yet.
Once selected projects issues are known, exceptions can be discussed
between the Python release manager and involved project maintainers on a
case by case basis. Not all issues deserve to block a Python release.
Selected projects
List of projects blocking a Python release (total: 27).
Projects (13):
- aiohttp
- cryptography
- Cython
- Django
- numpy
- pandas
- pip
- requests
- scipy
- Sphinx (needed to build Python)
- sqlalchemy
- pytest
- tox
Direct and indirect dependencies (14):
- certifi (needed by urllib3)
- cffi (needed by cryptography)
- chardet (needed by Sphinx)
- colorama (needed by pip)
- docutils (needed by Sphinx)
- idna (needed by Sphinx and requests)
- jinja2 (needed by Sphinx)
- MarkupSafe (needed by Sphinx)
- psycopg2 (needed by Django)
- pycparser (needed by cffi)
- setuptools (needed by pip and tons of Python projects)
- six (needed by tons of Python projects)
- urllib3 (needed by requests)
- wheel (needed by pip)
How projects are selected
Projects used by to build Python should be in the list, like Sphinx.
Most popular projects are picked from the most downloaded PyPI projects.
Most of project dependencies are included in the list as well, since a
single incompatible dependency can block a whole project. Some
dependencies are excluded to reduce the list length.
Test dependencies as pytest and tox should be included as well. If a
project cannot be tested, a new version cannot be shipped neither.
The list should be long enough to have a good idea of the cost of
porting a project to the next Python, but small enough to not block a
Python release for too long.
Obviously, projects which are not part of the list also are encouraged
to report issues with the next Python and to have a CI running on the
next Python version.
Incompatible changes
The definition here is large: any Python change which cause an issue
when building or testing a project.
See also the PEP 606: Python Compatibility Version <https://www.python.org/dev/peps/pep-0606/>
_ for more examples of
incompatible changes.
Examples
There are different kinds of incompatible changes:
- Change in the Python build. For example, Python 3.8 removed
'm'
(which stands for pymalloc) fromsys.abiflags
which impacts Python vendors like Linux distributions. - Change in the C extensions build. For example, Python 3.8 no longer links C extensions to libpython, and Python 3.7 removed
os.errno
alias to theerrno
module. - Removed function. For example, collections aliases to ABC classes have been removed in Python 3.9.
- Changed function signature. Reject a type which was previously accepted (ex: only accept
int
, rejectfloat
). Add a new mandatory parameter. Convert a positional-or-keyword parameter to positional-only. - Behavior change. For example, Python 3.8 now serializes XML attributes in their insertion order, rather than sorting them by name.
- New warning. Since more and more projects are tested with all warnings treated as errors, any new warning can cause a project test to fail.
- Function removed from the C API.
- Structure made opaque in the C API. For example, PyInterpreterState became opaque in Python 3.8 which broke projects accessing
interp->modules
(PyImport_GetModuleDict()
should be used instead).
Cleaning up Python and DeprecationWarning
One of the Zen of Python (PEP 20) <https://www.python.org/dev/peps/pep-0020/>
_ motto is:
There should be one-- and preferably only one --obvious way to do it.
When Python evolves, new ways emerge inevitably. DeprecationWarning
are emitted to suggest to use the new way, but many developers ignore
these warnings which are silent by default.
Sometimes, supporting both ways has a minor maintenance cost, but Python
core developers prefer to drop the old way to clean up the Python code
base and standard library. Such kind of change is backward incompatible.
More incompatible changes than usual should be expected with the end of
the Python 2 support which is a good opportunity to cleaning up old
Python code.
Distributed CI
Checking if selected projects are compatible with the master branch
of Python can be automated using a distributed CI.
Existing CIs can be reused.
New CIs can be added for projects which are not tested on the next
Python yet.
It is recommended to treat DeprecationWarning warnings as errors when
testing on the next Python.
A job testing a project on the next Python doesn’t have to be
“mandatory” (block the whole CI). It is fine to have failures during the
beta phase of a Python release. The job only has to pass for the final
Python release.
Copyright
This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.
–
Night gathers, and now my watch begins. It shall not end until my death.