Experience with Python 3.11 in Fedora

Hello,

I work for Red Hat as a member of the Python maintenance team in Fedora. My main scope of work is updating the main Python interpreter. Currently, I am rebuilding 3854 Python packages which we have in Fedora with development versions of Python 3.11 to evaluate the impact and report failures to maintainers and upstreams. We do this very early, usually from alpha 1, to make sure we can use the upcoming version of Python as the main interpreter when its final version is released.

I’d like to present some statistics and the most common breakages caused by changes in CPython we have encountered so far. At the time of writing of this, the latest released version of Python 3.11 is alpha 3. So far 2857 packages (86.2%) were rebuilt successfully and the rest failed for various reasons. Usually, it’s because of a missing dependency or some change in CPython which breaks the package. Let’s look at those I encountered repeatedly.

The most breaking changes:

  1. removal of unittest aliases (bpo-45162) - 61 packages
  2. removals from configparser module (bpo-45173) - 28 packages
  3. removal of asyncio.coroutine decorator (bpo-43216) - 16 packages
  4. removals from inspect module (bpo-45320) - 14 packages
  5. Py_TYPE() and Py_SIZE() l-value (bpo-39573) - 7 packages
  6. remove open() “U” mode (bpo-37330) - 4 packages

By far the most common failures are caused by the removal of unittest aliases, from Python 3.11, What’s new:

  • Removed many old deprecated unittest features:
  • TestCase method aliases failUnlessEqual, failIfEqual, failUnless, failIf, failUnlessRaises, failUnlessAlmostEqual, failIfAlmostEqual (deprecated in Python 3.1), assertEquals, assertNotEquals, assert_, assertAlmostEquals, assertNotAlmostEquals, assertRegexpMatches, assertRaisesRegexp (deprecated in Python 3.2), and assertNotRegexpMatches (deprecated in Python 3.5).
  • Undocumented and broken TestCase method assertDictContainsSubset (deprecated in Python 3.2).
  • Undocumented <unittest.TestLoader.loadTestsFromModule>TestLoader.loadTestsFromModule() parameter use_load_tests (deprecated and ignored since Python 3.2).
  • An alias of the TextTestResult class: _TextTestResult (deprecated in Python 3.2).
  • (Contributed by Serhiy Storchaka in bpo-45162.)

This simple change is affecting 61 packages in Fedora. Fixing them is easy, but it’s tedious to do it for all of them.

Affected packages by this change

css-parser
genshi
gpxpy
logutils
pysrt
libreport
nose2
alarmdecoder
uritemplate
httmock
pycurl
pyicu
pyrad
py-radix
geomet
chai
pycdio
scales
idstools
mygpoclient
pyzor
signedjson
python-etcd
libdnf
pycryptodomex
pyelftools
simplevisor
docker-squash
dirq
testresources
messaging
libsbml
ply
listparser
future
pathspec
netcdf4-python
oauthlib
icalendar
yapf
testtools
ephem
stomper
pycollada
gitdb
elements
flaky
nose
xmlrunner
bsddb3
mock
tornado
satyr
parsel
libmodulemd
javabridge
zim
liblarch
argcomplete
jmespath
fail2ban

Then there is a change regarding configargparse module, it affects 28 packages.

  • Removed from the configparser module: the SafeConfigParser class, the filename property of the ParsingError class, the readfp() method of the ConfigParser class, deprecated since Python 3.2.
  • (Contributed by Hugo van Kemenade in bpo-45173.)
Affected packages by this change

bokeh
constantly
ecdsa
plotly
click-spinner
quantities
transforms3d
conda-package-handling
offlineimap
stdlib-list
mom
spake2
grabbit
python-jsonrpc-server
libusb1
pytest-mpi
bids-validator
clyent
cvxopt
scripttester
iniparse
subversion
pastedeploy
setroubleshoot
classification-banner
annexremote
onionbalance
sphinx-math-dollar

A third most common problem is regarding the removal of @asyncio.coroutine decorator.

  • The @asyncio.coroutine decorator enabling legacy generator-based coroutines to be compatible with async/await code. The function has been deprecated since Python 3.8 and the removal was initially scheduled for Python 3.10. Use async def instead.
  • (Contributed by Illia Volochii in bpo-43216.)
Affected packages by this change

pexpect
pycec
rx
aiounittest
pyblackbird
osrf-pycommon
pyduofern
promise
txaio
brother
aiosasl
aioopenssl
aiohttp-cors
pyinsteon
ratelimiter
cached-property

Functions removed from the inspection module are affecting 14 packages.

  • Removed from the inspect module:
  • the getargspec function, deprecated since Python 3.0; use inspect.signature() or inspect.getfullargspec() instead.
  • the formatargspec function, deprecated since Python 3.5; use the inspect.signature() function and Signature object directly.
  • the undocumented Signature.from_callable and Signature.from_function functions, deprecated since Python 3.5; use the Signature.from_callable() method instead.
  • (Contributed by Hugo van Kemenade in bpo-45320.)
Affected packages by this change

bottle
grammalecte
pluginlib
pycha
easyargs
pycdlib
nyx
reportlab
pocketlint
asttokens
verboselogs
powerline
jwcrypto
wrapt

Changing Py_TYPE() and Py_SIZE() into inline static functions was initially supposed to land in Python 3.10. It got reverted and postponed into Python 3.11. We are down to 7 bugs like this since the last year’s 24 of them.

  • Since Py_TYPE() is changed to a inline static function, Py_TYPE(obj) = new_type must be replaced with Py_SET_TYPE(obj, new_type): see the Py_SET_TYPE() function (available since Python 3.9).
  • Since Py_SIZE() is changed to a inline static function, Py_SIZE(obj) = new_size must be replaced with Py_SET_SIZE(obj, new_size): see the Py_SET_SIZE() function (available since Python 3.9).
  • (Contributed by Victor Stinner in bpo-39573.)
Affected packages by this change

pypam
zstd
rdiff-backup
mercurial
pylibacl
python-snappy
zbar

4 packages are affected by the removal of open() “U” mode.

  • open(), io.open(), codecs.open() and fileinput.FileInput no longer accept ‘U’ (“universal newline”) in the file mode. This flag was deprecated since Python 3.3. In Python 3, the “universal newline” is used by default when a file is open in text mode. The newline parameter of open() controls how universal newlines works.
  • (Contributed by Victor Stinner in bpo-37330.)
Affected packages by this change

altgraph
sqlobject
pylibmc
apbs

27 Likes

Of these problems, pyupgrade can automatically resolve removal of unittest aliases and open(..., "U"), accounting for around half the packages you list. Might be useful for packaging, and/or upstream PRs?

(and as an upstream maintainer, thank you! The Fedora team’s work on advance support is a significant contribution to the health of the Python ecosystem :heart_eyes:)

7 Likes

Teyit is another handy tool for upgrading unittest asserts (including things like assertTrue(x in y)assertIn(x, y)).

I checked 20 unittest bugs, and half of them were already fixed. :slight_smile:

Summary

6 were already fixed and released upstream:

4 were already fixed upstream but not yet released:

I created 8 upstream PRs:

1 PR was already created:

1 not fixed, upstream archived:

4 Likes

Thank you both for your suggestions. Maybe it would be useful to mention those programs in the official documentation as a way how to migrate to Python 3.11. Had I known about them, I would include them in Bugzilla reports.

@hugovk thanks a lot for doing this, it’s very helpful.

3 Likes

Indeed, clearly giving people a working recipe that can automatically apply required updates to their Python codebase to run on 3.next is something we need to do more of.

Longer term it’d be ideal if a modernize tool able to easily be told to target fixing up things required in order to run on a specific version were available on PyPI by the time the CPython beta releases roll around.

2 Likes

There is the pyupgrade project for Python code.

I wrote upgrade_pythoncapi.py script in the pythoncapi_compat project for C code (C extension modules).

4 Likes

Followup:

After discussion on python-dev, the top two here have been reverted and will remain in Python 3.11 (October 2022), and instead be removed in Python 3.12 (October 2023) to give more projects time to update.

  1. removal of unittest aliases (bpo-45162) - 61 packages

Reverted in https://github.com/python/cpython/pull/30935

  1. removals from configparser module (bpo-45173) - 28 packages

Reverted in https://github.com/python/cpython/pull/30952

5 Likes

The Python 3.12 branch is now open, here’s a PR to remove the configparser deprecations in 3.12:

I checked the Fedora packages listed above for configparser, and 15/28 (54%) upstreams have fixes (or are unmaintained/removed from Fedora).

2 Likes

Until I carefully read the PR, I thought that it was actually removing the deprecations themselves, rather than the deprecated objects…it might be a good idea to clarify that PR title, heh.

2 Likes