Experience with Python 3.11 in Fedora

Core Development
1

Hello,

I work for Red Hat as a member of the Python maintenance team in Fedora. My main scope of work is updating the main Python interpreter. Currently, I am rebuilding 3854 Python packages which we have in Fedora with development versions of Python 3.11 to evaluate the impact and report failures to maintainers and upstreams. We do this very early, usually from alpha 1, to make sure we can use the upcoming version of Python as the main interpreter when its final version is released.

I’d like to present some statistics and the most common breakages caused by changes in CPython we have encountered so far. At the time of writing of this, the latest released version of Python 3.11 is alpha 3. So far 2857 packages (86.2%) were rebuilt successfully and the rest failed for various reasons. Usually, it’s because of a missing dependency or some change in CPython which breaks the package. Let’s look at those I encountered repeatedly.

The most breaking changes:

  1. removal of unittest aliases (bpo-45162) - 61 packages
  2. removals from configparser module (bpo-45173) - 28 packages
  3. removal of asyncio.coroutine decorator (bpo-43216) - 16 packages
  4. removals from inspect module (bpo-45320) - 14 packages
  5. Py_TYPE() and Py_SIZE() l-value (bpo-39573) - 7 packages
  6. remove open() “U” mode (bpo-37330) - 4 packages

By far the most common failures are caused by the removal of unittest aliases, from Python 3.11, What’s new:

  • Removed many old deprecated unittest features:
  • TestCase method aliases failUnlessEqual, failIfEqual, failUnless, failIf, failUnlessRaises, failUnlessAlmostEqual, failIfAlmostEqual (deprecated in Python 3.1), assertEquals, assertNotEquals, assert_, assertAlmostEquals, assertNotAlmostEquals, assertRegexpMatches, assertRaisesRegexp (deprecated in Python 3.2), and assertNotRegexpMatches (deprecated in Python 3.5).
  • Undocumented and broken TestCase method assertDictContainsSubset (deprecated in Python 3.2).
  • Undocumented <unittest.TestLoader.loadTestsFromModule>TestLoader.loadTestsFromModule() parameter use_load_tests (deprecated and ignored since Python 3.2).
  • An alias of the TextTestResult class: _TextTestResult (deprecated in Python 3.2).
  • (Contributed by Serhiy Storchaka in bpo-45162.)

This simple change is affecting 61 packages in Fedora. Fixing them is easy, but it’s tedious to do it for all of them.

Affected packages by this change

css-parser
genshi
gpxpy
logutils
pysrt
libreport
nose2
alarmdecoder
uritemplate
httmock
pycurl
pyicu
pyrad
py-radix
geomet
chai
pycdio
scales
idstools
mygpoclient
pyzor
signedjson
python-etcd
libdnf
pycryptodomex
pyelftools
simplevisor
docker-squash
dirq
testresources
messaging
libsbml
ply
listparser
future
pathspec
netcdf4-python
oauthlib
icalendar
yapf
testtools
ephem
stomper
pycollada
gitdb
elements
flaky
nose
xmlrunner
bsddb3
mock
tornado
satyr
parsel
libmodulemd
javabridge
zim
liblarch
argcomplete
jmespath
fail2ban

Then there is a change regarding configargparse module, it affects 28 packages.

  • Removed from the configparser module: the SafeConfigParser class, the filename property of the ParsingError class, the readfp() method of the ConfigParser class, deprecated since Python 3.2.
  • (Contributed by Hugo van Kemenade in bpo-45173.)
Affected packages by this change

bokeh
constantly
ecdsa
plotly
click-spinner
quantities
transforms3d
conda-package-handling
offlineimap
stdlib-list
mom
spake2
grabbit
python-jsonrpc-server
libusb1
pytest-mpi
bids-validator
clyent
cvxopt
scripttester
iniparse
subversion
pastedeploy
setroubleshoot
classification-banner
annexremote
onionbalance
sphinx-math-dollar

A third most common problem is regarding the removal of @asyncio.coroutine decorator.

  • The @asyncio.coroutine decorator enabling legacy generator-based coroutines to be compatible with async/await code. The function has been deprecated since Python 3.8 and the removal was initially scheduled for Python 3.10. Use async def instead.
  • (Contributed by Illia Volochii in bpo-43216.)
Affected packages by this change

pexpect
pycec
rx
aiounittest
pyblackbird
osrf-pycommon
pyduofern
promise
txaio
brother
aiosasl
aioopenssl
aiohttp-cors
pyinsteon
ratelimiter
cached-property

Functions removed from the inspection module are affecting 14 packages.

  • Removed from the inspect module:
  • the getargspec function, deprecated since Python 3.0; use inspect.signature() or inspect.getfullargspec() instead.
  • the formatargspec function, deprecated since Python 3.5; use the inspect.signature() function and Signature object directly.
  • the undocumented Signature.from_callable and Signature.from_function functions, deprecated since Python 3.5; use the Signature.from_callable() method instead.
  • (Contributed by Hugo van Kemenade in bpo-45320.)
Affected packages by this change

bottle
grammalecte
pluginlib
pycha
easyargs
pycdlib
nyx
reportlab
pocketlint
asttokens
verboselogs
powerline
jwcrypto
wrapt

Changing Py_TYPE() and Py_SIZE() into inline static functions was initially supposed to land in Python 3.10. It got reverted and postponed into Python 3.11. We are down to 7 bugs like this since the last year’s 24 of them.

  • Since Py_TYPE() is changed to a inline static function, Py_TYPE(obj) = new_type must be replaced with Py_SET_TYPE(obj, new_type): see the Py_SET_TYPE() function (available since Python 3.9).
  • Since Py_SIZE() is changed to a inline static function, Py_SIZE(obj) = new_size must be replaced with Py_SET_SIZE(obj, new_size): see the Py_SET_SIZE() function (available since Python 3.9).
  • (Contributed by Victor Stinner in bpo-39573.)
Affected packages by this change

pypam
zstd
rdiff-backup
mercurial
pylibacl
python-snappy
zbar

4 packages are affected by the removal of open() “U” mode.

  • open(), io.open(), codecs.open() and fileinput.FileInput no longer accept ‘U’ (“universal newline”) in the file mode. This flag was deprecated since Python 3.3. In Python 3, the “universal newline” is used by default when a file is open in text mode. The newline parameter of open() controls how universal newlines works.
  • (Contributed by Victor Stinner in bpo-37330.)
Affected packages by this change

altgraph
sqlobject
pylibmc
apbs

28 Likes
2

Of these problems, pyupgrade can automatically resolve removal of unittest aliases and open(..., "U"), accounting for around half the packages you list. Might be useful for packaging, and/or upstream PRs?

(and as an upstream maintainer, thank you! The Fedora team’s work on advance support is a significant contribution to the health of the Python ecosystem :heart_eyes:)

7 Likes
3

Teyit is another handy tool for upgrading unittest asserts (including things like assertTrue(x in y) → assertIn(x, y)).

I checked 20 unittest bugs, and half of them were already fixed. :slight_smile:

Summary

6 were already fixed and released upstream:

4 were already fixed upstream but not yet released:

I created 8 upstream PRs:

1 PR was already created:

1 not fixed, upstream archived:

4 Likes
4

Thank you both for your suggestions. Maybe it would be useful to mention those programs in the official documentation as a way how to migrate to Python 3.11. Had I known about them, I would include them in Bugzilla reports.

@hugovk thanks a lot for doing this, it’s very helpful.

2 Likes
5

Indeed, clearly giving people a working recipe that can automatically apply required updates to their Python codebase to run on 3.next is something we need to do more of.

Longer term it’d be ideal if a modernize tool able to easily be told to target fixing up things required in order to run on a specific version were available on PyPI by the time the CPython beta releases roll around.

2 Likes
6

There is the pyupgrade project for Python code.

I wrote upgrade_pythoncapi.py script in the pythoncapi_compat project for C code (C extension modules).

4 Likes
7

Followup:

After discussion on python-dev, the top two here have been reverted and will remain in Python 3.11 (October 2022), and instead be removed in Python 3.12 (October 2023) to give more projects time to update.

  1. removal of unittest aliases (bpo-45162) - 61 packages

Reverted in bpo-45162: Revert "Remove many old deprecated unittest features" by gpshead · Pull Request #30935 · python/cpython · GitHub

  1. removals from configparser module (bpo-45173) - 28 packages

Reverted in bpo-45173: Keep configparser deprecations until Python 3.12 by hugovk · Pull Request #30952 · python/cpython · GitHub

5 Likes
8

The Python 3.12 branch is now open, here’s a PR to remove the configparser deprecations in 3.12:

I checked the Fedora packages listed above for configparser, and 15/28 (54%) upstreams have fixes (or are unmaintained/removed from Fedora).

2 Likes
9

Until I carefully read the PR, I thought that it was actually removing the deprecations themselves, rather than the deprecated objects…it might be a good idea to clarify that PR title, heh.

3 Likes
10

Old thread, popping up with py 3.12 again.

Maybe a general remark from a maintainer: Some instructions like “use B instead of A” would help the transitioning a lot, and they should be in the doc of A and pointed to by the release notes. The way it is we often have to find them somewhere in the doc for A if they are there at all, or in PRs, or … From that experience, the best approach (in terms of time spent) is to ignore deprecations (if they don’t come with a guide), wait for the fall-out and hope for fixes at that point in time.

In an ideal world, we would reaction to deprecations, not to removals :slight_smile:

1 Like
11

I think we do it for most cases.
Which parts didn’t have “use B instead of A” document?

1 Like
12

The imp removal. See How do I migrate from imp?

1 Like
13

Sorry for sounding like “generic bashing”.

Miro mentioned a current pain point.

I remember that when I looked at SafeConfigParser deprecation first I did not know what to do (maybe because “my” package tried to be keep py 2.7 compatibility, too) - but that one was clearer when I had to revisit it now.

14

imp module document has many “Use XXX instead”. So it is not good example of not having “Use XXX instead” document.

As far as I read the thread, it is more difficult problem. In many case, there are no direct 1:1 mapping from deprecated API to recommended API.

Deprecated APIs may be have bad design. So it is difficult to provide 1:1 mapped new API. We need to provide case-by-case support, like the thread.

2 Likes
15

We could formally document those recommendations, listed by imp function, and link to that from the deprecation notice for each. For example, we could add a section to the old imp docs, (perhaps better) make a new “How-to” like e.g. “How to migrate from imp to importlib”. Of course, someone (who knows a lot more about the details of imp that I do) would have to volunteer to help provide the recommendations for each…

16

I’d like a single page listing current deprecations, with migration advice.

Similar to:

https://docs.pytest.org/en/stable/deprecations.html

Which I copied for:

This would be useful for users, who have a single page to see what’s deprecated, the planned removal releases/dates (if any), and how to upgrade.

And also useful for maintainers, who have a single page to see what needs removing when the time comes.

2 Likes
17

I like that idea.

I want to include planned incompatible changes too. For example,

  • Invalid escape sequence will become SyntaxError in the future Python. (not scheduled).
  • UTF-8 mode become default (scheduled in Python 3.15).
2 Likes
18

I’d expect this information to be in the “Porting notes” section of the What’s New page, e.g. https://docs.python.org/3.12/whatsnew/3.12.html#porting-to-python-3-12

That relies on people remembering to contribute those docs. Anyone is welcome to contribute to them, so if you’ve found a gap in the 3.12 docs, just file an issue about it. (I will say, just from skimming the current 3.12 section, a lot of those are not porting notes but simply changes that occurred in 3.12. The whole page usually needs a big cleanup pass before release, so I guess we’re waiting on that.)

19

There is a section like that: What’s New in Python 3.12: Pending Removal in Future Versions. We can add a similar sub-section under the “Porting to Python 3.12” section.

20

I was thinking of a single page the reader can check, instead of needing to trawl back through half a dozen (or more!) historical What’s New pages which become out-of-date, and list things that have since been removed.

A central page would be updated to reflect the current status: have deprecations with no removal date been given a removal date yet, has the planned removal date been changed, has the deprecated code been removed.

1 Like