PEP 594, take 2: Removing dead batteries from the standard library

I’m reviving this PEP w/ Christian. It has been updated to target Python 3.11. Since this has already been heavily discussed in PEP 594: Removing dead batteries from the standard library , my expectation is to take this to the steering council shortly, barring any major objections that come up in this topic.

PEP: 594
Title: Removing dead batteries from the standard library
Author: Christian Heimes <christian@python.org>,
        Brett Cannon <brett@python.org>
Discussions-To: https://discuss.python.org/t/pep-594-removing-dead-batteries-from-the-standard-library/1704
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 20-May-2019
Python-Version: 3.11
Post-History: 21-May-2019


Abstract
========

This PEP proposed a list of standard library modules to be removed from the
standard library. The modules are mostly historic data formats (e.g. Commodore
and SUN file formats), APIs and operating systems that have been superseded a
long time ago (e.g. Mac OS 9), or modules that have security implications and
better alternatives (e.g. password and login).

The PEP follows in the footsteps of other PEPS like :pep:`3108`. The
*Standard Library Reorganization* proposal removed a bunch of modules from
Python 3.0. In 2007, the PEP referred to maintenance burden as:

   "Over the years, certain modules have become a heavy burden upon python-dev
   to maintain. In situations like this, it is better for the module to be
   given to the community to maintain to free python-dev to focus more on
   language support and other modules in the standard library that do not take
   up an undue amount of time and effort."

The withdrawn :pep:`206` from 2000 expresses issues with the Python standard
library in an unvarnished and forthright manner:

   "[...] the standard library modules aren't always the best choices for a
   job. Some library modules were quick hacks (e.g. ``calendar``,
   ``commands``), some were designed poorly and are now near-impossible to
   fix (``cgi``), and some have been rendered obsolete by other, more complete
   modules [...]."


Rationale
=========

Back in the early days of Python, the interpreter came with a large set of
useful modules. This was often referred to as "batteries included"
philosophy and was one of the cornerstones to Python's success story.
Users didn't have to figure out how to download and install separate
packages in order to write a simple web server or parse email.

Times have changed. The introduction of PyPI (née Cheeseshop), setuptools,
and later pip, it became simple and straightforward to download and install
packages. Nowadays Python has a rich and vibrant ecosystem of third-party
packages. It's pretty much standard to either install packages from PyPI or
use one of the many Python or Linux distributions.

On the other hand, Python's standard library is piling up with cruft, unnecessary
duplication of functionality, and dispensable features. This is undesirable
for several reasons.

* Any additional module increases the maintenance cost for the Python core
  development team. The team has limited resources, reduced maintenance cost
  frees development time for other improvements.
* Modules in the standard library are generally favored and seen as the
  de facto solution for a problem. A majority of users only pick third-party
  modules to replace a stdlib module, when they have a compelling reason, e.g.
  ``lxml`` instead of ``xml``. The removal of an unmaintained stdlib module
  increases the chances of a community-contributed module to become widely
  used.
* A lean and mean standard library benefits platforms with limited resources
  like devices with just a few hundred kilobyte of storage (e.g. BBC
  Micro:bit). Python on mobile platforms like BeeWare or WebAssembly
  (e.g. pyodide) also benefit from reduced download size.

The modules in this PEP have been selected for deprecation because their
removal is either least controversial or most beneficial. For example,
least controversial are 30-year-old multimedia formats like the ``sunau``
audio format, which was used on SPARC and NeXT workstations in the late
1980s. The ``crypt`` module has fundamental flaws that are better solved
outside the standard library.

This PEP also designates some modules as not scheduled for removal. Some
modules have been deprecated for several releases or seem unnecessary at
first glance. However it is beneficial to keep the modules in the standard
library, mostly for environments where installing a package from PyPI is not
an option. This can be corporate environments or classrooms where external
code is not permitted without legal approval.

* The usage of FTP is declining, but some files are still provided over
  the FTP protocol or hosters offer FTP to upload content. Therefore,
  ``ftplib`` is going to stay.
* The ``optparse`` and ``getopt`` modules are widely used. They are mature
  modules with very low maintenance overhead.
* According to David Beazley [5]_ the ``wave`` module is easy to teach to
  kids and can make crazy sounds. Making a computer generate sounds is a
  powerful and highly motivating exercise for a nine-year-old aspiring developer.
  It's a fun battery to keep.


Deprecation schedule
====================

3.11
----

Starting with Python 3.11, deprecated modules will start issuing
``DeprecationWarning``. The estimated EOL of Python 3.10, the last
version without the warning, is October 2026.

3.12
----

There should be no specific change compared to Python 3.11.
This is the last version of Python with the deprecated modules,
with an estimated EOL of October 2028.

3.13
----

All modules deprecated by this PEP are removed from the ``main`` branch
of the CPython repository and are no longer distributed as part of Python.


Deprecated modules
==================

The modules are grouped as data encoding, multimedia, network, OS interface,
and misc modules. The majority of modules are for old data formats or
old APIs. Some others are rarely useful and have better replacements on
PyPI, e.g. Pillow for image processing or NumPy-based projects to deal with
audio processing.

.. csv-table:: Table 1: Proposed modules deprecations
   :header: "Module", "Deprecated in", "To be removed", "Added in", "Has maintainer?", "Replacement"
   :widths: 2, 1, 1, 1, 1, 2

    aifc,3.11 (3.0\*),3.13,1993,**yes (inactive)**,\-
    asynchat,**3.6** (3.0\*),3.12,1999,**yes**,asyncio_
    asyncore,**3.6** (3.0\*),3.12,1999,**yes**,asyncio_
    audioop,3.11 (3.0\*),3.13,1992,**yes**,\-
    cgi,3.11 (2.0\*\*),3.13,1995,no,\-
    cgitb,3.11 (2.0\*\*),3.13,1995,no,\-
    chunk,3.11,3.13,1999,no,\-
    crypt,3.11,3.13,1994,**yes (inactive)**,"legacycrypt_, bcrypt_, argon2-cffi_, hashlib_, passlib_"
    imghdr,3.11,3.13,1992,no,"filetype_, puremagic_, python-magic_"
    msilib,3.11,3.13,2006,no,\-
    nntplib,3.11,3.13,1992,no,\-
    nis,3.8 (3.0\*),3.13,1992,no,\-
    ossaudiodev,3.11,3.13,2002,no,\-
    pipes,3.11,3.13,1992,no,"subprocess_"
    smtpd,"**3.4.7**, **3.5.4**",3.12,2001,**yes**,"aiosmtpd_"
    sndhdr,3.11,3.13,1994,no,"filetype_, puremagic_, python-magic_"
    spwd,3.11,3.13,2005,no,"python-pam_"
    sunau,3.11 (3.0\*),3.13,1993,no,\-
    telnetlib,3.11 (3.0\*),3.13,1997,no,"telnetlib3_, Exscript_"
    uu,3.11,3.13,1994,no,\-
    xdrlib,3.11,3.13,1992/1996,no,\-

.. _aiosmtpd: https://pypi.org/project/aiosmtpd/
.. _argon2-cffi: https://pypi.org/project/argon2-cffi/
.. _ast: https://docs.python.org/3/library/ast.html
.. _astroid: https://pypi.org/project/astroid/
.. _asyncio: https://docs.python.org/3/library/hashlib.html
.. _bcrypt: https://pypi.org/project/bcrypt/
.. _Exscript: https://pypi.org/project/Exscript/
.. _filetype: https://pypi.org/project/filetype/
.. _hashlib: https://docs.python.org/3/library/hashlib.html
.. _importlib: https://docs.python.org/3/library/importlib.html
.. _legacycrypt: https://pypi.org/project/legacycrypt/
.. _passlib: https://pypi.org/project/passlib/
.. _puremagic: https://pypi.org/project/puremagic/
.. _python-magic: https://pypi.org/project/python-magic/
.. _python-pam: https://pypi.org/project/python-pam/
.. _simplepam: https://pypi.org/project/simplepam/
.. _subprocess: https://docs.python.org/3/library/subprocess.html
.. _telnetlib3: https://pypi.org/project/telnetlib3/

Some module deprecations proposed by :pep:`3108` for 3.0 and :pep:`206` for
2.0. The *added in* column illustrates, when a module was originally designed
and added to the standard library. The *has maintainer* column refers to the
`expert index <https://devguide.python.org/experts/>`_, a list of domain
experts and maintainers in the DevGuide.


Data encoding modules
---------------------


uu and the uu encoding
~~~~~~~~~~~~~~~~~~~~~~

The `uu <https://docs.python.org/3/library/uu.html>`_ module provides
uuencode format, an old binary encoding format for email from 1980. The uu
format has been replaced by MIME. The uu codec is provided by the ``binascii``
module.  There's also ``encodings/uu_codec.py`` which is a codec for the
same encoding; it should also be deprecated.


xdrlib
~~~~~~

The `xdrlib <https://docs.python.org/3/library/xdrlib.html>`_ module supports
the Sun External Data Representation Standard. XDR is an old binary
serialization format from 1987. These days it's rarely used outside
specialized domains like NFS.


Multimedia modules
------------------

aifc
~~~~

The `aifc <https://docs.python.org/3/library/aifc.html>`_ module provides
support for reading and writing AIFF and AIFF-C files. The Audio Interchange
File Format is an old audio format from 1988 based on Amiga IFF. It was most
commonly used on the Apple Macintosh. These days only few specialized
application use AIFF.

A user disclosed [8]_ that the post production film industry makes heavy
use of the AIFC file format. The usage of the ``aifc`` module in closed source
and internal software was unknown prior to the first posting of this PEP. This
may be a compelling argument to keep the ``aifc`` module in the standard
library. The file format is stable and the module does not require much
maintenance. The strategic benefits for Python may outmatch the burden.


audioop
~~~~~~~

The `audioop <https://docs.python.org/3/library/audioop.html>`_ module
contains helper functions to manipulate raw audio data and adaptive
differential pulse-code modulated audio data. The module is implemented in
C without any additional dependencies. The `aifc`_, `sunau`_, and `wave`_
modules depend on `audioop`_ for some operations.

The byteswap operation in the ``wave`` module can be substituted with little
extra work. In case ``aifc`` is not deprecated as well, a reduced version of
the ``audioop`` module is converted into a private implementation detail,
e.g. ``_audioop`` with ``byteswap``, ``alaw2lin``, ``ulaw2lin``, ``lin2alaw``,
``lin2ulaw``, and ``lin2adpcm``.


chunk
~~~~~

The `chunk <https://docs.python.org/3/library/chunk.html>`_ module provides
support for reading and writing Electronic Arts' Interchange File Format.
IFF is an old audio file format originally introduced for Commodore and
Amiga. The format is no longer relevant.


imghdr
~~~~~~

The `imghdr <https://docs.python.org/3/library/imghdr.html>`_ module is a
simple tool to guess the image file format from the first 32 bytes
of a file or buffer. It supports only a limited number of formats and
neither returns resolution nor color depth.


ossaudiodev
~~~~~~~~~~~

The `ossaudiodev <https://docs.python.org/3/library/ossaudiodev.html>`_
module provides support for Open Sound System, an interface to sound
playback and capture devices. OSS was initially free software, but later
support for newer sound devices and improvements were proprietary. Linux
community abandoned OSS in favor of ALSA [1]_. Some operation systems like
OpenBSD and NetBSD provide an incomplete [2]_ emulation of OSS.

To best of my knowledge, FreeBSD is the only widespread operating system
that uses Open Sound System as of today. The ``ossaudiodev`` hasn't seen any
improvements or new features since 2003. All commits since 2003 are
project-wide code cleanups and a couple of bug fixes. It would be beneficial
for both FreeBSD community and core development, if the module would be
maintained and distributed by people that care for it and use it.

The standard library used to have more audio-related modules. The other
audio device interface (``audiodev``, ``linuxaudiodev``, ``sunaudiodev``)
were removed in 2007 as part of the :pep:`3108` stdlib re-organization.


sndhdr
~~~~~~

The `sndhdr <https://docs.python.org/3/library/sndhdr.html>`_ module is
similar to the `imghdr`_ module but for audio formats. It guesses file
format, channels, frame rate, and sample widths from the first 512 bytes of
a file or buffer. The module only supports AU, AIFF, HCOM, VOC, WAV, and
other ancient formats.


sunau
~~~~~

The `sunau <https://docs.python.org/3/library/sunau.html>`_ module provides
support for Sun AU sound format. It's yet another old, obsolete file format.


Networking modules
------------------

asynchat
~~~~~~~~

The `asynchat <https://docs.python.org/3/library/asynchat.html>`_ module
is built on top of `asyncore`_ and has been deprecated since Python 3.6.


asyncore
~~~~~~~~

The `asyncore <https://docs.python.org/3/library/asyncore.html>`_ module was
the first module for asynchronous socket service clients and servers. It
has been replaced by asyncio and is deprecated since Python 3.6.

The ``asyncore`` module is also used in stdlib tests. The tests for
``ftplib``, ``logging``, ``smptd``, ``smtplib``, and ``ssl`` are partly
based on ``asyncore``. These tests must be updated to use asyncio or
threading.


cgi
~~~

The `cgi <https://docs.python.org/3/library/cgi.html>`_ module is a support
module for Common Gateway Interface (CGI) scripts. CGI is deemed as
inefficient because every incoming request is handled in a new process.
:pep:`206` considers the module as:

   "[...] designed poorly and are now near-impossible to fix (``cgi``) [...]"

Replacements for the various parts of ``cgi`` which are not directly
related to executing code are:

- ``parse`` with ``urllib.parse.parse_qs`` (``parse`` is just a wrapper)
- ``parse_header`` with ``email.message.Message`` (see example below)
- ``parse_multipart`` with ``email.message.Message`` (same MIME RFCs)
- ``FieldStorage``/``MiniFieldStorage`` has no direct replacement
- ``valid_boundary`` (undocumented) with ``re.compile("^[ -~]{0,200}[!-~]$")``

As an explicit example of how close ``parse_header`` and
``email.message.Message`` are::

  >>> from cgi import parse_header
  >>> from email.message import Message
  >>> parse_header(h)
  ('application/json', {'charset': 'utf8'})
  >>> m = Message()
  >>> m['content-type'] = h
  >>> m.get_params()
  [('application/json', ''), ('charset', 'utf8')]
  >>> m.get_param('charset')
  'utf8'


cgitb
~~~~~

The `cgitb <https://docs.python.org/3/library/cgitb.html>`_ module is a
helper for the ``cgi`` module for configurable tracebacks.

The ``cgitb`` module is not used by any major Python web framework (Django,
Pyramid, Plone, Flask, CherryPy, or Bottle). Only Paste uses it in an
optional debugging middleware.


smtpd
~~~~~

The `smtpd <https://docs.python.org/3/library/smtpd.html>`_ module provides
a simple implementation of a SMTP mail server. The module documentation
marks the module as deprecated and recommends ``aiosmtpd`` instead. The
deprecation message was added in releases 3.4.7, 3.5.4, and 3.6.1.


nntplib
~~~~~~~

The `nntplib <https://docs.python.org/3/library/nntplib.html>`_ module
implements the client side of the Network News Transfer Protocol (nntp). News
groups used to be a dominant platform for online discussions. Over the last
two decades, news has been slowly but steadily replaced with mailing lists
and web-based discussion platforms. Twisted is also
`planning <https://twistedmatrix.com/trac/ticket/9405>`_ to deprecate NNTP
support and `pynntp <https://github.com/greenbender/pynntp>`_ hasn't seen any
activity since 2014. This is a good indicator that the public interest in
NNTP support is declining.

The ``nntplib`` tests have been the cause of additional work in the recent
past. Python only contains client side of NNTP. The tests connect to
external news server. The servers are sometimes unavailable, too slow, or do
not work correctly over IPv6. The situation causes flaky test runs on
buildbots.


telnetlib
~~~~~~~~~

The `telnetlib <https://docs.python.org/3/library/telnetlib.html>`_ module
provides a Telnet class that implements the Telnet protocol.


Operating system interface
--------------------------

crypt
~~~~~

The `crypt <https://docs.python.org/3/library/crypt.html>`_ module implements
password hashing based on the ``crypt(3)`` function from ``libcrypt`` or
``libxcrypt`` on Unix-like platforms. The algorithms are mostly old, of poor
quality and insecure. Users are discouraged to use them.

* The module is not available on Windows. Cross-platform applications need
  an alternative implementation anyway.
* Only DES encryption is guaranteed to be available. DES has an extremely
  limited key space of 2**56.
* MD5, salted SHA256, salted SHA512, and Blowfish are optional extensions.
  SSHA256 and SSHA512 are glibc extensions. Blowfish (bcrypt) is the only
  algorithm that is still secure. However it's in glibc and therefore not
  commonly available on Linux.
* Depending on the platform, the ``crypt`` module is not thread safe. Only
  implementations with ``crypt_r(3)`` are thread safe.
* The module was never useful to interact with system user and password
  databases. On BSD, macOS, and Linux, all user authentication and
  password modification operations must go through PAM (pluggable
  authentication module), see `spwd`_ deprecation.


nis
~~~

The `nis <https://docs.python.org/3/library/nis.html>`_ module provides
NIS/YP support. Network Information Service / Yellow Pages is an old and
deprecated directory service protocol developed by Sun Microsystems. It's
designed successor NIS+ from 1992 never took off. For a long time, libc's
Name Service Switch, LDAP, and Kerberos/GSSAPI are considered a more powerful
and more secure replacement of NIS.


spwd
~~~~

The `spwd <https://docs.python.org/3/library/spwd.html>`_ module provides
direct access to Unix shadow password database using non-standard APIs.

In general it's a bad idea to use spwd. It circumvents system
security policies, does not use the PAM stack, and is only compatible
with local user accounts, because it ignores NSS. The use of the ``spwd``
module for access control must be considered a *security bug*, as it bypasses
PAM's access control.

Further more the ``spwd`` module uses the
`shadow(3) <http://man7.org/linux/man-pages/man3/shadow.3.html>`_ APIs.
Functions like ``getspnam(3)`` access the ``/etc/shadow`` file directly. This
is dangerous and even forbidden for confined services on systems with a
security engine like SELinux or AppArmor.


Misc modules
------------


msilib
~~~~~~

The `msilib <https://docs.python.org/3/library/msilib.html>`_ package is a
Windows-only package. It supports the creation of Microsoft Installers (MSI).
The package also exposes additional APIs to create cabinet files (CAB). The
module is used to facilitate distutils to create MSI installers with the
``bdist_msi`` command. In the past it was used to create CPython's official
Windows installer, too.

Microsoft is slowly moving away from MSI in favor of Windows 10 Apps (AppX)
as new deployment model [3]_.


pipes
~~~~~

The `pipes <https://docs.python.org/3/library/pipes.html>`_ module provides
helpers to pipe the input of one command into the output of another command.
The module is built on top of ``os.popen``. Users are encouraged to use
the subprocess module instead.


Modules to keep
===============

Some modules were originally proposed for deprecation.

.. csv-table:: Table 2: Withdrawn deprecations
   :header: "Module", "Deprecated in", "Replacement"
   :widths: 1, 1, 2

    colorsys,\-,"colormath, colour, colorspacious, Pillow"
    fileinput,\-,argparse
    getopt,\-,"argparse, optparse"
    optparse,**3.2**,argparse
    wave,\-,

colorsys
--------

The `colorsys <https://docs.python.org/3/library/colorsys.html>`_ module
defines color conversion functions between RGB, YIQ, HSL, and HSV coordinate
systems.

Walter Dörwald, Petr Viktorin, and others requested to keep ``colorsys``. The
module is useful to convert CSS colors between coordinate systems. The
implementation is simple, mature, and does not impose maintenance overhead
on core development.

The PyPI packages ``colormath``, ``colour``, and ``colorspacious`` provide more and
advanced features. The Pillow library is better suited to transform images
between color systems.


fileinput
---------

The `fileinput <https://docs.python.org/3/library/fileinput.html>`_ module
implements helpers to iterate over a list of files from ``sys.argv``. The
module predates the ``optparser`` and ``argparser`` modules. The same functionality
can be implemented with the ``argparser`` module.

Several core developers expressed their interest to keep the module in the
standard library, as it is handy for quick scripts.


getopt
------

The `getopt <https://docs.python.org/3/library/getopt.html>`_ module mimics
C's ``getopt()`` option parser.

Although users are encouraged to use ``argparse`` instead, the ``getopt`` module is
still widely used. The module is small, simple, and handy for C developers
to write simple Python scripts.


optparse
--------

The `optparse <https://docs.python.org/3/library/optparse.html>`_ module is
the predecessor of the ``argparse`` module.

Although it has been deprecated for many years, it's still too widely used
to remove it.


wave
----

The `wave <https://docs.python.org/3/library/wave.html>`_ module provides
support for the WAV sound format.

The module is not deprecated, because the WAV format is still relevant these
days. The ``wave`` module is also used in education, e.g. to show kids how
to make noise with a computer.

The module uses one simple function from the `audioop`_ module to perform
byte swapping between little and big endian formats. Before 24 bit WAV
support was added, byte swap used to be implemented with the ``array``
module. To remove ``wave``'s dependency on ``audioop``, the byte swap
function could be either be moved to another module (e.g. ``operator``) or
the ``array`` module could gain support for 24-bit (3-byte) arrays.


Discussions
===========

* Elana Hashman and Nick Coghlan suggested to keep the ``getopt`` module.
* Berker Peksag proposed to deprecate and removed ``msilib``.
* Brett Cannon recommended to delay active deprecation warnings and removal
  of modules like ``imp`` until Python 3.10. Version 3.8 will be released
  shortly before Python 2 reaches end-of-life. A delay reduced churn for
  users that migrate from Python 2 to 3.8.
* At one point, distutils was mentioned in the same sentence as this PEP.
  To avoid lengthy discussion and delay of the PEP, I decided against dealing
  with distutils. Deprecation of the distutils package will be handled by
  another PEP.
* Multiple people (Gregory P. Smith, David Beazley, Nick Coghlan, ...)
  convinced me to keep the `wave`_ module. [4]_
* Gregory P. Smith proposed to deprecate `nntplib`_. [4]_
* Andrew Svetlov mentioned the ``socketserver`` module is questionable.
  However it's used to implement ``http.server`` and ``xmlrpc.server``. The
  stdlib doesn't have a replacement for the servers, yet.


Rejected Ideas
==============

Creating/maintaining a separate repo for the deprecated modules
---------------------------------------------------------------

It was previously proposed to create a separate repository containing the
deprecated modules packaged for installation. One of the PEP authors went so far
as to create a `demo repository <https://github.com/tiran/legacylib>`_ . In the
end, though, it was decided that the added workload to create and maintain such
a repo officially wasn't justified, as the source code will continue to be
available in the CPython repository for people to vendor as necessary. Similar
work has also not been done when previous modules were deprecated and removed,
and it seemingly wasn't an undue burden on the community.


Update history
==============

Update 1
--------

* Deprecate parser module
* Keep `fileinput`_ module
* Elaborate why ``crypt`` and ``spwd`` are dangerous and bad
* Improve sections for `cgitb`_, `colorsys`_, `nntplib`_, and `smtpd`_ modules
* The `colorsys`_, ``crypt``, `imghdr`_, `sndhdr`_, and ``spwd`` sections now
  list suitable substitutions
* Mention that ``socketserver`` is going to stay for ``http.server`` and
  ``xmlrpc.server``
* The future maintenance section now states that the deprecated modules
  may be adopted by Python community members

Update 2
--------

* Keep ``colorsys`` module
* Add experts
* Redirect discussions to discuss.python.org
* Deprecate `telnetlib`_
* Deprecate compat32 policy of email package
* Add creation year to overview table
* Mention :pep:`206` and :pep:`3108`
* Update sections for ``aifc``, ``audioop``, ``cgi``, and ``wave``.

Update 3
--------
* Keep the legacy email API modules.  Internal deprecations will be
  handled separately.

Update 4
--------
* Add Brett as a co-author.
* Retarget the PEP for Python 3.11.
* Examples of how to replace the relevant parts of ``cgi``
  (thanks Martijn Pieters).


References
==========

.. [1] https://en.wikipedia.org/wiki/Open_Sound_System#Free,_proprietary,_free
.. [2] https://man.openbsd.org/ossaudio
.. [3] https://blogs.msmvps.com/installsite/blog/2015/05/03/the-future-of-windows-installer-msi-in-the-light-of-windows-10-and-the-universal-windows-platform/
.. [4] https://twitter.com/ChristianHeimes/status/1130257799475335169
.. [5] https://twitter.com/dabeaz/status/1130278844479545351
.. [6] https://mail.python.org/pipermail/python-dev/2019-May/157464.html
.. [7] https://discuss.python.org/t/switch-pythons-parsing-tech-to-something-more-powerful-than-ll-1/379
.. [8] https://mail.python.org/pipermail/python-dev/2019-May/157634.html


Copyright
=========

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.



..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End:
20 Likes

https://github.com/defnull/multipart is a proper replacement for cgi.FieldStorage, used by Zope, and thus by Plone, and so here to stay til the end of time :wink:

2 Likes

FYI since Python 3.1, 13 stdlib modules have been removed: Removed modules. I removed binhex and formatter after looking at PEP 594.

The asyncore module is also used in stdlib tests. The tests for ftplib , logging , smptd , smtplib , and ssl are partly based on asyncore . These tests must be updated to use asyncio or threading.

When I removed asyncore, asynchat and smtpd, I kept them as private modules in the test package. Technically, we can continue using them in tests, until someone steps in to rewrite tests with asyncio.

Recently, I had to debug a test_ftplib issue. It’s a pain to follow asyncore class inheritance and callbacks to understand where socket events come from, how they are handled, which callback is used (methods are overriden in sublcasses by design), etc. I really prefer asyncio coroutines!

Two nits from the current pep text: lib2to3 is listed as one to keep. That is already deprecated and removal is on track in its own bpo issue. Drop all mention of that or include it with plans matching the issue. (On mobile so I won’t search the # right now)

It also suggests jumping directly to DeprecationWarning without a PendingDeprecationWarning release cycle first. Why? Cover that. I’d suggest 3.11 be Pending for things without them already. 3.12 then makes those DeprecationWarning. For removal before our :pie: release if not in 3.13 itself.

2 Likes

https://bugs.python.org/issue40360

I suggest skipping PendingDeprecationWarning and going directly to DeprecationWarning.

Even with DeprecationWarning, it’s been a problem that warnings aren’t heeded or seen, and more than once removals have been bumped to a later release.

For example, using collections.Mapping instead of collections.abc.Mapping emitted a DeprecationWarning since Python 3.3 and was set to be removed in 3.9 but was bumped to 3.10 (and also to allow projects to drop 2.7 in concert).

https://docs.python.org/3/whatsnew/3.9.html#you-should-check-for-deprecationwarning-in-your-code

More recently, some unittest removals (deprecated since 3.5) and configparser (since 3.2) were reverted from 3.11 and deferred until 3.12.

https://discuss.python.org/t/experience-with-python-3-11-in-fedora/12911/7?u=hugovk

I also recommend mentioning in the warning which Python version removal is expected. This means people who see the warning have a visible timetable to work towards, rather than some vague point in the future.

PEP 387:

  1. Add a warning. If behavior is changing, the API may gain a new function or method to perform the new behavior; old usage should raise the warning. If an API is being removed, simply warn whenever it is entered. DeprecationWarning is the usual warning category to use, but PendingDeprecationWarning may be used in special cases where the old and new versions of the API will coexist for many releases [2]. Compiler warnings are also acceptable. The warning message should include the release the incompatibility is expected to become the default and a link to an issue that users can post feedback to.
2 Likes

Thanks for the reference. The Roundup Issue Tracker uses CGI and FieldStorage/MiniFieldStorage extensively as well as cgitb.
Sadly it does mean that Roundup will no longer work with only the standard Python library which has always been a goal.

Even though CGI is inefficient, the CGI deployment mode works fine for low volume trackers on cheap hosting environments.

Hopefully we can work around the depreciation and keep Roundup alive.

1 Like

Both cgi and cgitb appear to be single-file pure-Python modules; they would be easy enough to vendor into Roundup.

Yeah, I mentioned this on the ticket I opened to address this. One tricky part is that Roundup is multilingual and runs under both python2 and 3. I would need to vendor for python 3 only. But there is already a mechanism to handle issues like this.

Do you really need to support Python 2 still? If so, perhaps the solution would be to could just stick with Python 3.10 for the next decade. :slight_smile:

2 Likes

As you know, Roundup admins write Python code to implement workflows, integrations, custom interfaces, actions etc. There are libraries written in Python 2 that integrate with third party enterprise level tools.

Rewriting/modifying that code can be a major undertaking.
I assume that was part of the calculus for not upgrading B.P.O to Python 3. Moving to GitHub Issues was a better plan given the resources available.

So yeah I hope to stay multilingual for a few years yet. A different maintainer may make a different choice though.

Thanks for your interest though.

The docs for binhex are still there: binhex — Encode and decode binhex4 files — Python 3.11.0a0 documentation . I will remove it from the PEP, though, as the module itself is gone.

They seem to still be there, e.g., https://github.com/python/cpython/blob/main/Lib/asyncore.py and asyncore — Asynchronous socket handler — Python 3.11.0a5 documentation .

Removed!

That’s our deprecation policy: PEP 387 -- Backwards Compatibility Policy | Python.org .

But it’s documented elsewhere as this is how things are deprecated, so since this is following policy I don’t think it needs to be explicitly called out.

1 Like

Ah yes, asyncore, asynchat and smtpd are back in Python 3.11, the SC asked me to add them back: https://github.com/python/steering-council/issues/86

I just tried to explain that it is possible to remove these modules from the stdlib by hiding them into the test package. See my quote of PEP 594 in my previous message.

1 Like

It’s nice to have a core library replacement for removed modules. I have found urllib.parse.parse_qs a satisfactory replacement for cgi.FieldStorage

I’m not sure if it’s worthwhile to/how to add this to the PEP?

You could create a pull request, similar to this one:

1 Like

I have submitted the PEP to the SC for consideration: PEP 594 -- Removing dead batteries from the standard library · Issue #109 · python/steering-council · GitHub .

11 Likes

I realise I’m very late to this discussion, but scanning through the PEP, there are a couple of deprecations I was (mildly) surprised to see:

cgi: Yes, launching a new process for every request is inefficient, but there are cases where it’s good enough. It’s conceptually much easier than writing an application with a proper web framework - it’s the web equivalent of scripting. And there are (or used to be?) cheap web hosts would give you CGI support as the only way to run custom code on the server, and certainly don’t make it easy to install something from PyPI.

Reading bits of the previous discussion and making educated guesses, I think it will still be possible to write CGI scripts using only the standard library even without the cgi module, but I’m not 100% sure. If that is right, maybe it’s worth spelling it out in the PEP?

telnetlib: Telnet is old hat, but still used at times for devices on a local network. I ran into it when playing with an old e-reader: you could enable a telnet server to get shell access. A quick look on PyPI turns up two different packages to control VLC over telnet, packages for controlling network devices and other things, and a package adding SOCKS proxy support. These are all using telnetlib and released in the last couple of years.

There’s probably no particular reason this needs to be in the standard library, and I see there are already several alternatives on PyPI (telnetlib3 appears to have the most attention of the ones I found). Maybe the PEP can explain this - the current detail gives no rationale for removing the module, and its brevity implies that telnet is self-evidently useless, which doesn’t seem right.

2 Likes

Agreed, as someone who maintains an RFC-compliant Telnet server in
Python (a MUD framework specifically), I’m mildly disappointed to
see this basic implementation being dropped from the stdlib. In my
project’s case it’s not going to imply significant impact at least,
since I’m only using it for automated testing and even then only
rely on telnetlib’s client implementation and a few of its constants
for sending raw option negotiations, but as protocols go it’s no
more venerable than, say, ftplib or poplib.

Since this is for minimizing maintenance on Python itself we would need to explicitly know how widely CGI as a hosting platform is before considering this in reverting that part of the PEP.

If you have some suggested wording I would happily take a PR for that to clarify things.

No one said some of us wouldn’t like to drop those modules as well. :wink: But telnetlib is not widely used and shouldn’t be used unless you know what you’re doing due to the security risk. And everyone is going to have differing opinions for all of this as to whether is more used or not. But the key point with both cgi and telnetlib is we didn’t get enough pushback to exclude them from the PEP and we wouldn’t accept them into the stdlib today (which is what I believe Christian used as a guideline and I agree with).

3 Likes