Mitigating python deprecation message frustrations by improving the design of deprecation message handling

Ideas
1

Based on the discussion in How to ignore deprecation warnings in Python - Stack Overflow i created a bug report Never show deprectation messages via stderr - only log them only log once. · Issue #123307 · python/cpython · GitHub

and was pointed here to start a discussion first.

Bug report

Bug description:

Any python code i start might have some deprecated libraries that will make the actual output of my software hard to spot.
The stackoverflow question
Discusses this in all detail and there is no proper standard fix available especially if you install scripts with pyprojec.toml.

I think the design must be changed to never ever use stderr or stdout without being explicitly asked to do so. The deprecation info needs to go to a standard place and be available on request so that in any environment an active deprectation check can be made. These deprecation messages may e.g. show on pip install or any other upgrade operation and generally warn “x deprecation issues to be solved …” - if remember right npm works this way.

here is an example

wikiquery -d -l -s wiki -q "[[isA::Property]]" 2>&1 | grep -v "is deprecated"
  import pkg_resources
/usr/local/lib/python3.10/dist-packages/pkg_resources/__init__.py:3144: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  declare_namespace(pkg)
/usr/local/lib/python3.10/dist-packages/pkg_resources/__init__.py:3144: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('ruamel')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  declare_namespace(pkg)
/usr/local/lib/python3.10/dist-packages/pkg_resources/__init__.py:3144: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('ruamel')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  declare_namespace(pkg)
/usr/local/lib/python3.10/dist-packages/pkg_resources/__init__.py:3144: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('sphinxcontrib')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  declare_namespace(pkg)
/usr/local/lib/python3.10/dist-packages/pkg_resources/__init__.py:3144: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('sphinxcontrib')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  declare_namespace(pkg)
/usr/local/lib/python3.10/dist-packages/pkg_resources/__init__.py:3144: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('sphinxcontrib')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  declare_namespace(pkg)
/usr/local/lib/python3.10/dist-packages/pkg_resources/__init__.py:3144: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('sphinxcontrib')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  declare_namespace(pkg)
/usr/local/lib/python3.10/dist-packages/pkg_resources/__init__.py:3144: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('zope')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  declare_namespace(pkg)
/usr/local/lib/python3.10/dist-packages/pkg_resources/__init__.py:3144: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('zope')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  declare_namespace(pkg)
/usr/local/lib/python3.10/dist-packages/pkg_resources/__init__.py:3144: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  declare_namespace(pkg)
ask:
[[isA::Property]]
{

To mitigate the issue i have now added:

# avoid ugly deprecation messages see
# https://stackoverflow.com/questions/879173/how-to-ignore-deprecation-warnings-in-python
# and
import shutup
shutup.please()

to my main base class for command line tools.

Doing this defeats the purpose of the deprecation messages. There are better and less frustrating ways to do this and i hope this dicussion will lead to a change in the design that is much better than what was offered in the past decade.

1 Like
2

I don’t really understand what exactly is being asked here. Are you just asking for a way to hide DeprecationWarnings? You say the deprecation info should go to “a standard place” but that standard place is the standard error stream.

3 Likes
3

Sounds like you’re saying that the deprecation warnings should be issued when you install the package rather than when your code (indirectly) invokes the deprecated API?

4

What’s wrong with the warnings module? You can override the default filter like that:

Python 3.14.0a0 (heads/syntaxerr-location-basicrepl-121804:ef426d24cb, Aug 25 2024, 04:39:26) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import warnings
... warnings.filterwarnings("ignore", category=DeprecationWarning) 
... 
>>> complex(1+1j, 1+1j)  # no warnings!
2j
>>>

You also can override the showwarning() function to something else, that can e.g. log some warnings somewhere:

>>> import warnings
... warning_file = open("warnings.txt", "w")
... old_showwarning = warnings.showwarning
... def mywarning(message, category, filename, lineno, file=None, line=None):
...     old_showwarning(message, category, filename, lineno, warning_file, line)
... 
... warnings.showwarning = mywarning
... 
>>> complex(1j,1j)
(-1+1j)
>>> ^D
<sys>:0: ResourceWarning: unclosed file <_io.TextIOWrapper name='warnings.txt' mode='w' encoding='UTF-8'>

$ cat warnings.txt
<python-input-1>:1: DeprecationWarning: complex() argument 'real' must be a real number, not complex
  complex(1j,1j)
<python-input-1>:1: DeprecationWarning: complex() argument 'imag' must be a real number, not complex
  complex(1j,1j)
1 Like
5

Note that deprecation warnings are already ignored by default for modules other than __main__ (as per PEP 565 – Show DeprecationWarning in __main__ | peps.python.org), and have been since Python 3.2.

We turned them back on in __main__ since having them completely off by default meant that a lot of people ended up encountering unexpected breakages at the end of the deprecation period, since they were no longer getting the warnings during the deprecation period.

This does mean that authors of single file scripts need to either avoid using deprecated constructs, or else turn off deprecation warnings. That’s a matter of the feature working as intended, not an unfortunate accident.

Packaged libraries will only need to worry about the warnings if they have non-trivial code directly in __main__, and it’s genuinely better to move that code out into a support module and have __main__ import it from there (e.g. the “entry_points” feature for defining console scripts when publishing packages works that way).

8 Likes
6

Yes indeed. Doing the deprecation info at runtime via stderr is IMHO counter productive since the target might not be a human these days and other software that reads the outputmight breaks because of the non standardized content.

My main use case for command line tools in which this happens frequently is piping and unexpected pipe content is obviously annoying.

7

more specific a standard place in the users filesystem e.g. .python/deprecation_warnings.log or /var/log/python/deprecation_warnings.log … but not a place where the content might have unwanted undintendet side effects. Worse stderr might not even reach the intend target audience which is developers not users of a software.

8

This is really enlightening and would IMHO be a good answer to the stackoverflow question. It is also the basis for my current workaround in nicegui_widgets/ngwidgets/cmd.py at main · WolfgangFahl/nicegui_widgets · GitHub

It also means that tools such as argparser might be a good starting point to handle the deprecation messages effectively e.g. with a developer/production mode that makes sure deprecation messages never show up in production.

9

This approach does not work well/directly with console scripts. Ther there is a need for a standard handling mechanism that i would not known how to implement before this discussion started. I use console scripts a lot.

10

see also harmless deprecation warnings · Issue #676 · mkdocstrings/mkdocstrings · GitHub for a real life issue in a project where the current state is a problem.

11

The issue there was that some code in MkDocs was preventing the user from controlling the warnings, not an issue with Python warnings themselves. This has been fixed and will be part of the next release.

It sounds like you’re expecting the standard output/error streams of various tools to be “public API”, i.e. that they stay stable and backward compatible, but I honestly never saw any project that claimed or ensured such a thing, notably because it’s impossible[1] to actually deprecate what is written to the standard output/error streams without immediate breaking changes. Even the output of common utilities like ls cannot be relied on since it can change from one system/user to another.

Then, since your scripts break when the output/error streams change, you seem to blame deprecation warnings (or warnings generally), and want a way to hide them. Hiding warnings (and making that the default) is really not a good idea IMO. If I had something to say, I’d make them more visible by default.

In any case, Python warnings already provide ways to hide them (as shown by others here). Yes, sometimes third-party libs make mistakes that can prevent that, but that’s something to fix on their side :smile:

Maybe there’s something to add to Python to expose the warnings.showwarning functionality from the command-line though :thinking: Something like --warnings-output-file or PYTHONWARNIGS_OUTPUTFILE to choose a file to write the warnings to instead of writing them to the standard error.

  1. Or let say “very difficult”: you could technically add an option to opt into the new output mode, emit a deprecation warning that this option will be toggled on in the future, then once it is, emit a deprecation warning that the option will be removed in the future. That’s a lot of maintenance. ↩︎

4 Likes
12

Something like --warnings-output-file or PYTHONWARNIGS_OUTPUTFILE to choose a file to write the warnings to instead of writing them to the standard error.

I like this idea as a middle ground. Though personally believe the default should stay to stderr.

1 Like
13

The mkdocstrings-python problem with deprecation warnings being visible at runtime only occurs because python/scripts/make at main · mkdocstrings/python · GitHub contains non-trivial code.

The recommended approach to avoid allowing runtime objects to be potentially reachable under two different names (__main__.name and script_name.name) that refer to two different instances of “the same” object definition (whether that’s a function, type, or module level variable) is to keep the actual __main__ script as small as possible, along the lines of:

import subprocess
import sys

from mkdocstrings_handlers.python._cli import main

if __name__ == "__main__":
    try:
        sys.exit(main())
    except subprocess.CalledProcessError as process:
        if process.output:
            print(process.output, file=sys.stderr)  # noqa: T201
        sys.exit(process.returncode)

It’s a common enough pattern that the packaging standards allow wrappers along these lines to be automatically generated just by specifying CLI commands like my-cmd=my_package.my_submodule:my_function in the [project.scripts] or [project.gui-scripts] table in pyproject.toml: Writing your pyproject.toml - Python Packaging User Guide

The primary reasons for recommending this structural pattern when distributing packages have nothing to do with avoiding deprecation warnings:

  • it reduces the risk of hitting a known variant of the “double import” trap in the import system (described above)
  • it makes the code implementing the console scripts more amenable to unit testing with standard Python API testing tools rather than needing to be tested indirectly via the command line interface

However, it also has the effect of implicitly silencing deprecation warnings from the code that no longer resides in the __main__ namespace at runtime.

As a result, this is one of those situations where our reaction to a complaint of “Doing X (having non-trivial code directly in __main__) causes problem Y (deprecation warnings are visible at runtime)” is “Doing X not only causes problem Y, it can cause problems A, B, and C, too. Don’t do X, and not only will you avoid problem Y, you will avoid all those other potential problems you haven’t taken into account yet”.

(In the specific script involved here, there’s no .py extension, so the concern with reimporting __main__ doesn’t actually apply. However, the same structural technique can still be used to get rid of the deprecation warnings and make the code more amenable to automated testing)

2 Likes
14

Thanks for the analysis, however it’s incorrect :sweat_smile: Our make script is only used for development, never in “production”. mkdocstrings-python doesn’t even have a CLI.

What you’re writing about __main__ is correct though, and I do follow this approach. All my __main__ modules have the exact same contents:

# Entry-point module, in case you use `python -m pkg`.
#
# Why does this file exist, and why `__main__`? For more info, read:
#
# - https://www.python.org/dev/peps/pep-0338/
# - https://docs.python.org/3/using/cmdline.html#cmdoption-m

import sys

from pkg.cli import main

if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))

I learned and copied this from @ionelmc’s projects by the way :bowing_man:

MkDocs does suffer from a noisy __main__ module though, maybe this is what you meant?

15

Oh and this is the comment I have in all my cli.py modules:

# This module contains all CLI-related things.
# Why does this file exist, and why not put this in `__main__`?
#
# We might be tempted to import things from `__main__` later,
# but that will cause problems; the code will get executed twice:
#
# - When we run `python -m pkg`, Python will execute
#   `__main__.py` as a script. That means there won't be any
#   `pkg.__main__` in `sys.modules`.
# - When you import `__main__` it will get executed again (as a module) because
#   there's no `pkg.__main__` in `sys.modules`.
16

That’s a good summary, but note that
import __main__ is OK, since that does get a hit in sys.modules. It’s import pkg.__main__ (as well as any relative imports that resolve to that) that triggers the duplicate import.

1 Like
17

Same question, what’s exactly is wrong?

Above example could be adapted for console scripts as well:

# ./sitecustomize.py
import warnings
warning_file = open("warnings.txt", "w")
old_showwarning = warnings.showwarning
def mywarning(message, category, filename, lineno, file=None, line=None):
    old_showwarning(message, category, filename, lineno, warning_file, line)

warnings.showwarning = mywarning

$ cat ./a.py 
#! ./python
print(complex(1j, 1j))
$ export PYTHONPATH=. 
$ chmod u+x a.py 
$ ./a.py 
(-1+1j)
<sys>:0: ResourceWarning: unclosed file <_io.TextIOWrapper name='warnings.txt' mode='w' encoding='UTF-8'>
$ cat warnings.txt 
/home/sk/src/cpython/./a.py:2: DeprecationWarning: complex() argument 'real' must be a real number, not complex
  print(complex(1j, 1j))
/home/sk/src/cpython/./a.py:2: DeprecationWarning: complex() argument 'imag' must be a real number, not complex
  print(complex(1j, 1j))

Of course, if you want just filter out some warnings — much easier to override default filter with PYTHONWARNINGS.

I doubt it’s a popular feature request. This is also much less flexible (what if you want to redirect only some kind of warnings?) than shown above approach.

1 Like
18

you might want to read the stackoverflow issue and understand how many thousand people try this and it simply does not work as advertised.
see e.g.

grafik
grafik1330×98 12.7 KB

and other feedback

19

The warning system is very flexible and allows you to do many things you might want, e.g.:

  • Ignore all warnings: set the OS env var PYTHONWARNINGS=ignore

  • Only ignore some warnings, e.g the deprecation warnings: set the OS env var PYTHONWARNINGS=ignore::DeprecationWarning

  • Redirect all warnings to the logging module via a call to logging.captureWarnings(True) and configure this to write them to a file: see logging — Logging facility for Python — Python 3.12.5 documentation for how this can be configured

Where it’s not possible to configure these things via OS env vars, you can create a custom module and import this in the sitecustomize.py module, which is typically run at startup time (unless you use options to prevent this). See site — Site-specific configuration hook — Python 3.12.5 documentation for details.

It is also possible to only apply such settings to interactive REPL sessions by placing the import into PYTHONSTARTUP. See 1. Command line and environment — Python 3.12.5 documentation for details.

3 Likes
20

And what complaining at the right place and curing the disease instead of its symptoms? Have you filed a bug with wikiquery?

1 Like