Module-level __getattr__ and from imports

Ideas
1

Let’s say I have a attribute of a module that I deprecated and have removed. While the attribute was deprecated, the __getattr__ implementation looks like what’s in PEP 562:

from warnings import warn

bar = "hello_v2"

def __getattr__(name):
    if name == "foo":
        warn("foo is deprecated, use bar instead", DeprecationWarning)
        return "hello"
    raise AttributeError(f"module {__name__!r} has no attribute {name!r}")

Users who access the attribute or import it see a deprecation warning:

In [1]: from mymodule import foo
/home/nathan/Documents/test/mymodule.py:6: DeprecationWarning: foo is deprecated, use bar instead
  warn("foo is deprecated, use bar instead", DeprecationWarning)

In [2]: import mymodule

In [3]: mymodule.foo
/home/nathan/Documents/test/mymodule.py:6: DeprecationWarning: foo is deprecated, use bar instead
  warn("foo is deprecated, use bar instead", DeprecationWarning)
Out[3]: 'hello'

But what about after the deprecation has expired?

In that case, I want an AttributeError, but I still want my custom message to include a migration path for the deprecated item, in case users missed or ignored the warning:

from warnings import warn

bar = "hello_v2"


def __getattr__(name):
    if name == "foo":
        raise AttributeError("foo has been removed, use bar instead")
    raise AttributeError(f"module {__name__!r} has no attribute {name!r}")

In this case, I get the AttributeError I want when I do module-level access, but the import mechanism swallows the custom AttributeError message if I import the symbol using a from import (I checked and __getattr__ is still called and the error is raised, but the import internals discard my custom error message):

In [1]: import mymodule

In [2]: mymodule.foo
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[2], line 1
----> 1 mymodule.foo

File ~/Documents/test/mymodule.py:8, in __getattr__(name)
      6 def __getattr__(name):
      7     if name == "foo":
----> 8         raise AttributeError("foo has been removed, use bar instead")
      9     raise AttributeError(f"module {__name__!r} has no attribute {name!r}")

AttributeError: foo has been removed, use bar instead

In [3]: from mymodule import foo
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
Cell In[3], line 1
----> 1 from mymodule import foo

ImportError: cannot import name 'foo' from 'mymodule' (/home/nathan/Documents/test/mymodule.py)

This leads me to two questions: is there a way in Python right now to get a nice error message for both styles of accessing the foo symbol? If there isn’t a way to do this yet, would it be a reasonable change to Python to detect when an AttributeError has a custom error message and include that in the ImportError?

1 Like
2

Raising a custom ImportError will work.

mod.py:

def __getattr__(name):
    if key == "foo":
        import inspect
        import re
        message = "foo has been removed, use bar instead"
        frame = inspect.currentframe()
        context = inspect.getframeinfo(frame.f_back).code_context
        if context and re.search(r"from .* import", context[0]):
            raise ImportError(message)
        raise AttributeError(message)
    raise AttributeError(f"module {__name__!r} has no attribute {name!r}")

test.py:

from mod import foo

REPL:

>>> import mod
>>> mod.foo
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/chris/Sandbox/module_getattr/mod.py", line 10, in __getattr__
    raise AttributeError(message)
AttributeError: foo has been removed, use bar instead

>>> import test
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/chris/Sandbox/module_getattr/test.py", line 1, in <module>
    from mod import foo
  File "/home/chris/Sandbox/module_getattr/mod.py", line 9, in __getattr__
    raise ImportError(message)
ImportError: foo has been removed, use bar instead

Unfortunately, Trying to do from mod import foo directly in the REPL doesn’t work because there’s no code context:

>>> from mod import foo
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'foo' from 'mod' (/home/chris/Sandbox/module_getattr/mod.py)
1 Like
3

When I try that on Python 3.10, I find that frame.f_back is inside the importlib internals:

(Pdb) frame.f_back
<frame at 0x55b7d7194140, file '<frozen importlib._bootstrap>', line 1075, code _handle_fromlist>

But looking at the inspect docs, it seems getouterframes does what I want. Searching for a wrapping frame of context where the first non-whitespace content is from is sufficient, I think? Your regex wouldn’t work if someone had an import like:

from mod \
    import foo

which is valid syntax, if strange.

Still, relying on searching the code content of frames in the call stack directly feels kind of hacky and I think a more natural place to fix this is inside the import internals.

Would a PR that bubbles up the custom AttributeError message to the ImportError be welcome? That way code that’s already handling this in a natural way per NEP 562 will produce nicer error messages without any code changes.

1 Like
4

I agree that relying on things like frame inspection is fragile.

I’m not sure that’s a great idea, as there’s no real way to know what kinds of errors or error messages should or shouldn’t be bubbled up. An AttributeError with various messages could be raised for various reasons, and many of those would be confusing if bubbled up. (For instance, the module could be importing another module that somehow raises AttributeError on its own import.)

My main response to your original question, although maybe it’s not super helpful, is to just say that the default error message is good. It tells you that the name foo cannot be imported and that is true. If a user gets such an error, they’re likely to debug by doing import mymodule and poking around, and if they try mymodule.foo they’ll get the more informative error message.

To my mind, a real solution in the import mechanism would be something like a __reverse_all__ (__none__? :slight_smile: ) that lists names that should raise an error on import, possibly with a custom message for each one, like:

__removed_names__ = {
    "foo": "foo is deprecated, used bar instead",
    "junk": "junk has been moved to the othermodule library"
}
5

To the first question, there’s a very easy answer: use any flavor of error other than AttributeError.

For example, the following module works nicely:

# foo.py
def __getattr__(name):
    if name == "bar":
        raise RuntimeError("bar was renamed to baz")
    return name

import foo; foo.bar and from foo import bar both show the RuntimeError.

AttributeError is treated specially by the importer – which makes sense – so you can “cut through” the importer’s handling by using any other flavor of error.

Regarding this pattern, I just want to drop a brief note that it’s not even that strange.
black formatting of large import blocks is multiline, but of the form

from mod import (
    foo,
    bar,
    baz,
)

Not quite the same (I agree that the backslash continuation is a bit unusual), but pretty similar and breaks any regex based inspection.

If you really want to inspect the parent frame, use tokenize or ast.

If you wanted any language-level or stdlib-level change, I think you would have to justify why the error has to be AttributeError.

3 Likes
6

Yeah, I probably should have said I don’t really recommend the above. I just thought it was a fun problem. What I actually use for this use case is:

class ExpiredDeprecationError(RuntimeError): ...

I will keep that around for an additional major version before removing it and letting Python raise ImportError and AttributeError.

2 Likes
7

I like this idea!

Downsides:

  • Requires a PEP and new code in libraries to support producing a better error

Upsides:

  • Low-effort to opt into and with a good dunder name could be self-documenting
  • Doesn’t force downstream users of the library doing the deprecation and removal to catch multiple exception types if they need to support multiple versions of the library
  • Allows removing the custom __getattr__ completely after the deprecation is completed.
8

I think this is not as strong of a story as it might first appear.

The typical recommendation is to try a new method first and then failover to the old one. This situation is only even possibly a concern if downstream users write their compatibility code “backwards” from the norm. Even stipulating to that as a possibility, it feels like this arises from trying to remove something before you’re really ready to remove it.

The normal pattern for the library is that you first want to intercept the old name and emit a deprecation warning, then at some point simply drop the old name (with no special error behavior).

If you want to go the extra mile with ExpiredDeprecationError or similar, cool! But any user who encounters that error has broken code and needs to fix it in a way such that they aren’t seeing the error – not even trying to catch it. At a certain point you do have to let your users bear some of the responsibility for this.

The hardest thing, IMO, is communicating out and documenting norms like “try the new way first”. It’s well-socialized within a certain class of library maintainers, but not as familiar to users who aren’t package maintainers. So you need to document how you want your users to handle compatibility and upgrade concerns, or they won’t know what you expect of them.

1 Like
9

I forgot to address the main thing; sorry for replying by parts. :sweat_smile:

I think it has more downsides than this, which is why I’d rather help think through and find solutions within the current paradigm for module behaviors.

You have to think about how attribute access (__getattr__, __getattribute__) are supposed to behave in detail, and whether or not this is a dunder name which will be supported on non-module objects.

I would 100% be there with you using “easy mode” for module attribute lifecycle if the feature were added to the language. But I think the amount of value added vs the cost of adding it is highly unfavorable to this kind of change.