Heisenbug involving attrs, ABCs, and inspect (Broken environment?)

Python Help
1

Consider the following code:

from __future__ import annotations
from abc import ABC
import inspect
from typing import ClassVar
import attr


@attr.define
class Abstract(ABC):
    INDEX: ClassVar[int]

    @classmethod
    def make(cls, index: int, value: int) -> Abstract:
        for klass in Abstract.__subclasses__():
            if klass.INDEX == index:
                print(f"{klass=}")
                return klass(value)
        raise ValueError(f"Unknown index: {index}")


@attr.define
class Concrete(Abstract):
    INDEX: ClassVar[int] = 1
    foo: int


print("Concrete.__init__:")
print(inspect.getsource(Concrete.__init__))
print()

print(Abstract.make(1, 42))

When I run this on macOS 11.7.2 using CPython 3.11.5 (installed via Homebrew), with attrs 23.1.0 installed in the system site packages directory, I get the following output & error:

Concrete.__init__:
def __init__(self, foo):
    self.foo = foo


klass=<class '__main__.Concrete'>
Traceback (most recent call last):
  File "/Users/jwodder/work/dev/tmp/attrs-bug-20230923/tests/inspect-init.py", line 31, in <module>
    print(Abstract.make(1, 42))
          ^^^^^^^^^^^^^^^^^^^^
  File "/Users/jwodder/work/dev/tmp/attrs-bug-20230923/tests/inspect-init.py", line 17, in make
    return klass(value)
           ^^^^^^^^^^^^
TypeError: Abstract.__init__() takes 1 positional argument but 2 were given

However, if I create a fresh virtualenv using the same Python and attrs versions, the script succeeds. If I remove the inspect lines, the script fails inside & outside a virtualenv. If I keep the inspect lines and remove the (ABC) and INDEX: ClassVar[int] bits, the script passes inside & outside a virtualenv.

I initially assumed that some other package in Pythonā€™s system site packages was messing things up somehow, so I compiled a topologically-sorted list of all system site packages and installed them one by one in a fresh virtualenv, rerunning the above script after each installation; unfortunately, the script succeeded each time, so I failed to reproduce the error.

I also tried uninstalling & reinstalling attrs; no change.

I also have Python 3.10.13 (with attrs 22.2.0) and Python 3.9.18 (with attrs 22.1.0) on my system; the script succeeds with both.

My current theory is that my Python 3.11 installation is broken somehow, but I donā€™t know how to diagnose it. For the record, the output of pip list is:

Package                 Version
----------------------- ---------
annotated-types         0.5.0
argcomplete             3.1.2
attrs                   23.1.0
autocommand             2.2.2
black                   23.9.1
bleach                  6.0.0
build                   1.0.3
CacheControl            0.13.1
cachetools              5.3.1
certifi                 2023.7.22
cffi                    1.15.1
cfgv                    3.4.0
chardet                 5.2.0
charset-normalizer      3.2.0
click                   8.1.7
colorama                0.4.6
colorlog                6.7.0
contourpy               1.1.1
coverage                7.3.1
cryptography            41.0.3
cycler                  0.11.0
Deprecated              1.2.14
distlib                 0.3.7
docutils                0.20.1
filelock                3.12.4
flake8                  6.1.0
flake8-bugbear          23.9.16
flake8-builtins         2.1.0
flake8-unused-arguments 0.0.13
fonttools               4.42.1
ghrepo                  0.7.0
headerparser            0.4.0
identify                2.5.29
idna                    3.4
importlib-metadata      6.8.0
in-place                0.5.0
inflect                 7.0.0
iterpath                0.4.0
jaraco.classes          3.3.0
jaraco.context          4.3.0
jaraco.env              1.0.0
jaraco.functools        3.9.0
jaraco.text             3.11.1
Jinja2                  3.1.2
keyring                 24.2.0
kiwisolver              1.4.5
lockfile                0.12.2
markdown-it-py          3.0.0
MarkupSafe              2.1.3
matplotlib              3.8.0
mccabe                  0.7.0
mdurl                   0.1.2
mercurial               6.5.2
more-itertools          10.1.0
msgpack                 1.0.5
mypy                    1.5.1
mypy-extensions         1.0.0
nh3                     0.2.14
nodeenv                 1.8.0
nox                     2023.4.22
numpy                   1.26.0
packaging               23.1
path                    16.7.1
pathspec                0.11.2
pbr                     5.11.1
Pillow                  10.0.1
pip                     23.2.1
pip-run                 12.2.2
pipdeptree              2.13.0
pipx                    1.2.0
pkginfo                 1.9.6
platformdirs            3.10.0
pluggy                  1.3.0
pre-commit              3.4.0
pycodestyle             2.11.0
pycparser               2.21
pydantic                2.3.0
pydantic_core           2.6.3
pyflakes                3.1.0
PyGithub                1.59.1
Pygments                2.16.1
PyJWT                   2.8.0
PyNaCl                  1.5.0
pyparsing               3.1.1
pyproject-api           1.6.1
pyproject_hooks         1.0.0
python-dateutil         2.8.2
PyYAML                  6.0.1
readme-renderer         42.0
requests                2.31.0
requests-toolbelt       1.0.0
rfc3986                 2.0.0
rich                    13.5.3
setuptools              68.2.2
six                     1.16.0
stevedore               5.1.0
tox                     4.11.3
trove-classifiers       2023.8.7
twine                   4.0.2
txtble                  0.12.0
typing_extensions       4.8.0
urllib3                 2.0.4
userpath                1.9.1
virtualenv              20.24.5
virtualenv-clone        0.5.7
virtualenvwrapper       4.8.4
wcwidth                 0.2.6
webencodings            0.5.1
wheel                   0.41.2
wrapt                   1.15.0
zipp                    3.17.0

and the contents of /usr/local/lib/python3.11/site-packages are:

2ec0e72aa72355e6eccf__mypyc.cpython-311-darwin.so*
Deprecated-1.2.14.dist-info/
Jinja2-3.1.2.dist-info/
MarkupSafe-2.1.3.dist-info/
PIL/
Pillow-10.0.1.dist-info/
PyGithub-1.59.1.dist-info/
PyJWT-2.8.0.dist-info/
PyNaCl-1.5.0.dist-info/
PyYAML-6.0.1.dist-info/
Pygments-2.16.1.dist-info/
__pycache__/
_black_version.py
_cffi_backend.cpython-311-darwin.so*
_distutils_hack/
_yaml/
annotated_types/
annotated_types-0.5.0.dist-info/
argcomplete/
argcomplete-3.1.2.dist-info/
attr/
attrs/
attrs-23.1.0.dist-info/
autocommand/
autocommand-2.2.2.dist-info/
black/
black-23.9.1.dist-info/
blackd/
bleach/
bleach-6.0.0.dist-info/
blib2to3/
bugbear.py
build/
build-1.0.3.dist-info/
cachecontrol/
cachecontrol-0.13.1.dist-info/
cachetools/
cachetools-5.3.1.dist-info/
ced4bbd844d3a34b6fc2__mypyc.cpython-311-darwin.so*
certifi/
certifi-2023.7.22.dist-info/
cffi/
cffi-1.15.1.dist-info/
cfgv-3.4.0.dist-info/
cfgv.py
chardet/
chardet-5.2.0.dist-info/
charset_normalizer/
charset_normalizer-3.2.0.dist-info/
click/
click-8.1.7.dist-info/
clonevirtualenv.py
colorama/
colorama-0.4.6.dist-info/
colorlog/
colorlog-6.7.0.dist-info/
contourpy/
contourpy-1.1.1.dist-info/
coverage/
coverage-7.3.1.dist-info/
cryptography/
cryptography-41.0.3.dist-info/
cycler-0.11.0.dist-info/
cycler.py
dateutil/
deprecated/
distlib/
distlib-0.3.7.dist-info/
distutils-precedence.pth
docutils/
docutils-0.20.1.dist-info/
filelock/
filelock-3.12.4.dist-info/
flake8/
flake8-6.1.0.dist-info/
flake8_bugbear-23.9.16.dist-info/
flake8_builtins-2.1.0.dist-info/
flake8_builtins.py
flake8_unused_arguments-0.0.13.dist-info/
flake8_unused_arguments.py
fontTools/
fonttools-4.42.1.dist-info/
ghrepo/
ghrepo-0.7.0.dist-info/
github/
headerparser/
headerparser-0.4.0.dist-info/
hgdemandimport/
hgext/
hgext3rd/
identify/
identify-2.5.29.dist-info/
idna/
idna-3.4.dist-info/
importlib_metadata/
importlib_metadata-6.8.0.dist-info/
in_place-0.5.0.dist-info/
in_place.py
inflect/
inflect-7.0.0.dist-info/
iterpath/
iterpath-0.4.0.dist-info/
jaraco/
jaraco.classes-3.3.0.dist-info/
jaraco.context-4.3.0.dist-info/
jaraco.env-1.0.0.dist-info/
jaraco.functools-3.9.0.dist-info/
jaraco.text-3.11.1.dist-info/
jinja2/
jwt/
keyring/
keyring-24.2.0.dist-info/
kiwisolver/
kiwisolver-1.4.5.dist-info/
libsvn@
lockfile/
lockfile-0.12.2.dist-info/
markdown_it/
markdown_it_py-3.0.0.dist-info/
markupsafe/
matplotlib/
matplotlib-3.8.0.dist-info/
mccabe-0.7.0.dist-info/
mccabe.py
mdurl/
mdurl-0.1.2.dist-info/
mercurial/
mercurial-6.5.2.dist-info/
more_itertools/
more_itertools-10.1.0.dist-info/
mpl_toolkits/
msgpack/
msgpack-1.0.5.dist-info/
mypy/
mypy-1.5.1.dist-info/
mypy_extensions-1.0.0.dist-info/
mypy_extensions.py
mypyc/
nacl/
nh3/
nh3-0.2.14.dist-info/
nodeenv-1.8.0.dist-info/
nodeenv.py
nox/
nox-2023.4.22.dist-info/
numpy/
numpy-1.26.0.dist-info/
packaging/
packaging-23.1.dist-info/
path/
path-16.7.1.dist-info/
pathspec/
pathspec-0.11.2.dist-info/
pbr/
pbr-5.11.1.dist-info/
pip/
pip-23.2.1.dist-info/
pip-run.py
pip_run/
pip_run-12.2.2.dist-info/
pipdeptree/
pipdeptree-2.13.0.dist-info/
pipx/
pipx-1.2.0.dist-info/
pkg_resources/
pkginfo/
pkginfo-1.9.6.dist-info/
platformdirs/
platformdirs-3.10.0.dist-info/
pluggy/
pluggy-1.3.0.dist-info/
pre_commit/
pre_commit-3.4.0.dist-info/
pycodestyle-2.11.0.dist-info/
pycodestyle.py
pycparser/
pycparser-2.21.dist-info/
pydantic/
pydantic-2.3.0.dist-info/
pydantic_core/
pydantic_core-2.6.3.dist-info/
pyflakes/
pyflakes-3.1.0.dist-info/
pygments/
pylab.py
pyparsing/
pyparsing-3.1.1.dist-info/
pyproject_api/
pyproject_api-1.6.1.dist-info/
pyproject_hooks/
pyproject_hooks-1.0.0.dist-info/
python_dateutil-2.8.2.dist-info/
readme_renderer/
readme_renderer-42.0.dist-info/
requests/
requests-2.31.0.dist-info/
requests_toolbelt/
requests_toolbelt-1.0.0.dist-info/
rfc3986/
rfc3986-2.0.0.dist-info/
rich/
rich-13.5.3.dist-info/
setuptools/
setuptools-68.2.2.dist-info/
six-1.16.0.dist-info/
six.py
stevedore/
stevedore-5.1.0.dist-info/
svn@
tox/
tox-4.11.3.dist-info/
trove_classifiers/
trove_classifiers-2023.8.7.dist-info/
twine/
twine-4.0.2.dist-info/
txtble/
txtble-0.12.0.dist-info/
typing_extensions-4.8.0.dist-info/
typing_extensions.py
urllib3/
urllib3-2.0.4.dist-info/
userpath/
userpath-1.9.1.dist-info/
virtualenv/
virtualenv-20.24.5.dist-info/
virtualenv_clone-0.5.7.dist-info/
virtualenvwrapper/
virtualenvwrapper-4.8.4-py3.11-nspkg.pth
virtualenvwrapper-4.8.4.dist-info/
wcwidth/
wcwidth-0.2.6.dist-info/
webencodings/
webencodings-0.5.1.dist-info/
wheel/
wheel-0.41.2.dist-info/
wrapt/
wrapt-1.15.0.dist-info/
yaml/
zipp/
zipp-3.17.0.dist-info/
2

Hi, Iā€™m a maintainer of attrs.

You didnā€™t paste your imports so I did my best to recreate them. Hereā€™s how you can repro your issue:

from __future__ import annotations

import gc
import inspect
from abc import ABC
from typing import ClassVar

import attr

gc.disable()


@attr.define
class Abstract(ABC):
    INDEX: ClassVar[int]

    @classmethod
    def make(cls, index: int, value: int) -> Abstract:
        for klass in Abstract.__subclasses__():
            if klass.INDEX == index:
                print(f"{klass=}")
                return klass(value)
        raise ValueError(f"Unknown index: {index}")


@attr.define
class Concrete(Abstract):
    INDEX: ClassVar[int] = 1
    foo: int


print("Concrete.__init__:")
print(inspect.getsource(Concrete.__init__))
print()

print(Abstract.make(1, 42))

I can explain the issue. When you use attrs.define, youā€™re asking attrs to create a new, slotted class for you. The class you give to attrs isnā€™t slotted (itā€™s just a regular dict class), so attrs has to create a new class for you. (Dataclasses does the same thing, thereā€™s currently no other way.)

For whatever reason this creates some garbage with cyclic references; this garbage includes the original class. The original class stays in there until a garbage collection pass happens, and thatā€™s what youā€™re seeing in Abstract.__subclasses__. So if a garbage collection pass happens before you call make, itā€™ll succeed, and if not, itā€™ll fail. You can fix your issue by manually triggering a gc pass (this needs to be done only once):

gc.collect()

print("Concrete.__init__:")
print(inspect.getsource(Concrete.__init__))
print()

print(Abstract.make(1, 42))
1 Like
3

On MacOS 13.5.2, using Python 3.10.12 or Python 3.11.5 (conda installs), with attrs-23.1.0, Iā€™m not able to reproduce this issue.

4

So, if Iā€™m understanding you correctly, Abstract.__subclasses__ contains a ā€œpre-attr-ifiedā€ copy of Concrete until the point at which garbage collection happens? Are there any other ways to get around this besides messing with GC? For example, you said that a new class needs to be created to implement slots, so if I disable slots on both classes, would the code be guaranteed to always succeed?

5

Yeah, you got it.

If you create both classes using @attr.define(slots=False) the problem disappears (as far as I can see). Then youā€™d have dict classes though, which are slightly different and that may or may not matter to you.

But Iā€™m not saying you should mess with the gc (the gc.disable() call is just to reliably demonstrate the problem, you shouldnā€™t actually do this). Iā€™m saying you can just trigger a collection by using gc.collect(). Thatā€™s part of normal Python operation, youā€™d just be triggering it manually once.

2 Likes
6

Thank you very much! This ended up addressing the original bug in a project of mine that I was trying to create an MVCE for when I ran into the behavior here.

1 Like
7

Technically you can make a slotted class if you manage to rewrite the class before compilation. I have an implementation that does this at the AST level, but it does require an import hook and a ā€˜meaningfulā€™ comment[1]. Itā€™s not pretty and I probably wouldnā€™t recommend doing it that way, but it is possible. (The implementation itself is largely an experiment in optimising import time).

  1. Maybe it could be avoided if we get something like PEP-638? Not sure. ā†©ļøŽ