Module 'importlib' has no attribute 'metadata'

Can someone explain what can cause this?

I have run into a situation where:

import importlib.metadata 
importlib.metadata # Error!

But:

import importlib.metadata as importlib_metadata
importlib_metadata # NO error!

I have seen other similar odd issues with this module. What is going on? I can provide more details if necessary, it would just be a bit of a slog to write up. Mostly I’m hoping that to zeroth order someone can just shed light about any special assumptions this module has, or under what circumstances the metadata submodule would just not exist.

Edit: Python 3.10 FYI

Are you sure you did

import importlib.metadata

and not

import importlib

Most of the time I have seen people run into that is when they do the latter and then try to use importlib.metadata. But if you only import importlib you don’t get importlib.metadata, you have to import the submodule explicitly.

@saaketp Yes, 100% sure, which is why I am very confused.

Edit:

bk301-py10 ❯ bokeh serve --show foo.py
2022-11-10 14:23:59,567 Starting Bokeh server version 3.0.1 (running on Tornado 6.2)
2022-11-10 14:23:59,900 User authentication hooks NOT provided (default user enabled)
2022-11-10 14:23:59,903 Bokeh app running at: http://localhost:5006/foo
2022-11-10 14:23:59,903 Starting Bokeh server with process id: 30450
> /Users/bryan/tmp/foo.py(28)<module>()
-> import pyproj
(Pdb) import importlib.metadata
(Pdb) importlib.metadata
*** AttributeError: module 'importlib' has no attribute 'metadata'
(Pdb)

FWIW Bokeh server is in the unusual position of being a Python application that runs other Python applications, which sometimes introduces weirdness. I can believe there is something we might need to change or start doing on our end to make things more “normal” for the app code, but I also don’t know what that might be.

I cannot reproduce this on my Windows Store install of Python 3.10:

C:\>python3.10
Python 3.10.8 (tags/v3.10.8:aaaf517, Oct 11 2022, 16:50:30) [MSC v.1933 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import importlib.metadata
>>> importlib.metadata
<module 'importlib.metadata' from 'C:\\Program Files\\WindowsApps\\PythonSoftwareFoundation.Python.3.10_3.10.2288.0_x64__qbz5n2kfra8p0\\lib\\importlib\\metadata\\__init__.py'>

Is it possible that Bokeh server is invoking a different interpreter version than the 3.10 you think it is?

What’s the output of import sys; sys.version_info in that same context?

No, the Bokeh server runs the app code in-process (no subprocess invocation), so it’s the same:

(Pdb) import sys
(Pdb) sys.version
'3.10.6 (main, Oct 24 2022, 11:04:34) [Clang 12.0.0 ]'
1 Like

What is the repr of importlib, and sys.modules, and sys.path?

Anyone likely to be doing reloads or messing with sys.modules?

Often the second import of a module can do weird things like this, though there shouldn’t be a second import because it should be cached. But then, people like to try and “clear the cache,” which is much harder than del sys.modules[name]

What’s the output of import sys; sys.version_info in that same context?

short ones:

(Pdb) importlib
<module 'importlib' from '/Users/bryan/anaconda/envs/bk301-py10/lib/python3.10/importlib/__init__.py'>
(Pdb) sys.path
['/Users/bryan/tmp', '/Users/bryan/anaconda/envs/bk301-py10/bin', '/Users/bryan/anaconda/envs/bk301-py10/lib/python310.zip', '/Users/bryan/anaconda/envs/bk301-py10/lib/python3.10', '/Users/bryan/anaconda/envs/bk301-py10/lib/python3.10/lib-dynload', '/Users/bryan/anaconda/envs/bk301-py10/lib/python3.10/site-packages', '/Users/bryan/anaconda/envs/bk301-py10/lib/python3.10/site-packages/GDAL-3.4.1-py3.10-macosx-10.9-x86_64.egg']

sys.modules is big, has to go in a gist: sys.modules · GitHub

@steve.dower Sort of, Bokeh programmatically creates a module for the app code to be executed in for each session, then installs that module in sys.modules. But nothing is deleting modules at the point this error happens. [1] This has all worked fine for many years but it seems like some recent developments mean we need to update some aspect of how this is handed.

Edit: @steve.dower oh wait, do you mean this:

We could stop doing that…


  1. and we only clean up the modules we, ourselves, installed, some time after a session ends. ↩︎

1 Like

Can you try

import sys
print("importlib" in sys.modules and hasattr(sys.modules["importlib"], "metadata"))
print("importlib.metadata" in sys.modules)

Also possibly related
https://mail.python.org/archives/list/python-dev@python.org/thread/VIPXZRK3OJNSVNSZSAJ7CO6QFC2RX27W/

@saaketp

bk301-py10 ❯ bokeh serve --show foo.py
2022-11-10 15:01:01,348 Starting Bokeh server version 3.0.1 (running on Tornado 6.2)
2022-11-10 15:01:01,574 User authentication hooks NOT provided (default user enabled)
2022-11-10 15:01:01,577 Bokeh app running at: http://localhost:5006/foo
2022-11-10 15:01:01,577 Starting Bokeh server with process id: 30849
BokehDeprecationWarning: tile_providers module was deprecated in Bokeh 3.0.0 and will be removed, use add_tile directly instead.
> /Users/bryan/tmp/foo.py(28)<module>()
-> import pyproj
(Pdb) import sys
(Pdb) print("importlib" in sys.modules and hasattr(sys.modules["importlib"], "metadata"))
False
(Pdb) print("importlib.metadata" in sys.modules)
True
(Pdb)

@saaketp @pradyunsg @steve.dower

It seems that removing the del importlib.metadata in the bokeh/__init__.py module might fix:

(Pdb) import importlib.metadata
(Pdb) importlib.metadata
<module 'importlib.metadata' from '/Users/bryan/anaconda/envs/bk301-py10/lib/python3.10/importlib/metadata/__init__.py'>
(Pdb)

Okay then seems like the exact same problem as that mailing list post.
If importlib.metadata is already in sys.modules then import importlib.metadata does nothing, so importlib doesn’t get the metadata attribute. It should have got the attribute set when you imported the first time, but then that del statement deleted it.

2 Likes

Just to be clear (for my understanding), the metadata attribute (submodule?) is created dynamically by importlib import, but if it is deleted, then subsequent, explicit import importlib.metadata does not re-create it, because importlib is still cached in sys.modules?

That seems understandable in hindsight, but if I am honest, also seems like a foot-gun. :slight_smile:

1 Like

Well, actually I guess this is just Python 101 and I am just over way overthinking things. I really almost never, ever do import foo.bar without aliasing (or importing from).

When you try to import a submodule, if it succeeds it gets added to sys.modules and then set as an attribute on the parent package (that’s how the attribute access in your own code works). If the import fails, the name gets set to None in sys.modules to avoid trying a known-bad import again. Then future imports pull from sys.modules.

You can follow the code starting at cpython/_bootstrap.py at 3a1dde8f29215418ec4e27fd6234cfa19a5407c6 · python/cpython · GitHub as to how this all works.

@brettcannon Sure, all of that makes sense. I guess what surprised me here is the mode of failure. If I do del foo.bar, and then subsequently do import foo.bar I guess I would have expected that to fail immediately, rather than silently “work” (if it is not going to be automagic’ed to “actually work”). Like I said I almost always “import as” or “import from” so my intuition just got caught off guard.

The trick here is that del foo.bar isn’t doing anything with modules.

It translates into (approx) foo.__delattr__(bar). So all you’re doing is removing the attribute from foo, but foo.bar is still imported just as it ever was.

2 Likes