PEP 562 (Module __getattr__ and __dir__) and issue with sub-interpreters

Python Help
1

Hi, a project I work on encountered an issue with CPython freezing on an import of a submodule of scipy, but CPython is embedded inside a Java process using Jep and it is using sub-interpreters. Jep has some tricks to work around the fact that most CPython modules were not coded to work correctly in sub-interpreters and the shared modules one is being used on my project. An analysis of the cause of the freezing issue is available on Jep #487.

I emailed the scipy developers about potentially importing importlib in __getattr__ and they pointed out that they are following the example given on PEP 562 so the problem is likely more widespread than just scipy. They also suggested I reach out to CPython developers.

I have the following questions:

  1. Is it possible for PEP 562 to be changed or amended? If so, is this something worth changing it for? (The issue is rather esoteric and specific to Jep so I won’t be surprised if you say no).
  2. Are there any ideas of improvements that could be done to importlib to either solve or detect this issue better?
  3. Are there any ideas for how Jep could fix this or handle it better?

Thanks.

2

PEP 562 has been around for 6 years now, so no. But I think what you’re really asking is whether Python could be changed, to which the answer is “maybe”. You don’t say what change you’re asking for, so it’s hard to be specific, but assuming what you want is a breaking change, Python’s backward compatibility policy applies.

But basically, describe what change you want, and it’ll be easier to know how to respond.

3

For the PEP, I was just interested in if we could move the import of importlib in the example code. The example code for lazy submodule imports is

# lib/__init__.py

import importlib

__all__ = ['submod', ...]

def __getattr__(name):
    if name in __all__:
        return importlib.import_module("." + name, __name__)
    raise AttributeError(f"module {__name__!r} has no attribute {name!r}")

I was just wondering if we could move the import of importlib into __getattr__ as that fixes the Jep issue with sub-interpreters and organizations like scipy are following that code almost exactly. But it sounds like the PEP can’t be changed.

So then I don’t have any other changes to suggest, I was just hoping someone here might have the expertise to know how importlib could be improved or Jep could do things differently.

4

That’s just example code. If it doesn’t work in your situation, just fix your code - by moving the import, by the sound of it. There’s no need to update the PEP, as example code in a PEP isn’t intended to be production ready.

But of course in this case, the scipy developers are correct. There’s no way you can guarantee that every bit of 3rd party code is safe for this usage - especially as importing within a function is generally considered bad practice. This may be something to cover in a document on “how to write code that’s safe for use in multiple subinterpreters”, but I’m not aware of such a document existing (yet).

1 Like