Clarification status priority between extensions and .py files?

Python Help
1

If there are in the same directory an extension and a source file (.py) with the same name, the interpreter imports the extension and not the source file.

This behavior has been stable since at least Python 1.5. It is used by Python compilers like Cython (in “pure Python mode”) and Pythran (now used for example in Scipy build).

This behavior is documented in Built-in Package Support in Python 1.5 | Python.org

(Tip: the search order is determined by the list of suffixes returned by the function imp.get_suffixes(). > Usually the suffixes are searched in the following order: “.so”, “module.so”, “.py”, “.pyc”. Directories
don’t explicitly occur in this list, but precede all entries in it.)

However, imp is deprecated and this behavior does not seem to be explicitly documented in the current documentation.

@uranusjr wrote in Problem with adding a Python file next to the universal extension to make it importable · Issue #203 · hpyproject/hpy · GitHub :

imp is deprecated, and the current documentation says:

Deprecated since version 3.3: Use the constants defined on importlib.machinery instead.

But importlib.machinery does not have an equivalent constant; the only thing close to it is all_suffixes(). And its value does not match the implemented import logic :

  • _get_supported_file_loaders() lists extensions first, then .py and .pyc
  • all_suffixes() lists .py , .pyc , and extensions last.

On the one hand, changing this behavior in future Python versions would break a lot of packages (at least those using Cython in pure Python mode and/or Pythran), for example Scipy.

On the other hand, the hack now used by HPy to be able to import universal extension (adding during install a dummy .py file next to the universal extension) is not compatible with this priority behavior. However, it is just a temporary hack and another solution will have to be found.

It would be good to get a clarification about the status of the priority behavior of Python (extension chosen first). Is this behavior intended to be relied on?

I don’t know if this is the right place to ask for this clarification?

2

I think we need to clarify whether all_suffixes() is only intended to list all the available extensions, or ifthe ordering actually means something. If it’s the former, we can just change the ordering. Personally I suspect the ordering is not meaningful (the current ordering is not useful in any scenarios afaict), a b.p.o. with a PR to change that might be the best way to drive the discussion forward.