Is shipping mod.<platform>.{pyd,so} and mod.py with the same name, in the same wheel an intended pattern?

Python Help
1

charset-normalizer includes the module charset_normalizer.md, native wheels provide it as both

  1. a native extension, e.g. charset_normalizer/md.cpython-314-darwin.so
  2. a pure python file, charset_normalizer/md.py

Is this an intended/documented way to ship an extension module with a pure python fallback, or a quirk of one project’s packaging that I’ve read too much into?

Python’s importlib appears to transparently use the native extension if it’s compatible (by platform, major.minor, …) and fallback to the pure python if necessary. There’s no explicit try: import …; except ImportError: import …_fallback as … in charset_normalizer that I could spot.

For example if I manually create a situation where the .so is compiled for a non-matching Python version (3.14 vs 3.13) then the import succeeds without complaint/warning, using the pure Python

alex@d13:~$ unzip charset_normalizer-3.4.4-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl
...
  inflating: charset_normalizer/md.cpython-314-x86_64-linux-gnu.so
  inflating: charset_normalizer/md.py
  inflating: charset_normalizer/md__mypyc.cpython-314-x86_64-linux-gnu.so
...
alex@d13:~$ python3
Python 3.13.5 (main, Jun 25 2025, 18:55:22) [GCC 14.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import charset_normalizer.md
>>> charset_normalizer.md
<module 'charset_normalizer.md' from '/home/alex/charset_normalizer/md.py'>

Context: I’m investigating fixes for RequestsDependencyWarning: Unable to find acceptable character detection dependency (chardet or charset_normalizer). · Issue #1405 · mitogen-hq/mitogen · GitHub and might end up using this behaviour in a fix. Mitogen serves pure python modules over the wire to child processes as they import them.

2

Given that the wheels are also platform+Python ABI tagged, it should be impossible for that .py fallback to ever be used without circumventing the package manager’s wheel selection in some way.

It looks like like their aiming for an opt-in compiled extension if building from source and the prebuilt wheels are built with that opt-in enabled. In which case the .py file in the wheel would be dead weight but presumably nobody’s bothered enough to try and cajole setuptools into conditionally excluding one .py file from wheels if mypyc is enabled.

3

That’s exactly what I’m considering for Mitogen - control node A serves md.py to client node B, because A and B may be different OS, instruction arch, and/or Python version.

Agreed, it’s more likely there by chance than design. If so, at best limited benefit to rely on it. Thanks.

4

They’re using mypyc but optionally compiling a .py file is something you can do with Cython too. In Cython, keeping the .py file around is useful because it is used to add the source lines to any tracebacks. I don’t know if mypyc works in the same way though.

Edit: I suspect it probably does work the same because most of the work is just providing a code object with the source line and source file - it’s Python that actually reads that when printing the traceback.

1 Like