How-To publish an f2py project?

Hi all. I wrote a numpy extension and I’d like to publish it but I’m not sure how to author a distributable for all the various OS’s. Anybody here able to help me locate a guide for this?

For more context, you can find the repo here (a readme is forthcoming): GitHub - rm-rf-etc/ehlers

The project is based on GitHub - scivision/f2py-examples: Examples of using f2py to get high-speed Fortran integrated with Python easily, meaning it compiles fortran code and wraps it using f2py. The machine I’m using right now is running Windows 10, so the output file I get is ehlers.cp39-win_amd64.pyd. Curious to know how to output the proper file for all the various OS’s and Python versions and numpy versions.

@henryiii and friends would likely have some good advice here. Mostly likely it will involve cibuildwheel building the various wheels for the OS and Python versions you’re targeting, and typically what is done AFAIK is targeting the oldest Numpy version you want to support and building against that. But I’m not intimately familiar with the process as others are, especially with f2py, so I’d defer to them.

2 Likes

f2py is annoying for distribution, because it’s tied to numpy, means if you select oldest-supported-numpy, to get a compatible extension, you are stuck with an older version of f2py. If you select a newer numpy, you can’t be loaded on an older version of numpy (not due to f2py, but due to the numpy headers). (NumPy extensions can load on any newer version of numpy, but not older versions (except for most patch versions). See GitHub - scipy/oldest-supported-numpy: Meta-package providing the oldest supported Numpy for a given Python version and platform for help with selecting the lowest NumPy, but that might not be as helpful with f2py.

Developer information - Scikit-HEP has a guide for packaging related things, including distribution. cibuildwheel (already mentioned above) is the main tool you’d likely want to use to build wheels for distribution.

Edit: f2py itself is not at all annoying, it’s very nice, it’s just tricky to distribute because upgrading even for bug fixes force you to use newer NumPy’s)

1 Like

Thanks for all this info. What would you recommend as an easier way to create a distributable? Would it be easier if I switch to C?

Taking a step back to clarify some of the base assumptions, sorry if I am over-splaining. Skip over this if the information is already clear to you.

f2py (1) outputs a pyd which is a python c-extension module, and (2) requires NumPy to work.

Any project that does (1) needs to build the project for the 3d matrix of (python_version) X (platform) X (python implementation) since it is tightly tied to all of those parameters. Usually people build for CPython, which is the cp in your file name, you could also build for PyPy. Then the 39 means python==3.9, and the win_amd64 means 64-bit-windows-on-x86 (as opposed to 64-bit-windows-on-arm or any other platform). The number of combinations is sometimes overwhelming. NumPy releases about 20 such packages. cibuildwheel can create this matrix of builds on top of CI infrastructure and make the whole process of building, testing, and publishing packages easier. There is a way to build one package for various versions of python, but NumPy and f2py do not support the stable API so that is not relevant here.

As for (2): any c-extension that depends on NumPy must build against the minimum version of NumPy it is prepared to support. NumPy (so far) is backward compatible: once you build against that minimum version, it will support future versions. To use f2py means you are now restricted to using the f2py that comes with that minimum version.

Bottom line: cibuildwheel makes (1) tolerable, and if you use C instead of f2py (2) goes away.

1 Like