Building an object file during installation from source

I’ve got a package builds an extension with CFFI. That extension needs to be linked with an object file to work.

Currently I build the object file separately, then add it to extra_objects. This works nicely for binary/wheel distributions, even on Windows.

However, it doesn’t work for source distributions. pip install doesn’t know that it needs to build the object file first. Can I make it do that?

Are there other options? I considered:

  • Use subprocess to build the object file - I would need separate Unix/Windows command for this, which is unappealing
  • Include the object files in the sdist - this is against the spirit of the source distribution, plus I would need all platforms with unique names

There seems to be documentation here: Preparing and Distributing modules — CFFI 1.17.0.dev0 documentation

Maybe there are setuptools plugins for this. Otherwise you might need to override some setuptools commands for this: Extending or Customizing Setuptools - setuptools 68.2.2.post20231016 documentation

Another possibility is to find an alternative build backend (instead of setuptools) that has support for cffi. I do not know if there are such build backends.

Is this a CFFI specific issue? I’ve been assuming it’s more of a general “how can I make setuptools (or something else) build an object file for linking to an extension” problem. It seems like that should be possible, but finding a method that works for source distributions is proving difficult.

1 Like

Building things that are not pure Python seems to always be somewhat problematic (as far as I have heard, I guess I am lucky that I never needed to do such a thing). But you are also not the first one to want to build non-pure Python project, there are solutions and documentation around. It is true that the quality of solutions and documentation seems to often be quite poor, so you might need to read a lot of different sources and try a lot of different techniques before finding the one that is right. If I were you I would try with the CFFI documentation page I linked first. Maybe maturin is a solution you could try (this would replace setuptools): Distribution - Maturin User Guide

Sorry, I should have specified: I’m coming here because I’ve exhausted the documentation sources that seem relevant.

The CFFI guide is great for the binary package: I used it! I’m successfully distributing the CFFI module using this extra compilation and linking step! I just can’t make it work for the sdist.

My hope is that this forum will have somebody who had accomplished building a thing prior to the extension who could point me to how it’s done. Fingers crossed!

What sort of command exactly are you using to build this object file?

On Unix:

$(CXX) -c "ada_url/ada.cpp" -fPIC -std="c++17" -O2 -o "ada_url/ada.o" $(ARCHFLAGS)

On Windows:

C:\\Program Files\\Microsoft Visual Studio\\2022\\Enterprise\\VC\\Auxiliary\\Build\\vcvars64.bat" && cl "ada_url\\ada.cpp" /c /nologo /Fo"ada_url\\ada.o" /O2 /GL /MD /W3 /EHsc /std:c++17

OK, I believe it is too far from my area of knowledge, I do not think I will be able to help. I suggest presenting a “minimal reproducible example” if that makes sense, maybe it will make it easier for others to help you.

But part of me wonders if what you are trying to do is necessary… Source distributions are meant to be multi-platform, right? And am I understanding right that you want to add compiled object binary files in the source distribution? I do not know much about CFFI, but I am wondering if that makes sense. Sorry if it is a stupid question.

Let me see if I understand your problem. You want to build two C/C++ files and link them together to form your extension, and you’ve understood that you can do that in setup.py, but the compiler commands are platform-specific, and this is what you want to avoid?

setuptools definitely has support for building C/C++ extensions (auto-detecting compilers and such), but I don’t know anything about it personally. Also, setuptools is somewhat widely recognized to suffer from complication and legacy burden.

If I were you, I’d look at meson-python. Personally, I’ve found Meson quite a joy to use (YMMV).

1 Like