Documentation for writing C extensions needs improving

As a developer not so familiar with Python, I found it very hard and disappointing to read the official documentation for writing C extension, which lacks a minimal working example and leads to many errors when following those guides.

To begin with, I would like to create a Python wrapper for my library, and as a common practice for C projects, I organized the files as

.
├── inlcude
│   └── foo.h
└── lib
    └── libfoo.so

To create a Python wrapper, I prefer the official documentation, which I think, unlike some outdated blogs, it would be up-to-date and correct. So I began to read the Extending Python with C or C++.

The code part for the documentation is very detailed, and I quickly write the necessary C glue codes including PyMethodDef, PyModuleDef, and the PyMODINIT_FUNC initialization function.

Then how can I build this project? From the Compilation and linkage subsection, it recommends to go to the building section, which further guided me to the setuptools documentation, where I found the related docs are in Building Extension Modules part.

In that documentation, it tells me to create a pyproject.toml alongside my C wrapper. So now, the project layout is

.
├── foo_python_wrapper.c
├── inlcude
│   └── foo.h
├── lib
│   └── libfoo.so
└── pyproject.toml

And the content of pyproject.toml is

[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

[project]
name = "mylib-foo"
version = "0.1"

[tool.setuptools]
ext-modules = [
  {name = "mylib.foo", sources = ["foo_python_wrapper.c"]}
]

This would obviously not work, since I have not written the header path and library path. And unfortunately, the documentation does not contain the solutions, and searching google did not help :frowning:

By looking at the “Extension API Reference” at the bottom of the documentation, I found the include_dirs and library_dirs may be what I want, but how can I use them in a TOML file? After some trails, I managed to write the pyproject.toml as

[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

[project]
name = "mylib-foo"
version = "0.1"

[tool.setuptools]
ext-modules = [
  {name = "mylib.foo", sources = ["foo_python_wrapper.c"], include-dirs=["./include"], library-dirs=["./lib"]}
]

Now can we proceed? After running python3 -m build, another error occurs:

error: Multiple top-level packages discovered in a flat-layout: ['lib', 'inlcude'].

To avoid accidental inclusion of unwanted files or directories,
setuptools will not proceed with this build.

If you are trying to create a single distribution with multiple packages
on purpose, you should not rely on automatic discovery.
Instead, consider the following options:

1. set up custom discovery (`find` directive with `include` or `exclude`)
2. use a `src-layout`
3. explicitly set `py_modules` or `packages` with a list of names

To find more information, look for "package discovery" on setuptools docs.

To be honest, I still don’t know what to do by looking at those three options. So I followed its last line, and went to the Package Discovery and Namespace Packages part of official documentation.

In that documentation, it explains the two common project layouts used for Python packages. However, the examples are all pure-python packages, and I still don’t know what to do with my library wrapper.

Then there is no clue at official documentations, leaving me confusing and lost in strange blogs found in google…

2 Likes

I’m sorry that you are not finding what you need in the docs. This is an area that has been pretty fluid around packaging. The scientific python ecosystem uses C-extensions quite a bit. Here are a few pages that should help:

  1. Compiled packaging - Scientific Python Development Guide
  2. Either scikit-build-core 0.10.8.dev19 documentation or meson-python
  3. If you choose meson python, a tutorial: Tutorial - meson-python

Additionally, pysheets has a very clear example as well C Extensions — pysheeet

2 Likes

Thank you so much for providing those valuable links. These save my world!

And since I put the question here (Documentation hub instead of Help hub), I not only seek for help, but also hope someone could help improving the official docs, because we all want the official documentations become better. :slight_smile:

2 Likes

I would suggest we want to finish Packaging binary extensions - Python Packaging User Guide and then update the core docs to point to that for most users, and only document the lowest level aspects (to help the people who build the build tools documented on the packaging.python.org page).

There are existing issues on the tracker for this, somewhere. Historically we didn’t think anywhere was good enough to link to from the core docs, but hopefully the packaging.python.org site is “safe” enough for us, and it just needs the content fleshed out to make it useful.

3 Likes