Include header above project root

Python Help
1

I have been trying to modernize Linux GPIB Support / git / [0fc6e3] /linux-gpib-user/language/python to pyproject.toml with setuptools as build backend. However, it fails to build since it expects a header that is above the project root:

{ /usr/bin/python3 -m build && touch build; } || { rm -f -r build; exit 1; }
* Creating venv isolated environment...
* Installing packages in isolated environment... (setuptools)
* Getting build dependencies for sdist...
/tmp/build-env-a11cqjed/lib/python3.12/site-packages/setuptools/config/pyprojecttoml.py:72: _ExperimentalConfiguration: `[tool.setuptools.ext-modules]` in `pyproject.toml` is still *experimental* and likely to change in future releases.
  config = read_configuration(filepath, True, ignore_option_errors, dist)
running egg_info
creating gpib.egg-info
writing gpib.egg-info/PKG-INFO
writing dependency_links to gpib.egg-info/dependency_links.txt
writing top-level names to gpib.egg-info/top_level.txt
writing manifest file 'gpib.egg-info/SOURCES.txt'
reading manifest file 'gpib.egg-info/SOURCES.txt'
writing manifest file 'gpib.egg-info/SOURCES.txt'
* Building sdist...
/tmp/build-env-a11cqjed/lib/python3.12/site-packages/setuptools/config/pyprojecttoml.py:72: _ExperimentalConfiguration: `[tool.setuptools.ext-modules]` in `pyproject.toml` is still *experimental* and likely to change in future releases.
  config = read_configuration(filepath, True, ignore_option_errors, dist)
running sdist
running egg_info
writing gpib.egg-info/PKG-INFO
writing dependency_links to gpib.egg-info/dependency_links.txt
writing top-level names to gpib.egg-info/top_level.txt
reading manifest file 'gpib.egg-info/SOURCES.txt'
writing manifest file 'gpib.egg-info/SOURCES.txt'
running check
creating gpib-1.0
creating gpib-1.0/gpib.egg-info
copying files to gpib-1.0...
copying Gpib.py -> gpib-1.0
copying README -> gpib-1.0
copying gpibinter.c -> gpib-1.0
copying pyproject.toml -> gpib-1.0
copying gpib.egg-info/PKG-INFO -> gpib-1.0/gpib.egg-info
copying gpib.egg-info/SOURCES.txt -> gpib-1.0/gpib.egg-info
copying gpib.egg-info/dependency_links.txt -> gpib-1.0/gpib.egg-info
copying gpib.egg-info/top_level.txt -> gpib-1.0/gpib.egg-info
copying gpib.egg-info/SOURCES.txt -> gpib-1.0/gpib.egg-info
Writing gpib-1.0/setup.cfg
Creating tar archive
removing 'gpib-1.0' (and everything under it)
* Building wheel from sdist
* Creating venv isolated environment...
* Installing packages in isolated environment... (setuptools)
* Getting build dependencies for wheel...
/tmp/build-env-fv8woooz/lib/python3.12/site-packages/setuptools/config/pyprojecttoml.py:72: _ExperimentalConfiguration: `[tool.setuptools.ext-modules]` in `pyproject.toml` is still *experimental* and likely to change in future releases.
  config = read_configuration(filepath, True, ignore_option_errors, dist)
running egg_info
writing gpib.egg-info/PKG-INFO
writing dependency_links to gpib.egg-info/dependency_links.txt
writing top-level names to gpib.egg-info/top_level.txt
reading manifest file 'gpib.egg-info/SOURCES.txt'
writing manifest file 'gpib.egg-info/SOURCES.txt'
* Building wheel...
/tmp/build-env-fv8woooz/lib/python3.12/site-packages/setuptools/config/pyprojecttoml.py:72: _ExperimentalConfiguration: `[tool.setuptools.ext-modules]` in `pyproject.toml` is still *experimental* and likely to change in future releases.
  config = read_configuration(filepath, True, ignore_option_errors, dist)
running bdist_wheel
running build
running build_py
creating build/lib.linux-x86_64-cpython-312
copying Gpib.py -> build/lib.linux-x86_64-cpython-312
running egg_info
writing gpib.egg-info/PKG-INFO
writing dependency_links to gpib.egg-info/dependency_links.txt
writing top-level names to gpib.egg-info/top_level.txt
reading manifest file 'gpib.egg-info/SOURCES.txt'
writing manifest file 'gpib.egg-info/SOURCES.txt'
running build_ext
building 'gpib' extension
creating build/temp.linux-x86_64-cpython-312
x86_64-linux-gnu-gcc -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O2 -Wall -fPIC -I/tmp/build-env-fv8woooz/include -I/usr/include/python3.12 -c gpibinter.c -o build/temp.linux-x86_64-cpython-312/gpibinter.o
gpibinter.c:12:10: fatal error: gpib/ib.h: Datei oder Verzeichnis nicht gefunden
   12 | #include <gpib/ib.h>
      |          ^~~~~~~~~~~
compilation terminated.
error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1

ERROR Backend subprocess exited when trying to invoke build_wheel
make: [Makefile:496: build] Fehler 1 (ignoriert)

Is there a way I can point it to this header ? I searched far and wide and consulted the docs but I couldn’t find any way to do this.

This is my patch to switch to pyproject.toml:

commit 20e3bc6425960fc4d38eece1efd193a54e00dbd8
Author: Matthias Geiger <werdahias@debian.org>
Date:   Fri Aug 8 16:45:20 2025 +0200

    Switch build to pyproject.toml

diff --git a/linux-gpib-user/language/python/Makefile.am b/linux-gpib-user/language/python/Makefile.am
index 758f3aac..dbed010b 100644
--- a/linux-gpib-user/language/python/Makefile.am
+++ b/linux-gpib-user/language/python/Makefile.am
@@ -7,12 +7,12 @@
 #   the Free Software Foundation; either version 2 of the License, or
 #   (at your option) any later version.
 
-EXTRA_DIST = gpibtest.py setup.py Gpib.py gpibinter.c srq_board.py srq_device.py
+EXTRA_DIST = gpibtest.py Gpib.py gpibinter.c srq_board.py srq_device.py
 
 all-local: build
 
 build: gpibinter.c
-	-{ $(PYTHON) setup.py build && touch build; } || { $(RM) -r build; exit 1; }
+	-{ $(PYTHON) -m build && touch build; } || { $(RM) -r build; exit 1; }
 
 install-data-local:
 	-$(PYTHON) -m pip install . --prefix=$(DESTDIR)$(prefix) --root-user-action=ignore
diff --git a/linux-gpib-user/language/python/pyproject.toml b/linux-gpib-user/language/python/pyproject.toml
new file mode 100644
index 00000000..7dcd576b
--- /dev/null
+++ b/linux-gpib-user/language/python/pyproject.toml
@@ -0,0 +1,16 @@
+[build-system]
+requires = ["setuptools>=61"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "gpib"
+version = "1.0"
+description = "Linux GPIB Python Bindings"
+
+[tool.setuptools]
+py-modules = ["Gpib"]
+ext-modules = [
+{name = "gpib", sources = ["gpibinter.c"], include-dirs = ["lib"], py-limited-api = true}
+]
+
diff --git a/linux-gpib-user/language/python/setup.py b/linux-gpib-user/language/python/setup.py
deleted file mode 100644
index 7df5d175..00000000
--- a/linux-gpib-user/language/python/setup.py
+++ /dev/null
@@ -1,18 +0,0 @@
-#!/usr/bin/env python
-import sys
-if sys.version_info < (3,10):
-    from distutils.core import setup,Extension
-else:
-    from setuptools import setup, Extension
-setup(name="gpib",
-	version="1.0",
-	description="Linux GPIB Python Bindings",
-	py_modules = ['Gpib'],
-	ext_modules=[
-		Extension("gpib",
-		["gpibinter.c"],
-		include_dirs=["../../include"],
-		library_dirs = ['../../lib/.libs'],
-		libraries=["gpib", "pthread"]
-	)]
-)

Python version: 3.12

OS: Ubuntu 24.04

Worst case I’d just symlink something, but I’d rather have a cleaner solution.

2

Add the headers to a MANIFEST.in Controlling files in the distribution - setuptools 80.9.0 documentation

I’m not sure how well Hatch works with a C-extension, but as well as being included, the header needs to be unpacked at: gpib/ib.h, and hatch definitely lets you include files from anywhere, and gives more control over where they end up: Build configuration - Hatch

3

This is a fairly common problem when you have say a standalone C library as well as a Python package with bindings to the C library in the same repository. A relative lookup of a header (or any other build target) outside the Python package’s source tree is never portable - which you’re finding out here when building a wheel from an sdist, since the sdist doesn’t contain the header.

The question to answer is whether that C library with the header that’s going missing is (a) part of the Python package, or (b) an external dependency. If (a), can you move it to inside the Python package’s source tree? Symlinking is kind of a hack ot achieve that if you cannot reorganize your repo; setuptools may not fully support that, not 100% sure - you’ll have to try. If (b), then you have to make sure that the C library is installed and the headers on the search path whenever you build a wheel.

1 Like
4

but as well as being included, the header needs to be unpacked at: gpib/ib.h,

I have this as MANIFEST.in:

include ../../include/gpib/ib.h

and this as updated pyproject.toml:

[build-system] 
requires = [“setuptools”, “hatchling”] 
build-backend = “setuptools.build_meta”

[project]
name = “gpib”version = “1.0”
description = “Linux GPIB Python Bindings”

[tool.setuptools]
py-modules = [“Gpib”]ext-modules = [{name = “gpib”, sources = [“gpibinter.c”], py-limited-api = true}]

[tool.hatch.build.targets.wheel.force-include]
“../../include/gpib/ib.h” = “gpib/ib.h”

It still fails though:

{ /usr/bin/python3 -m build && touch build; } || { rm -f -r build; exit 1; }
* Creating venv isolated environment...
* Installing packages in isolated environment... (hatchling, setuptools)
* Getting build dependencies for sdist...
/tmp/build-env-j8ugjtwp/lib/python3.12/site-packages/setuptools/config/pyprojecttoml.py:72: _ExperimentalConfiguration: `[tool.setuptools.ext-modules]` in `pyproject.toml` is still *experimental* and likely to cha
nge in future releases.
  config = read_configuration(filepath, True, ignore_option_errors, dist)
running egg_info
writing gpib.egg-info/PKG-INFO
writing dependency_links to gpib.egg-info/dependency_links.txt
writing top-level names to gpib.egg-info/top_level.txt
reading manifest file 'gpib.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'gpib.egg-info/SOURCES.txt'
* Building sdist...
/tmp/build-env-j8ugjtwp/lib/python3.12/site-packages/setuptools/config/pyprojecttoml.py:72: _ExperimentalConfiguration: `[tool.setuptools.ext-modules]` in `pyproject.toml` is still *experimental* and likely to cha
nge in future releases.
  config = read_configuration(filepath, True, ignore_option_errors, dist)
running sdist
running egg_info
writing gpib.egg-info/PKG-INFO
writing dependency_links to gpib.egg-info/dependency_links.txt
writing top-level names to gpib.egg-info/top_level.txt
reading manifest file 'gpib.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'gpib.egg-info/SOURCES.txt'
running check
creating gpib-1.0
creating gpib-1.0/gpib.egg-info
copying files to gpib-1.0...
copying Gpib.py -> gpib-1.0
copying MANIFEST.in -> gpib-1.0
copying README -> gpib-1.0
copying gpibinter.c -> gpib-1.0
copying pyproject.toml -> gpib-1.0
copying ../../include/gpib/ib.h -> gpib-1.0/../../include/gpib
copying gpib.egg-info/PKG-INFO -> gpib-1.0/gpib.egg-info
copying gpib.egg-info/SOURCES.txt -> gpib-1.0/gpib.egg-info
copying gpib.egg-info/dependency_links.txt -> gpib-1.0/gpib.egg-info
copying gpib.egg-info/top_level.txt -> gpib-1.0/gpib.egg-info
copying gpib.egg-info/SOURCES.txt -> gpib-1.0/gpib.egg-info
Writing gpib-1.0/setup.cfg
Creating tar archive
removing 'gpib-1.0' (and everything under it)
* Building wheel from sdist
* Creating venv isolated environment...
* Installing packages in isolated environment... (hatchling, setuptools)
* Getting build dependencies for wheel...
/tmp/build-env-u5rcrfmf/lib/python3.12/site-packages/setuptools/config/pyprojecttoml.py:72: _ExperimentalConfiguration: `[tool.setuptools.ext-modules]` in `pyproject.toml` is still *experimental* and likely to cha
nge in future releases.
  config = read_configuration(filepath, True, ignore_option_errors, dist)
running egg_info
writing gpib.egg-info/PKG-INFO
writing dependency_links to gpib.egg-info/dependency_links.txt
writing top-level names to gpib.egg-info/top_level.txt
reading manifest file 'gpib.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '../../include/gpib/ib.h'
writing manifest file 'gpib.egg-info/SOURCES.txt'
* Building wheel...
/tmp/build-env-u5rcrfmf/lib/python3.12/site-packages/setuptools/config/pyprojecttoml.py:72: _ExperimentalConfiguration: `[tool.setuptools.ext-modules]` in `pyproject.toml` is still *experimental* and likely to cha
nge in future releases.
  config = read_configuration(filepath, True, ignore_option_errors, dist)
running bdist_wheel
running build
running build_py
creating build/lib.linux-x86_64-cpython-312
copying Gpib.py -> build/lib.linux-x86_64-cpython-312
running egg_info
writing gpib.egg-info/PKG-INFO
writing dependency_links to gpib.egg-info/dependency_links.txt
writing top-level names to gpib.egg-info/top_level.txt
reading manifest file 'gpib.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '../../include/gpib/ib.h'
writing manifest file 'gpib.egg-info/SOURCES.txt'
running build_ext
building 'gpib' extension
creating build/temp.linux-x86_64-cpython-312
x86_64-linux-gnu-gcc -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O2 -Wall -fPIC -I/tmp/build-env-u5rcrfmf/include -I/usr/include/python3.12 -c gpibinter.c -o build/temp.linux-x86_64-cpython-312/gpibinter.o
gpibinter.c:12:10: fatal error: gpib/ib.h: Datei oder Verzeichnis nicht gefunden
   12 | #include <gpib/ib.h>
      |          ^~~~~~~~~~~
compilation terminated.
error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1

ERROR Backend subprocess exited when trying to invoke build_wheel
make: [Makefile:496: build] Fehler 1 (ignoriert)
5

I don’t think using multiple build back ends is supported. The hatchling setting requires picking hatchling for everything and ditching setuptools. Setuptools could be doing other useful stuff for C extensions that hatchling doesn’t do by default

6

Hmm, I see. Looks like I need to use the hatch-cython hook then to build the extension

7

Hm, so I wasn’t abele to set the build up with hatch and hatch-cython; any other solution I can try ?

8

What is the answer to this question:

To put it another way, if a Linux distro was packaging this up would the C library be a separate distro package from the Python bindings?

Is the intention that someone would first have the C library installed and would then build/install the Python bindings to use that installed C library?

Does the sdist for the Python package contain the code for the C library?

There is (potentially) a difference here between how the setup works for development compared to how it works for the “end user” who installs these things from source packages but without directly accessing the git repo.

1 Like
9

The C library would not be separate from the python code. However, during build the C header is not available directly from a package. I intend to ship the C library as well as the python module from the same source tree.

The idea is to build the python module at the same time as the C library.

As workaround, I adjusted the makefile so the header is copied into the same directory as pyproject.toml; so it would be included in the sdist.

However, even with include gpib/* in MANIFEST.in the header is not found.

10

It’s just a linker error, this is a solved problem. It should just require having the build dependencies available, and configuring the build correctly.

If setuptools is too restrictive, have you looked at packaging using Anaconda, or scikit-build (and CMake) instead?

1 Like
11

Or is the issue just configuring setuptools using pyproject.toml? Using setup.py it looks like Extension.include_dirs need to be set?

12

Even with include-dirs set like this

[tool.setuptools]
py-modules = ["Gpib"]
ext-modules = [
{name = "gpib", sources = ["gpibinter.c" ], include-dirs = ["gpib"], library-dirs = ["gpib"], py-limited-api = true}
]

the header is not found. I know this should be possible to achieve with just setuptools.

13

I’m still a bit unclear how this software works. It sounds like what you are saying is something like:

  • This is a Linux only project and that someone would install it by running e.g. configure, make, make install
  • That would build both the C library and the Python bindings and install both system-wide.
  • There is no intention to interact with other Python packaging e.g. this won’t be uploaded to PyPI or installed with pip or installed into a venv and there is no need to e.g. distribute wheels or a Python-style sdist.

Is that correct?

Please understand that what you are doing here is not very typical of Python packages and so you should not assume that anyone else understands any of these things implicitly.

If what I said above is correct then I think maybe trying to use the build backend to compile the C code is just getting in the way. You can just add the relevant compiler commands to build the extension modules in the main Makefile.

2 Likes
14

Yes, sorry if that was unclear. This is a Linux-only build that should build both the C library and the bindings at the same time, and they likely won’t end up on PyPI.

Would I add a pyproject.toml in the toplevel directory then ? As far as I understand the problem here is pointing setuptools to the header; I think it expects them by default under /usr/share/include

15

I would only put the pyproject.toml at top level if the intention is that someone can do pip install . to install the whole thing as a Python package. I don’t think that is your intention though as it would not install the C library in the usual system-wide way.

Everything would be cleaner and simpler if you treat the C library as an external dependency that is expected to be installed first before building the Python bindings. Then the installation instructions would be:

  • Run configure/make/install.
  • Run pip install ./bindings/python if you want to install the Python bindings.

Then the header files will be in the relevant search path when building the bindings. This way you could also use python -m build --sdist to build an sdist that could be uploaded to PyPI so that someone could do pip install linux-gpib-user to install the bindings and you could also upload manylinux wheels that bundle the C library if you wanted. A Linux distro could use the sdist as the source package for the Python bindings as a separate distro package.

If you don’t want to separate the install steps then I suggest adding the compiler commands in your main Makefile. You can see the command that setuptools is running in your first post above:

x86_64-linux-gnu-gcc -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O2 -Wall -fPIC -I/tmp/build-env-fv8woooz/include -I/usr/include/python3.12 -c gpibinter.c -o build/temp.linux-x86_64-cpython-312/gpibinter.o

You just need to adapt this and the linker command to something that compiles the extension module and has the right include path to find your header file as well as Python’s header files. Assuming you generally have an in-place autotools build I would just have the commands that build the .so in place and then you can tell setuptools to bundle the .so file.

The advantage of letting setuptools figure out the compiler commands is that it has some understanding of what the compilers are supposed to be on different platforms (although newer build backends such as meson-python are much better at this). If you only care about Linux and CPython and gcc it is not hard to put together the compiler command yourself and then you can let the user configure it with ./configure --foo=bar as you would for the C library.

2 Likes
16

Thanks, then I’ll manually hack everything into the makefile. Separating the C headers and the python code is unlikely to happen.