Compile and include shared object file during pip install

Python Help
,
1

Hi All,

I am currently working on a small project which requires the use of a shared object file for access to C functions through the use of the ctypes package. I have a created a smaller test case package for this discussion. My goal is to have the shared object file included with the other package files after a pip install . I find that a single pip install . will compile the shared object file, but will not include them at the site-packages location. A subsequent call to pip install . will however.

The package layout is:

top
├── c_files
│   └── test.c
├── compile.sh
├── MANIFEST.in
├── mypkg
│   ├── __init__.py
│   └── mod1.py
└── setup.py

The contents of the setup.py file are:

import os
import subprocess
from setuptools import setup, Extension
from setuptools.command.install import install


class CustomInstall(install):
    def run(self):
        with open("install.log", "w") as f:
            subprocess.run(["./compile.sh"], stdout=f)
        install.run(self)

setup(
    name="mypkg",
    packages=['mypkg'],
    package_dir={'': '.'},
    package_data={'': ['libc.so']},
    cmdclass={"install": CustomInstall},
    include_package_data=True,
    zip_safe=False,
)

and the MANIFEST.in file are:

global-include mypkg/libc.*

The compile.sh contains:

#!/bin/bash

c_dir=$PWD/c_files
py_dir=$PWD/mypkg
cd $c_dir
gcc -fPIC -c test.c -o libc.o
gcc -shared libc.o -o libc.so
cp libc.so $py_dir

I will gladly provide the contents of any other files if needed for context. I assume I am missing something that might be obvious, any and all help is greatly appreciated.

2

The setuptools maintainers would know for sure, but I believe that what’s happening is that you’re doing steps at install time (when the package is installed) which you actually want to happen at build time (when the package is built).

The flow for pip install . is roughly

  • setup build environment (in your case, get setuptools)
  • run the build tool (again, setuptools) to populate dists/
  • install from the built artifact in dists/

The separation between those last two steps is what’s tripping you up, if I’ve spotted the issue accurately.
(Thanks for the small example, BTW.)

I’m going to move this thread to Python Help and apply the packaging tag. This part of the forums is primarily for discussion of the packaging ecosystem itself (e.g., specs), but it’s hard to know that a priori.

1 Like
3

Hi @sirosen thank you so much for looking into this! I think you are correct. After some more searching (and some help from a LLM) I found that I need to customize the build_py command instead. (For posterity) The updated setup.py would now be:

import subprocess
from setuptools import setup
from setuptools.command.build_py import build_py


class CustomBuild_Py(build_py):
    def run(self):
        with open("install.log", "w") as f:
            subprocess.run(["./compile.sh"], stdout=f)
        build_py.run(self)

setup(
    name="mypkg",
    packages=[“mypkg”],
    cmdclass={"build_py": CustomBuild_Py},
    include_package_data=True,
    zip_safe=False,
)
1 Like