Can I use the same setup.py in checked-out repo for developing and installing a non-cythonized package, as well as building a cythonized package?

Packaging
1

Given a company-internal git repository for a pure Python3 application (multiple packages inside single root package, fwiw) is it possible to support the following development/deployment usecases with a single unified setup.py? Even if not, I’m thankful for constructive directions as to how to handle such use cases in a sane manner; please bear in mind that I’m new to this forum and I’m not a terribly experienced Python aficionado.

Note bene: this is a “pure” Python3 source code base without any external C/C++ modules; cythonization is only used for container deployment to “obfuscate” the sources.

  1. in-place development: pip3 install -e . inside the repository base; for working on the source code.

  2. source install: pip3 install . inside the repository base; to install the unobfuscated sources into the system/venv, no cythonization, no “external modules”.

  3. cythonized install: install only the obfuscated cythonized files into the system/venv, but not any sources.

  4. build binary wheel: python3 setup.py bdist_wheel, then install the wheel and use only the installed binary files in the final container image.

So far, I’ve managed to get use cases 1. and 4. working, where 4. is basically done via a custom build_py class.

# https://stackoverflow.com/a/56043918
from setuptools.command.build_py import build_py as build_py_orig
try:
    from Cython.Build import cythonize
except:
    cythonize = None

# https://stackoverflow.com/a/56043918
extensions = [
    Extension('spam.*', ['spam/**/*.py'],
              extra_compile_args=["-O3", "-Wall"]),
]

cython_excludes = ['spam/**/__init__.py']

def not_cythonized(tup):
    (package, module, filepath) = tup
    return any(
        fnmatch.fnmatchcase(filepath, pat=pattern) for pattern in cython_excludes
    ) or not any(
        fnmatch.fnmatchcase(filepath, pat=pattern)
        for ext in extensions
        for pattern in ext.sources
    )

class build_py(build_py_orig):
    def find_modules(self):
        modules = super().find_modules()
        return list(filter(not_cythonized, modules))

    def find_package_modules(self, package, package_dir):
        modules = super().find_package_modules(package, package_dir)
        return list(filter(not_cythonized, modules))

setup(
    name='spam',
    packages=find_packages(),
    ext_modules=cythonize(
        extensions,
        exclude=cython_excludes,
        compiler_directives={
            "language_level": 3,
            "always_allow_keywords": True,
        },
        build_dir="build",  # needs to be explicitly set, otherwise pollutes package sources
    ) if cythonize is not None else [],
    cmdclass={
        'build_py': build_py,
    },
    include_package_data=True,
    install_requires=[...]
)

How can I support also use case 2 and 3? I would suspect to use a custom install class but then I’m stuck: how do I prevent setuptools’ install class from trying to build a source wheel which then always includes both the source as well as the cythonized shared libraries? I’m fine if there is only use case 2 possible at a time and then document that otherwise for use case 3 two steps are required: python3 setup.py bdist_wheel && pip3 install spam-*.whl.

How to properly deal with these multiple use cases? Admittedly, this is most probably not a much-needed situation, but specific to sometimes obfuscated private python projects. Any help, remedies?

2

Can you also share the error you are seeing for pip install . as that should work if building your wheels is working.

But installing your built wheels is a totally reasonable thing to do.

3

I don’t get errors, building and installing the wheel works, as well as a development install (as far as I understand this goes via the unchanged develop class). What I would like to add now are use cases 3 (and maybe 4). I added my own install class:

from setuptools.command.install import install as install_orig

# ...

class install(install_orig):
    def finalize_options(self):
        super().finalize_options()
        self.distribution.ext_modules = None

setup(
   # ...
    cmdclass={
        'build_py': build_py,
        'install': install,
    },
   # ...
)

Now when I execute pip3 install . I end up with both the sources and the cythonized shared libs getting installed. From the logging it seems that the original setuptools.command.install I inherit from now triggers a source wheel build (I think that’s in its run method). I tried to disable at least the cythonized .so’s by setting self.distibution.ext_modules=None but that doesn’t seem to be correct.

Can I tell the original install class to not build a source distribution wheel on pip3 install . as before when I hadn’t added my own derived install class?

4

I have no idea. Are you using PEP 517 at all? Otherwise this is between you and setuptools. :wink:

5

pip doesn’t call setup.py install in any code paths except for one (and that prints a warning that it’ll be removed in the near future). If you want use case 2 to work, you’ll want to make all the customisations in build_* commands.