Custom build steps / Moving Bokeh off setup.py

Packaging
15

First off, to make this as useful for @bryevdv quickly… Broadly, I’m suggesting changing your release build process from:

npm make build
python setup.py sdist --install-js
python setup.py bdist_wheel --install-js

To:

npm make build
BOKEH_COPY_LOCALLY_BUILT_BOKEHJS=1 python -m build

My concrete suggestions are:

  • Don’t try to remove setup.py for now. The blog post you’d linked to as motivation is literally titled “Why you shouldn’t invoke setup.py directly” and not “Why you should get rid of setup.py from your project”. There’s a good reason for the specific wording there.

  • Stop invoking python setup.py ... and instead use python -m build/pip directly.

  • Use an environment variable instead of the --install-js flag. When the environment variable is set and BokehJS is not built locally in the relevant location, error out. If it isn’t set, you can keep the existing behaviour of invoking npm make build.

    The build-system tooling for Python has build configuration mechanisms, but you don’t need them for your usecase (as far as I can tell) – you can move the responsibility of passing this configuration “boolean” to the OS, instead of the Python packaging tooling.

  • There are alternatives to setuptools available but Bokeh doesn’t need them – they can provide a developer experience improvement but switching to them is not a requirement and can bring its own “growing” pains + migration costs.

As for improvements you could make to your build system, I have a one suggestion: Move the logic that invokes npm make build in setup.py and performs the copy of the built JS, into a build_py subclass and override the default build_py class with it (using setuptools.setup's cmdclass argument) – see “Extending the build through an override” below for details.

There’s a few things in the discussion already, so I’m gonna try and group them:

  • Moving off of setup.py

    Realistically, setup.py is not going away as a way to configure Python package builds. It has been here for more than a decade, and will be around for likely longer. OTOH, it gives every user a Turing-complete mechanism to describe every possible key-value pair, which is far from ideal.

    That said, we do want people to stop doing setup.py install and setup.py sdist bdist_wheel and move to pip install . and python -m build – they do a few more things to ensure that builds happen correctly and are better solutions in terms of interoperability and available maintainance bandwidth. See also blog post noted above.

    Personally, I’d like package authors to describe as much of their metadata statically as feasible, in files that don’t need to be executed with a Turing-complete thing to parse and for dependency resolution mechanisms for Python to be able to get this information cheaply.[1]

    Today, this information can be specified statically in the [project] table in pyproject.toml (which is backed by a interoperability standard) and setup.cfg (which is implementation-defined, as are most legacy things in Python Packaging) but neither is used during dependency resolution today. There’s some tooling advantages, eg: it’s easier to parse/modify those files than a setup.py file using an automated tool.

  • Adding a custom build step to setuptools

    1. Moving the build-logic into a dedicated project

      A demonstration for doing this is available in setuptools’ issue tracker, written by one of the maintainers: Support for custom build steps · Issue #2591 · pypa/setuptools · GitHub.

    2. Extending the build through an override (this is what I recommended above)

      You can extend an existing build_py command in setuptools, using cmdclass and do additional build work in there. This has the advantage of being an intended point of extension for the setuptools build system and eliminates the need to look at sys.argv at any point. :slight_smile:

      I recently did something like this in Memray for an example of that (full disclosure: that’s an OSS project from work). That project builds JS assets using an npm run-script build command – it extends build_ext, you can extend build_py since you don’t have extension code. That project has C++, JS and Python build systems and was a fun one to get building correctly.

  • Changing to an alternative build-backend

    As noted by a bunch of folks already, there’s a lot of alternatives available for setuptools today. None of them were popular late last year, except for Poetry which does not have the extensibility you need anyway (AFAIK).

    In my opinion, what you’re seeing is well-meaning enthusiasm (and skepticism) from the folks around here, about the new build-backends in Python’s packaging ecosystem. In broad strokes, it took a lot of effort to get to this point and folks prefer the newer build-backends over setuptools for both “simple” and “complex” use cases; since they’re being built without the backwards compatibility constraints of setuptools and are able to innovate + improve various aspects of the developer experience.

    I’m not familar with any of the ones relevant for this discussion as a regular user though – so no real suggestions on that front. Mostly just wanted to provide context for why alternatives to setuptools are being enthusiastically mentioned. :slight_smile:

  1. I know a bunch of other folks want this too but I don’t wanna speak for anyone else. ↩︎

6 Likes