Custom build steps / Moving Bokeh off setup.py

Packaging
17

I just wanted to circle back on this and give an update for anyone who might be interested in the details, since we did manage to get to a happy place eventually:

A few other miscellaneous cleanups were part of that PR, here are the relevant bits related to this discussion:

  • Conda 3.22 was just released, adding load_file_data function that can read toml files, so we can template the conda recipe dependencies directly from pyproject.toml. The conda recipe also now just builds from the wheel, which is simpler.

    Combined with setuptools support for pyproject.toml this means that all of our runtime dependencies can finally be specified in one place. We even have a Sphinx extension for including dependencies list in the docs.

  • We got rid of versioneer and all the vendored files it adds. We switched to using setuptools-git-versioning instead. We did look at setuptools-scm but could not make its automagical dev version behavior work for us.

  • We updated our setup.py to control the JS build via a BOKEHJS_ACTION environment variable in stead of command line args. This is important to be able to have single source of truth for BokehJS across all package types.

  • Lots old code could be deleted from setup.py, so we rolled all our previous helper functions in a _setup_support.py directly back into setup.py and then made a custom build sub-command to wrap it up, now that it is possible to customize build directly.

    Even pulling everything into one file, the setup.py is still <150 LOC (including the colorama reporting code I can’t give up :smile:)

  • Since the only thing left in setup.cfg was Flake8 configuration, we moved the file to .flake8 which is hidden at least. Maybe some day Flake8 will add pyproject.toml support but I’m not too hopeful since they seem pretty aggressively against it. Still the top level of the repo is much cleaner now!

Putting things together, our dev workflow looks like:

Non-editable local install

# Build and use fresh BokehJS
pip install . 

# Use existing (already built) BokehJS
BOKEHJS_ACTION="install" pip install .

Editable install (editable for Python modules only)

# Build and use fresh BokehJS 
pip install -e . 

# Use existing (already built) BokehJS 
BOKEHJS_ACTION="install" pip install -e .

My only very tiny complaint is that in order to see our nice BokehJS build report output without being buried in the voluminous output from setuptools, we have to invoke like:

BOKEHJS_ACTION="install" pip -v install . --config-settings quiet=true

I do hope maybe this could be improved in the future (e.g. via some logging API we could hook into instead of just using print)

For releases, our automation does something more or less like this:

cd bokehjs
node make build
cd ..

# use the single already-built BokehJS for both
BOKEHJS_ACTION=install python -m build -s -w .

# conda-build using the wheel with the already-built BokehJS
VERSION="3.0.0" conda build conda.recipe  --no-test --output-folder /tmp

which satisfies our main critical requirement. If I could improve one thing here, it would be to find a way to not need to coordinate the VERSION to conda build explicitly, but so far I have not found a way.

All in all, it was a net deletion of ~2700 LOC which is always a good feeling! Many thanks to @pradyunsg and @abravalheri for their help and guidance about the custom build command.

8 Likes