I’ve been talking to the AstroPy project about removal of setup.py.
I think several of us have the (perhaps mistaken) impression that we should aim for a static pyproject.toml build declaration, and eventually get rid of setup.py. However, most scientific Python projects have a large number of extensions, so incremental builds are essential.
We currently do this using: python setup.py build_ext -i. But, the setuptools docs
states:
Changed in version 21.3: The project being installed is no longer copied to a temporary directory before invoking the build system, by default.
You will have to run the pip install --no-build-isolation --editable . command every time the source code of a Cython file is updated (ending in .pyx or .pxd). Use the --no-build-isolation flag to avoid compiling the whole project each time, only the files you have modified.
On my system, the above command does not do incremental builds.
So, my two questions:
(1) Should removing setup.py be a goal, in favor of a static definition in pyproject.toml?
(2) What is the correct way of invoking an incremental build without a setup.py file?
setuptools isn’t very well suited to deal with large native projects like yours IMO. You might want to consider moving to something like meson-python or scikit-build. They were created specifically for those kinds of projects.
Hi @FFY00! Personally, I support that strategy (you’ll note that we’ve already ported skimage, numpy, and scipy). Meson is fantastic, and with it incremental, parallel builds are a dream. That said, I don’t have control over what sklearn, astropy, and other projects use, but I am trying to come up with best practices to recommend to the scientific Python community.
setup.py is going nowhere, you need it today (at least if you have C code, like for the projects you refer to) and you will still need it next year and the year after as long as you keep using setuptools. There is no replacement.
For (1), the message from the setuptools maintainers has been to stop using the UX of python setup.py ..., and prefer pip & co for build/install commands. That’s a very different thing than removing setup.py altogether - that’s just not a thing.
For (2), I’m seeing the same behavior as you do with pip 23.0.1 and setuptools 67.5.1 (= most recent versions). pip is still, or again, creating a tmpdir:
Created temporary directory: /tmp/pip-unpack-dvo87xii
Building wheels for collected packages: scikit-learn
Created temporary directory: /tmp/pip-wheel-83tqj9l7
Given the pip docs you linked to, which say: “Changed in version 21.3: The project being installed is no longer copied to a temporary directory before invoking the build system, by default.” this looks like a bug in pip. I can’t find an open one for this.
With regard to (1), the recommendation is to define everything you can define statically in pyproject.toml. But for metadata that you need to define at build time, you should use whatever mechanism your backend provides for that. And setup.py is the mechanism in setuptools. For that purpose, as code that gets invoked at build time, it’s not going anywhere.
What setuptools has done, though, is deprecate the use of setup.py to run build commands (e.g. setup.py build_ext), so there is no longer a way of invoking setuptools other than via a build frontend like pip or build. As a consequence, there is no way of “invoking an incremental build” because there is no standard for incremental builds, and setuptools hasn’t (to my knowledge) implemented a tool-specific way via the standard config_settings mechanism.
Anything more than this is something you need to discuss with the setuptools project directly.
There’s no need for this. Why doesn’t pip do exactly what it says it will do in its docs (“… no longer copied to a temporary directory” … “When provided with a project that’s in a local directory, pip will invoke the build system “in place”)? That automatically gives you an incremental build with every single real-world build backend.
Ah, sorry, I’d missed the comment above suggesting this wasn’t working properly. Can someone create a pip issue for this? Ideally, with an easier reproducer than building scikit-build, as it’s unlikely many of the pip developers will be able to do that (I assume building scikit-build needs more than just a C compiler , which is all I have, for example).
I was assuming that setuptools needed some special instructions to invoke an incremental build, hence my confusion.
We’re doing what is documented: running the build backend in-tree. setuptools however doesn’t perform incremental builds, and instead does a build from scratch. I’m not sure how/why this worked earlier and someone providing a reproducer of this working with an older version of setuptools would be helpful.
You can check pip’s behaviour by asserting the pwd by hard-coding it within a setup.py.
The issue here is that setuptools isn’t doing an incremental build, and that there’s no way to tell it to do so via pip.
When you remove -e from the install command, then indeed we do see an in-tree build. There is a difference in behavior here between editable and regular installs. I’m not sure if that’s intentional or not - but I’d think not. pip uses the --dist-dir in the same way, so I guess indeed it’s setuptools that fails to create an in-tree build/ directory.
Without the -e, an incremental build is indeed obtained:
$ time pip install --no-build-isolation . # clean build
...
real 3m41,443s
user 3m37,409s
sys 0m8,696s
$ time pip install --no-build-isolation . # second build
real 0m5,846s
user 0m6,811s
sys 0m3,208s
That said, that incremental build still has quite a bit of overhead; >80% of the runtime is pip rather than setuptools (not strange, pip has to do metadata related things, while setuptools can simply check timestamps of source files and build targets, and then do nothing):
real 0m0,931s
user 0m1,637s
sys 0m1,413s
This matters for @stefanv’s use case, because when rerunning the build each time one performs another development step (tests, docs, etc.), 5 seconds is still a lot. So for that use case, ideally setuptools gets fixed to do the editable build in-tree, and then the next invocations should still be python setup.py build_ext -i.
I am afraid that so far we still don’t have a good plan for how we are going to replace this functionality.
Maybe the idea with config_settings is enough, but if we have to regenerate a wheel file that might be already cause some overhead…
I would recommend opening an issue or at least a discussion in the setuptools repository, specially if anyone has an implementation plan in mind (that is not too high level).
I can also investigate better if there is a problem with the editable installs (I would also appreciate if anyone can create an issue with the reproducer ).