Setuptools: removing setup.py, keeping incremental builds?

Packaging
1

Hi all,

I’ve been talking to the AstroPy project about removal of setup.py.

I think several of us have the (perhaps mistaken) impression that we should aim for a static pyproject.toml build declaration, and eventually get rid of setup.py. However, most scientific Python projects have a large number of extensions, so incremental builds are essential.

We currently do this using: python setup.py build_ext -i. But, the setuptools docs
states:

Changed in version 21.3: The project being installed is no longer copied to a temporary directory before invoking the build system, by default.

This gives the impression that:

pip install --no-build-isolation --editable .

will do an incremental build. In fact, the scikit-learn documentation makes that claim explicitly:

You will have to run the pip install --no-build-isolation --editable . command every time the source code of a Cython file is updated (ending in .pyx or .pxd). Use the --no-build-isolation flag to avoid compiling the whole project each time, only the files you have modified.

On my system, the above command does not do incremental builds.

So, my two questions:

(1) Should removing setup.py be a goal, in favor of a static definition in pyproject.toml?
(2) What is the correct way of invoking an incremental build without a setup.py file?

3 Likes
2

For the record, I’m not sure if it’s expected to be supported long term but you can invoke setuptools commands without setup.py by doing e.g.:

python -c "from setuptools import setup; setup()" build_ext -i
1 Like
3

This works, but isn’t particularly friendly to type on the CLI and isn’t exactly a particularly great UX either. :slight_smile:

1 Like
4

Thanks! That’s definitely something I can wrap as a command in devpy. Perhaps python -m setuptools would make more sense from a UX perspective.

1 Like
5

(disclaimer: I am a maintainer of meson-python)

setuptools isn’t very well suited to deal with large native projects like yours IMO. You might want to consider moving to something like meson-python or scikit-build. They were created specifically for those kinds of projects.

3 Likes
6

Hi @FFY00! Personally, I support that strategy (you’ll note that we’ve already ported skimage, numpy, and scipy). Meson is fantastic, and with it incremental, parallel builds are a dream. That said, I don’t have control over what sklearn, astropy, and other projects use, but I am trying to come up with best practices to recommend to the scientific Python community.

3 Likes
7

I’d greatly appreciate any sense of where setuptools packaging is going with setup.py. An authoritative answer would be even better!

8

Or, indeed, scikit-build-core, though @henryiii can weigh in on whether it’s ready for something like astropy to start depending upon it just yet.

1 Like
9

Maybe we can reserve this thread for the issue in question, though.

10

setup.py is going nowhere, you need it today (at least if you have C code, like for the projects you refer to) and you will still need it next year and the year after as long as you keep using setuptools. There is no replacement.

For (1), the message from the setuptools maintainers has been to stop using the UX of python setup.py ..., and prefer pip & co for build/install commands. That’s a very different thing than removing setup.py altogether - that’s just not a thing.

For (2), I’m seeing the same behavior as you do with pip 23.0.1 and setuptools 67.5.1 (= most recent versions). pip is still, or again, creating a tmpdir:

Created temporary directory: /tmp/pip-unpack-dvo87xii
Building wheels for collected packages: scikit-learn
  Created temporary directory: /tmp/pip-wheel-83tqj9l7

Given the pip docs you linked to, which say: “Changed in version 21.3: The project being installed is no longer copied to a temporary directory before invoking the build system, by default.” this looks like a bug in pip. I can’t find an open one for this.

2 Likes
11

Sure! I wanted to add to @FFY00 comments, but I can appreciate that the threading model of Discourse is not as amenable to isolated conversations.

3 Likes
12

Maybe related: Proposal: Adding a persistent cache directory to PEP 517 hooks

13

With regard to (1), the recommendation is to define everything you can define statically in pyproject.toml. But for metadata that you need to define at build time, you should use whatever mechanism your backend provides for that. And setup.py is the mechanism in setuptools. For that purpose, as code that gets invoked at build time, it’s not going anywhere.

What setuptools has done, though, is deprecate the use of setup.py to run build commands (e.g. setup.py build_ext), so there is no longer a way of invoking setuptools other than via a build frontend like pip or build. As a consequence, there is no way of “invoking an incremental build” because there is no standard for incremental builds, and setuptools hasn’t (to my knowledge) implemented a tool-specific way via the standard config_settings mechanism.

Anything more than this is something you need to discuss with the setuptools project directly.

14

There’s no need for this. Why doesn’t pip do exactly what it says it will do in its docs (“… no longer copied to a temporary directory” … “When provided with a project that’s in a local directory, pip will invoke the build system “in place”)? That automatically gives you an incremental build with every single real-world build backend.

1 Like
15

Ah, sorry, I’d missed the comment above suggesting this wasn’t working properly. Can someone create a pip issue for this? Ideally, with an easier reproducer than building scikit-build, as it’s unlikely many of the pip developers will be able to do that (I assume building scikit-build needs more than just a C compiler :wink:, which is all I have, for example).

I was assuming that setuptools needed some special instructions to invoke an incremental build, hence my confusion.

1 Like
16

I don’t think this is related to pip.

We’re doing what is documented: running the build backend in-tree. setuptools however doesn’t perform incremental builds, and instead does a build from scratch. I’m not sure how/why this worked earlier and someone providing a reproducer of this working with an older version of setuptools would be helpful.

You can check pip’s behaviour by asserting the pwd by hard-coding it within a setup.py.

The issue here is that setuptools isn’t doing an incremental build, and that there’s no way to tell it to do so via pip.

17

It’s a mix - pip is creating tmpdirs and then asking setuptools to write into them:

  • metadata: ['setup.py', 'dist_info', '--output-dir', '/tmp/pip-modern-metadata-op6f8gvu', '--keep-egg-info']
  • regular wheel build command: ['setup.py', 'bdist_wheel', '--dist-dir', '/tmp/pip-wheel-1fwwm560/.tmp-62rf9fm0']
  • editable install command: ['setup.py', 'editable_wheel', '--dist-dir', '/tmp/pip-wheel-axqh1178/.tmp-wepjt068']

When you remove -e from the install command, then indeed we do see an in-tree build. There is a difference in behavior here between editable and regular installs. I’m not sure if that’s intentional or not - but I’d think not. pip uses the --dist-dir in the same way, so I guess indeed it’s setuptools that fails to create an in-tree build/ directory.

Without the -e, an incremental build is indeed obtained:

$ time pip install --no-build-isolation .  # clean build
...
real    3m41,443s
user    3m37,409s
sys     0m8,696s

$ time pip install --no-build-isolation .  # second build
real    0m5,846s
user    0m6,811s
sys     0m3,208s

That said, that incremental build still has quite a bit of overhead; >80% of the runtime is pip rather than setuptools (not strange, pip has to do metadata related things, while setuptools can simply check timestamps of source files and build targets, and then do nothing):

real    0m0,931s
user    0m1,637s
sys     0m1,413s

This matters for @stefanv’s use case, because when rerunning the build each time one performs another development step (tests, docs, etc.), 5 seconds is still a lot. So for that use case, ideally setuptools gets fixed to do the editable build in-tree, and then the next invocations should still be python setup.py build_ext -i.

1 Like
18

Because the OP used build_ext -i, the flag explicitly asks setuptools to do an incremental build instead of from scratch.

So it seems config_settings is the right thing for this, if setuptools provides a way to pass that flag via the mechanism.

1 Like
19

Calling over @abravalheri , the resident Setuptools maintainer and expert who’s most active on here currently.

20

I am afraid that so far we still don’t have a good plan for how we are going to replace this functionality.

Maybe the idea with config_settings is enough, but if we have to regenerate a wheel file that might be already cause some overhead…

I would recommend opening an issue or at least a discussion in the setuptools repository, specially if anyone has an implementation plan in mind (that is not too high level).

I can also investigate better if there is a problem with the editable installs (I would also appreciate if anyone can create an issue with the reproducer :sweat_smile:).