Adding the old `distutils.ccompiler` to the stdlib

Distutils has been gone for a little while now, but it did come with a handy platform-independent interface for working with a C compiler that could be used for things unrelated to packaging.

Since distutils is gone in recent versions, I’ve found myself adding setuptools as a dependency, and using their vendored setuptools._distutils.ccompiler as a replacement. The distutils.ccompiler module used to be in the standard library anyway, so why not just add it back, independent from Python packaging?

It would work the same way it did in distutils, and require minimal, if any, changes to add it back to the standard library:

import ccompiler

compiler = ccompiler.new_compiler()
compiler.compile(["./whatever.c", "./another.c", ...])
1 Like

Maybe someone will have a compelling reason not to, but I like the idea of build backends having a common interface to C compilers.

On the other hand, maybe a better design for such is possible. We’re talking about a pretty old attempt, after all.

Sure, maybe some things could be added. Personally, I haven’t had any complaints with the distutils C compiler approach (in fact, I quite like it!), so rewriting it seems like a lot of extra work.

Why does it need to be in the stdlib? If it’s purpose is unrelated to packaging and it’s just a way to interface with C compilers, it seems like it could evolve into a new package on PyPI.

1 Like

It is already on PyPI, under a vendor in setuptools. I just find it a bit inconvenient that you now have to add an external package (most of which, you won’t even need in your project if all you’re after is the C compiler) for something that used to be available in the standard library. Has there ever been a standard library module that’s been removed, and added to PyPI under a separate maintainer?

Everything that was removed in PEP 594 is up on PyPI with the standard- prefix. I don’t know that any of it is being actively developed, but it’s there.

There isn’t any standard-distutils package on PyPI, but that’s beside the point, since you can steal it from setuptools anyway. Nonetheless, bringing the entirety of setuptools into your project solely for the C compiler doesn’t sound great to me.

Technically speaking, someone could fork the C compiler implementation and try to maintain it themselves – but distutils.ccompiler is long and complicated. CPython maintained it for years – why not again?

Because it’s long and complicated? You can’t have it both ways - either it’s easy to maintain, so someone can maintain it on PyPI, or it’s hard to maintain so why would the core devs have enough time to do so?

3 Likes

I’m assuming that one of the core devs knows how the implementation works, so easy to maintain for them. Based on the distutils repository, the C compiler implementation has very few changes yearly, but from the looks of it, changes like those would still be difficult for a third party to try and implement.

Okay, now assume that this isn’t true. Or simply that this core dev doesn’t want to do it anymore.

1 Like

Even if it’s a great little feature for those who already have a C compiler installed, the vast majority of Python users on Windows probably don’t have a C compiler installed, let alone glibc or musl etc. This new addition to the stdlib would be broken right out of the box for them.

At the very least it would require additional setup. However PyPi and pip are ideal tools to use for this. So what’s the advantage of taking this away from Pypi?

Well, distutils was deprecated only a few years ago, so I would be quite surprised if there was nobody around that knew how it worked. I’m not at all trying to put extra stress on the core devs, but to me distutils.ccompiler seems pretty low maintenance, and I think it makes more sense for it to exist with the same people who wrote it as opposed to some third party. However, if there’s nobody around to maintain it, then this idea is no good.

Well, someone has to be maintaining it for setuptools anyway – I was just under the assumption that it was a core developer who was doing that. Correct me if I’m wrong, though.

I don’t think that not having a C compiler installed counts as being “broken”. This still applied to it when distutils was in the standard library. AFAIK, it just throws an error saying that there is no compiler available.

The main problem I mentioned before was that in order to use it now, you need to include setuptools as a dependency for your project, which is a bit overkill if you aren’t building packages. Note that even here, you’re using a private API – setuptools._distutils isn’t guaranteed to always be there. I doubt they plan on removing it anytime soon, but it’s worth noting.

I mean, I don’t know what the current status is. But this stuff was removed from the stdlib on purpose, to reduce maintenance. As you’ve noted, it’s a complicated piece of code. I don’t really understand the “well I think it might be easy for [unknown dev] to maintain this for me” argument. You don’t know and also it’s their decision?

I didn’t realize that distutils was removed for maintenance reasons – I thought it was just due to setuptools being the recommended alternative and because of PEP 517. If that’s the case, then yeah, scrap this idea (although, I don’t see much about maintenance in PEP 632).

I’ll see if I can reach out to whoever is in charge of maintaining distutils.ccompiler currently, and ask about turning it into a library instead (or, if they’re a core developer, then adding it back to the standard library I guess?)

To be fair, I’m not really asking for “[unknown dev]” to maintain anything for me. I just thought this would be a nice feature to have again, considering it was once in the standard library. I’m not totally sure the extent of what has to be changed yearly, so I think arguing about whether or not it’s possible to maintain it is a bit pointless right now. For all I know, you very well could get away with just uploading a copy of distutils.ccompiler with no maintenance whatsoever to PyPI and calling it a day, but then we’ve just gone full circle.

How about you go ahead and upload only ccompiler as a standalone PyPl Package then? The license surely allows you to do this.

1 Like

I know next to nothing about the C compiler implementation – why would it be a good idea for me to maintain it?

Because you want it. You don’t need to actively maintain it - you can just regularly copy out whatever changes happen from within the setuptools version.

My primary point is that if you big complained is that you can’t install it independently, then that is a very much fixable problem.

2 Likes

So, make a library that vendors an already vendored version of distutils. I guess it works, but it seems more like a band-aid rather than a solution.

I’ll confirm for you then, no-one (at least, no-one who was willing to continue being a maintainer for it) really knew how it worked by this point. It just worked, and we hoped we wouldn’t have to do much to it.

Also, the lack of changes is misleading, if you’re trying to use it as a measure of maintenance effort required. To an extent, the lack of change reflects the lack of anyone comfortable with making non-trivial changes, and hides the work needed to manage users’ expectations and respond to issue tracker items that don’t get addressed as a result.

… and that’s ultimately the point here. Maybe that is enough, but someone needs to volunteer to do it. The core devs are no longer willing to, and I get the feeling you’re not volunteering. And whoever does volunteer is taking a risk by doing so - what if it isn’t as simple as you hope it will be, and there is a lot of maintenance required? (Even simply responding to feature requests and bug reports with “sorry, I won’t be adding/fixing that” can be a burden - saying “no” isn’t always easy). The code’s available if someone wants to, though.

3 Likes

Before we start recommending the standard- packages, are they vetted?

The xz situation is still fresh on the mind. Don’t want to wake up one day to backdoors hidden in the wheels.