Need help with resolving has_function issue in distutils/setuptools

Packaging
1

We are trying to transition the Fedora distribution to a system compiler that does not support implicit function declarations. This is based on the observation that treating implicit declarations as warnings (the current default) quite regularly wastes programmer times if the warnings get overlooked.

I still hope that this change can land in GCC 14 upstream, so that even more C programmers benefit. Clang 16 is also going to make similar changes. They plan to proceed even if distributions and upstream projects are not fully ready because Clang is typically not used as a distribution system compiler.

I have identified one blocker that affects a few packages: The has_function check in the distutils/setuptools universe is conceptually broken, and has always been. I have submitted a PR to fix this as far as possible:

But the PR isn’t make any progress. I understand that this may be obsolete/deprecated technology, but we really need the fix upstream so that distributions can backport it and transition to newer system compilers.

I would appreciate guidance how to proceed with this. Thanks.

1 Like
2

I’d for sure propose the same change to setuptools. Side note in support: I work on SCons, which has its own Autoconf-ish system in Python, and the check-for-function logic had to make essentially this change a while ago, it now builds up a stub that looks like this:

#include <assert.h>
  
#ifdef __cplusplus
extern "C"
#endif
char printf(void);
  
#if _MSC_VER && !__INTEL_COMPILER
    #pragma function(printf)
#endif
  
int main(void) {
#if defined (__stub_printf) || defined (__stub___printf)
  fail fail fail
#else
  printf();
#endif
  
  return 0;
}
3

Thank you for your comments. I got the impression that I need to get this fixed in distutils before it will be merged in setuptools. Quoting Anderson Bravalheri:

Would you like to submit the PR to pypa/distutils so the distutils maintainers can have a look?

Alternatively you can try to implement the changes without touching the _distutils folder (I don’t know if it is possible…)

Thank you for fixing SCons. Your approach looks okay as well (although I would probably use #error). <assert.h> includes <features.h> on glibc, so it gets the __stub_ defines. But I didn’t want to make it more complicated than absolutely necessary.

1 Like
4

Yeah, setuptools maintain their distutils fork in that repo, so it’s the place to start. Hopefully Anderson or Jason will get to it.

If you’re planning to backport to the old standard libraries, you’ll likely have to just do that yourself. Nothing from the PyPA repo gets backported to the CPython repo, and we won’t take it into the standard library at this stage (keeping up with compiler changes is literally why we deprecated it). So probably just go ahead with your own patches there.

1 Like
5

Certainly not my call on how it gets in (I’m mainly an “observer” here :slight_smile: ) . I do see every change to setuptools._distutils has been via merge from distutils so I see where the “don’t know if it is possible” comment came from.

6

There’s a bunch of copies of “distutils” flying around.

The one that end-users will end up seeing is the one in setuptools._distutils which is largely pypa/distutils. This copy is separate from the CPython one; which has been removed from CPython’s current main. So… depending on what you’ve done to your Python environment, an import distutils could pick up any of those three copies.

Realistically, ~everyone is going to see the copy that’s with setuptools unles you’re building the interpreter or bootstrapping a Python ecosystem. :slight_smile:

3 Likes
7

Would setuptools come from the system’s Python installation for users who use pip or virtualenv, or would it be downloaded separately? Unfortunately, I’m not very familiar with the native Python software distribution mechanisms (I’m used to distribution packaging only). I’m trying to figure out how far patching things locally in distributions would get us. It doesn’t really our users if we patch our local setuptools version in the distribution if common workflows actually bypass it. It would still lead to breakage once we upgrade to GCC 14 or 15 (any future version that no longer supports implicit function declarations by default).

8

The future workflow is that it will be installed on demand when pip (or another build tool) needs to compile the package (and it’ll only be downloaded if specified in the pyproject.toml). It was only ever bundled in core CPython because we believed that most sdists would require it without declaring that they required it - I believe 3.12 is the version it gets removed from core.

There’s a pretty good chance you’ll have users who explicitly bypass setuptools._distutils on 3.10/3.11 (e.g. because numpy required it, and so they bypassed it for everything via environment variable), so if you’re actively supporting those users, you’ll want to patch.

(Need a virtualenv dev to speak to that one. venv uses whatever was bundled with CPython.)

9

I think this means that we really need this fixed in the official repositories. Just patching it downstream won’t be enough.

10

Yes, you’ll want it fixed in pypa/distutils. That’s the one that impacts what people will download on demand (assuming they download directly from PyPI).

However, if your users are using Python 3.11 or earlier and set the environment variable telling setuptools to use the stdlib distutils, then you’ll also need to patch it in your own lib/python3.X/distutils. That one doesn’t need to go into CPython’s repo, because it isn’t downloaded on demand.