Compiling 3.9.1 on Bizarre Linux System

Hi,

I am struggling to compile Python 3.9.1 on a very custom Linux system (ancient core with tons of hand-compiled newer versions of stuff, can’t do anything about it) where I need to basically specify, for almost every module, their location individually. To that end, I ran things like

$ ./configure --with-openssl=/path/to/my/openssl --with-bz2=/other/path/to/libbz2 --with-sqlite3=/yet/another/path …

where, under these paths, there are usually ./lib, ./include and ./bin directories with the appropriate libraries, both shared and unshared. But somehow, the build process does not seem to find it, and looking into setup.py into find_modules() didn’t make me any wiser, too. In the end, I get messages that several _module s were not found, and even, that _ssl has been removed because it could not be imported. Furthermore, documentation on this topic is unfortunately a bit scant.

Also, does Python support the pkg-config standard? I could not see anything, but that would help.

Can anyone please help me out?

The --with-somelibrary option only add header and library paths for the compile and link step. They do change the library search path for shared library. The imports are probably failing because dlopen() cannot locate and load the shared libraries from the custom paths.

You have multiple options to address the issue. You can either define LD_LIBRARY_PATH=/path/to/openssl/lib:/path/to/libbz2/lib at runtime or define a rpath at build time (LD_RUN_PATH or -Wl,-rpath,/path/to/openssl/lib -Wl,-rpath,/path/to/libbz2/lib).

1 Like

Also, Python configure in general does not support pkg-config nor --with-<pkg> options other than for openssl. In particular, there are no --with-bz2 or --with-sqlite3 options to configure:

$ ./configure --with-openssl=/path/to/my/openssl --with-bz2=/other/path/to/libbz2 --with-sqlite3=/yet/another/path
configure: WARNING: unrecognized options: --with-bz2, --with-sqlite3
[...]
1 Like

Thank you! Well, compilation still fails, due to “could not load _uuid” and similar error messages. What is the easiest method to figure out what this thing wants -because I’ve added the paths to libuuid.so both with -L and -Wl,–rpath ? This is not the only module where I have this problem, but one of few stragglers, although I configured them all the same way, adding their respective directories to -I and the linker directives.

When you are setting the RPATH using -Wl,--rpath, I am fairly certain you need to set that to the directory which contains the .so, not to the .so itself. You are providing a search path to the linker, not providing the actual path to the library.

1 Like

That’s what I am doing. The dependencies are in directories of the form
/path/to/thing-1.2.3/{include,lib,bin,…}, and I am basically telling GCC to use
-I/path/to/thing-1.2.3/include -L/path/to/thing-1.2.3/lib -Wl,–rpath,/path/to/thing-1.2.3/lib
with 1.2.3 being the version number of ‘thing’.

1 Like

I’m still having some trouble, one of which is why I get different results everytime I run the build process. Eg. running it two times in a row on the same system, I get binaries for python3 that seem to be vastly different, as suggested by ‘cmp’. Is there a way to make two compilations yield identical binaries?

Also, how do I change components from optional to mandatory?

I’ve done more testing, this time with Python 3.9.5. It looks like this problem may somehow be related to this bug: https://github.com/pypa/pip/issues/9568, but I don’t understand the relationship. In 3.9.1, things sort of work. I can run Python from the command line and then run ‘import ssl’ without a problem. But in Python 3.9.5, the same script to compile it, doesn’t work, and when I try to ‘import ssl’, I get an ImportError. This also happens with some other modules:

_sqlite3, _uuid, and _tkinter (I don’t need that).

At this point, my questions are, why do the various Python versions exhibit so different behaviour during compile, and how can I best understand this without becoming an expert in core Python?