Sharing packages between different Python versions

Python Help
1

I am using KDE Neon jammy 22.04 (Ubuntu based). It cames with Python3.10 installed. I also installed Python3.11 with sudo apt install python3.11. I wrote a program using PyQt5. It works with Python3.10, but not Python3.11. My error is ModuleNotFoundError: No module named 'PyQt5.sip'.

PyQt5 is in /usr/lib/python3/dist-packages/. It is present neither in /usr/lib/python3.10 nor /usr/lib/python3.11. /usr/local/lib/python3.10/dist-packages/ and /usr/local/lib/python3.11/dist-packages/ are empty. When I do :

$ python3.11 -c "import sys; print(sys.path)"

I can see

['', '/usr/lib/python311.zip', '/usr/lib/python3.11', '/usr/lib/python3.11/lib-dynload', '/home/richard/.local/lib/python3.11/site-packages', '/usr/local/lib/python3.11/dist-packages', '/usr/lib/python3/dist-packages']

So apparently my Python3.11 goes to look for packages in /usr/lib/python3/dist-packages where PyQt5 is… the folder with PyQt5 is present. Why my PyQt5 program doesn’t work with my Python3.11 ? Why it says No module named 'PyQt5.sip' ? There is no difference between my package in python3.10 and python3.11.

Obviously if I create a virtual environment with Python3.11 and install PyQt5 in it, it works… but it’s not my point and I don’t want to add multiple installs for that.

Thanks

2

Every installation of Python has its own set of third-party libraries. dist-packages isn’t a standard location for this stuff; Linux distros hack the standard library site.py (which doesn’t normally care, although theoretically you could use a .pth file in site-packages to add a dist-packages folder explicitly) to add it.

For example, the docstring for site.py from my system’s Python (3.8, from Mint 20.3 on Ubuntu 20.04) tells me:

This will append site-specific paths to the module search path.  On
Unix (including Mac OSX), it starts with sys.prefix and
sys.exec_prefix (if different) and appends
lib/python3/dist-packages.
On other platforms (such as Windows), it tries each of the
prefixes directly, as well as with lib/site-packages appended.  The
resulting directories, if they exist, are appended to sys.path, and
also inspected for path configuration files.

For Debian and derivatives, this sys.path is augmented with directories
for packages distributed within the distribution. Local addons go
into /usr/local/lib/python<version>/dist-packages, Debian addons
install into /usr/lib/python3/dist-packages.
/usr/lib/python<version>/site-packages is not used.

which is different from how it was originally written up.

Yes, that’s normal. I’m not sure why they create such folders at all for Python installed via the system package manager. Normally, AFAIK, Python-related stuff only ends up in /usr/local because you compiled from source and used make altinstall.

Interesting. It certainly doesn’t work this way with Python compiled from source with make altinstall (like I did it).

Are you sure? if you did install a newer version of PyQt5 for Python3.11 separately, it would use that in preference to the one in the /usr/lib/python3/dist-packages - because it would be installed into /home/richard/.local/lib/python3.11/site-packages, earlier on the sys.path. (You can’t install multiple versions of a package to the same folder, but you can end up with multiple versions of the same package in the same environment - after all, Python doesn’t have a way to know they’re the same package.) And in this case, the newer version might simply not have the sip subpackage:

What do you get with python3.11 -c "import PyQt5; print(PyQt5.__file__)"? Similarly with 3.10? And what about if you import PyQt5.sip explicitly in the 3.10 environment?

Honestly, if you’re ever planning to distribute the code, I’d recommend it anyway. It’s much saner for testing, ensuring that your package specifies the right dependencies, etc.

1 Like
3

Ubuntu packages for the system python, that is 3.10 in your case.
If you want to use a different python you will need to find the packages from somewhere else, not ubuntu.
FYI this is also the case for debian and Fedora as well.

In the general case packages for python 3.10 may not be usable with python 3.11, especially if they are C/C++ extensions, as is the case with
PyQt.

You can try installing them from PyPI into a venv.
Which I think will work for PyQt5, but you could try PyQt6 is that does not work.

1 Like
4

Could Python 3.11 have looked in the main '/usr/lib/python3/dist-packages' folder, found the stuff that was packaged with Python 3.10, but then concluded that it wouldn’t work in 3.11 and reported it missing? I thought that finding a binary-incompatible .so file would cause a generic ImportError immediately, rather than skipping the file and leading to a later ModuleNotFoundError.

5

This is no such logic in python. So no.
The packaged pythons will no share any directories.

6

Thanks for your great answer :). It helped me to understand some things. So I think it’s related to the Debian-based distribution. I don’t have any dist-packages on my other laptop with Manjaro-linux (Arch-based).

All package are in /usr/lib/python3/dist-packages/. Just few are in /usr/lib/python3.XX/such as tkinter, sqlite3, distutils… In .local/lib/python3.XX/site-packages/there are just few packages I explicitly installed (tkdesigner, faker…).

I noticed I can’t install PyQt5 with python3.11. Here what I get :

$ python3.11 -m pip install PyQt5
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: PyQt5 in /usr/lib/python3/dist-packages (5.15.10)
Requirement already satisfied: PyQt5-sip<13,>=12.13 in /usr/lib/python3/dist-packages (from PyQt5) (12.13.0)

That is… interesting… So python3.11 know it’s in here but don’t access it when I import it ? Weird. However in a virtual environment no problem to install it.

Python3.11 were installed using sudo apt get install python3.11.

I typed your command and get :

$ python3.11 -c "import PyQt5; print(PyQt5.__file__)"
/usr/lib/python3/dist-packages/PyQt5/__init__.py

I don’t understand the Debian-logic to have this dist-packages… saving memory space I guess. But for now, I’d just would like to use any version of Python with its packages without any problem or having to install each time specific packages. I don’t want to have to install PyQt5 specifically every time when it’s supposed to be there. :confused:

7

Why don’t you just install PyQt5 from the distribution repositories using apt?
There is a separate package python3-pyqt5.sip.

8

The OP wants to use python 3.11 and the OS is only packaging for 3.10
so that does not work.

1 Like
9

I tried but it didn’t change anything. It install it for python3.10 apparently. I still have the same error… I think for Debian all packages are packed for python3.10 (not sure about that, it just the behavior that makes me think that). I’d just would like to use a python3.11 (or now I saw there is the python3.12) with all packages. Python3.12 is not available on apt. Maybe I should try a fresh install from source code… It doesn’t seem very clean the way Python works on Debian :confused:

10

Its clean in the sense that they only support one system python version.
If you do not want to work with that version then you have to do all the hard work. This is the tradeoff in using an LTS OS - the software on it is old.

If being on a newer python version is important to you then you might consider using Fedora that uses python 3.12 as its system python.

11

Debian stable has Python 3.11. But you seem to be using Ubuntu 22.04. I wish you would not keep talking about Debian.

I work with PyQt5 on Debian/Devuan stable with system Python 3.11 and have Python 3.9 and 3.10 installed with pyenv (which I like a lot).
Everything works very clean.

12

It does access PyQt5, and it does count as “installed” for Python 3.11 when it’s in that folder.

This shows that Python 3.11 finds the same PyQt5 that Python 3.10 does, in the common dist-packages. Which is all well and good. There’s a common folder like this exactly so that, when you use the system package manager to install Python, they can share some packages that the distro thinks you should have. But you get specific versions of those packages, which don’t necessarily work with every Python version.

In the version you have, there isn’t a PyQt5.sip, but there is a PyQt5.QtWidgets that expects PyQt5.sip to be there. I think this is an issue for the distro maintainer. But there are more diagnostics you should try first (note the versions carefully):

$ python3.11 -c "import PyQt5.sip; print(PyQt5.sip.__file__)"
$ python3.10 -c "import PyQt5; print(PyQt5.__file__)"
$ python3.10 -c "import PyQt5.sip; print(PyQt5.sip.__file__)"
$ python3.10 -c "from PyQt5.QtWidgets import *"

And we should probably also see what import you actually tried in the “program using PyQt5” that “works with Python3.10, but not Python3.11”.

Agreed. From what I’ve seen, the Python they provide you is really there to support the OS, not to enable development.

13

Thanks everyone, I understand better how it works in background, I wanted to know for the intellectual curiosity because I didn’t understand. I can use Python3.10 or use Pyenv or create venv. I have not a necessary need of the last version, but I like to understand things deeper.