Unable to import module installed from PyPI with `pip` unless force-reinstalled?

OneCD · March 2, 2023, 8:32am

I experience this behaviour with 3 modules (sabctools, sabyenc3, brotli) - all installed from PyPI, and all have manylinux2014 in their filename. I’ll concentrate on a single module for support-purposes.

Environment is a QNAP NAS running QTS (QNAP’s heavily-modified version of Ubuntu). Python 3.10.7 is installed via Entware.

First, created a new virtualenv:

[/share/Public] # python3 -m virtualenv venv
created virtual environment CPython3.10.7.final.0-64 in 2527ms
  creator CPython3Posix(dest=/share/CACHEDEV2_DATA/Public/venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/share/CACHEDEV2_DATA/Public/pip-cache)
    added seed packages: pip==23.0, setuptools==67.1.0, wheel==0.38.4
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

[/share/Public] # . venv/bin/activate

(venv) [/share/Public] # pip list
Package    Version
---------- -------
pip        23.0
setuptools 67.1.0
wheel      0.38.4

Then, I installed the new sabctools module via pip:

(venv) [/share/Public] # pip install sabctools
Collecting sabctools
  Downloading sabctools-6.1.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (174 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 174.2/174.2 kB 1.5 MB/s eta 0:00:00
Installing collected packages: sabctools
Successfully installed sabctools-6.1.1

Then test. I’m unable to import it. Python can’t see this module, although it IS listed as installed by pip.

(venv) [/share/Public] # python
Python 3.10.7 (main, Dec  4 2022, 09:29:05) [GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sabctools
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/share/CACHEDEV2_DATA/Public/venv/lib/python3.10/site-packages/sabctools/__init__.py", line 2, in <module>
    from sabctools.sabctools import *
ModuleNotFoundError: No module named 'sabctools.sabctools'

If I then perform a --force-reinstall via pip (which appears to run some sort of local compiling process), I can then import this module without issue.

(venv) [/share/Public] # pip install --force-reinstall --no-binary :all: sabctools
DEPRECATION: --no-binary currently disables reading from the cache of locally built wheels. In the future --no-binary will not influence the wheel cache. pip 23.1 will enforce this behaviour change. A possible replacement is to use the --no-cache-dir option. You can use the flag --use-feature=no-binary-enable-wheel-cache to test the upcoming behaviour. Discussion can be found at https://github.com/pypa/pip/issues/11453
Collecting sabctools
  Downloading sabctools-6.1.1.tar.gz (228 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 228.7/228.7 kB 1.8 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: sabctools
  Building wheel for sabctools (pyproject.toml) ... done
  Created wheel for sabctools: filename=sabctools-6.1.1-cp310-cp310-linux_aarch64.whl size=35493 sha256=d9d6be1d4c906b348e16486c9b995ecb647603e658eb1a5fcc94ab8d4ea2ab25
  Stored in directory: /share/CACHEDEV2_DATA/Public/pip-cache/wheels/af/7e/24/6987efea469e00564a548e783b96a246c97b391c7837ce7fe8
Successfully built sabctools
Installing collected packages: sabctools
  Attempting uninstall: sabctools
    Found existing installation: sabctools 6.1.1
    Uninstalling sabctools-6.1.1:
      Successfully uninstalled sabctools-6.1.1
Successfully installed sabctools-6.1.1

(venv) [/share/Public] # python
Python 3.10.7 (main, Dec  4 2022, 09:29:05) [GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sabctools
>>> import sys
>>> for path in sys.path: print(path);
... 

/opt/lib/python310.zip
/opt/lib/python3.10
/opt/lib/python3.10/lib-dynload
/share/CACHEDEV2_DATA/Public/venv/lib/python3.10/site-packages
>>>

Hoping someone can please advise me further. Thank you.

steven.daprano · March 2, 2023, 9:15am

When people have this sort of issue:

install a package using pip
but then importing the package fails

the two common causes are either

the distro package name that pip sees and the name that Python uses to import the package as a module are different;
or the pip command uses a different version of Python to the one the user is trying to import with.

The first case clearly isn’t the issue here.

Instead of using pip install sabctools use python3 -m pip install sabctools and see if that fixes the problem.

You can also check the output of python3.10 -m pip -V and compare it to pip -V.

I notice that you used python3 to create the virtualenv, but python inside the virtualenv. That might mean you are using two different interpreters.

OneCD · March 2, 2023, 8:56pm

Hi Steven, and thank you for your detailed answer.

My-bad: the actual “production” script has all the correct syntax (full paths to the Python interpreters, as well as python3 -m pip instead of just pip). I used shorted commands when building my examples to make them a bit easier to read.

I’ve just double-checked: yes, each time, pip is the same, and the Python interpreter is also the same.

The other common cause sounds interesting. In fact, it was suggested as a possible issue by the developer of sabctools. However, neither of us know how to check this.

If this helps, I had also examined this problem in two completely separate environments: the first where the target module installed via a basic pip install, the second is after force-reinstalling it.

The only difference appeared to be in the Python shared-object (which I assume is installed by pip for this module).

On the original (failed) environment, this file is called “sabctools.cpython-310-aarch64-linux-gnu.so” and is 591kB in size.

# file sabctools.cpython-310-aarch64-linux-gnu.so 
sabctools.cpython-310-aarch64-linux-gnu.so: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, BuildID[sha1]=9578059cde79019bce57d05e3d28a2d4b47b77e3, with debug_info, not stripped

On the working setup (where ‘sabctools’ has been force-reinstalled), this file is called “sabctools.cpython-310.so” and is 70kB in size.

# file sabctools.cpython-310.so 
sabctools.cpython-310.so: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, not stripped

And the pycache directory for both environments has a single file called:

 __init__.cpython-310.pyc

steven.daprano · March 2, 2023, 9:20pm

Do both methods install this sabctools file into the same location?

What happens if you skip the venv and install as normal? Do you get the 591kB file or the 70kB file? Which one is correct?

What I meant was that there are some third-party libraries that you install like this:

python -m pip install SpamLibrary

but import like this:

import spam

and if you try import SpamLibrary you get an error. But that’s not the case here, as far as I can see.

OneCD · March 5, 2023, 6:34pm

Hi Steven, good questions.

I think both methods install to the same location (relative to their specific virtualenv). I can see the shared object file in:

venv/lib/python3.10/sitepackages/sabctools

If I skip the virtualenv(s), and install as normal, I get the 591kB file which fails to import. The correct (working) one is the 70kB file that seems to be locally built when I force-reinstall.

Re - import name: ah, thank you. Yes, agree it sounds unlikely, but I’m not a Python guru, so I don’t know how the import process works.

OneCD · March 10, 2023, 9:29am

Are there any thoughts on what I should check for next?

clinton-hall · June 8, 2023, 9:08am

Hi One CD,

I have had the same issue for a while. I too was using the --no-binary option for sabctools but this has also started failing due to gcc issues with recent entware updates.

What I have discovered is that the sabctools.cpython-310-aarch64-linux-gnu.so is valid and all I need to do is rename it to sabctools.cpython-310.so

In fact, for me it was sabctools.cpython-311-aarch64-linux-gnu.so…

in my start script, where it updates sabnbz then checks for requirements I have the following:

cd $PKG_DIR
python_version="$(python --version | sed 's/Python //g')"
py1="$(echo $python_version | cut -d'.' -f1)"
py2="$(echo $python_version | cut -d'.' -f2)"
source env/bin/activate
python -m pip install --upgrade pip
cd $PKG_DIR/sabnzbdplus
python -m pip install -r requirements.txt
mv $PKG_DIR/env/lib/python${py1}.${py2}/site-packages/sabctools/sabctools.cpython*.so $PKG_DIR/env/lib/python${py1}.${py2}/site-packages/sabctools/sabctools.cpython-${py1}${py2}.so

note: the last line just renames the .so file so that it can be correctly imported.
$PKG_DIR is the installation location for the app.

OneCD · June 8, 2023, 8:00pm

Hey Clinton,

I’ve long had an issue with gcc on ARMv5 NAS models, but all other arches appear to be OK. I generally try to prevent any compilation on ARMv5.

Ah, good pickup on the filename, I’ll give that a go here. Just to confirm: the solution is to remove everything between the Python version and the period in that shared object filename?

Cheers!

clinton-hall · June 11, 2023, 6:12am

Yes, solution here is to remove everything after the python version and before the file extension.
I am not sure if it is “normal” for the shared object file to have the architecture in it, but for this module it seems to not want that.

OneCD · June 17, 2023, 5:23pm

Hey Clinton,

I’ve finally been able to get a few minutes to try this. Yes, works fine!

This is certainly going to save a lot of time - I’ll no-longer need to rebuild the wheels for these problem modules. Just need to build this logic into the sherpa service scripts.

Cheers mate.