How to use pip install with local tar.gz source file?

Packaging
1

As a beginner in packaging a module containing an extension module, it doesn’t work the first time.
I must try different setup.cfg and setup.py parameters. Uploading the tar.gz file generated by “python3 -m build” to test.pypi.org is annoying because it requires to change the version at each upload and it slows the testing process.

Is it possible to do a pip install by specifying the local tar.gz file to use to test if building and installation works ?

What command should I use ? I couldn’t find an example in the documentation.

What command should I use to fully remove the package (and cache if any) so that I can retry the pip install with a new version of the tar.gz file ?

2

pip install <path-to-tar-gz> should work. If it does not, your tar.gz is probably packaged incorrectly, and you should either contact package maintainers, or (if you are the maintainer), look for help by posting here how it went wrong for you.

2 Likes
3

Thank you very much for your help. You suggested command worked, but building failed.

Please ignore the rest of this response if the question is inappropriate for this forum.

The problem is the following.

I have a C file named qjson.c that includes a header file qjson.h (#include "qjson.h"). Both files are stored in the subdirectory src. The compilation fails because the include path to qjson.h is invalid.

I used the example provided here.

This is my setup.py file.

from setuptools import setup, Extension

setup(
  ext_modules=[Extension("qjson.ext",
                       ["src/qjson.c"],
                       depends=["src/qjson.h"],
                       include_dirs=["src/"],
              )],
)

This is the pip install output:

...
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.8
  creating build/lib.linux-x86_64-3.8/qjson
  copying qjson/__init__.py -> build/lib.linux-x86_64-3.8/qjson
  running build_ext
  building 'qjson.ext' extension
  creating build/temp.linux-x86_64-3.8
  creating build/temp.linux-x86_64-3.8/src
  x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Isrc/ -I/usr/include/python3.8 -c src/qjson.c -o build/temp.linux-x86_64-3.8/src/qjson.o
  src/qjson.c:1:10: fatal error: qjson.h: Aucun fichier ou dossier de ce type
      1 | #include "qjson.h"
        |          ^~~~~~~~~
  compilation terminated.
  error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

The error is that the file qjson.h can’t be found. It seam that src, src/ or . are not valid include path. Though it seam that the cwd is the directory above my src.

The examples I could find with inlude_dirs use an absolute path like /usr/local/include.

4

What command should I use to fully remove the package (and cache if any) so that I can retry the pip install with a new version of the tar.gz file ?

You should be testing installation of your package inside a virtualenv. That way, when you need to try again, you can just delete the virtualenv and create a new, fresh one.

If you need to delete a cached wheel built from your .tar.gz file (not sure if it matters when installing from a .tar.gz on disk), run pip cache remove $PROJECT_NAME. To delete all cached wheels, run pip cache purge.

5

Thank you very much.

In the mean time, I found the explanation of the compilation error. The header file src/qjson.h is not added to the tar.gz file. The Extension variable name depends is apparently ignored.

I could not find how to get python3 -m build to add the header file.

6

To add the header file, you probably need to add it to the MANIFEST.in file, which specifies additiona files to add to the sdist.

1 Like
7

Side note: doing that using virtualenvwrapper makes it far, far easier:

$ mktmpenv
created virtual environment CPython3.8.5.final.0-64 in 240ms
...
This is a temporary environment. It will be deleted when you run 'deactivate'.
8

Thank you very much.

That is the cause of my problem. python3 -m build uses sdist and by default sdist won’t include header files. A MANIFEST.in file is required. The files to be added in the tar.gz file must then be specified explicitly.