I trust the Python folks and their decisions (PEP’s). This post is not about that I am against PEP668. I just don’t understand the big picture and don’t know how it is supposed to fit into my workflow or how I have to modify and adapt my workflow. My opinions are not hard coded and I am willing to modify them.
In the past I avoided the use of virtual environments without any problems because I do know what I am doing and don’t destroy or shadow things. And IMHO I don’t need protection from myself (in that case). Of course I do run virtual environments on CIs, build- and test-machines (e.g. TravisCI) but not on my productive and development machine.
One problem is that I am not able (without hacks/tricks) to run pytest on such projects when installed in Development Mode via --editable. As an example see my demo project. I installed it in Developer Mode and then could run it from everywhere in my shell because it installed an entry point script in $HOME/.local/bin/.
That is why a run of pytest do not work. pytest can not import the package/application because pytest do run globally outside of the virtual environment.
_________________________________________________________ ERROR collecting tests/test_dummy.py _________________________________________________________
ImportError while importing test module '/home/user/tech-demo-python-packaging/01a_terminal_helloworld_setuptools/tests/test_dummy.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.11/importlib/__init__.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/test_dummy.py:7: in <module>
import helloworldcli
E ModuleNotFoundError: No module named 'helloworldcli'
Technically It makes sense. But I don’t know how to deal with it.
I mean I don’t know how the designers of pipx (and the principles behind it) intended to handle such situations.
I think you may not be fully realizing what this entails.
It means that before doing pip install --user foo, you need to ensure that not only foo, but also all of its transitive dependencies, are not already installed on the OS level (or are installed with the same version).
It also means that when you do apt/dnf install foo, you need to check that foo does not have a dependency that is already installed by pip.
pytest needs to be run in a virtual environment where the project is installed. The purpose of pipx is to manage environments that make CLI tools globally available, but pipx doesn’t go farther than that. For running tests, either create the environment with venv yourself, or use an environment manager like the one integrated to Hatch, or tox, or Nox.
Basically, current practice is to use two different environments managed by two different tools. (Although Hatch might in the future gain pipx-type functionality and could potentially reuse the same environment.)
To elaborate on why testing against a pipx-installed package doesn’t work (and, in fact, shouldn’t work and isn’t supposed to work): When you run pipx install foo, pipx creates a virtualenv just for foo and its dependencies, installs foo in the venv, and creates symlinks in ~/.local/bin pointing to foo’s commands/scripts that were installed in the venv. pipx is meant for end-users to install Python programs without getting into dependency hell. In particular, after pipx install completes, the virtualenv in question is not activated, and so foo cannot be imported.
In contrast, installing & testing in a virtualenv looks like this at its most basic:
# Create the venv in the folder venv_folder:
python3 -m venv venv_folder # Or use virtualenv
# Activate the venv.
# This assumes you're using bash or a similar shell.
# Activation in different shells is beyond the scope of this answer.
source venv_folder/bin/activate
# Any Python commands run while the venv is active will only be
# able to install packages to and import packages from venv_folder.
# Packages installed outside the venv are unaffected.
# Install your project inside the venv:
pip install -e .
# Install pytest or whatever other test dependencies you need in the venv:
pip install pytest
# Run pytest on the version of your project installed in the venv:
pytest
Tools like tox and nox can be used to automate creation of venvs and installing packages inside them.
Note that “activating” a venv/virtualenv isn’t typically necessary
as long as you don’t mind specifying the path to installed
executables (python3, pip, pytest, et cetera) on the command line.
Thanks a lot for your answers and examples. Am I right to say, that “pytest” should be a dependency (via pyproject.toml) in my demo project? Then it would work?
I think my main problem is my refusal to accept virtual environments. I will sleep over that.
I have the feeling that this sentence is important. But I do not understand it. Can you explain it a bit more in details please?
No, it would not “work” in the sense that just running pytest in the project source directory would do the right thing. When you run pytest outside of a virtual environment, i.e., the pytest installed in your global environment, it won’t pick up stuff that you haven’t installed in the global environment. Given that you should not use pip to install things in the global environment, you should simply not run pytest from the global environment.
Also, your project dependencies are for the runtime requirements of your package. Testing requirements should never be included.
I just mean that pipx is a tool that follows the UNIX mantra: do one thing and do it well. The one thing pipx does is, when you run pipx install foo, to create a fresh virtual environment, to install foo in it, and to link the commands that foo makes available (typically declared in [project.scripts]) into some location on $PATH. The use case for pipx is to install commands that are implemented in Python, isolating them from each other and from the system. It has a few more bells and whistles, e.g., pipx run can create a one-off temporary environment to run just one command and remove it afterwards, or, since very recently, execute scripts with inline script metadata. But it’s not a general-purpose tool that you could use to invoke your tests or such.
Did you try installing pytest inside the virtual environment and running it from there?
(Normally, PipX would not be part of a development workflow. It’s intended for letting end users install the code as an “application” to make it easily runnable, without caring about any development stuff. The usual “simple” way for development is to create your own virtual environment explicitly, manually; install pytest there; and pip install -e . or similar into that same virtual environment.)
I really appreciate your contributions to this discussion and dealing with a stubborn one like me.
You gave me a lot new insides and things to think about.
It will take some time to come to a conclusion for my self but I will modify my tutorial repo.
I will sleep over that.