Confused by virtual environments

Python Help
1

I began trying to install this program as per the instructions:

sudo apt update && sudo apt install -y python3-venv bluez git
git clone https://github.com/kbennett2000/linux-lifepo4-bms-monitor.git
cd linux-lifepo4-bms-monitor
python3 -m venv venv && source venv/bin/activate
pip install -r requirements.txt
cp config.example.json config.json
# edit config.json with your batteries' MAC addresses
python3 dashboard.py

It made the virtualenv because after running the 4th line it then says ‘venv’ in the command line. Next I run the pip command and am greeted with this error:

(venv) [hello@blah]:~/bms-monitor/linux-lifepo4-bms-monitor$ sudo !!
sudo pip install -r requirements.txt
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.

    If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
    sure you have python3-full installed.

    If you wish to install a non-Debian packaged Python application,
    it may be easiest to use pipx install xyz, which will manage a
    virtual environment for you. Make sure you have pipx installed.

    See /usr/share/doc/python3.11/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

I am using Debian and got that same message when I was trying to install a pip package for something else a few days ago and just gave up. Can you tell me how to untangle this? When I used python and pip in the past I was on Arch and didn’t come across such errors. Instead of using venv, for isolation purposes, I used pip install --user. I was aware of venv and was always advised by other python users online to use that as the first choice but it always seemed too complicated for what I was doing, having every project and related packages isolated in their own environment, so never saw the point and this is just an example of that but being on Debian it seems to be forcing me to learn it as won’t let me install stuff as --user like I used to plus this project using venv.

I see it recommends pipx but how would I use that while also following these instructions as venv has made an environment, doesn’t pipx also make one? I really have no idea what is going where. That is the problem with isolation I don’t know where things are isolating to and if these different commands will put things in different places which is exactly why I avoided it in the past in favour of --user which still allowed isolation from the system wide python and just kept everything together in my userspace, which worked fine with me but now it seems venv is being foisted upon me.

Seems a good idea to learn it since it appears to be the ‘proper’ way recommended by pythonistas and many using it as the default, as in this case.

2

The problem might be sudo pip?
“super user” might use the system python instead of the venv you activated.

(In general sudo pip install should be avoided)

3

On a modern linux system you should only install python packages system wide using your systems package manager. In your case the debian/ubuntu apt tool.

For anything that else install into a your venv.

Note that the activate script is a convience and not required to use your venv.
You can run python and pip directly without using activate.

In your case I think this is the command you can use:

venv/bin/pip
4

I of course tried non sudo first which failed, which is why I tried sudo next. As you can see from the command sudo !! indicating the last command was not sudo. Here is the output from none sudo:

$ pip install -r requirements.txt
Collecting bleak>=3.0.2
  Using cached bleak-3.0.2-py3-none-any.whl (146 kB)
ERROR: Ignored the following versions that require a different python version: 0.1.0 Requires-Python >=3.12; 0.10.0 Requires-Python >=3.13; 0.11.0 Requires-Python >=3.13; 0.12.0b1 Requires-Python >=3.13; 0.12.0b2 Requires-Python >=3.12; 0.12.0b3 Requires-Python >=3.12; 0.12.0b4 Requires-Python >=3.13; 0.12.0b5 Requires-Python >=3.13; 0.12.1 Requires-Python >=3.12; 0.12.2b1 Requires-Python >=3.12; 0.13.0 Requires-Python >=3.12; 0.14.0 Requires-Python >=3.12; 0.14.0b1 Requires-Python >=3.12; 0.14.0b2 Requires-Python >=3.12; 0.14.1 Requires-Python >=3.12; 0.14.2 Requires-Python >=3.12; 0.15.0 Requires-Python >=3.12; 0.16.0 Requires-Python >=3.12; 0.17.0 Requires-Python >=3.12; 0.18.0 Requires-Python >=3.12; 0.18.0b5 Requires-Python >=3.12; 0.18.1 Requires-Python >=3.12; 0.19.0 Requires-Python >=3.12; 0.19.1 Requires-Python >=3.12; 0.19.2 Requires-Python >=3.12; 0.2.0 Requires-Python >=3.12; 0.2.1 Requires-Python >=3.12; 0.2.2 Requires-Python >=3.12; 0.2.3 Requires-Python >=3.12; 0.20.0 Requires-Python >=3.12; 0.21.0 Requires-Python >=3.12; 0.22.0 Requires-Python >=3.12; 0.23.0 Requires-Python >=3.12; 0.24.0 Requires-Python >=3.12; 0.25.0 Requires-Python >=3.12; 0.26.0 Requires-Python >=3.12; 0.27.0 Requires-Python >=3.12; 0.28.0 Requires-Python >=3.12; 0.29.0 Requires-Python >=3.12; 0.3.0 Requires-Python >=3.12; 0.4 Requires-Python >=3.12; 0.4.1 Requires-Python >=3.12; 0.5.0 Requires-Python >=3.13; 0.7.0 Requires-Python >=3.13; 0.9.0 Requires-Python >=3.13; 0.9.1 Requires-Python >=3.13; 0.9.2 Requires-Python >=3.13; 0.9.3b1 Requires-Python >=3.13; 0.9.3b2 Requires-Python >=3.13; 0.9.3b3 Requires-Python >=3.13; 0.9.3b4 Requires-Python >=3.13
ERROR: Could not find a version that satisfies the requirement aiobmsble>=0.28.0 (from versions: none)
ERROR: No matching distribution found for aiobmsble>=0.28.0

Can you explain what the command you suggest is doing? I still have no clue what venv does. This one you suggest I guess is just running pip inside the venv? but how do you then access this venv again once you exit out of it? it is the management that I don’t understand if they are all discreet, that is why I don’t see the point when I can have one user python/pip environment which will be separate from the system pip and that makes sense because I mostly use the same libraries from project to project, historically at least.

5

The error says you do have a version of python that the aiobmsble supports.

I think the problem is that you have a very old version of python installed.
Can you confirm the version of python please?

6 
$ python --version
Python 3.11.2
7

For this it seems it didn’t see the files in the current directory:

$ venv/bin/pip -r requirements.txt

Usage:
  pip <command> [options]

no such option: -r

EDIT: I see I missed install in the last command but got the same error:

$ venv/bin/pip install -r requirements.txt
Collecting bleak>=3.0.2
  Using cached bleak-3.0.2-py3-none-any.whl (146 kB)
ERROR: Ignored the following versions that require a different python version: 0.1.0 Requires-Python >=3.12; 0.10.0 Requires-Python >=3.13; 0.11.0 Requires-Python >=3.13; 0.12.0b1 Requires-Python >=3.13; 0.12.0b2 Requires-Python >=3.12; 0.12.0b3 Requires-Python >=3.12; 0.12.0b4 Requires-Python >=3.13; 0.12.0b5 Requires-Python >=3.13; 0.12.1 Requires-Python >=3.12; 0.12.2b1 Requires-Python >=3.12; 0.13.0 Requires-Python >=3.12; 0.14.0 Requires-Python >=3.12; 0.14.0b1 Requires-Python >=3.12; 0.14.0b2 Requires-Python >=3.12; 0.14.1 Requires-Python >=3.12; 0.14.2 Requires-Python >=3.12; 0.15.0 Requires-Python >=3.12; 0.16.0 Requires-Python >=3.12; 0.17.0 Requires-Python >=3.12; 0.18.0 Requires-Python >=3.12; 0.18.0b5 Requires-Python >=3.12; 0.18.1 Requires-Python >=3.12; 0.19.0 Requires-Python >=3.12; 0.19.1 Requires-Python >=3.12; 0.19.2 Requires-Python >=3.12; 0.2.0 Requires-Python >=3.12; 0.2.1 Requires-Python >=3.12; 0.2.2 Requires-Python >=3.12; 0.2.3 Requires-Python >=3.12; 0.20.0 Requires-Python >=3.12; 0.21.0 Requires-Python >=3.12; 0.22.0 Requires-Python >=3.12; 0.23.0 Requires-Python >=3.12; 0.24.0 Requires-Python >=3.12; 0.25.0 Requires-Python >=3.12; 0.26.0 Requires-Python >=3.12; 0.27.0 Requires-Python >=3.12; 0.28.0 Requires-Python >=3.12; 0.29.0 Requires-Python >=3.12; 0.3.0 Requires-Python >=3.12; 0.4 Requires-Python >=3.12; 0.4.1 Requires-Python >=3.12; 0.5.0 Requires-Python >=3.13; 0.7.0 Requires-Python >=3.13; 0.9.0 Requires-Python >=3.13; 0.9.1 Requires-Python >=3.13; 0.9.2 Requires-Python >=3.13; 0.9.3b1 Requires-Python >=3.13; 0.9.3b2 Requires-Python >=3.13; 0.9.3b3 Requires-Python >=3.13; 0.9.3b4 Requires-Python >=3.13
ERROR: Could not find a version that satisfies the requirement aiobmsble>=0.28.0 (from versions: none)
ERROR: No matching distribution found for aiobmsble>=0.28.0

I just read an outdated pip might be the cause of such errors. Can’t recall where/how I installed pip now.

I just checked and states: pip 23.0.1

8

3.11 is too old. Can you upgrade your system to get a newer version of python that the software you want to install supports?

Maybe you can install an extra new version of python on your system?
Check what is available from apt. (I’m not a debian/ubuntu user and I do not know the details)

9

So you could have gotten a long way by reading this error message :wink:
It tells you explicitly you need Python 3.13 or later.
The venv was created using your system Python (3.11) which I understand could be a nasty surprise, but does also make sense as default behaviour.

If you install and use Python 3.13 or 3.14, your ‘personal’ Python will automatically be separate from the system python.
Personally I’d use uv or compile the cutting edge Python version from git, but there’s also specialised tools like pyenv apparently. (Never used, but it’s the top search result for me.)

10

I will have a look. also as above I read it might alternatively be pip?

Can someone explain venv to me and what is the point, and why is it used for this project? I know generally it is to ‘keep things isolated’ but I don’t see why you want to isolate every single script run. I am sure that is useful for some use cases but not for my general usage.

I would like a single user instance where I install libraries just once, not install them every single time in new instances. Can venv also do that? and is it adviseable to use is for my purposes? what about using this project? should I still run it in venv as the instructions are indicating? or somehow can I make it run in a user install of python? would I want to or keep it as is? how does one call venvs? I am imagining something like different containers or virtual machines which are an added layer one has to negotiate - usually unnecessarily for everyday use.

11

No it’s not pip that is the issue it is the version of python.

FYI to avoid issues like pip and python commands not being matched you can run pip like this:

python3.13 -m pip
12

I am just reading that uv is the package manager to replace them all, including pip, pipx, and also pyenv. So if that is pegged as the master to succeed all others I will try that. First time I have come across it.

EDIT: seems a bit ironic that uv is written in Rust rather than Python? sacrilege?

13

pip is not being replaced. But there are a lot of wonderful projects like uv that give you alternative workflows.

14

I am not liking the sound of uv now I read more. Pyenv seems more my style: pyenv made with Unix philosophy in mind of do one thing well, uv made with do everything in one package to save time and be more efficient. I never had any problem with speed or effeciency with using pip which it seems to slate so much. Sounds like just hype for a problem I never had.

15

yes that makes sense. Uv shines when you have different venvs that you switch between regularly, and that also have their requirements change regularly.

I remember back in the day I also used to have a single personal ‘venv’ that I used for everything

16

Ok so venv could also be used as an ‘all in one’ env like I used to use python user instance? I see the python user has been discouraged for some time now in favour of venv.

17

Yes you can do that. I have a single venv that I use for all my personal tools.

I added a script called ~/bin/tool-python3 that I use in the ‘#!’ of my scripts.
That script contains:

#!/bin/bash
# run inside of the tools venv
exec ~/.local/tools.venv/bin/python3 "$@"

And the venv is in ~/.local/tools.venv.

Then my scripts start:

#!/usr/bin/env tool-python3
import sys
import os
...
18

I am just reading up on pyenv and it looks like pyenv global might be just what I want to replace old python user behavior - the global still being restricted to the user environment.

EDIT: yea buddy! This is a nice workflow! I have now learned about pyenv and pyenv-virualenvs and just installed the original program in a virtualenv within pyenv. This is really neat.

One thing I am noticing though is the terminal seems to lag quite a bit on commands now though since installing pyenv, even for commands outside of pyenv. Could it be with the .bashrc changes?

# Load pyenv automatically by appending
# the following to
# ~/.bash_profile if it exists, otherwise ~/.profile (for login shells)
# and ~/.bashrc (for interactive shells) :

export PYENV_ROOT="$HOME/.pyenv"
[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init - bash)"

# Load pyenv-virtualenv automatically by adding
# the following to ~/.bashrc:

eval "$(pyenv virtualenv-init -)"
19

The recommended place for these sorts of bash config is in .bash_profile, not .bashrc. This is because .bashrc changes can block you doing override in the shell.

Usually the .bash_profile is run once when you login.

20

What do you mean by block you doing override in the shell?

Looking up the issue online and it seems a common one and the solutions have not stated what you put. They are saying it is about having a $ in the bashrc which slows things down considerably. There are 2 here. That is all well and good but I don’t know what I should put instead of the $s as I don’t know what those commands are doing.

I have another issue now, related to the script itself. When I try and run it I get the error returned:

read failed: TypeError: type TempSensor doesn't define __round__ method

I was looking that up and it seems a common python error. I found this article which says to define round as a workaround but I have no idea where I would do that? I did a grep in the folder where the script is and didn’t get any hits for ‘TempSensor’ to see where the code for round might be placed so not sure where that error is coming up from or how to suppress it? The output unhelpfully doesn’t give the file or line number either as would normally be the case for python traceback - only what I have given above.