Virtualenv always active

I have a python class that runs on virtualenv and I need this virtual environment to be restarted automatically, without having to disable virtualenv, every time there are updates to the python script, how can I get around ?

It depends on what you mean by “restarted automatically”. Can you show
us what you currently do to “restart” the virtualenv? And describe when
you think you need to do it i.e. under what circumstances?

A virtualenv is essentially a little install area with a bin/ containing
executables like “python” etc whose property is that when you run them,
the python which runs uses the things installed in the virtualenv
instead of using (for example) the system python install. The virtualenv
also has a lib subdirectory with all the modules etc, and is where
modules for that virtualenv get installed (again, instead of trying to
install against the “system” python).

So for me, I usually use a virtualenv by sticking the path to the “bin”
directory in my $PATH ahead of the default path comonents. Example:

export PATH=$HOME/var/venv/3/bin:$PATH

which puts my current python 3 based personal virtualenv ahead of
everything else. I do this in my .profile, so that it is always done
that way when I log in. That is all you need to do.

And I do not use the “activate” command. That is something intended for
interactive shells - it does the above and also hacks around your shell
prompt to remind you of the change. I find it intrusive and complex.

Those are UNIX instructions; there will be a parallel process for
Windows but I do not know what it is.

Finally, when you say “every time there are updates to the python
script”, what do you mean? I change scripts all the time, and never have
to fiddle my virtualenv setup - the virtualenv is just what is used to
run a Python script. Changing a script does not require any “restart”
of a virtualenv.

Cheers,
Cameron Simpson cs@cskk.id.au

1 Like

In my case, I run a webserver.py on port 80 depending on virtualenv to execute the file, using the command (python webserver.py). The problem is that when closing the terminal, webserver.py is interrupted and virtualenv is deactivated (stops working, no longer fulfilling the http request) and in this case I need to make virtualenv persistent so that it keeps webserver.py functional.

Ah.

I’m assuming you’re on UNIX of some kind.

The standard answer to this is the nohup command, causing a programme to
ignore the SIGHUP (hangup) signal, sent when a terminal is closed:

nohup python webserver.py

You should output the programme output to a file, eg:

nohup python webserver.py >>webserver.out 2>&1

so that you have something to look at if things fail.

Alternatively, and perhaps more conveniently for debugging, you can run
the web server in a terminal not associated with a window. Usually tmux
or screen is used for this.

Supposing tmux, you’d type “tmux” at the shell prompt. This starts a new
shell in a new terminal, displayed in your current window. Yopu can run
your command there. Typing ^B d (control-b, then “d”) will detach from
that terminal, dropping back to your original shell. You could now close
your window. The nice thing with tmux or screen is that the terminal is
still around, and can be reattached to later.

Try this:

https://edricteo.com/tmux-tutorial/

Cheers,
Cameron Simpson cs@cskk.id.au

I did the following, I decided to make an alias for the python interpreter, giving up virtualenv provisionally. but I came across the following error when running webserver.py:

root@webstrucs:/var/net# python webserver.py
No module named socketserve
Traceback (most recent call last):
     File "webserver.py", line 8, in <module>
     from http.server import BaseHTTPRequestHandler, SimpleHTTPRequestHandler
ImportError: No module named http.server

I tried to handle the error using the Try / Except block:

try:
    from socketserver import BaseRequestHandler, ThreadingTCPServer
except ImportError:
    print("No module named socketserver")

But the same error persists

I did the following, I decided to make an alias for the python
interpreter, giving up virtualenv provisionally. but I came across the
following error when running webserver.py:

Sounds like your alias breaks things. Can we see the alias?

I like aliases or shell functions like this (which divert a “normal
command” like python) to show me what they’re doing, eg:

python(){
  ( set -x
    /path/to/some/specific/python $@
  )
}

root@webstrucs:/var/net# python webserver.py
No module named socketserve

Missing trailing “r” in socketserver" above.

Traceback (most recent call last):
File “webserver.py”, line 8, in
from http.server import BaseHTTPRequestHandler, SimpleHTTPRequestHandler
ImportError: No module named http.server

There’s no http.server in Python 2. What python are you invoking? These
days it should always be Python 3 absent some very special reason.

I tried to handle the error using the Try / Except block:

try:
from socketserver import BaseRequestHandler, ThreadingTCPServer
except ImportError:
print(“No module named socketserver”)

This conceals the real error, potentially. Always include the exception
itself in the message, eg:

except ImportError as e:
    print("No module named socketserver:", e)

But you’re not “handling” the exception, in that you’re not doing
anything to make things work, just printing and not failing.

But the same error persists

Without knowing the python version, seeing the whole code (at least up
to the code which fails) and the full traceback it is hard to help.

Cheers,
Cameron Simpson cs@cskk.id.au

alias python=/usr/local/bin/python3.9

here it was a typo of mine when I posted it, sorry.

Thought as much based on the other code. But cut/paste is your friend
here - that particular (typo) accident is hard if you cut/paste from a
programme exhibiting what you’re describing.

Cheers,
Cameron Simpson cs@cskk.id.au

Thanks. Looks sane to me. Personally, I’d do it like this:

python()
{ ( set -x
    /usr/local/bin/python3.9 $@
  )
}

instead, as stated, so that I know what’s being invoked.

Cheers,
Cameron Simpson cs@cskk.id.au

What is being invoked and exactly the path for the python interpreter :slight_smile:

/usr/local/bin/python3.9

What I was able to identify was that when I see the python version as a normal user I get the version in which the alias was configured (python3.9.4) and when I run webserver.py as a normal user I get the following output:

/var/net$ python webserver.py

Traceback (most recent call last):
  File "/var/net/webserver.py", line 57, in <module>
    with ThreadingTCPServer((HOST, PORT), HTTPTCPIPv4.HTTPProtocol) as server:
  File "/usr/local/lib/python3.9/socketserver.py", line 452, in __init__
    self.server_bind()
  File "/usr/local/lib/python3.9/socketserver.py", line 466, in server_bind
    self.socket.bind(self.server_address)
PermissionError: [Errno 13] Permission denied

And logango as root and specifying the python version I get the version (2.7 …) and without a doubt the script.py does not run:

  File "var/net/webserver.py", line 2, in <module>
    from socketserver import BaseRequestHandler, ThreadingTCPServer
ImportError: No module named socketserver

What is being invoked and exactly the path for the python interpreter
:slight_smile:

/usr/local/bin/python3.9

Via your alias I presume. Good.

What I was able to identify was that when I see the python version as a normal user I get the version in which the alias was configured (python3.9.4) and when I run webserver.py as a normal user I get the following output:

/var/net$ python webserver.py

Traceback (most recent call last):
 File "/var/net/webserver.py", line 57, in <module>
   with ThreadingTCPServer((HOST, PORT), HTTPTCPIPv4.HTTPProtocol) as server:
 File "/usr/local/lib/python3.9/socketserver.py", line 452, in __init__
   self.server_bind()
 File "/usr/local/lib/python3.9/socketserver.py", line 466, in server_bind
   self.socket.bind(self.server_address)
PermissionError: [Errno 13] Permission denied

That will probably be because you’re using port 80 for the listening
port. On UNIX systems ports below 1024 may only be used by the superuser
(root).

DO NOT test code as the superuser!

Just pick a spare port over 1023. 8000 is a common choice.

And logango as root and specifying the python version I get the version (2.7 …) and without a doubt the script.py does not run:

 File "var/net/webserver.py", line 2, in <module>
   from socketserver import BaseRequestHandler, ThreadingTCPServer
ImportError: No module named socketserver

Yes, Python 3 scripts are often incompatible with Python 2 - the 2->3
transition was chosen as an opportunity to make a lot of breaking
changes which improved the Python ecosystem - before and after that
transition point Python updates take GREAT pains to be backward
compatible with earlier versions.

Try using “python3” if you must run stuff as root. But my VERY STRONG
recommendation is that you DO NOT run things are root, particularly hand
grown webservers. They are a gateway to disaster.

Cheers,
Cameron Simpson cs@cskk.id.au

Really as it is a web server (http and https) that I am dedicating to develop. Now I don’t understand why running the script on virtualenv, port 80 listens ? And how it would be to run on port 80 without using virtualenv, as it implies that virtualenv breaks the user privileges protocol.

During development/debugging, use port 8000.

For production, if you’re really exposing this, I recommend running the
webserver not as root, on a port which does not require root
provileges.

Instead, configure a system reverse proxy such as haproxy, traefik,
nginx which listens on the privileged port and which directs connections
to your web server’s nonprivileged port.

This means that bugs in your webserver will at least not be running as
root. Instead they will be running as whichever user runs your webserver
python programme. ideally that user (a) also is not you and (b) has
very limited access to data on your machine, particularly no write
access.

I installed (iptables-persistent) to be able to write the rules for entering and leaving netfilter on port 80 and being able to attend the http requests of any user. When running python as non-root I get PermissionError: [Errno 13] Permission denied ?

I would guess your app is stil trying to listen on port 80. Still
forbidden - pretty sure iptables does not “magic permissioning” in this
regard.

What you could to is write a rule to redirect inbound connections on
port 80 to an internal high port, and have your app listen on that high
port.

Cheers,
Cameron Simpson cs@cskk.id.au

What can I not understand? Is that if you run the same script on virtualenv port 80 listen.

Please provide more detail.

The basic situation is:

  • nonroot users may not bind to ports below 1024, so you can’t listen on
    port 80

  • it is a very bad idea to run a web service as root

  • adding a firewall rule permitting inbound access to port 80 does not
    grant ordinary users (i.e. you) permission to bind a listen to such a
    port

  • having your web service listen on a high port might be fine, if you
    create a firewall rule which allows inbound port 80 and diverts it
    to the high port your web service is listening on

None of these permissions have anything to do with virtualenvs;
virtualenvs just provide a little environment with a particular Python
and libraries. They don’t affect your permission to do things.

Cheers,
Cameron Simpson cs@cskk.id.au

Okay, I managed to rotate by redirecting to a high door. Now, I need to make the script (webserver.py) persistent (always active), because when I end the session at the terminal, the server stops serving web requests?

As I mnetioned earlier, my approach for this is often tmux: make a new
tmux session, start your script, leave tmux. You can reattach later to
hand restart you script or to see its error output etc.

Cheers,
Cameron Simpson cs@cskk.id.au