[Split] How to install a single Python application while keeping the dependencies up-to-date?

@barry-scott true but much same adding system dependencies as the shebang (I am also thinking to the command of a docker-compose file for example).

@pf_moore there could be a requirements.txt which is system-wide (since it is considered as an environment) with all the versions, if I am not requiring a downgrade of a package included in there (and cascading to dependencies, which is probably already there or a system tool could break another system tool for exact same reason you mentioned), then I’m not gonna break anything.

Well, this is the pip behavior now. It changed once, probably to improve it. I think it will keep evolving

To be clear - I’m a pip maintainer, and I can confirm that yes, pip will continue evolving. But this particular behaviour will change, if at all, only if distribution maintainers approve any change. Furthermore, the behaviour was defined in PEP 668, and it would need a further PEP to change it.

Unless you are planning on writing a new PEP and getting distro maintainers to support it, I can confidently state that pip will not drop its support for PEP 668.

1 Like

I’m not in the position to go writing a PEP or something.
As said previously, I already am (me and the whole team) going through the process of adapting to new specs.
I was just pointing out this could probably be less disruptive, by limiting it to the case when something is really about to happen (not just a possibility) and I also understand this may not be due to the package installer itself but a higher requirement.
Therefore, was trying to understand which is the best suggested alternative, in terms of effort, reusability and also style and which has less overhead in term of work and resources (in some cases also MBytes matters) but without “forcing”

That is a hard problem to solve. You might pip install a package that is not installed from the distro at the moment. But when you install a new app it will be installed from the distros packaging system.

Well, I installed it with pip, which will manage that file, so it could work the opposite (the new app installation tells me that the package xyz will be updated and then I have to pay attention to my own scripts).
Also because that happens for any app, also the system apps themselves, or no?

That’s quite a burden to put on the system package managers – effectively forcing them to implement importlib.metadata in whatever (usually quite low level) language they’re written in just to track whether some unexpected file came from an externally managed Python distribution.

Ale - it doesn’t have to be terribly complicated, even if your distro defaults to not letting you use pip to install things. Let’s hypothetically say you have a simple script which happens to use rich, and your distro doesn’t package rich. Here’s the dumb script (really, this is just an example off Rich’s GitHub page):

from rich import print

print("Hello, [bold magenta]World[/bold magenta]!", ":vampire:")

Using uv to set up (this is just an example, so I’m happily picking a tool that makes this easy):

$ uv init --script myscript.py
Initialized script at `myscript.py`
$ uv add --script myscript.py rich
Updated `myscript.py`
$

uv updates the script with these two requests, so afterwards it looks like:

# /// script
# requires-python = ">=3.8"
# dependencies = [
#     "rich",
# ]
# ///

from rich import print

print("Hello, [bold magenta]World[/bold magenta]!", ":vampire:")

You can now run it as:

$ uv run myscript.py

Or if you don’t want the extra typing and want to run it as an “ordinary command”, you can make it executable and add a shebang line to it:

#!/usr/bin/env -S uv run --script`

Notice this doesn’t encode any paths in the script, and you can put it anywhere. Managing the dependencies is done by asking uv to update the embedded information, you don’t need to manually edit things. For dependencies which are satisfiable by the standard library, or distro packages, there’s no need to add anything; if you do, you’ll get copies fetched (and then cached) from PyPI instead of the distro ones - so it’s your choice. There is a virtualenv involved, but (for this case - there are other operation modes besides the “simple script” one) it’s created on the fly and not retained (except for the cache of fetched packages), so you don’t have to fuss with creating it or keeping it up to date.

1 Like

As i said, I prefer to not rely on external tools and libraries and try to avoid it as long as possible (I don’t know if somewhere else it is, in the distro we use, which is an embedded one based on linux, uv is not included).
On the fly creation makes me think of an overhead during startup, which may be not acceptable in time critical systems (or anywhere low startup time is a requirement). This specific case then, better a configuration overhead than startup (this has both, I have to change installation scripts and it will slow down startup).
Anyway, I think this fall in the case of “use an external tool” in the above mentioned possibilities.
I didn’t get though where it takes dependencies (not that initial comment, right?)

Use python3 -m venv; it is provided by python’s stdlib.

1 Like

So, which is the golden rule for a deployed environment, that can either be containerized (i.e. may require to be initialized from a Dockerfile)?
The metrics are, overhead in terms of anything (required work, startup time, memory) and keep configuration files outside from source or scripts in case I want to cleanup after it is configured
UV? VENV? PIPX?

For build processes (including container build processes): python -m venv is usually enough (you may want other tools like uv or pip-compile to get reproducible builds and other niceties, but you don’t strictly need them).

Specifying --system-site-packages may or may not be useful depending on the use case. Using a venv for this keeps the app code (and dependencies) in the venv clearly separated from the components coming from the underlying Linux distro image.

The check for venv setup is an inherent part of the Python interpreter startup process, so it doesn’t make much difference performance wise whether that answer comes back as “yes” or “no”. (venvs that are isolated from the system site packages don’t even end up with a longer import path, since the venv dir gets added, but the system dir doesn’t. And even if both site package directories are included, while absurdly long import paths can impact runtime performance, the difference from adding a single extra directory on a typical linux distro is going to be lost in the noise)

For ad hoc Python tool usage on a mutable developer system: that’s where you want something like pipx or uv tool install (since those will also help with keeping dependencies up to date and in migrating the tool environments to newer Python versions).

2 Likes

Thanks for the completeness of the comment.
The concern about startup time was more for a tool recreating the environment (or making checks about it) each time the script has to be ran. In case of venv, I won’t expect longer times.
For the venv the only pain, is to change each script (and hope the pre-built images we use have already changed the scripts as well).
Yet until now, it seems the best option, having a requirements.txt or similar, separated from source, is something we really would like to keep.
Still thinking this change could be less disruptive and same effective, that’s a complete different story though and it may be the effect of different causes, all summed up

Using the standard tooling, which is venv, seems like probably the least disruptive option. It’s probably a good idea to recognize that where we are now is the result of around 20 years iteration to make other problems less disruptive.

Have you considering using a makefile, or a simple shell alias, or any number of options that would allow easily abstracting the behavior you want (or the recommended behavior) into something that downstream devs don’t have to think about? I would recommend using a standardized base image for your docker builds anyway as it makes things more efficient in a few ways, and you could trivially bake it into there.

Maybe if minimal overhead is a critical priority you could consider shipping golang binaries and not using an interpreted language at all? Then you can ship images that don’t even have a distro with packages to break at all. So, so many options.