Creating Vanilla Environments from Polluted Python Installations

Hi everyone,

My apologies if I’m late to the conversation, but I’d like to understand why we don’t ship all Python versions with a vanilla-requirements.txt file. This would allow users to create a clean (vanilla) environment even from the most polluted installations.

Anticipated Concerns and Counter-Arguments:

  1. Technical Feasibility and Maintainability:
  • I understand that maintaining a vanilla-requirements.txt across different OSes and Python versions could be challenging. However, if distributions like Ubuntu are concerned about maintainability, they should already be using virtual environments for their purposes rather than polluting the system-wide settings. The same applies to other operating systems.
  1. Best Practices with venv:
  • While I completely agree that using venv is the best practice for creating isolated environments, we must acknowledge that many beginners start by installing packages directly into their global Python environment. Once this environment is polluted, there is currently no straightforward way to revert to a clean state, as a vanilla-requirements.txt doesn’t exist to undo the damage.

For the remainder, I am open to hearing your thoughts and suggestions. Has this idea been considered before, and are there any other reasons why it hasn’t been implemented?

Thank you for your insights!

What exactly would vanilla-requirements.txt contain? And who would consume it when? It feels like you have a fundamental misunderstanding of what the relation between the python core/stdlib, distributors and python packaging utils (like pip) is.

Thank you for your quick response. Allow me to clarify my proposal:

Content of vanilla-requirements.txt:

  • This file would list the core dependencies and standard libraries that come with a default, clean Python installation. Essentially, it would contain the minimal set of packages and their specific versions that are part of a pristine Python environment without any additional user-installed packages.

Who Would Use It and When:

  • Users: Particularly beginners who have inadvertently polluted their global Python environment with various packages and versions, leading to conflicts and broken environments.
  • Scenario: When a user realizes their environment is no longer functioning correctly due to too many or conflicting packages, they could use this file to restore their environment to a clean state.

How It Would Be Used:

  • Users would create a virtual environment and then use pip to install only the packages listed in vanilla-requirements.txt. For example:

    python -m venv clean_env
    source clean_env/bin/activate
    pip install -r vanilla-requirements.txt
    

Purpose and Benefits:

  • This approach provides a way to reset an environment to a known good state, which is especially useful for educational purposes and for users who are new to Python.
  • It offers a straightforward method to recover from a polluted environment without needing to reinstall Python or manually figure out which packages to remove.

I hope this clarifies my suggestion. I’d be interested to hear your thoughts on whether this approach is technically feasible and if it aligns with the goals of the Python community.

Thank you!

You mean restore to this, right?

$ python3.8 -m pip list
Package    Version
---------- -------
pip        24.0
setuptools 69.2.0
wheel      0.43.0

I feel like there should be a command for this, this is not easy to remember:

$ pip uninstall -y -r <(pip freeze)
$ pip install -r requirements.txt
1 Like

So yes, you do have a slight misunderstanding of what python’s core is. Because by default, in a “pristine” python distribution, there is nothing installed by pip and so vanilla-requirements.txt would be empty. The only thing that would belong in there is the packages @Nineteendo listed, which are the requirements for pip, meaning you can’t easily use pip to install them. But that is what the stdlib module ensurepip is there for, so the state you want to reach is achived by running the already existing (and somewhat recommended) python -m ensurepip instead of pip install -r vanilla-requirements.txt.

4 Likes

For the vanilla python: Yes!
That’s exactly what I had in mind.

Building derived environments is easy as I can stack the dependencies.

  1. vanilla-requirements.txt
  2. ubuntu-requirements.txt # builds upon the venv for vanilla.
  3. myapp-for-ubuntu-requirements.txt # builds upon the venv for ubuntu

You may want to have a look at pip-tools, which makes all this fairly easy.

1 Like

There is a perfectly straightforward way to revert the base environment to the original state: uninstall everything that was installed from PyPI. There is neither any need to list any “core dependencies and standard libraries”, nor any possible way to do so - because these did not come from PyPI and were not and cannot be installed by Pip. A vanilla-requirements.txt could not possibly “undo the damage” - because it doesn’t specify packages to remove, only more packages to install.

There is also a perfectly straightforward way to create a new, clean environment: via python -m venv, which you are apparently already perfectly aware of. It will, by default, not contain any of the packages that were installed into the base environment, except for the Pip bootstrap. From here, there is nothing to install to get “the core dependencies and standard libraries that come with a default, clean Python installation” - so such a vanilla-requirements.txt would be empty, and useless. Exactly nothing that can be installed from PyPI is a “core dependency” or “standard library”. There are no package names for any of the standard libraries, and it is not possible for Pip or any similar tool to restore them - because there is nothing available for them to use as a source for such restoration.

In short:

It probably hasn’t been considered for very long, if anyone actually thought of it - because it makes quite little sense, and demonstrates serious misunderstandings about how all of this actually works.

Thanks for your quick responses. I see it may have been premature of me to suggest a solution before the problem is clear. For that I apologise.

Perhaps a practical use case where things break down can prompt a link to a “how-to”:

John Doe discovers that package x.y.z is incompatible with pandas version a.b.c and asks for help. After executing pip list hundreds of packages appear and the system wide environment has been polluted. John Doe has not come across python -m venv myenv yet.

What is the recommend solution to undo-the pollution?

Update: John Doe doesn’t know the difference between packages that are required by Ubuntu and packages that s/he installed. So pip uninstall *everything will probably also uninstall dependencies that his ubuntu desktop require.

How can python help though? This is a distro-specific problem. The solution is probably to check each package whether it’s a dependency of something and remove it. If that still doesn’t solve the issue, he should learn venv. Actually, he should learn venv regardless, because it’s the solution for this kind of problem, so that he doesn’t get dependency issues in the first place, instead of fixing it after it happened.

How did John even get into this situation, though? Pip won’t install into a distro-managed system Python installation, so they presumably were using apt to install the Python packages. So they need to look to apt to address this issue. (And I’m pretty sure apt would object if you tried to uninstall something that is required by Ubuntu…)

This has been fixed in 3.12 by the way. Bjorn, are you still on 3.11?

I have to use Python 3.10 and lower, what has been fixed?

By default you can’t use pip with home-brew for example:

wannes@Stefans-iMac ~ % pip install -r requirements.txt
error: externally-managed-environment

× This environment is externally managed
╰─> <homebrew error message>

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
1 Like

Great to hear! Unfortunately I see all version between 3.7 and 3.12.

I believe Joe installed using sudo bash-from-colleague.sh :frowning: Hence my search for the “undo” option.

Trust me, Joe will learn venv now.

It’s nothing to do with the Python version, that “externally managed environment” check is part of recent versions of pip (which should be present in all supported versions of Python).

  1. Using sudo without knowing what you are doing is unsupported and ill-advised (and frankly dangerous). Maybe Joe doesn’t know any better, but at some point we have to assume a minimal level of understanding from the user.
  2. Joe should be asking “colleague” to tell them how to fix the mess that “colleague”'s script made of their system.
2 Likes

Oh I do agree, but I can still wish for the “undo” feature on something that runs on sudo.

pip-tools was mentioned already. I wonder what its pip-sync tool would do with an empty requirements.txt file, or maybe a requirements.txt that only contains pip (and setuptools and wheel`?).