Hi @potiuk, thank you for giving feedback. The PEP is still in draft, so it is possible to change it. Unfortunately, I think all of the authors are fairly busy, so I think probably the best move forward is to open a PR with your proposed changes and ask for reviews.
I have read your post when you original commented, but haven’t had time to properly review it. I am now skimming it to give you some feedback. Here are some of my initial thoughts:
While I acknowledge this might not be the best practice, not following it will break a lot of setups. This was the main motivation behind the recommendation.
Perhaps the PEP does not accurate reflect this situation, so it could be updated to better do so, and maybe only recommend this action when desired to keep backwards compatibility. Though, I am not really sure of what this might mean in the long term, this is just a thought.
One thing to note here is that Airflow has a very specific use-case, which is not at all representative of the majority of the users, who are the main target. It does, however, represent the needs of more complex projects, which we should still attend to.
The PEP is already very extensive, so perhaps this proposal could be split up to its own informational PEP. I think documenting your experience would be incredibly valuable, I am just not sure if this PEP is the best place to do so.
It’s worth mentioning Spack I think. It works in a very similar way to Conda.
(Conda  is a bit of a special case, as the conda command can install much more than just Python packages, making it more like a distro package manager in some senses.
I don’t actually think about it like this at all. The conda base environment is like a Linux distro, and you should never mess with it as a user. However, user-created conda environments are conceptually similar to Python’s virtual environments, just more powerful. They do environment activation, and have a “one environment for one development task” philosophy. The environment activation does a bunch of magic, that’s why you can’t layer a virtualenv on top of an activated conda env. And a major part of the reason why you need environments is because the package manager/repository provides multiple versions of every package - just like PyPI, and unlike Linux distros and Homebrew.
The above all applies to Spack environments too I believe.
This makes the “using a virtual environment is best-practice” advice more generalizable. We have two kinds of Python installs:
“base” or “system”: macOS system, Linux distros, Homebrew, Python.org installers, Python built from source, conda base env Python, etc.
The advice for all of these is: never mess with (1), always work in (2).
The NixOS Python docs page linked from the PEP is kind of illustrative in this regard: it has a python-with-my-packages where everything is in the “base” OS packages and that’s immutable so you can’t pip install anything, and then further down it explains how to get virtual/conda/micromamba/mach-nix environments - all user-level dev envs where you can install whatever you want.
I like this characterisation. Apart from being a good model for this discussion, it also leads nicely into the (completely off-topic for this discussion, but maybe worth having a separate thread on at some point) question of shipping Python “applications”, which is essentially the question of how you take your code and bundle up the essential aspects of your “dev environment” in a way that allows your code to be deployed to a “base” environment without violating the “never mess with the base environment” rule.
If I remember correctly that statement was less about conda’s capabilities with the ability to create virtual environments, and more about the fact that it cannot rely solely on the Python level metadata because a number of its Packages have nothing to do with Python at all. Thus, in this case, it’s operating more like a Linux distribution in that it has its own database of installed packages that exists outside of the Python level metadata.
Granted conda itself has integration with pip, but pip doesn’t integrate with conda.
Though if I recall correctly the PEP doesn’t expect conda to write the “don’t touch me” file, just the sysconfig split.
MacPorts is a major distributor of Python, third-party Python packages, and many other open source packages on macOS using their own package manager. It was based on the FreeBSD ports package manager also still in use.
In Nixpkgs/NixOS there is not really any difference between system packages and user packages. With NixOS and Nix in general the system and users can have profiles where they can install packages/environments, or they can open a shell with whatever packages/environments. The built packages are exactly the same and will be shared as well. Everything built with Nix is read-only. Using pip or conda outside of Nix means you have to give it a writable folder so it can work.
If I understand the PEP correctly in Nixpkgs we will add a EXTERNALLY-MANAGED file to our CPython build. For NixOS we will not add any directories for system-wide local installation because that goes against our way of working.
We are trying to move people to environments so they don’t modify the existing python installation directory (as it’s likely that the python install is a dependency of many things with different needs). Ideally it’d be read-only, and they could use pip in an env. We’d probably mark the installed python as externally managed and not the python in the spack environment
So, if I’m understanding correctly, this would be useful to Spack!
I’d like to get a better sense if there’s any need to reach out more redistributors for establishing that this PEP would be useful+beneficial overall. Primarily because I don’t think we want to go to every potential redistributor of Python.
From what I understand based on past interactions, Homebrew doesn’t specify a groups of people working on policy decisions like this. Homebrew maintainers (as in ensuring brew to work as intended) aren’t that interested in ecosystem-specific things, and generally don’t really know who to talk to either. So what I would do is to simply make the decision, and contribute the externally-managed marker to Homebrew afterwards. Hopefully “the community” will welcome the change, but if not, things continue as-is there and we won’t get complaints either.
Although the [pyenv-virtualenv](https://github.com/pyenv/pyenv-virtualenv) plugin does manage virtualenvs. Those would not be marked as ‘externally managed’ though, as the normal packaging tools are used to maintain them.
Hi, Homebrew maintainer here. We welcome discussion about big changes in projects we distribute, especially those aimed at the distro level like this. If a project wants to talk with us, a discussion or issue or PR in our homebrew-core repo is the perfect place. We just don’t have dedicated people for parts of the packages, so we can’t point out to one person and say “this is our Python person”.
Regarding the PEP, I have read through it and I think it addresses an important issue. We will discuss how to integrate that in Homebrew in future releases, but we probably want to mark the base environment in such a way. Probably some users will complain at the beginning, as will all changes, because it disrupts some workflows, but in the long term it will be less pain for everyone.