I’ll try to weigh in from the Spack perspective here.
I guess I should do this more, as I don’t get the impression that folks over here in Python packaging understand the package installation model in Spack/Nix/Guix/etc. particularly well, and that’s probably our fault. I don’t try to impose Spack opinions too much on Python packaging, because we’re really more like a distro than a Python tool, and many of the things we implement (like environments) are mostly distinct from core python. So it can seem futile to speak up here. I’ve been inspired by recent packaging discussions, though, as it seems like the community is starting to realize that scientific computing, HPC, and AI use cases (where we have to integrate with C, C++, Fortran, Lua, Julia, etc.) are not so niche after all. We really are attacking a more general problem (albeit with a smaller user base).
Also, people use Spack to develop Python code, so we share a lot of similarities with tools like pip as well as distros. It would be nice to provide a familiar user experience to Python people in Spack.
Using Python Packages
I think most folks in the community think there about two main ways to install Python packages:
- Installing in the python interpeter, and
- Installing in a virtual environment.
I agree the second one’s way better – and people should isolate their python package installations (especially from the system one), but there are other frequently used mechanisms for isolating Python installations. In particular, many HPC sites, as well as Spack, nix, and guix will install every package into its own prefix, and the user can pick what to load using one of:
How Spack does it
In Spack, we also support isolated environments, but they’re implemented by symlinking (or hardlinkng, or outright copying and relocating)
--prefix installs into the environment’s prefix (i.e., into site-packages). We use the old virtualenv trick of copying the interpreter and
os.py, but we might switch to the
venv mechanism eventually. In all of these cases, you can have arbitrarily many Spack environments with arbitrary combinations of packages in them, and we always start with a
--prefix install that Spack then links into some Spack env.
If a user just wants to try out different versions of some Python package, they might install a few and load one:
$ spack install firstname.lastname@example.org
$ spack install email@example.com
$ spack load py-black@22
$ which black
Spack users are used to this; they get that they need to
spack load things to use them, and it’s nice because they can have as many versions of any package they like installed at once.
We specifically don’t support installing things into any particular Python interpreter prefix – we don’t want that. If users want Python installed with a bunch of packages, they make a Spack environment, which might look like, e.g., this:
- python @3.9.15
- py-torch @1.12.1 +cuda +cudnn +mpi
That is then concretized (resolved), we spit out a
spack.lock with all the specific dependency configurations (you can use this to reproduce the build), and the whole env gets linked together in what we call a view – a single prefix. The user can activate/deactivate the env with:
$ spack env activate .
$ spack env deactivate
And they’ll get all of those packages installed into a prefix that they can load/unload on demand.
Some points to note here are that Spack would never install “into” a virtualenv like
pip does, and Spack sets up the build env independently (and reproducibly) for every package. We don’t want a stateful environment for package installations. We want every package to be isolated from every other package, which is IMO more aggressive than requiring a venv to do an installation. So the requirement for a
venv for us is superfluous.
The way this gets implemented is as you might expect – we set up
PYTHONPATH, etc. in the build environment and we currently run
pip install with
--prefix and a bunch of other args (see here for the rest of them).
We also do things like rewriting shebangs for each script to point to the specific Python that the script was installed with. We generally do not use
/usr/bin/env python3. If we end up copying or hardlinking an environment into place, we’ll relocate the shebangs to point to the environment prefix, not the canonical installation directory.
Nix and Guix have similar installation models – there may be slight differences, and they generally only support using things within a user “profile” (which is kind of like an environment), not through theone-off load/unload mechanism Spack has.
Thoughts on this PEP
I had a strong negative reaction to this PEP when I first read it, but I read it over a few times more and tried to follow the discussion here, and in the end I don’t think it affects Spack too much.
So here are some thoughts and concerns specific to the PEP:
The PEP doesn’t currently say anything about the many installation modes that tools like
pip support. In particular, I think it should say something about
--prefix installs, and I would love it if it specifically called those modes out as exempted from the venv recommendation. If I’m doing a
--prefix install, I really don’t care about whether there is a venv loaded. I know what I’m doing. With this added, I don’t think the PEP will break us at all.
The PEP concerns me a little bit in that I think it can be misread (as I initially misread it) as encouraging every Python package to assume a virtualenv. This fear is possibly unfounded, but if people read the PEP this way, it will be really problematic for our model.
I am imagining a future where Python developers misinterpret this PEP to mean that installing every Python package in a virtual environment is somehow the “best practice”. Then I worry that packagers are going to start assuming a
venv in their
setup.py / whatever other build tool they’re using, and they’re going to tell us Spack people that our
--prefix installs are nonstandard and stop supporting them. I don’t know how they would make their packages break without a virtualenv (maybe they would require that the
.venv directory exists or something), but Hyrum’s law tells me we may start to see packages that simply don’t install outside of a
IMO that would be bad, and would stifle innovation by de-facto requiring projects like Spack and Conda to implement only the “standard” venv semantics, when we’re trying to do something more versatile. To be clear, I don’t think that’s the intent of the PEP, but I think it could be an effect of this PEP if we’re not careful how we couch it for packagers.
(minor) This language is a bit confusing to me:
This PEP recommends that package installers like pip require a virtual environment by default on Python 3.13+.
What’s an “installer like pip”? Is Spack one? Kind of… if you asked me for “tools like pip” I’d probably name
spack. But neither of those is likely to implement this PEP. So how should I interpret the first line of the PEP there if I want to try to play nicer with the Python community?
That’s it. I hope this is helpful and I’m happy to answer more questions. I’ll try to be better about participating in the discussions here .