Apologies. I knew the terminology was going to cause confusion, but I couldn’t come up with a better way to express things without being even more verbose than I normally am
When I said “scripts working on the same environment”, I meant “environment” in the sense of “managing the same database” or “providing utilities for the same server cluster” or “controlling the same service component”. I was not talking about Python virtual environments - in particular my point was precisely that there’s no reason such a set of scripts would necessarily use the same dependencies or the same virtualenv.
Sorry for the confusion.
Right. I was thinking some more about this last night, and I realised that this is the key point - the
[run] section assumes a single-venv-per-project model, which is certainly common, but which I’m strongly against mandating.
At the moment, our tools are relatively agnostic about project structure. Higher level tools like PDM, Poetry and Hatch are exploring project management approaches, but nothing is yet really settled - some people really like one or more of the tools, but there’s no clear consensus on either a tool or a project model (especially outside the “build a wheel” workflow, where I’ve searched for good “best practices” advice for years, and found very little that doesn’t essentially feel like “try to force the build a wheel approach to fit your needs”).
In addition, PEP 582 explored the idea of not using virtual environments at all. It wasn’t accepted, but my reading of the discussions was that the problem was abandoning the venv mechanism, rather than the idea that users shouldn’t have to deal with managing venvs.
I’ve personally been thinking about whether it might be possible for a tool like pipx to manage virtual environments differently, starting from requirements and combining different requirement sets where they are compatible to minimise the need for virtual environments. So users would just specify (per-script) requirements, and the tool would ensure that a minimal set of venvs was available to run them. It’s not even reached the stage of writing code yet, but it’s an illustration of a model where “one venv per project” is more constraining than I’d want.
Looking to the long term, my ideal Python development toolset would be one or more tools that used virtual environment technology “under the hood” to allow me to freely state, on a per script basis, what 3rd party packages I need to run the script. Environments would be managed for me, but would be visible if I want to look at the lower level (which I probably never would, but I’m averse to the idea of tools that work on a “no user serviceable parts inside” basis…) Imagine something like git, where no-one is expected to know how the data is stored, but the information and tools are there if you have the need.
Tools that bundle a project into some sort of distributable artefact would need to collect together dependencies - but this might be one main project script, or it may be a number of smaller scripts - imagine something like
coreutils, which bundles many commands into a single distributable project. It’s the responsibility of the build tool to merge the script dependencies into a single “project environment” (or maybe more than one, if that’s needed to handle incompatible requirements) - the user certainly shouldn’t be expected to worry about resolving the requirement set for themselves. Such tools might want a way to define the “project main script(s)” in
pyproject.toml, so they know what scripts to collect requirements for, but separating the two concepts like that is better (IMO) than forcing them into a single “project requirements” idea and losing the extra granularity. Maybe it’s my database design background showing through, but for me, the design is “a project IS MADE UP OF one or more scripts, and a script HAS zero or more runtime requirements”. Merging the two levels is a normalisation error.
The way I’ve framed the above, PEP 723 addresses everything needed here - each script records its own dependencies, and tools combine as needed. But in reality, projects do need other “groups of requirements”. A testing group is an obvious one. A debugging group might be another. Or doc builds. Many people use an environment manager like nox or tox for these, but having a central place to record the data that those tools use would be convenient for introspectability. So I don’t think it’s only scripts that need requirement sets. But I very strongly do believe that there’s no inherent reason to assume that there’s any privileged set of “project requirements” either.
So basically I’m -1 on the
[run] proposal because it implicitly ties us to the model of “one virtual environment per project”, which at the moment is simply convention and convenience, not policy. And I don’t think we have enough experience at this point to be sure that one venv per project is the right long-term solution. As I’ve already noted, hatch assumes and allows for multiple environments per project - do we really want to exclude hatch from the standard mechanism?
Looking at this another way, aren’t we really just talking about
requirements.txt? (I know we are, it’s been mentioned explicitly a number of times in this thread). But there’s never been a need for a project to only have a single
requirements.txt file, and in practice, many projects do have multiple requirements files. Sometimes they build on one another, sometimes they are independent. And sometimes requirements get hidden elsewhere, like in tox or nox configuration, or even in README files or “how to set up your dev environment” instructions. Trying to force all of those various requirement sets into a single
[run] section - or even singling out one such requirement set as deserving of being called
[run] and ignoring all the others - seems to me to be very short-sighted. It may solve an immediate problem (tools wanting to manage “the project venv”) but it will hold us back in the longer term. And I thought people were arguing that we should think more about the long term impacts of our standards - if this isn’t a case where we need to be cautious about whether we’re making short-term choices or looking at the longer term, I don’t know what is.