Projects that aren't meant to generate a wheel and `pyproject.toml`

Packaging
81

Perhaps I’m looking at this too broadly (not just in the context of “projects that aren’t meant to generate a wheel”; though one way of looking at this discussion is that pyproject.toml has outgrown the “just the wheel, please” pattern); I was referring to

My point was that a putative [run] table could also be a candidate for projects that do create wheels.

Basically I responded to what I saw as a possible avenue for further unifying the various diverging scenarios. Ideally we could avoid something “like project.optional-dependencies”, and actually provide the same interface regardless of use-case.

I don’t want to turn your proposal into something you did not intend, but I found the [run] idea very appealing, and a new table is also an opportunity to make this API (more) consistent. Also, [library] and [application] already got mentioned, so I figured a bit of a wider angle is fair game.

We currently have at least: build-system.requires, project.python-requires, project.dependencies & project.optional-dependencies. and now potentially run.{...}. What I’m saying is that it might be possible to slice & dice this in a better way[1]. Obviously touching this for existing use-cases would mean migration pain (and we’d have to manage that), but I’m trying to take the long-term view here.

As a complete strawman for unifying the things under discussion:

[dependencies.build]     # optional; for projects building a wheel; ~= [build-system]
backend = "setuptools"   # PEP 517
requires = [...]         # PEP 518
[dependencies.run]       # PEP 621 & PEP 723
python = ">=3.9"
requires = [...]         # for wheel metadata (if applicable); for applications:
                         # base constraints that can be used to compile lock file
[dependencies.optional]  # PEP 621, resp. your proposal's run.dev-dependencies
tests = ["pytest"]
# your proposed extension for self-references
coverage = [".[tests]", "spam[tests]", "coverage"]

I was even tempted to put [dependencies.lock] in there, if only for exposition[2] :stuck_out_tongue_winking_eye:

That way, [dependencies.run] could be shared between libraries, applications & scripts, and scripts could reuse [dependencies.optional] where necessary.

It would be a fair criticism to say that the above strawman bites off more than a PEP might reasonably chew, but this is the kind of unification that I think pays off even such an effort in the long term (and that users are asking for).

  1. I re-read PEP 517/518/621 again, and those were solving crucial problems at the time, though that space was still brandnew, so it’s unrealistic to expect an eternal API from v1. ↩︎

  2. scripts will want to be able to lock dependencies too right? Should still stay single-file presumably, and with TOML we could inject the equivalent of PEP 665’s .pylock.toml very easily. ↩︎

2 Likes
82

I’m getting a bit confused on what use cases are under discussion here, so I’m going to back up a bit.

I find it useful to think of things like the dependencies in terms of how the mind of the programmer (i.e., the person writing pyproject.toml or similar info) engages with them. Let’s suppose I sit down and write some code that uses, say, numpy and requests. I write some imports and realize “hey, I am using numpy and requests”, and so I know I want to list numpy and requests among my dependencies in some metadata file. At some point I may realize that I need to specify a certain lower bound on the numpy version to ensure I have access to some feature that was added in a certain version, so then I know I need to update my dependency list to reflect that. And at some point (perhaps in a later release of my own code) I may discover that numpy made a breaking change, so I need to add an upper bound as well.

All of this is the same whether I am writing a library or an application. That’s why I’m uncertain about the need to separate the two cases. Now, it may be that for a library vs. application there is a difference in when or how this information is utilized: for a library I may send along just my dependency list, and the user’s resolver will calculate, at install time, precisely what to install; whereas for an application I may run a resolver beforehand and generate a precalculated set of all versions to ensure that different users won’t get different behavior due to having different environments. But still in both cases all of what happens flows from me as a human thinking “I am using this library”, and possibly some version restrictions, and me deciding to add those to my metadata.

In particular, I don’t think at any time “oh I am writing an application so I need exactly numpy version a.b.c”[1]; if that ever happens at all, it is derived from the more abstract, human decision of “what does my code need”. As far as I can introspect, I don’t at any time think about the dependencies differently depending on whether I’m releasing an app or a library, because I think about the dependencies as all being “what does this code need to run”. (In some sense it seems some of the app-vs-library question is maybe actually about the reverse, namely “what else might need this code in order to run”, and for an app the answer is “nothing because you don’t depend on apps” and maybe that affects the way the metadata is used.)

I’d imagine that what will be useful to users is a system that allows them to specify dependencies, as much as possible, in the way that they think about them in the course of development. Based on my own thinking above, to me this means not having to conceptualize application and wheel dependencies as different at the “root” level (i.e., a top level key like [project] or [run]), but rather to layer on some additional information about when, how, and by whom/what that information to be used. For instance some metadata that means “I just told you what my code needs to run, now I’m telling you that it is an app, so that means resolve that at build time before I distribute anything to anyone”.

That’s certainly possible! :slight_smile: So what I’m wondering is, in my thinking above, is there something that is “wrong”? If so, independent of this discussion, it might be something for me to note as a possible future doc improvement that explains how to think about dependencies and writing metadata.

I’m not sure I grasp that. For the user, an application cannot be run without first being distributed somehow, so I feel like we have to at least somewhat peer over the boundaries here to understand how this metadata will be used.

Based on what I said above, I still don’t really see the difference in terms of what needs to be given to a resolver. I do see that there may be a difference in when that is given to a resolver (i.e., at the build/distribution stage vs. at the end-user install stage), but in both cases, to me, the dependency information itself is the same at a conceptual level.

  1. unless maybe due to some odd combination of bugs there is only one version that has the combination of features I need, but that’s unlikely and in any case is really just a limit case of specifying upper and lower bounds ↩︎

1 Like
83

Because you’re looking at it from your perspective of an author, not a user of your code (regardless of whether we’re talking about a library or application).

If you’re using a library, then you’re not using pyproject.toml. A pyproject.toml file has zero interest to users of your library because they won’t have it; the information in the pyproject.toml got transformed into a METADATA file and that’s what gets shipped in the wheel.

But users of your application, if you send them an, e.g., zip file, will see the pyproject.toml file and potentially use it to see what extras it provides. And all of that stuff in [project] that has nothing to do with running your application (e.g., keywords, summary, description, etc.) are just a distraction or something that will probably confuse users of your application when they do a search online for python pyproject.toml project table and don’t understand why all of this wheel-related metadata stuff could be there but isn’t.

There’s a big difference in my mind between distributing in an official packaging sense and sending someone some code. A wheel is a distribution as it’s a standard binary artifact in the Python packaging ecosystem. Sending someone a zip file of source code is not a distribution in my head.

2 Likes
84

Well, an archive of source code is a source distribution at any
rate (and while it’s less popular these days, “users” do still
sometimes build and install applications from source… I mean, I do
that and then use the resulting software).

3 Likes
85

And “clone this repo and install it” is a common way of sharing code (in my experience at least), especially before it’s deemed ready for publication.

3 Likes
86

But in my conception, the idea is that the author writes down the metadata needed to make the code work, and then someone (either the author or the user) uses a tool to convert that into what the user needs. What we’re describing here is a sequence of steps that an author will take to record metadata and/or use tools to process that metadata. The user shouldn’t need to know anything whatsoever about any of these internal details; they may know literally nothing except the name of the application/library. So from the user perspective I would say all of what we’re talking about here is immaterial; they’ll mostly only care about whatever tools eventually use the metadata, not the metadata itself.

I tried to be careful to say “metadata”: rather than “pyproject.toml”, although I may have slipped up and said the latter (since that’s what we currently use for metadata). I thought the purpose of this discussion was to figure out whether pyproject.toml or something else is the way to express some metadata (like application metadata). If that is the case I prefer to think conceptually about what information is present at what stage, and then after getting clear on that we (or at least I, if I’m the only one who’s confused on this) can think about whether that is the same for the current library-focused pyproject.toml workflow as for a putative application-oriented system.

The person writing the code (hopefully! :slight_smile: ) knows what libraries their code needs to run, and they note that in some kind of metadata file. The person using it need not know any of that, neither from pyproject.toml nor from the wheel metadata (i.e., they as a human likely will not look inside those files). A tool which builds and/or installs the code may make use of the metadata the author created to somehow facilitate creating a valid environment on the user’s end to run that code. As I see it that accurately characterizes the current workflow with pyproject.toml. What I’m saying is that I see that workflow as compatible with distributing an application, perhaps with some changes in which tools are used at different stages, or how they work.

So, again, are we specifically considering only the “send someone a zip file” situation here? It seemed to me in the thread that people were considering application distribution, and I don’t consider “sending someone a zip file” as the only or even the best way to do that.

Even so, most of the stuff you describe is just as unrelated to using a library as it is unrelated to running an application. Keywords are just as unrelated to doing “import foo” as to doing “python foo.py”. And surely an application can have keywords, a description, and so on, just as much as a library can.

Well, for my part, there’s a difference (although maybe not so big) in my mind between “an application” and “some files I zipped up and sent to someone”. :slight_smile: It seems maybe there’s some misalignment in terminology.

One key thing is that for me an “application” is still something that is “deliberately” distributed, in the sense that the person distributing it is fine performing some build steps, assigning a version number, etc. So I think almost all of the metadata that’s in [project] is quite relevant for applications as well.

Another is that I interpret “distribution” pretty broadly. To me it just means “I have some code running in one environment and I want to get it successfully running in another environment”. So sending someone a zip file is something I absolutely consider as distribution.

Whether we call that distribution or not, I do think we need to think about it as a step that needs consideration, and that may require for instance the running of tools, even if those tools don’t do quite what pip install currently does. I’ve frequently used “guerrilla distribution” mechanisms (e.g., sending a zip file, cloning a repo) in the past, and in my experience it’s usually overoptimistic to think that literally just getting the code is going to be enough. There’s almost always some kind of configuration or setup required (e.g., “edit this file to point at your data directory”).

So when I think of “not generating a wheel” I really am just thinking of “not generating a wheel”; I’m still considering that the author may still need to take steps to prep the code for being sent to someone[1], and that the user may still need to take steps to make use of the code rather than just unzipping a file and immediately typing “python dostuff.py”.

Which makes me wonder about something I didn’t see in this thread, and I don’t remember seeing it in the thread this was split from either, which is: When someone has a project that they don’t want to generate a wheel for, why don’t they want to generate a wheel? What are the practical use cases that aren’t handled by wheels? I think we all have some notions about this but maybe it would be helpful to lay them out to get clear on what problems we’re trying to solve.[2]

  1. those may be “build” steps like packaging an application, or “maintenance” steps like keeping some metadata file up to date ↩︎

  2. As an example, I have seen people shy away from making a wheel because they don’t want to publicly publish their code, and they associate wheel-building with publishing to PyPI, not realizing they can do the former without the latter. That’s something that could maybe be handled just with documentation improvements. ↩︎

87

I have definitely written many applications that have never been distributed, and were never intended to be. In both professional and hobby contexts, and in Python and other languages. But maybe our definitions of “distribution” differ.

Well, I wouldn’t consider moving an application from a test environment to production as “distribution”, to give a specific example. If you do, then I guess you’d say a lot of my applications get distributed, but I disagree.

I’d distinguish based on intent. If I send them a zip of my working code, I’d consider that more like “sharing” than “distribution”. For me, if I’m distributing something, I’d expect to send something that includes instructions on how to set it up in the user’s system. I’d expect to commit to supporting the user if they have problems getting the application to run. I’d even assume a certain responsibility for helping if the application doesn’t work the way they expected (although how much responsibility varies case by case, based on my relationship with the recipient). Maybe that’s because I’m used to working in a more “commercial” context, but I do think that it’s important to distinguish between a more formal idea of “distribution” as distinct from a more informal “sharing”.

For me, the biggest reason is usually because I don’t want to require the user to install the code. I’ll often send something that only needs to be run once, or occasionally. Download and run is very definitely the model I’m aiming for. The user I’m sending it to may not understand Python environments, or know about virtualenvs. I might want to avoid having to deal with a user having an incompatible version of one of my dependencies. I might not even want the user to have to know the application is written in Python, for a larger project.

Let’s put this another way. Mercurial is an application, written in Python. They don’t (primarily) ship as a wheel, because their users don’t want them to. That’s a perfectly valid use case, and one I’d like to see supported by the “packaging ecosystem”. It means getting some sort of integration with, and buy in from, projects like pyInstaller and BeeWare, but that’s the reality of “packaging Python applications”. If we ignore this, it’s very hard to credibly say that we’re listening to Python users - survey or not.

3 Likes
88

I’ve been talking from the perspective of replacing pip’s requirements files. I personally never got into application distribution ala Briefcase. Worrying about distributions somehow is an expansion of what this topic was originally about, stemming from, I think, the analogy to wheels and how this is all meant to be a different use case.

I personally just want a way to write down the requirements my code has in order to run. People objected to reusing the [project] table back in A look into workflow tools - package management in the Python extension for VS Code because it felt like a re-purposing of [project] for something it wasn’t meant for. This discussion got split off to try and come up with something separate from [project] for this use case of replacing pip’s requirements files.

I proposed a simple strawman idea in Projects that aren't meant to generate a wheel and `pyproject.toml` - #56 by brettcannon . Some people didn’t like the suggested table and key names, so a slightly tweaked alternative was also proposed by me. I think the only other proposal is from @h-vetinari at Projects that aren't meant to generate a wheel and `pyproject.toml` - #81 by h-vetinari .

Unfortunately, even after taking the time to write a blog post to try and motivate why I didn’t think reusing [project] was necessarily the right thing, I’m still spending most of my time trying to clarify this point. That tells me that either I’m personally failing horribly at explaining all of this or the concept is just not clear enough on its own to explain in general and thus the idea should just be dropped due to difficulties in teaching it.

I was hoping this could all get resolved before I needed to make a decision about those PEPs to potentially help inform what PEP 723 would want embedded a single-file script. But if that’s simply not going to happen then so be it as I’m not holding up making a decision about those PEPs for this because if we aren’t heading towards consensus/decision after 87 posts and a month of very active discussions then we might not ever come to a conclusion here and thus status-quo wins.

89

I guess what you call “sharing” is more like what I’d call “distribution” and what you call “distribution” is more like what I’d call “publishing”.[1]

Regardless of the terminology, I agree that the factors you mention are relevant. At least for me, though, even the informal “sharing” situation often has to involve some amount of instruction for how to get things working (even if it’s just “unzip this and run dostuff.py”). I suppose there is a gradient of how much apparatus (instructions, support, etc.) is attached to code that is shared/distributed/published.

When you say “don’t want require the user to install”, are you referring only to an install via Python mechanisms like pip? I think I know what you mean, but there is a sort of uncanny valley in terms of how things may be installed. At the level of casually sharing code among tech-savvy users, the “simplest thing” may just mean “copy this file and run it”. At the intermediate level of “formally” sharing code with tech-savvy users is where we live in the pip/wheel/pyproject.toml world, which seems to be the world we’re trying to step slightly outside of in this discussion.[2] But often, in the wide world of people who know nothing about Python or programming, installing the software is exactly what the users do want to do, and is viewed by those users as more simple than just opening a zip file — and that’s why tools like pyInstaller and Beeware exist. As you say, these users don’t want to know or care that they’re installing Python code, but there is definitely an install process, and it may in some ways be even more technically complex than pip-type installs in that it has to integrate smoothly into user expectations across different OSes, etc.

I totally agree that this latter kind of installation (what we might call Python-agnostic installation) is currently separated from the world of Python packaging by an immense chasm, and it would be great to bridge that. One way is integration with tools that package Python apps to be installed as “normal” programs for whatever platform, as you mention. Another idea is some kind of distribution platform for Python apps that is easy to use even for non-technical users; this model has been very successful for games via Steam, and Anaconda Navigator has a similar mechanism for installing applications (as distinct from Python packages).

Getting bad to the pyproject.toml matter, though, the question for me is again whether anything really needs to be different in terms of how the author initially specifies the dependencies. That is, if we are aiming at something like pyInstaller or Beeware, I see it as feasible for such tools to take the same kind of dependency information that’s currently specified for a wheel, and use it to package up a Python-agnostic installer.[3]

  1. Publishing, of course, can still mean commercial publishing, in addition to things like PyPI. ↩︎

  2. And, personally, often the reason I don’t want to subject users to that install process is precisely because of the problems that exist with Python packaging :upside_down_face: . ↩︎

  3. It’s been a while since I’ve looked at those tools, but my recollection is that this is more or less how they currently do things anyway. ↩︎

90

At the same time, it’s good if project.dependencies constraints remain abstract instead of pinned. And when we do that, it is the same again for applications and libraries.

A trickier part here I think is that (as I wrote in an earlier comment) when we’re not building a wheel, the application may not contain dynamic dependencies, but when building a wheel we do.

91

A project may contain a library and an application. They may not have the same dependencies. E.g., the application may only be a subset of the library dependencies (or the other way around if you use packages such as click for your cli).

Maybe we’re going a bit too deep here but it does matter. E.g., in Nixpkgs we often split packages, moving the executables elsewhere, because depending on your use case you may not need them and thus also not certain dependencies.

4 Likes
92

On the assumption that with the provisonal acceptance of PEP 723, we’re going to revisit this discussion, can I suggest that we start by thinking about what workflows we want to support here. I can see the following potential arrangements of “a directory with a pyproject.toml and some Python code” that could be candidates, and I think it’s worth being very explicit about which we support. Ideally, we could also do with agreeing some terminology - “project” is over-used, “application” seems to mean different things to different people, and “project that will be built into a wheel” is clumsy…

  • A directory that will be built into a single wheel. That wheel may be intended as a library, or as an executable application that is run via an entry point and installed using something like pipx, but either way the key is that there is a build process that generates a wheel.
  • A directory that contains the source code for a standalone application. This is also built, but not using a typical build backend, and the resulting distributable artefact is a runnable binary of some sort. GUI applications often work like this, and CLI programs would probably work well this way, but the overheads involved often mean developers prefer to distribute CLI applications as wheels (see previous case).
  • A directory that contains an application that is run “in place”, typically using some form of management script. Web applications often take this form.
  • A directory where the developer executes many inter-related tasks, all aimed at one fundamental goal, but often independent. Some tasks may not even be written in Python, and the tasks may have very little in common beyond the overall goal (data acquisition, cleansing, analysis and reporting tasks are very different, for example). Data science projects often take this form.
  • A directory containing scripts all working on the same environment, but often with very little else in common. This may be the same as the previous case, although the lack of a unified goal may be significant. Automation projects often work like this.
  • A monorepo project, containing multiple sub-projects. I don’t know much about this workflow, except that such projects seem to have trouble even with the existing capabilities of pyproject.toml.

There are probably other common workflows (scientific analysis may well be another - I’m assuming it’s similar to data science, but my experience of scientific programming is very limited, so that’s a guess).

Key features of these different project types that we need to consider include the following:

  1. Some types want fully pinned dependencies, where being able to reproduce one or more exact runtime environments is critical. Other types want broad dependencies, to work in as many environments as possible. Pinning is part of the build (or project management) step for the former, and the install step for the latter (at least in types that have build or install steps). Some types may not particularly care - reproducibility is not always a key feature in every project.
  2. Some types have one clear “thing” that is executed to use the project. Others have multiple scripts, commands and processes.
  3. Some projects may have a single license/readme/maintainer, others may not.

We may or may not choose to support all of these project types with pyproject.toml. I don’t think it’s necessarily a given that it’s a suitable solution for everything. But we do need to be clear what our target use cases are, and right now, I get the feeling that a lot of the confusion in the discussion comes from people talking from the perspective of different (and possibly even incompatible) use cases, without realising it.

6 Likes
93

Yes, please!

To clarify, are we talking about what workflows we want pyproject.toml to support now, or what we eventually want pyproject.toml to support? I’m going to answer for the “now” case to keep the scope down (i.e., specifying dependencies for something that isn’t meant to result in a wheel).

This is when I think of the use cases for pip’s requirements files:

  • Listing the top-level dependencies
  • Nesting dependencies (e.g., requirements-dev.txt references requirements.txt)
  • Pinning via pip-tools

I consider these separate concerns and from the scope of PEP 723 not critical to solve right now. I do acknowledge, though, that the execution bit has come up before.

My choices would be:

I assume we consider this taken care of.

+1

+1

If we have to care about other languages, -0, otherwise +0 (although I’m not sure how much that differs from the “automation” example in that instance).

+1

-0

This one is tricky simply because there isn’t a universal definition of how a monorepo should be structured or work. Plus there isn’t a guarantee the monorepo is Python-oriented which ties into your “data science” project, just at a bigger scale.

From a PEP 723 perspective, these all require some way to specify at least your top-level dependencies.

94

I disagree. The “collection of scripts and processes” model, in my experience, does not require a single set of top-level dependencies, or a single “do everything” virtualenv. It’s often managed that way, because handling multiple venvs is difficult, but we should not be designing formats which mandate that single-venv structure. Rather, we should be trying to make it easier to use a venv per script/process, when appropriate. Consider hatch environments, where a project could have many environments - in PEP 723 terms, which environment gets to be the one that counts as “running the project”?

4 Likes
95

But isn’t that different from “a directory containing scripts all working on the same environment” (emphasis mine). A collection of scripts that each have their own dependencies and environments is a different beast.

Maybe “one pyproject.toml == one environment” is a reasonable scope to start with. If something doesn’t fit into that mold it can wait until later.

5 Likes
96

Doesn’t PEP 723 already handle a collection of scripts workflow. Each script defining a PEP 723 can be their own environment or be resolved into one environment—left up to the PEP 723-aware script installer/runner.

97

James’ interpretation is how I took that workflow. I was also commenting on the ones I personally thought pyproject.toml should support (so +0 and +1 in my list); sorry for not explicitly stating my comment that way.

I think it definitely could, but the discussion here is whether that is how we want to always solve this use case or whether we want at least the option to solve it with pyproject.toml so we know what goals we have for pyproject.toml.

98

I’d say the more on that list that can be supported, the better. :slight_smile:

To slice it another way, I find it useful to think about what the user of the distribution is going to do, and maybe also what they expect to get and how. I’m going to use the term “bundle” here just to refer to this thingamajig that the user gets.

Specifically, we can think about:

  • Does the user know Python?
  • Does the user have Python already installed?
    • If so, do they want to use an already-installed Python to use the bundle?
  • Are there other non-Python things that need to be installed (e.g., system libs)?
    • If so, does the user responsible for making sure those are available, or do they expect the bundle to provide whatever is needed?
  • Will the user use the bundle by. . .
    • writing import blah in a Python source file?
    • typing python somefile.py a command prompt (or maybe something like jupyter somenotebook.ipynb)?
    • typing blahcmd at a command prompt?
    • using some standard GUI that the OS provides (e.g., clicking an icon on the desktop or start menu)?
    • connecting it up to a webserver to run as a web app?
    • accessing it via some other software (e.g., as a plugin within some enclosing app)?
    • something else?
    • some/all of the above?
  • Is the user intending to use the bundle “systemwide” (e.g., as an installed app) or might they want to have multiple instances of the bundle that are independent of one another (e.g., different versions of a library in different environments)?
  • Does the user expect/hope to get the bundle by. . .
    • having someone email them a zip file?
    • typing a command like pip install at a command prompt?
    • doing something like a git checkout and working directly with the resulting files (e.g., no separate “install”)?
  • Is the user expecting to be able to modify the bundle somehow and send it to someone else (this is for instance not uncommon in data science or academia)?

Maybe some of these are too hairy to be handled as part of Python packaging, but I think if we’re talking about “applications” it’s worth at least considering each combination closely.

Right now as far as I can see, the wheel/PyPI system always assumes that:

  • The user already has Python installed (and wants to use an existing install for the bundle)
  • The user intends to use the bundle by typing import blah in a Python source file
    • . . . or by typing blahcmd at a command prompt, but to do this in a non-venv-bound way requires requires the user to be aware of and use a separate component like pipx to install it (that is, the bundle itself cannot say “I’m probably supposed to be installed into my own environment”).

In practice the process also mostly assumes that the user knows some amount of Python; in theory they could know nothing but how to type pip install blah, but I don’t see many cases where someone who knew only this would thereby be able to install something they could realistically use without knowing Python.

Obviously, as has been mentioned, an important omission here is the case of “I wrote a full-fledged GUI app (or even just a terminal app) and I want to be able to give it to people who don’t even know what Python is”. Of course, it’s possible to do that, but to do it you have to venture rather far afield from the “official” Python packaging tools.

1 Like
99

Apologies. I knew the terminology was going to cause confusion, but I couldn’t come up with a better way to express things without being even more verbose than I normally am :slight_smile:

When I said “scripts working on the same environment”, I meant “environment” in the sense of “managing the same database” or “providing utilities for the same server cluster” or “controlling the same service component”. I was not talking about Python virtual environments - in particular my point was precisely that there’s no reason such a set of scripts would necessarily use the same dependencies or the same virtualenv.

Sorry for the confusion.

Right. I was thinking some more about this last night, and I realised that this is the key point - the [run] section assumes a single-venv-per-project model, which is certainly common, but which I’m strongly against mandating.

At the moment, our tools are relatively agnostic about project structure. Higher level tools like PDM, Poetry and Hatch[1] are exploring project management approaches, but nothing is yet really settled - some people really like one or more of the tools, but there’s no clear consensus on either a tool or a project model (especially outside the “build a wheel” workflow, where I’ve searched for good “best practices” advice for years, and found very little that doesn’t essentially feel like “try to force the build a wheel approach to fit your needs”).

In addition, PEP 582 explored the idea of not using virtual environments at all. It wasn’t accepted, but my reading of the discussions was that the problem was abandoning the venv mechanism, rather than the idea that users shouldn’t have to deal with managing venvs.

I’ve personally been thinking about whether it might be possible for a tool like pipx to manage virtual environments differently, starting from requirements and combining different requirement sets where they are compatible to minimise the need for virtual environments. So users would just specify (per-script) requirements, and the tool would ensure that a minimal set of venvs was available to run them. It’s not even reached the stage of writing code yet, but it’s an illustration of a model where “one venv per project” is more constraining than I’d want.

Looking to the long term, my ideal Python development toolset would be one or more tools that used virtual environment technology “under the hood” to allow me to freely state, on a per script basis, what 3rd party packages I need to run the script. Environments would be managed for me, but would be visible if I want to look at the lower level (which I probably never would, but I’m averse to the idea of tools that work on a “no user serviceable parts inside” basis…) Imagine something like git, where no-one is expected to know how the data is stored, but the information and tools are there if you have the need.

Tools that bundle a project into some sort of distributable artefact would need to collect together dependencies - but this might be one main project script, or it may be a number of smaller scripts - imagine something like coreutils, which bundles many commands into a single distributable project. It’s the responsibility of the build tool to merge the script dependencies into a single “project environment” (or maybe more than one, if that’s needed to handle incompatible requirements) - the user certainly shouldn’t be expected to worry about resolving the requirement set for themselves. Such tools might want a way to define the “project main script(s)” in pyproject.toml, so they know what scripts to collect requirements for, but separating the two concepts like that is better (IMO) than forcing them into a single “project requirements” idea and losing the extra granularity. Maybe it’s my database design background showing through, but for me, the design is “a project IS MADE UP OF one or more scripts, and a script HAS zero or more runtime requirements”. Merging the two levels is a normalisation error.

The way I’ve framed the above, PEP 723 addresses everything needed here - each script records its own dependencies, and tools combine as needed. But in reality, projects do need other “groups of requirements”. A testing group is an obvious one. A debugging group might be another. Or doc builds. Many people use an environment manager like nox or tox for these, but having a central place to record the data that those tools use would be convenient for introspectability. So I don’t think it’s only scripts that need requirement sets. But I very strongly do believe that there’s no inherent reason to assume that there’s any privileged set of “project requirements” either.

So basically I’m -1 on the [run] proposal because it implicitly ties us to the model of “one virtual environment per project”, which at the moment is simply convention and convenience, not policy. And I don’t think we have enough experience at this point to be sure that one venv per project is the right long-term solution. As I’ve already noted, hatch assumes and allows for multiple environments per project - do we really want to exclude hatch from the standard mechanism[2]?

Looking at this another way, aren’t we really just talking about requirements.txt? (I know we are, it’s been mentioned explicitly a number of times in this thread). But there’s never been a need for a project to only have a single requirements.txt file, and in practice, many projects do have multiple requirements files. Sometimes they build on one another, sometimes they are independent. And sometimes requirements get hidden elsewhere, like in tox or nox configuration, or even in README files or “how to set up your dev environment” instructions. Trying to force all of those various requirement sets into a single [run] section - or even singling out one such requirement set as deserving of being called [run] and ignoring all the others - seems to me to be very short-sighted. It may solve an immediate problem (tools wanting to manage “the project venv”) but it will hold us back in the longer term. And I thought people were arguing that we should think more about the long term impacts of our standards - if this isn’t a case where we need to be cautious about whether we’re making short-term choices or looking at the longer term, I don’t know what is.

  1. Actually, pipenv has also been working in this area for years. ↩︎

  2. Yes, [run] could model hatch’s default environment, but we’d still need [run] equivalents for other environments, so why not put them all on an equal footing with the default just being a distinguished name, like hatch does? ↩︎

4 Likes
100

This would be amazing. I’m the “casual user” and have never plunged into using venvs.

I can remember to specify dependencies, but please let’s get a tool included that handles the rest for me!