Storing requirements for tasks in pyproject.toml (drafting a PEP)

[kknechtel (Karl Knechtel)]

@rgommers

you should very clearly and concisely write up the current problems / pain points it’s trying to solve.

Of course. A PEP is supposed to have a Motivation section, after all. I’ll go over the bullet points here while I think about how to write it properly (I’m not sure if this is exhaustive, but it should be pretty close):

• pyproject.toml seems like the one obvious place to put metadata about a project.
  • Everyone’s already using it that way anyway.
  • Nothing about the name [project] exactly screams “this corresponds one-to-one with entries in the METADATA file of a wheel”, but according to PEP 621 it all does (except for [project.entry-points], but that’s still specifically wheel-oriented).
    • On the other hand, many of those names make perfect sense in other contexts anyway.
  • Consequently, people expect to be able to store general metadata there - in particular, lists of dependencies.
• Lots of users have “projects” that don’t fit the mold of the standard packaging flow.
  • “Project” is quite loosely defined overall, but we have some ideas about common forms they take.
    • Different kinds of projects will involve varying tasks, but again there are a few easily identified common tasks.
  • There’s a broad perception that packaging is hard.
    • Learning about wheel metadata is intimidating and unnecessary if you only care about requirements.
    • It’d be nice to be able to ease the transition - this was a huge point of contention in the discussion surrounding PEP 722/723.
    • One great way that could work is if users didn’t have to worry about the classic concept of packaging before starting to use pyproject.toml for its apparent natural purpose.
  • Many people don’t want or need to build wheels at all.
  • Even more people need to do other things besides build wheels.
    • Practically any “task” one could name here could reasonably have a list of dependencies.
• However, pyproject.toml was designed to store specifically information used by the wheel-building process. It isn’t readily extensible to describe contexts other than wheel-building.
  • Some people try to work around this with requirements.txt; this comes across as clunky and ad-hoc (and Pip-specific).
  • Some try using Pipenv and the corresponding pipfile and pipfile.lock - this has its limitations, but more importantly it still scatters the metadata to separate, tool-specific files.
  • Some try [ab]using the “extras” mechanism - this comes with numerous downsides.
  • Various tools use the completely free-form [tool] table to express their opinions about what the metadata should look like. There’s no standardization or collaboration; compatibility is needlessly sacrificed.
  • Basically, every workaround feels “proprietary” even though it’s in plain sight.
  • Even many of these workarounds don’t make distinctions that would be useful to make, e.g. between various kinds of “dev dependency”.

Probably not all of that is directly relevant; I’ll need to go through some revisions, of course. Some of these thoughts probably belong in the “Rejected Ideas” section instead.

---

@rgommers

This proposal looks like it adds a lot of things to pyproject.toml, many redundant with keys already present.

It looks this way because I’ve detailed semantics for a bunch of keys, all of which fundamentally work the same way.

First I should show an example, in case I accidentally oversold the complexity:

[required-to]
install = ["pandas"]
# Web requests will be mocked in testing; don't include this in test envs.
use-wheel = ["requests"]
build-doc = ["sphinx>=7"]
test = ["pytest>=6,!=7.1.0"]

# Define the "make-fancy-videos" extra for our project.
[required-for.make-fancy-videos]
# This also empowers e.g. `future-pyrun ourproject[make-fancy-videos]`.
install = ["Pillow", "imageio[ffmpeg]"]
# We need sophisticated algorithms to verify the accuracy of video output.
test = ["scipy"]

In the description I used some more dots in the names; that’s just an alternative way of describing the TOML structure.

This proposal is primarily in response to:

@pf_moore

I’d much prefer it if we simply tried to standardise requirements files:

…

• Focus solely on lists of requirements. Introduce equivalents of pip-specific options cautiously, if at all.
• Define a section in pyproject.toml that contains a series of key = [req, req, req, ...] entries.
• Don’t worry about the idea of a “default” set of requirements for now, make all sets be named.

… except that I realized that the conditions to select a list of requirements could depend on both the “task” (the fundamental reason why the requirements are required) and the extras configuration, and it seems reasonable to treat them as mostly orthogonal.

Initially I basically just wanted to say that here is a namespace where you can put some lists of requirements; the lists work the same way as existing examples; here’s how the namespace is organized. But I got a pretty clear impression that that would not fly, e.g. (admittedly in a different context):

@sirosen

The most radical version of this view is that we should have no restrictions on project.toml because it’s just a key-value store – it’s just metadata and tools can do whatever!

That’s obviously a very poor experience for developers, who must learn that “field foo means X under build tool Y” rather than “field foo means X”.

Giving fields clearly specified meanings ensures that developers learn one concept and carry that same meaning across tools. That doesn’t mean that the defined behavior can’t be expansive or even intentionally non-specific, but it must be defined. Vagueness is not a virtue in specifications, after all.

As regards the redundancy, this is because I see the existing design as non-extensible in a place that needs to be extensible; since solving the problem requires building something new anyway, I designed it for parallelism. In this approach, the task of building a wheel is treated essentially the same way as other tasks, rather than assuming that wheel building is the default, dependencies related to installed wheels

It’s also because I think it’s important to distinguish between wheel contents that are vs. are not expected in a test environment. ([project.dependencies] can’t do that; so currently users are stuck either creating test environments that include libraries for functionality that will be mocked out anyway, or using some totally custom, parallel setup to describe dependencies in the test environment vs. the wheel.)

That said, I’m open to considering other ways to arrange the data. For example, I considered doing it this way to avoid duplication:

[project]
dependencies = ["pandas"]
# Web requests will be mocked in testing; don't include this in test envs.
wheel-dependencies = ["requests"]
doc-dependencies = ["sphinx>=7"]
test-dependencies = ["pytest>=6,!=7.1.0"]

[project.optional-dependencies]
make-fancy-videos = ["Pillow", "imageio[ffmpeg]"]

[project.optional-test-dependencies]
make-fancy-videos = ["scipy"]

But:

• I felt these names were less clear and read less naturally;
• Aesthetically I don’t like that non-wheel tasks come across as second-class citizens;
• The “dependencies” tag gets spammed (to avoid conflict with other [project] keys and make the purpose clear) - namespaces are one honking great idea etc.;
• The extension to extras seemed especially inelegant.