Projects that aren't meant to generate a wheel and `pyproject.toml`

Packaging
164

In my personal view (as opposed to PEP delegate, which was more the perspective of my previous post):

  • The momentum is absolutely a good thing, getting people motivated to be involved is crucial, we want as many perspectives as we can get.
  • PEP 723 is provisional, so IMO[1] it’s OK for us to require changes to that PEP if this discussion goes in a different direction.
  • The script use case is similar, but separate, from the case of a “project directory”. In particular, having PEP 723 enabled scripts inside a project is entirely reasonable, so if the two ideas overlap (which may or may not be the case) then we need to define the rules for how they interact.

As far as proposals are concerned, the ones I’m aware of are basically Karl’s, plus a number of variants of a section in pyproject.toml. It may be called run, but other names have also been suggested (for example, application). It may be of the form of run.dependencies.<name>, or run.<name>.dependencies. Other keys may be allowed, or not. Personally, I’m using [run] as a convenient shorthand for “put the data in pyproject.toml”. I don’t consider anything about how it gets stored in there as having been agreed on, PEP 723 notwithstanding.

And most importantly of all, no-one has yet done any significant work on describing actual use cases and how they would be handled with any of the proposals. There’s been a lot of talk about syntax, but very little about semantics.

For example:

  • When building a standalone application, how much of [project] makes sense, and where do we need a new key like [run]? So do we need to allow both? Do we need to look at what tools like pyInstaller do? If they infer dependencies from imports, how does that fit with the model of putting dependencies in pyproject.toml? Do we count that as out of scope, but if so, how do people transition from a “private” application to one they want to distribute?
  • How do we handle “run in place” projects with many ways of being run? For example, a webapp may have a debug version, or versions that run async or multi-process.
  • How do tools like tox get their configuration data from this new metadata? Not just requirements, but other information (like python version) needed to set up a run environment.
  • How do projects with multiple independent tasks structure their dependency data? Do we need some sort of inheritance (core data plus task-specific data)?
  • Requirements files are often used in projects that do build wheels (for test dependencies, or doc builds). How do we allow that use case (it would mean that [run] and [project] can both be present in a single pyproject.toml under the [run] model)?

I do not think that the argument “we define syntax here, that says what data is available, but it’s up to tools what they choose to do with it” is sufficient here. Like it or not, we have to consider semantics. And yes, it will be hard to do so, but going back to the survey, users want that level of well-defined behaviour from the packaging ecosystem.

To be honest, though, I think we would be better not trying to aim for the big picture here - I know other people want to think longer term, so I’m trying not to block that approach, but it does lead to long, complex and rambling debates where we find it hard to get consensus (the way this discussion is going!) So all of the above is not actually what I think we should be doing. I’d much prefer it if we simply tried to standardise requirements files:

  • Forget the whole “run a project” terminology and semantics debate. Actually, ignore the question of “what is a project” altogether.
  • Remove workflow and project types from the discussion, and focus solely on a scope of “if you currently use a requirements file, this is the replacement”.
  • Focus solely on lists of requirements. Introduce equivalents of pip-specific options cautiously, if at all.
  • Define a section in pyproject.toml that contains a series of key = [req, req, req, ...] entries.
  • Don’t worry about the idea of a “default” set of requirements for now, make all sets be named.
  • See how far that gets us.

Which is more or less what you suggested above. I’m sure someone will object to this on the basis that we need to think longer term, but :person_shrugging:

  1. although Brett is PEP delegate on that one, not me ↩︎

3 Likes