Projects that aren't meant to generate a wheel and `pyproject.toml`

Packaging
141

I meant that would be a changed requirement for non-pyproject.toml config files. But probably I got some details wrong in my understanding of the existing PEPs. Basically I want it so that pyproject.toml keeps working basically as it already does; and other config files have the same restrictions when they have a clear wheel target (which I define by: is there a [build-backend] table?) and a more relaxed spec otherwise (since a script runner realistically doesn’t need the name and version info). That is, other config files will not assume a default Setuptools backend, because the point is to use them for non-wheel purposes.

142

I think this proposal has a lot of interesting components but I’m worried that there’s too much other stuff going on in this thread. There’s also some questions about backwards compatibility that aren’t clear–in all it’s a pretty big change in how python metadata could work.

1 Like
143

Maybe I should split to a separate thread for my proposal, then? And maybe trim out most of the background discussion, and try to justify/explain it on its own terms instead?

144

Recently I’m thinking that just relaxing [project] seems like one of the simplest solutions.

One big lesson I see here for the future is that it pays to take some time to pick good names up front (a lesson we’ve also seen in the related discussion about stuff being referred to by PEP numbers instead of some actual descriptive name). It’s kind of annoying that we have this section that is called [project] but it really should be called [wheel] if it’s only meant for building wheels. At the least it seems like [wheel] could be added as an alias (even if the timeline for actually not accepting [project] is so distant as to be purely theoretical).

2 Likes
145

Thanks Stephen, I want to bring particular attention to the trade-off expressed here.

  1. I want to say very directly and bluntly: those who want a standardized way to define environments and want that defined in the same file as everything else are simply wrong. You do not want a single configuration file that is hundreds of lines long or more. You do not want to standardize the features that environment managers provide. Could this work for a subset of cases, maybe 20%? Sure, but that is a subset. I don’t think anyone that is expressing this view has actually seen a real world example configuration file for environments of an at least moderately sized project.
    • Black - 98 lines - tox
    • Hatch - 149 lines - Hatch
    • cryptography - 308 lines - nox
    • dd-trace-py - 2442 lines - custom environment manager because they outgrew tox, in the process of migrating to Hatch which will reduce line count by 70%
    • Twisted - 221 lines - tox
    • Flask - 42 lines - tox - note how much is going on there
    • Django - 89 lines - tox
    • Rich - 51 lines - tox - note how much is going on there
    • openai-python - 9 lines - nox - single-digit line count yet now you have to standardize a way to join dependencies from multiple sources
    • setuptools - 102 lines - tox
    • urllib3 - 167 lines - nox
  2. Why would we do this? Who is benefiting exactly? The only use case I have heard about this topic over the past few years is redistributors that wish to “test” a project as part of their pipeline.

@davidism Please split this into a different topic!

2 Likes
146

I’m okay with splitting this to a new thread, but it was my intent to contribute to the discussion of the run table – which seems like an important sub-topic.

You’re saying it’s wrong to try to put everything in a single file is wrong. I agree with this.
But what about just the dependency specifications?

It’s common with tox to have a number of extras defined in package metadata, and then for tox to define a matrix over those extras to test. That’s the separation I’d like to see – only, I don’t want to use extras for it because

  • extras require installing the package, which is often not correct
  • extras are published as metadata, which produces undesirable pressure to hide or remove them, or keep them small in number

Is the run table not a good place to tackle this? It seems like it to me, so if not, I have missed something important.

147

The reason why those tables and fields are optional is backwards compatibility and/or facilitate using modern API-driven build processes instead of legacy distutils-style without breaking completely (as much as possible) build for old distributions that have been published and are not 100% maintained.

Also, some tools have other configuration files that preceeded the existence of pyproject.toml , so forcing project on them would not be a good move.

I see that you say the "original pyproject.toml stays as it is which is good to avoid breaking the ecosystem. However trying to use the same “semantics” for multiple files, but that actually follow different rules, may be confusing.

Btw, could you please give some hypothetical examples of such files and their names?

148

Maybe not a perfect example – certainly not a MRE – but I would point you to the config we have in tox for webargs as an example of “mindeps” testing:

One of my areas of interest under this topic/thread is applicability to projects which do build wheels.
Libraries and applications often include run contexts like tests, linters, and support scripts which look similar to projects with no build.

149

I’d just like to point out how Jupyter projects handle testing against minimum dependencies. We have an Action that creates a wheel and parses the dependencies to create a pip constraints file, which we then use in our tests, allowing us to use >= and have the min versions only specified in one place.

1 Like
150

That looks really cool, and much more sophisticated than what I’ve been doing. I’ll take a look!

It’s worth noting that this is an ultra-specific case of a general problem:

Even projects which aim to produce wheels may contain components, tools in support of the main project, which may benefit from the addition of new ways to declare dependencies.

If there is a new space for explicitly unpublished metadata – that is, stuff that won’t go into the wheel for my library – doesn’t that sound like a good place for extras with purely internal purposes?

Today this can be achieved with requirements files. But those are not discoverable for tools or humans (worse for tools) without some explicit prompting.

Perhaps a reframing of this all to a question is useful:
What is different between the desired new section of project.toml and the currently possible contents of requirements files which only list dependency specs and no pip options?

What should be doable which can’t be done with requirements.txt, and what can be done with (one or more) requirements files which is not in scope?

151

I think this is a good point. There’s a clear and well-understood use case for requirements files, and while “no pip options” may be an issue in some cases (specifying an extra index for example) I think that we would get a lot of mileage from something that was nothing more than a mapping from a “name”[1] to a set of requirements. I don’t think it is directly linked to the idea of “projects that aren’t meant to generate a wheel”, as it’s just as useful for projects that do generate a wheel, but I think that’s less relevant if we’re talking about a feature that standardises existing functionality, rather than inventing something from scratch.

People have made a good case for some variation on requires-python. Although exactly what that means isn’t 100% clear to me yet. Is it meant to be linked to a specific set of requirements, or is it independent? To an extent that does depend on the use case, because it’s not something where we can copy existing functionality. If we’re thinking of each requirement set as its own environment, then having a requires-python per environment does make sense. If we’re thinking about test matrices, or layering requirement sets on top of one another, maybe not so much. I don’t know, because there’s no equivalent in requirements.txt.

One thing that’s definite, though, is that if we’re looking at this as being a standardised form of a requirements file, there’s absolutely a need for having multiple independent sets of requirements. There may also be a need for building one requirement set on top of another (the equivalent of using -r in a requirements file), although I don’t know how common that is, and it falls outside the “no pip options” restriction that you imposed.

  1. replacing the requirements filename ↩︎

2 Likes
152

A large part of the point of my proposal is that I don’t think that a [run] table is necessary if we have multiple config files. A wheel-builder could understand [project] to hold project metadata that it needs to build a wheel (and possibly other metadata that it doesn’t care about), and a script-runner could understand [project] to hold project metadata that it needs to run a script (and possibly other metadata that it doesn’t care about). It’s perfectly ordinary that metadata works this way: there’s one mechanism for specifying a whole structured piece of information about the core data, and different tools can take interest in the information that they need.

My idea is to eliminate the second drawback by having the sdist build process generate different metadata that takes out things irrelevant to the end user. (While I’m on the topic: it’s great that we can use the [build-system] table to specify a backend for building a wheel. It’s not so great that when I publish an sdist, I’m then demanding the end user to use the backend that I specified, which might be completely unnecessary. It might even still be a pure Python project!) Because a proliferation of config files, combined with the understanding that Pip only cares about pyproject.toml, would force that to happen. (In particular, in the case of a monorepo, I should be able to build separate sdists for the separate packages, and have each one end up with a pyproject.toml that’s based on the otherwise-named config file I used to describe the package.)

(I also don’t see how simply designing and using a [run] table would avoid that drawback…)

I think the first drawback is fake, and only a consequence of Pip’s current design. This metadata is still just metadata about dependencies - not a list of directives to install them, and certainly not an implicit directive to install the main package. Hypothetically we could have an --only-deps option for pip install, for example, or other tools could just take interest in this metadata and do what’s needed. (Although it’s not clear to me why you would set up a testing matrix, iterate to create environments, install dependencies in those environments, but not want to install the package being tested in those environments?)

I don’t think anything about my design really forces anything on anyone, except for sdist builders.

There would be no restriction on their names, except perhaps a .toml suffix. I will make a new thread and try to clarify the idea and give examples.

I think we may be at cross purposes here. I’m not trying to make testers, linters or support scripts put their specific config data into any particular TOML file, especially not into pyproject.toml specifically. Initially I thought it might be a good idea to think about enabling that, but now I don’t really like that.

On the other hand, here I think we’re on exactly the same page. To the extent that what you need to config is a dependency set, we already have a mechanism for that, and in my mind this is a perfectly reasonable use. Conceptually, “test whether the code will work with minimal dependencies” sounds to me like just a generalization of “build a wheel that specifies to use minimal dependencies, install it in a fresh environment (such that the minimal dependencies are installed, according to what the wheel says), and test in that environment”. Of course we can easily imagine ways to create the same environment that don’t have a wheel as an intermediate step. Since, again, the metadata is just metadata - not a directive to build a wheel - I think it should look the same for both cases. Conversely, if there are different metadata ideas to communicate about the same code - such as different sets of files/Python packages to consider, or different sets of requirement specs - then it makes sense to write separate files.

My answer is “so far, I can see no useful, consequential difference from the [project] section” - which is why I don’t want to design a separate new section.

Compared to current pyproject.toml, requirements files handle Pip-specific things, hashes and provisioning (which might also be Pip-specific in a sense). If requirements specifiers themselves are extended to include hashes, we’re IMO already most of the way there. Pip-specific stuff seems like it naturally belongs in the [tool.pip] section (at least for things that should be shared in sdists). Maybe provisioning does too.

In terms of purely installing dependencies, I can’t think of anything that should be doable that Pip doesn’t already do (if I could, I would be off making a new thread about it, I’m sure). By my understanding, requirements files are basically just automating the process of giving command-line arguments to Pip; I conclude that there is nothing that “should be doable which can’t be done with requirements.txt”.

At the risk of redundancy: In terms of proposed contents of a hypothetical [run] table, there are many things that aren’t covered by requirements.txt, but I think they are all already covered by [project].

153

But if I’m building a wheel with my project but I also want to record my test dependencies, where do I put them? If I put them in [project] as extras, they will end up in the wheel metadata, but if we’re going to force everyone to do that, that’s a whole other debate (there are people with strong feelings on whether things like test dependencies should be published in the wheel).

Instead of test dependencies, think of documentation dependencies, like Sphinx. You wouldn’t want to install the project just to build the docs.

No. Extras are not the same as requirement files, precisely because they aren’t published. I understood that when @sirosen said “extras with purely internal purposes” he was referring to non-published data that was like extras, but weren’t actually specified as extras.

Extras are a complicated mechanism with a lot of weird edge cases and problems (a huge part of pip’s resolver code is designed to deal with issues around extras). I would very definitely not recommend using them for this functionality. An unpublished “list of requirements” (like a requirements file) is a far better model IMO (because it’s simpler, while still being sufficient).

What would be the purpose of a project.license entry in one of your additional TOML files? How would you explain a project that had 3 such files, with 3 different licenses in them? Similarly for most of the other fields in [project] - they make no sense when you have multiple TOML files with different values in them.

The whole point of the discussion around a new section is that it doesn’t have to worry about the meaning of all the fields that are already defined in the project section. It just has to define the fields it needs and that’s enough.

1 Like
154

Either:

  • They go in a separate config file, that includes the necessary information for building for test purposes, while the main pyproject.toml includes the necessary information for building for distribution purposes. In this design, any such config file (not just pyproject.toml) can be used to build a wheel, as long as it specifies a build system (pyproject.toml also gets to use the default).

  • Wheel-builders simply add the functionality to skip [project.optional-dependencies.test] (or more general entries) when computing wheel metadata. Which largely amounts to the same thing, except that tooling is generating a virtual config file with everything except the “private” test dependencies, on the fly.

Even with the existing design, I don’t see why writing something in pyproject.toml should mandate that a wheel-builder includes some corresponding information in the metadata. By my reading of the relevant PEPs, tools aren’t being hand-held through the process of building a wheel. The PEPs say “here’s what may/must be in pyproject.toml and what it means; here’s what may/must be in a wheel’s dist-info folder and what it means; here’s the API that the frontend will call” - but connecting the dots is still the backend’s responsibility. In particular, PEP 517 says build_wheel must build a wheel and put it in a specific place, but AFAICT it doesn’t mandate any kind of correspondence between the contents of pyproject.toml and the contents of the wheel’s dist-info/METADATA. Similarly for prepare_metadata_for_build_wheel.

But tools already assume they’re entitled to write pyproject.toml, anyway, so there’s a loophole the size of the entire system there. (Most of them also seem to think they’re both the frontend and backend.)

Regarding people with strong opinions, I think we should support both styles regardless.

True. I don’t think this causes a major problem for my idea, but I’ll take it into consideration when trying to re-explain.

If I have a monorepo, I should be able to apply a separate license to each package that I want to distribute separately.

I would need to be convinced of this on an individual basis. Please let me write up a new thread (it will take some time, since I should go over all the relevant existing PEPs much more carefully, and also create some examples) and then we can see where the model doesn’t fit.

There are two completely separate topics here. One is the idea of having multiple configurations of code defined within the same set of .py files. The other is having the ability to specify dependencies for a script runner, as opposed to a wheel builder.

If I have a repository that contains one package, and it’s meant to be both runnable as an application (without installing a wheel and invoking an entry point) and installable as a library, I would normally expect to use just one config file for that, under my proposal. The name, version and license for the application should normally be the same as for the library. The trove classifiers mean nothing to the application, but that’s fine because a script runner can just ignore them.

If I have a folder with a loose collection of foo.py and bar.py and baz.csv, and I want to be able to share foo.py and bar.py with others separately, under separate licenses (and the reason that foo.py and bar.py are in the same folder is just that they both are meant to work on the same data), and they have separate requirements, then I could use separate config files for them (or inline updated-PEP-723 data). If they both only make sense as command-line tools, fine. If they both make sense to build into wheels and get imported by the next person, also fine; I just add the [build-system] tables as needed.

155

This came up in the PEP 722 thread, but I also lack some clarity on what it means – especially if it can’t be satisfied on the current system.

It seems like a directive for an environment manager. Perhaps proponents are expecting failures if a missing interpreter is specified, as you would get if you used a basepython in tox config which can’t be found.
Or maybe they’re wanting automatic installation (e.g., pyenv integration).

The field could be specified with a short list of possible behaviors, but I don’t think it should be specified without restricting/defining the behavior at all. Perhaps there’s a counterpoint here about allowing tools to experiment, but someone would need to champion it and put forth a way to define the field in the spec which is clear to users.

I think we should plan for this as a future extension but not include it from the start. I’ve seen -r in requirements files often enough to believe the utility is there, but rarely enough that I think many cases are covered without it. Unless someone wants to push for this, I think it will be easier to plan out as a standalone design effort once we agree on where the data even goes.

I see now that Karl asked a similar question about alternative solutions to [run] – my answer holds. It should be explicitly accounted for as a future extension, but I wouldn’t try to tackle it until the format and location of dependency data is established and settled.

It’s possible that after everything else is settled, we could return to this topic before a PEP is accepted. But the discussion is still too fluid, IMO, for us to be able to talk productively about this case.

The other item which I’ve seen mentioned for options in requirements data is package hash. To me that seems like a job for a lockfile, not the human-edited dependency data. So I’d tend to exclude that possibility for a tighter spec today.

My counterpoint would be that multiple files wouldn’t be necessary if we added the appropriate new section to the existing file with a standardized name.

Making the data tool-discoverable is important. I think that if you are going to explore and propose a new file structure, you will need to either define a directory name which must be used or define a way for project.toml to list the other files used.

Perhaps I’ve lost the boundary between what standards define and what pip does, but I believe that the issue is that there is no defined developer UX for referring to an extra of an arbitrary package but not the package itself. A flag to pip can’t change the fact that foo[bar] is a syntax with its own meaning. There’s no way to refer to the [bar] part without foo – and I’ve seen people asking for this run pretty hard into responses from the pip maintainers that even if we do add syntax for it, there are issues (which, disclaimer, I don’t understand) implementing it.

So in order to specify three dependencies – pytest, coverage, and pytest-xdist (as an example) – I must include all of the information from pyproject.toml in duplicate?

My reaction is “that can’t be what you’re saying… can it?!” Either I’ve misunderstood or we’re very, very far apart on this.

Some project needs have basically trivial requirements and some have elaborate ones. I don’t believe that they should all be weighted equally.

1 Like
156

I just want a useful failure if the script-runner can’t satisfy that requirement. Different tools might have different failure modes–something that can handle multiple versions can say “I can’t find python 4” or “you don’t have that one installed”. A tool that doesn’t do versions might say “this requires 3.14 and you only have 3.12”. As long as it communicates this requirement to the user, who is potentially not the person who wrote the script in question.

My impression is: you are allowed to define test.toml in your tests/ directory (or somewhere) and it can have a [project] section that only defines test dependencies. Either it inherits everything else from higher up the project tree, or it overrides things as needed, but it doesn’t need to repeat anything.

1 Like
157

My take is still that metadata is just metadata, and it’s up to any given tool to infer a course of action according to that metadata. I’m a proponent, and what I’m expecting is that the tool should do something that makes sense according to the tool’s purpose and which is explained in the tool’s documentation (in case there’s any possibility that the right thing to do for that tool isn’t obvious).

If the config says that python 3.12 is required - however that’s specified - possible behaviours are:

  • python and python3.11 commands would simply not care;
  • A hypothetical future version of py might treat the code as if it had a corresponding shebang line;
  • pipx might complain that 3.12 couldn’t be found on the system, or (in the future) automatically grab a 3.12 PyBI compatible with the system;
  • pyenv might try to activate an existing 3.12 venv before running the code;
  • python3.11 -m venv might issue a warning that the environment it’s about to create won’t meet the code’s advertised needs, and request confirmation[1];
  • wheel-building commands (including build and pip build, indirectly via whatever backend) would presumably just make the corresponding note in the wheel’s METADATA;
  • if trove classifiers are omitted or marked dynamic, a wheel builder could also generate a Programming Language :: Python :: 3.12 classifier;
  • if a contradictory python-version trove classifier is explicitly specified, a wheel builder could complain about that;
  • a linter could complain that certain __future__ imports are redundant if 3.12 is assumed;
  • A more complex CI/CD system could determine that it doesn’t have to build wheels that target earlier Python versions;
  • etc.

I don’t mean those to be authoritative! Just to give an example of the possible range of interpretation, and why I think it’s perfectly find to “specify it without restricting/defining the behaviour” - determining how to react to “this code is written for Python X.Y” depends way too much on what one is trying to accomplish with the code, and actually there are way too many possibilities for that, not all of them obvious; and whatever tool you’re using, it should be that tool’s responsibility to make the decision. I don’t understand why there is any difficulty in “defining” this. The word “requires” is perfectly ordinary English.

…And as it happens, we already have a key for this: [project.requires-python].

My proposal (I’m sure I had it somewhere in the original post) is that if you want the tool to use information from a different file than pyproject.toml, you use a command-line option to tell it which file to use. If it wants to generate its own config files (like Poetry currently does with pyproject.toml), keeping track of them is its problem.

That said, I’m not opposed to a mechanism for such files to reference, include, or partially include each other. I just haven’t designed one.

Okay. This is something I’ll have to ask them about, then. I was hoping this could be fixed at the Pip level. Alternatively, this could be a reason why people would actually want to try to develop alternatives to Pip specifically :wink:

In the worst case, if other approaches can’t be made to work. So in the next thread, I’d like to figure out what’s needed to make them work :wink:

There’s a strong case here that inheritance of metadata will be necessary. I’d like for the mechanism to be explicit, so I’ll have to think about it more.

  1. Well, of course this one wouldn’t actually happen as described, because the venv module for Python 3.11 already exists, and a change this significant wouldn’t be made in a patch release. But you get the idea. ↩︎

1 Like
158

In general I would expect python to not care because it doesn’t read config files–it might error on new syntax, of course. But we already have an example of a version-specific tool here: pip refuses to install a package when it isn’t running the minimum required version.

1 Like
159

The most radical version of this view is that we should have no restrictions on project.toml because it’s just a key-value store – it’s just metadata and tools can do whatever!

That’s obviously a very poor experience for developers, who must learn that “field foo means X under build tool Y” rather than “field foo means X”.

Giving fields clearly specified meanings ensures that developers learn one concept and carry that same meaning across tools. That doesn’t mean that the defined behavior can’t be expansive or even intentionally non-specific, but it must be defined. Vagueness is not a virtue in specifications, after all.

The current Requires-Python field (showing my bona fides by using the name in the metadata! :wink:) is well defined vis-a-vis dependency solvers. It makes a package version an invalid solution for certain target interpreter versions. If you want to reuse the same field in project.toml which populates this for a different purpose… well, I’m pretty strongly against it, but I think you’re going to be to reckon with the fact that you’re making the same config field mean two different (related! but different) things.

At present, I’m feeling pretty skeptical of the idea of multiple configs as standard (if a tool wants multiple configs of it’s own, that’s the tool’s prerogative). Given the breadth of impact here – the whole community, more or less – this is seeming to me like a major and complex change involving inheritance, overrides, append operations, changing field requirements, and some file discovery mechanism.

I’d like to try to stay open to this, but I’m weighing all of this complexity against a new table with one or two fields. By comparison a much more conservative change.

1 Like
160

I think I just don’t understand the idea of “different meaning” the same way you do :wink: That name means exactly what it sounds like, in all contexts: doing anything that involves bytecode-compiling and running the code, requires that a particular version (or range of versions) of python shall be used for those tasks. Obviously that has a different implication for a tool that tries to locate an existing version of python to run the code, versus a tool that tries to provision a new one, versus a tool that bundles up the code for someone else to run it. But those tools are all still understanding the metadata in the same way.

Well sure, but the point is that this thread exists because there are many more use cases to support than the ones you’re talking about - again referring back to post 92. (It’s certainly about a lot more than just whatever needs to be decided in order to unblock PEP 723.) I’m trying to describe something that can support everyone - with the understanding that 0 and 1 are legal values of “0 or more”.

1 Like