Please reconsider using [project] table for "project not meant to be a wheel" specification

There have been various opinions that rejects pyproject.toml [project] table for specifying the needs of a projects not meant to be a wheel (e.g., website, applications, a script file) — let’s call it application projects for short.

The dominant opinion is the [project] table is unsuitable fore applications because it requires the name and version field:


I don’t believe this opinion is strong enough to warrant rejecting [project] as a way to specify application projects. The stronger discontent with the status quo is not that these two metadata fields are awkward to use (in fact applications can have meaningful names and versions[1]), but because there are no tool out there that supports managing an application project specified with [project] table without these limitations:

  1. Intends the application project be packaged and requires a [build-system] table:
    • Rye’s rye sync which creates the project’s environment will fail without a [build-system] because it must install the project as a package
    • Hatch hatch env create which creates the project’s environment(s) ignores the project metadata when (see GitHub issue) if you do not intend to install the project (skip-install=true) as a package requiring the [build-system] table.
  2. Ignores relevant metadata in [project] table needed for the application environment:
    • Poetry’s poetry install --no-root does not need a [build-system] defined but ignores [project] outright and instead uses their own [tool.poetry] (see this popular feature request from 2020 to use [project] .
    • Pip-tools’ pip-compile and pip-sync can use [project] table for respectively defining and creating the application’s environment but it cannot enforce the python version as specified in project.requires_python .
  3. Requires extra runtime information to treat the project as an application project

From the list of tools, the closest one to use [project] table for application is PDM; Poetry is a tiny step away since a minimal [tool.poetry] table satisfies a minimal [project] table [2]). With the --no-self flag (--no-root in Poetry), they effectively specifies, creates, and installs the application environment.

I hope that convinces folks to reconsider and use [project] to specify application projects. In recent, the story has changed from wanting to “specifying runtime environment of an application” (i.e., [run] table) to generally accommodate different use-case environments (i.e., dependency groups). I understand that solving this application project would take multiple PEPs, but they do limit future options (see name and version is required). PEP 735, if accepted as-is, will change the landscape where people see [project] table only for packaging and [dependency-group] meant for application projects and others[3].


  1. Many have claimed their application project does not have a meaningful name and version. I find it hard to believe there isn’t a meaningful name for your application and we should encourage versioned deployments of applications (albeit not force; it’s probably too late now). Despite this difference of opinion, there are not real side effects of including this metadata in the use or deployment of your application. ↩︎

  2. As I’m writing this, I am happily surprised. I have never thought of Poetry as application-friendly when in the past I felt it was very packaging-oriented. ↩︎

  3. Would anyone care to entertain the idea of a [project].type field? ↩︎

2 Likes

+1 on better support for projects that aren’t intended to be packaged as wheels. But I don’t think [project] is necessarily the right way of doing this. Most of the fields in the [project] table are designed around building and publishing wheels and sdists. I think a different table - maybe something like [application], although not all projects are applications, either - would be better.

What’s your use case for the [project] table? Specifically, what fields would you normally use and why is it useful that they go under [project] specifically?

1 Like

My own answer: I use most of the required fields, and only version is particularly superfluous as I’m not really cutting releases. Personally I don’t find the version requirement that onerous, since it’s easy to put something there and leave it alone. The rest of the required fields are good practice for any kind of project, in my opinion: every project needs a name, a README, a brief description, etc.

It’s less about project being especially useful and more about the cognitive burden of remembering and deciding that there are multiple options, if something else takes over this use-case. Once I got used to the ecosystem, it felt natural to use the project table for any kind of project, and the fact that it does or doesn’t create a wheel is incidental.

Plus, this pattern makes it simple to convert to a package later, if that ever makes sense.

I don’t think this has much to do with PEP 735, though. I think dependency groups are compatible with any use of the project table.

7 Likes

Poetry seems to be working currently actively on a feature in this area: Introduce non-package-mode by radoering · Pull Request #8650 · python-poetry/poetry · GitHub

3 Likes

(Aside: I really need to make time to do more work on PEP 735, since I have ideas about allowing includes to cross with project.dependencies which need to turn into a proper spec.)

I contest this reading. [dependency-groups] doesn’t establish a space specifically for non-package projects!

Rather, it establishes a new namespace for dependencies – regardless of your project type – and carefully avoids requiring that a project be a package in order to use it. Necessarily, when you go from only having project to having project + dependency-groups, then anything which dependency-groups supports but which project does not is “only supported by dependency-groups.”

But that’s not the same as the new table being designed to satisfy “the same needs as project, but for different project types”. (And one of the problems with even trying to do that is that it’s been difficult to pin down exactly how non-package projects are shaped.)

There’s plenty of space to debate project table usage, or the addition of some parallel application or run table. But dependency-groups is intentionally positioned to be mostly orthogonal to those questions. I see it as a way for us to make progress even without trying to answer “what shapes of projects do we need to support”, and “can we support them all in one table, or one table per project shape?” and “are all of these project shapes mutually exclusive?” and so forth.

5 Likes

They are compatible with applications use case. PEP 735 (motivation section and others) is quite upfront on suggesting its proposal as (one of) the solution to the applications project. @sirosen (just now saw your comment as I’m typing this. I think the verbiage in PEP 735, as I interpret it, strongly suggests this)

So it comes down to which table does the project managing tool support for use with application projects. It could be both. I am not against dependency groups in general, but I’m afraid of the tooling picking sides. For example, Hatch only supports dependency groups (via their own table [tools.hatch.envs]) as a solution and recently removed using project table if your project intend to not be installed/packaged. Changes like these detrimentally affects my workflow (as it did last week) — application deployment and discussion and PEPs influences tooling to change (why I’ve made a seperate topic instead of continuing in this one)

To be fair to the tooling, I don’t criticize the them for taking a reactive stance changing their features as the climate around this topic develops. I do love those that attempt to set the standard by being proactive especially proposing PEPs.

Plus, this pattern makes it simple to convert to a package later, if that ever makes sense.

Big +1. BTW have you seen the amazing addition to the Packaging Guide on application packaging! :heart: you guys are awesome!!

Being part of a solution for these projects doesn’t mean that it replaces the existing project table–it is designed to augment the existing capabilities while maintaining compatibility.

I don’t think there’s any reason to expect the bifurcation you’re worried about.

Thanks for outlining its general intent. I very much support dependency groups and PEP 735 if you were to express that same intent more explicitly on your section on why project table is rejected in PEP 735 (by you, I of course mean to include all those involved in the draft)

If I were to rephrase why I wrote this topic, short, this is an request to tool maintainers to not forgo the project table as a valid path in the future. PEP 735 makes it so easy to do so.

You may be only thinking of [project] in terms of dependencies. If so, then I can somewhat see your point (I don’t think it’s a significant concern, but I can see why you’re asking the question). But the majority of the values in [project] are unrelated to dependencies, and [dependency-groups] doesn’t affect them.

2 Likes

You’re right, it’s mostly dependencies because that’s the functional parts of the project table for applications. Let me take the perspective of someone who has to deploy a Python application.

Wrapping back to your initial question…

Specifically, what fields would you normally use and why is it useful that they go under [project] specifically?

…for application deployment

  • Python version[1]
  • Python top-level dependencies (to produce lock files)
  • Perhaps controversial, the entrypoint field for global installs of non-packaged source code.
  • Tools table for anything else

Not surprisingly this looks a lot like inline script metadata spec (PEP 724).

For development of the application, dependency groups covers this use case[1:1].


  1. PEP 735 lacks specifying python version. ↩︎ ↩︎

I would recommend this post by Brett which sort of captures how the thought process around this has evolved until now: Differentiating between writing down dependencies to use packages and for packages themselves

1 Like

That’s an interesting perspective. Last year I also wrote a rant on what I think is misunderstood in what we declare as dependencies and where: multiple pip-compile runs with multiple python versions -> single requirements.txt · Issue #1326 · jazzband/pip-tools · GitHub. Perhaps, this would extend the context of the topic…

This all confusion is also causing us problems in updating a PyPUG guide that is attempting to explain some of those concepts: Update "install_requires vs. requirements files" discussion by jeanas · Pull Request #1427 · pypa/packaging.python.org · GitHub.

1 Like

Brett did a great job highlighting the disparity most who’ve deployed Python applications and developed packages are familiar with.

However I don’t agree with the conclusion:

It might also mean we either need to define a new table for pyproject.toml […] or we need a new file entirely separate from pyproject.toml […]

It’s a conclusion you’d make if you’re unwilling to change the purpose of project table or pyproject.toml:

pyproject.toml is truly meant for: as a TOML representation for the core metadata of a distribution.

Here I’m saying, there is a world where we can expand the purpose of pyproject.toml. Calling project table or pyproject.toml “is meant for only packages” is as awkward and confusing as requiring name and version for applications workflow.

3 Likes

On PyPI, the term “project” refers to a package name basically. It has never really had a direct relation with a term “application”. Instead, it relates to dists: Glossary - Python Packaging User Guide. There are a lot of problems in the ecosystem already that come from recycling the same terminology over and over again. In a disconnected context, it might make sense, but as things are right now — I don’t think it would.

2 Likes