PEP 621: round 3

I don’t want to unilaterally act on setuptools’ behalf (particularly since, especially lately, @jaraco has done something between the lion’s share and all of the work), but to the extent that my opinion is what sways things, I plan to advocate for implementing the configuration file spec from PEP 621 regardless of whether or not it’s accepted (modulo table name). I think it’s a Good Enough Design™ (especially considering that it was designed by committee…).

The only reason I’m suddenly down on this PEP is that the latest iteration changed it into an sdist standardization PEP.

If we remove the sdist standardization stuff, I can see some marginal benefits of PEP 621 being accepted, but I can also see Paul’s point that standardizing the input is not that important, since the interop benefits are kinda weak, and without the interop benefits, it doesn’t need to be a standard, since anyone interested can just adopt it or not.

1 Like

It wasn’t a typo, it was me misremembering that pipenv doesn’t produce any binary artifacts.

Thanks, and I understand you can’t necessarily unilaterally speak for setuptools (unless @jaraco also chimes in :wink:).

@takluyver what do you think of the PEP?

1 Like

I would have thought one of the biggest reasons for standardizing the specification of a project’s metadata is the benefit to users, both project authors and library users. If in 5 years’ time 98% of all Python projects had the same way of specifying author/dependencies/entry-points/etc, then less Python-literate visitors to repos would have a better time analysing a prospective package to include in their application. Was this ever the case with setup.py?

1 Like

98% (made up number :-)) or projects already have a standardised way of specifying metadata, it’s setuptools. (Apologies to flit and Poetry if that 2% I left you is too small :slightly_smiling_face:).

What’s useful about PEP 621 is that it’s an introspectable standardised way.

My argument is that introspection is more often done on sdists than on original source trees, so having PEP 621 data be reliable only in source trees is relatively unimportant. @pganssle’s argument is that making PEP 621 data be as complete as possible in sdists subsumes sdist metadata standardisation.

1 Like

If a file is considered reliable in source trees, why wouldn’t this same un-updated file also be reliable in sdist ?

1 Like

To avoid derailing PEP 621, I’ve made a proposal here for taking the idea of dynamic and adding it to the core metadata for use in sdists.

If we can agree on that, I’m fine with abandoning the idea of backends writing to pyproject.toml. (If we can’t agree on that proposal, I may have to abandon backends writing to pyproject.toml anyway, but there’s less risk of PEP 621 being caught in the fallout if we can deal with sdist metadata separately).

1 Like

It will. But my point is that the amount of data that’s reliable will be too small to be helpful, if it’s just what the user specified in the source tree. I don’t expect to see a rush to switch from setup.py/setup.cfg to pyproject.toml, unless setuptools aggressively deprecates the older files, which I assume is unlikely. So most projects will remain all-dynamic for quite some time.

Hmm… I feel like we should allow for pip to reliably depend on the information in pyproject.toml and not mandate that the file be updated as part of the build process (I’d missed that in my initial reading of this).

This is possible by basically saying something like “if the metadata is specified in the pyproject.toml file, the build tools MUST fail if they would generate metadata that does not match that declaration (changed dependencies and whatnot). Other tooling may use the information in this table, as long as they respect the dynamic key and invoke the build tool for getting those values”.

That way, if pip sees the project.dependecies key in a pyproject.toml file, it can reliably depend on it for sdists (because sdist’s generated metadata would be tied to that user input) and it doesn’t introduce the quirks that are making people react as they are reacting right now.

Compared to iteration 2, then difference would then be “sdist metadata MUST match metadata specified by the user in the project table. Otherwise, the build backend MUST raise an error” replacing the sentences that were to the effect of “this metadata is not for anyone except the build backend”.

(yea, forgive my fairly rough phrasing here)


How does that sound to folks here?

My position (for clarity) is that I agree that we shouldn’t try to update this TOML file as a way to communicate sdist metadata. OTOH, until we have a proper mechanism that’s portable, it’ll be nice to allow tooling to depend on these static declarations made through this mechanism that we’re introducing here.

This also ties in nicely with @pf_moore’s proposal over at Sdists (again): Metadata standardisation incremental update because it’ll then allow the sdist metadata file itself to use the same model as this file around the dynamic key.

As I said, I think the reality is that (setuptools) users won’t migrate to pyproject.toml fast enough for it to be of any practical use. But if we can get somewhere with getting this into the sdist metadata instead then that becomes a non-issue.

Yeah, I think there’s no controversy about “anything specified in PEP 621 is canonical”. Without the part where pyproject.toml is an output of the build process, I think we’re all in agreement about all the other details of PEP 621 and the only question remaining is whether or not PEP 621 as an opt-in way for backends to accept metadata is worth standardizing at all.

I also agree with Paul that PEP 621 itself probably won’t have an appreciable impact on pip's ability to read static metadata for quite some time to come, whereas an output file should scale nicely and quickly.

Any users adopting PEP 621 (either — if it is accepted — in the project table or — if it is rejected — in a tool.backend table) would presumably make it easier for more of the sdist metadata to be reliable, and of course to the extent that people adopt it, it will make it easier for tools to scrape the data when available (if that’s the best way to get your stuff counted for vanity metrics it might actually spur some adoption of the format :sweat_smile:).

1 Like

Awesome. So, assuming that the sdist metadata discussion goes through, and we drop the “update pyproject.toml file” section from this draft; would we be good to go here?

My main reservation remains “is this sufficiently useful to standardise”? Remember, this is being proposed as a standards-track PEP, so it’s saying that backends must respect it. We know that Poetry won’t be doing so for some time yet, and setuptools has major backward compatibility reasons to take quite some time to fully switch, so it feels like it could be another PEP that sits around with limited support for ages.

Even without backends enriching the data, I don’t think there’s anything wrong with the proposal.

I guess I’m just concerned that as a community, Python packaging is generating PEPs faster than we’re implementing them, and I don’t think that gives us a good image in terms of stability.

1 Like

setuptools has major backward compatibility reasons to take quite some time to fully switch

I think setuptools can add support for this fairly quickly[citation needed] but the adoption will obviously be slower, which is expected and A-OK to me. I don’t expect that all the world will be on this format in 20251 even if we adopt+implement+publicise it in 2020. I think improvements taking a while to percolate out is the nature of things here and shouldn’t be why we can’t introduce a fully-backwards-compatible improvement (we’re still supporting Python 2 in pip!).

Even if poetry doesn’t adopt this, it’ll be a yet-another case of poetry not complying with the specs, which is also fine by me. flit, hatch etc can still add support for it.

1 Yes, of course, I’m assuming that human civilization makes it to 2025 fairly intact. :slight_smile:

This is true, though it would be good to message it as “each PEP is a discussion about one aspect, not recreating the whole system”.

This is probably something to clarify, but I never thought that this PEP should be mandatory and binding on backends, just that this is the spec for the [project] table. In other words, in order to be a backend you don’t need to support PEP 621, but if you use the [project] table you must implement the whole spec.

I don’t really think it needs to be mandatory — I think that the benefits of standardization will accrue to both the authors of backends (free documentation, possibly free implementation, etc) and individual users (their data is easier to parse for their own personal reasons or for things like dependabot) to a sufficient degree that using the PEP 621 spec will be a natural choice for most if not all backends anyway, with possibly poetry and flit being the exceptions, since they already have declarative configuration in pyproject.toml that they may already be satisfied with.

I don’t think setuptools will be able to make this “the only way to do things” for an absurdly long time if ever because setuptools still needs to be able to build random projects on PyPI and github that are unmaintained or under-maintained. That said, we can try to provide tools that would automatically convert setup.cfg and/or setup.py over to pyproject.toml inputs, and generally try and make some noise about converting.

My reading of the “room” is that even with the sdist bit pulled out, PEP 621 is, at best, a recommended format for tools to use in their pyproject.toml, but otherwise lacks enthusiasm enough to keep it going. As such, I am planning to reject my PEP by Friday before I finish at the core dev sprints unless people can convince me not to do that.

If projects choose to take PEP 621 for inspiration for how to lay out their metadata, great! If the dynamic idea and this discussion leads to finally standardizing sdists, great! Either of those outcomes makes the past 7.5 months of working on this not (entirely) wasted.

But my takeaway will be that trying to standardize anything that involves UX is a no-go in Python’s packaging ecosystem. And that’s fine and at least a lesson learned in general. That does mean that probably focusing standards towards build artifacts like standardizing sdists and APIs like PEP 517 to drive tools are a better use of time going forward.

2 Likes

Can we at least turn it into an Informational PEP that can be updated? It’s bonkers that when new metadata fields are added all backend maintainers will individually be burdened with how to name, represent, etc. or form some mechanism to talk to each other to form consensus.

I’d say that there is a standing expectation that “all build tools are expected to accept this format as input”. If they don’t, they’re not PEP 621 compliant and cannot use the [project] for metadata. This is similar to how certain tools don’t follow certain interoperability specifications (eg: poetry + PEP 440).

The difference between your phrasing and mine, AFAICT, is that tools are expected to implement this standard.

@brettcannon I don’t think this PEP should be rejected, and would request you to reconsider. It is already status-quo that certain tools may not adopt new standards immediately, and we don’t need to make this the “only possible way” for specifying metadata for tooling.

I’ve been thinking of this as a “tools are expected to implement this” standard and serving as the way we all agree to represent metadata in a pyproject.toml file. If some backends don’t implement them, they’re not following an interoperability standard, which is a choice all backends have always had a choice to make. They can’t use the project table in exchange.

How quickly this gets adopted is also a function of the amount of effort + disruption that the changes could have (eg: PEP 592 – yanked – took quite a while to get implemented on PyPI’s end), and not really a reason to reject this PEP.

All the reasons for working on and collaborating on this specification are still unchanged as far as I can tell. Certain build backend have stated that they’ll be slow to implement this but I don’t think anyone has out-right opposed this PEP in it’s overall form. I still think with a few changes, this PEP will be good-to-go.

2 Likes