Pronouncement on PEPs 631 and 633
After much deliberation, I’m happy to announce that I am approving PEP 631 - “Dependency specification in pyproject.toml based on PEP 508”.
As a consequence, PEP 633 - “Dependency specification in pyproject.toml using an exploded TOML table” is rejected.
Thanks to the authors of both PEPs for all the work they put in, and to the many participants in the various discussions. This has not been an easy decision to make, and I don’t imagine it will be to everyone’s liking, but I hope that now the question has been resolved, we will all be able to work towards finalising PEP 621 and moving forward from there.
Clarifications on the wording of PEP 631
PEP 631 includes some rather confusing wording, in that it says values must be an “array of strings or inline tables”, but then goes on to say that inline tables are for future expansion, and backends MUST error if they encounter inline tables “unless the specification is extended”. I consider this a pointless distinction, as making inline tables allowed can be done when (if!) the specification gets extended, so there’s no need to make an explicit provision now.
As a result, I interpret the spec as effectively saying that only an array of strings is allowed, and I would encourage @ofek (as PEP author) to clean up that part of the PEP to make that explicit. Doing so will not affect the decision on the PEP, which is based on my reading of the practical implication of the text.
(Note that the inline table provision is left over from a rejected option to have a link to a file of requirements, and was never intended to allow for any form of “exploded table” format).
Reasons for the decision
Surprisingly, I did not in the end find that “readability” was a significant factor in the decision. Both approaches looked “OK” for straightforward uses, and both got messy for more complex cases. The level of punctuation needed is down to the fact that PEP 621 (and pyproject.toml as a whole) uses TOML syntax, and that applies equally to both proposals.
Unfortunately, neither PEP had particularly compelling arguments motivating their proposal - PEP 631 was particularly disappointing here as it lacked any “Motivation” section at all. As a result, I relied heavily on the overall “tone” of the discussions, along with my own views, to infer motivating arguments.
In addition, PEP 633, as the more complex proposal, had some flaws which concerned me (for example the translation rule noted here is not covered in the PEP - maybe the intention was to disallow the git@github.com:aio-libs/aiohttp.git form, but that was never stated in the discussion or in the PEP). I’m sure they could all be addressed, but the point of submitting a PEP for approval is that in the authors’ view, it has covered all of the details, so I did consider that a weakness of PEP 633.
The key things that ultimately affected the decision were consistency and compatibility.
In terms of consistency, as an existing standard, PEP 508 strings are used in a lot of places throughout the Python packaging ecosystem - package metadata as stored in METADATA files and reported by importlib.metadata, requirements files, installer command line arguments, tox configuration, etc. PEP 633 doesn’t propose any change for those use cases (nor should it, those areas are out of scope), but as a consequence, under that proposal users would need to deal with both formats. That’s not a showstopper, but I believe that it’s easier for users if we stick to a single format.
As far as compatibility is concerned, setuptools and flit already use PEP 508 strings for dependency specification. For users of those tools, moving to PEP 631 simply involves copying the existing value to a new location, whereas PEP 633 involves learning a new format and translating. Again, that’s not a major issue, but it is a hurdle for users, and in a world where Python packaging gets negative feedback for the level of change that’s going on, I’d prefer to avoid it if possible. Poetry users already use a TOML format, but it’s not the same as PEP 633, so they’d have to change either way.
Overall, “status quo wins”, plus PEP 631 being a significantly simpler proposal, which fits more closely to existing tools and with fewer rough edges, was the deciding factor.
What’s next for PEP 621?
I’d like to see PEP 621 updated to include the proposed form from PEP 631 directly, rather than including PEP 631 “by reference”. I don’t think there is sufficient complexity to warrant keeping the dependency syntax in a separate document.
With the question of dependencies settled, I think the biggest remaining task is that PEP 621 needs to strengthen its statement of what benefits it provides. If all it offers is a common user interface for backends to use, then I think that making it a mandatory standard is probably too strong. What I’d like to see (and what, in my view, would put PEP 621 firmly into the area of being an “interoperability standard”) would be if we could allow consumers to treat metadata read from pyproject.toml as canonical. That would extend the benefits significantly beyond “people who want to be able to change backend without rewriting the metadata definition”. It would also remove a significant awkwardness in the current PEP, where the data is present, but we’re telling people not to use it unless they are writing a backend…
With a sufficiently strong set of benefits, I think PEP 621 will then be pretty much ready for submission (although ultimately that’s for @brettcannon to decide).