What new Reference or Explanation would usefully reduce confusion? I am asking to see if there is a consensus about the scope of new documents that would be helpful in today’s stage of packaging evolution.
Also, I don’t see a PyPA Specification which really describes the pyproject.toml file. There is a PyPA Declaring build system dependencies specification, but it is a stub which refers out to PEP 518. PEP 518 defines the pyproject.toml file overall, and the [build-system] and [tool] tables therein. PEP 517 defines the build-backend key in the [build-system] table. The Pypa Declaring project metadata specification describes the metadata parts of pyproject.toml well, but not the basic format or the [build-system] and [tool] tables.
I suggest a need for a Reference specifying everything about pyproject.toml files in one place. Maybe the shortest path is to add to the existing Declaring project metadata Specification everything from PEP 518 and PEP 517 about pyproject.toml that is not already there. It could continue with the existing format. The metadata heading would become a section of the overall Reference. It would need a more general title. Alternatively, there could be a new Specification about pyproject.toml overall, and it could delegate the metadata fields specification to the existing document. In either case, the Declaring build system dependencies stub gets replaced by another document.
I don’t see a PyPA Specification which really describes PEP 517’s “format for source trees”. PEP 517 doesn’t seem to describe that format clearly either, except to say it is “a new style of source tree based around the pyproject.toml file”. I suggest a need for a Reference describing the “format for source trees”. Perhaps it would be quite short, just saying, structure it however works for you and your build tools, and put pyproject.toml at the root. Or, maybe we need discussion to crystallise the actual current consensus for required source trees format.
It seems clear from a couple of recent Discuss threads that PEPs are not intended for the general developer, and should not be changed just to make them clearer. The present sdist/wheel build documentation, however, asks them to take on that role. Part of my “definition of done” for these Reference documents is that they are complete enough that no reader needs to resort to the underlying PEPs.
There is probably a need for a Explanation document. I can imagine rejecting parts of the PEPs which are not Reference, but wanting to put them somewhere, because they are helpful for the app or tool developer to understand. However, I see Explanations as less important than the References.
Is there a consensus that Reference documents like these could be helpful, and are achievable to write?
If there is, and if it seems that I could succeed, I am willing to try writing them. (I like making contributions which directly address obstacles which I have recently encountered in my own work.) I imagine that the next step would be to make a formal proposal, following the Specification Update Process. But, I am new to the Python packaging space. I will make ignorant mistakes. Maybe someone else would prefer to try? I am happy to defer, or to collaborate.
Your suggestions have actually been very helpful in getting me thinking about how to better structure this. After some more thought, I’ve proposed a detailed structure for how all the various normative sections of the PEPs can fit together as PyPA specifications, that combines the best parts of the various ideas while avoiding most of the potential issues. See my reply on the issue for more on that.
IMO, the title and section names of PEP 517 is, as you point out, don’t really accurate describe its final contents. In reality, the normative sections of the PEP are essentially concerned with two related but separate concerns, namely specifying how package authors declare build system dependencies, entry points and local paths, and how backends and frontends interact. Source trees are only even mentioned in ≈two sentence, describing them in general as “something like a VCS checkout”, their legacy format (an arbitrary directory with a setup.py) and stating the new format (an arbitrary directory with a pyproject.toml).
Given that is the extent of the specification of “source trees” and the packaging specs are not otherwise concerned with them, such a document would only be at most a few sentences, unless a new PEP was drafted, reviewed and accepted further specifying them. Threfore, it IMO makes much more sense to just add a Glossary entry defining the term “Source Tree” as defined/used in PEP 517, PEP 639 and the existing “Project” glossary entry. I’ve proposed this as part of the above.
Yes; in my re-review of them for proposing the above structure, I certainly noticed some sections that do not below in the specifications (Reference) but would be good candidates for an Explanation page (e.g. much of PEP 517 Appendix A, particularly the diagrams). However, as you mention, this is lower priority than both migrating the specs and updating/expanding the user-facing Tutorials and Guides (and the proposed top-level Explanation).
Indeed, and it would be particularly valuable to have a newer but interested and motivated user like yourself contributing to that, which would be beneficial to a lot broader cross section of users than a just the specification while providing something closer to the “Explanation” type content you are looking for, that explain how things work and fit together without delving too deep into the gory implementation details that only a tiny proportion of packaging tool authors need to worry about (since the specs are very strictly highly technical “Reference” material).