PEP 639, Round 2: Improving license clarity with better package metadata

Packaging Standards
102

This I can totally do. For all of the context and motivation can I refer to 639 or should I copy some stuff over?

103

Personally, I’m fine if PEP 639 gets referenced if you think what it currently says makes sense. Otherwise I’m also fine with copying with attribution back to PEP 639.

104

Sorry for the delay in replying; I meant to respond two weeks ago, but right around the time this was posted my poor cat got very sick and I’ve had to constantly care for him, and I had a number of urgent projects to complete for Spyder.

To update everyone, after several rounds of detailed discussion getting @sethmlarson and @ksurma on board, familiarizing them with the PEP and drafting a work plan to get this done, Karolina took the first month to read and get familiar with the PEP in detail, while Seth later ended up stepping back to an advisory role due to other commitments and Karolina seeming to have things well under control.

After some back and forth questions and discussions, she developed a solid plan for revising, cutting down and greatly simplifying the PEP, and then begun drafting PRs in separate branches to pull out/remove non-essential sections from the PEP as well as the previous migration guidance from the spec [1], and greatly simplify and clarify the remaining language, respectively, after which we went back and forth at a regular pace with several rounds of questions and reviews of her drafts to get them ready to merge. As she merged my latest changes into her own branch, I was going to propose that she just submit that directly as part of her PR to avoid any further blockage and my end, and we could iterate from there.

However, after answering her latest questions and making some additional suggestions, I haven’t heard back for about a month, so I’m not sure what the status is on her end—hopefully everything’s all right; I’ve reached out again to check.

See my most recent post just above for a list of the main items; we have a more detailed work plans in our PEP 639 working group, which I can also share here (or just add you there). I’d be more than happy to add you as a PEP author and help spin you up and drive this forward; it would be extremely valuable to have a tool author who’s implemented this in practice on board, particularly since one of the two main action items listed there is simplifying, updating and refining the specification to reflect what is practical for tools to implement in practice:

The second main work item, cutting down and simplifying the PEP text and moving out all the non-essential sections, is what Karolina’s been focusing on and made a lot of progress with; hopefully she can pick that up and finish it, or if not either you or I can at least finish tackling the low-hanging fruit, polishing that up and submitting it as-is for now.

Finally, the third item is just a handful of relatively minor, atomic updates that I (or you, or others) can easily make as small PRs.

Particularly if Karolina’s still up for it, ISTM that gives us a workable plan with clear division of responsibilities to each of our strengths while also having a fallback plan for each if any one of us (particularly myself or Karolina) should prove unable to follow through.

The latest plans and drafts for PEP 639 seem to be converging on basically what you are looking for, with either a dramatically simplified or else mostly untouched Motivation, Rationale, etc., all the non-essential sections moved out of the PEP, and simplifying the specification based on what’s actually practical for implementations, and there’s been quite a bit of existing discussion, planning and work toward that end already.

Therefore, ISTM it makes more sense to just invite Ofek as an author and have him apply his specification updates, simplifications and revisions directly to the PEP. Additionally, together we would finish/merge the nearly-complete PR moving the various non-essential sections and appendices out of the PEP entirely, and either finish or merge as-is (if Karolina is unavailable) the branch that substantially simplifies and cuts down the motivation, rationale, etc. required sections. Combined with a handful of other minor updates, that gets us to the desired end state without having to hack something new together from scratch or abandoning all the existing years of work on the PEP and that already done to take it in the direction you’re looking for.

If you’re okay with it, @ofek I can add you to the PEP 639 working group right now and you can start getting up to speed, going through the specification and figuring out what needs simplifying, and reviewing the status of the other remaining bits. Just lmk!

  1. it was something I inherited, and kept wanting to move to an informational appendix instead of clogging the spec with it ↩︎

1 Like
105

That sounds like it would take more time than I have unfortunately. I wouldn’t be able to thoroughly read the current version to know how to simplify (and simplifying often takes more time than the main writing).

What I can do is more or less reiterate the current technical specification and show that Hatchling does it completely and setuptools has a partial implementation.

Let me know how you and Brett want me to proceed.

106

Okay, gotcha. Just to be clear, I wasn’t expecting you to need to help simplify anything but the actual technical specification [1], and that can be as simple as just taking the existing spec and deleting the parts you didn’t actually implement or that are overly complicated, and making any other updates/changes based on implementation experience. Or, if you’d rather just clean-sheet rewrite a spec from scratch based on your implementation, we could just integrate that (or worst comes to worst, just stick it directly in the PEP, merge the rest and call it a day). The rest is mostly either already simplified/removed from the PEP in the latest drafts (cutting it down to only a third its prior length), or not essential to change.

Just to note, PEP 639 covers this near the top:

While dramatically simplifying and improving the present Python license metadata story, this specification standardizes and builds upon existing practice in the Setuptools and Wheel projects. Furthermore, an up-to-date version of the current draft of this PEP is already successfully implemented in the popular PyPA Hatch packaging tool, and an earlier draft of the license files portion is implemented in Setuptools.

  1. minus the whole “conversion” section, which the latest drafts (linked above) completely removes from the PEP, along with the other non-essential sections which together comprise around 2/3rds of its total length ↩︎

107

Oh that’s fine then, sure add me to whatever.

1 Like
108

So quick update with good news—@ksurma got back to me to let us know she’d been busy preparing for and going on a major international trip, and should be back and working again on the PEP next Monday, and plans to have her section by section rewrite to simplify and cut down the PEP finished by the end of the month.

In the meantime, this week to (finally…) finish the most important glossary changes (or worst comes to worst, move ahead with it as is), and as time allows help complete and merge Karolina’s branch removing non-essential sections from the PEP (and referencing them instead where relevant for interested readers) so she can focus on finishing the rewrite once she gets back.

Thanks; sent you a friend request so I can add you :+1:

4 Likes
109

Hi all,
The work on PEP 639 has been done in PEP 639: Move ancillary parts to separate pages by befeleme ¡ Pull Request #3727 ¡ python/peps ¡ GitHub (merged) and PEP 639: Post-split edits (language simplification, deeper edits of sections) by befeleme ¡ Pull Request #3743 ¡ python/peps ¡ GitHub

Currently, the draft is split into the main specification part and a bunch of auxiliary documents.
Its total size is about 1/5 smaller than it was initially, with the core specification being 5,138 words.

When it comes to the proposal, everything the previous draft recommended has been retained. There are no new factual changes.

I’d like to ask all interested folks to take a look at the open PR and review it. What’s the next step to make it (finally) proceed?

4 Likes
110

My suggestion would be to create a new “PEP 639, Round 3” topic with the first post containing one or more of:

  • a summary of what the current design is
  • what has changed since the initial round 2 draft
  • any particular area where you want people to focus on/provide feedback on

I expect that that there are not a lot of details that need hashing out (famous last words), and having a new thread will make it easier to navigate the discussion as well.

5 Likes
111

Agreed. The discussions here have been extremely long and fragmented, and I think the whole proposal needs a new start:

  • A new thread, as @pradyunsg suggested.
  • A summary of the design that does not require prior knowledge of previous iterations of the proposal.

I’m not personally sure that “what has changed since the previous draft” is that important. I know I can’t remember anything about the previous draft, and a statement of what’s changed would be useless for me.

One specific point that I think needs to be addressed is the fact that the PEP basically requires tools to be able to parse and normalise SPDX expressions (for example, it’s invalid to write a License-Expression field without normalising it using rules that aren’t part of the PEP but are presumably covered by the SPDX specification). How are tools expected to do that? Is there an official SPDX library for Python? I did a quick search of PyPI and found a couple of candidates, but nothing that included a normalisation function as far as I could see. spdx-tools seems like the most comprehensive, but I couldn’t follow its API (I’m not a SPDX expert). Hatchling seems to have a custom implementation - is that expected to be the norm?

Basically, I’d be concerned about the maintenance implications of this PEP if there’s no easy, well-understood and canonical way of implementing its requirements. The SPDX format seems way too complex to leave it up to packaging tool implementers to get everything right.

2 Likes
112

It’s hilarious that it works:

I implemented it in the days when people would complain about any additional dependency. People still do but now I don’t care so much. I’m definitely interested in a dedicated, small package that only does parsing.

113

Thank you, @pradyunsg and @pf_moore for the suggestions, I’m about to start a new thread.

Just to quickly address the parsing tooling question: the PEP recommends license-expression ¡ PyPI which can parse, compare, simplify and normalize license expressions (such as SPDX license expressions) using boolean logic (per its README).

4 Likes
114

Thank you very much for the effort here, I’m so happy it’s finally happening!

Just a heads up that Hatchling probably will not use that library for a few reasons:

  • It has a dependency (this is its own reason).
  • The dependency appears to be unmaintained.
  • The current package is very bloated compared to what is actually required by the PEP. The data file is 28K lines long and in JSON format rather than a Python data structure. The generated file Hatchling ships is 700 lines long and a Python dictionary. The wheel is over 100 KiB whereas Hatchling’s is about 80 KiB.
115

The new thread: PEP 639, Round 3: Improving license clarity with better package metadata

3 Likes