PEP 639, Round 2: Improving license clarity with better package metadata

Delays due to moon launch responsibilities are acceptable I guess :stuck_out_tongue_winking_eye:

5 Likes

As long as he’s not saying Laika ate his homework I guess that’s a good enough excuse.

3 Likes

Any update?

Hey @CAM-Gerlach congrats for sending the rocket towards the Moon and not crashing it!
@thatch and I are back on the saddle and willing to help usher this through the finish line.
You have done an enormous amount of work there!
What can we do to help?
What is your latest PR?

3 Likes

One thing that came up in hatch’s issue tracker: Custom files under `dist-info` as part of a build hook · Issue #620 · pypa/hatch · GitHub

Currently, this PEP defines a directory for storing license files under the .dist-info folder, in a sub-folder. In the discussions back when installer’s scope/API was being figured out, @dholth pointed out that (at least, by design) the files in dist-info should be treated like a dictionary of name-of-file: text-contents-of-file with no directories expected in there (and, that was somewhat baked-into the API design of installer).

I’m mentioning this to elevate from those discussions that this is a semantic change we’d be making to how dist-info works, which could lock us out of potential data structures for storing metadata info in that, and other stuff like that, in the future.

I don’t personally have strong opinions on this; the pip/installer side of this is relatively straightforward to deal with, and I’m fairly certain that they already do – but I’m not sure if everyone is fine with making this change for tooling. It’s certainly worth calling out in the PEP as well. :slight_smile:

I also don’t have a strong opinion here, beyond that we shouldn’t just make assumptions, we should agree and document a position, one way or another.

The spec for the .dist-info directory says “Additional installer-specific files may be present”. That doesn’t explicitly prohibit directories, but it could be viewed as implying “files only, no directories”. The wheel spec says nothing about this, as far as I can tell.

Does anyone know of any existing wheels or installed projects that have subdirectories in .dist-info? My database of PyPI data doesn’t currently include wheel contents, so I can’t check this myself.

If we’re allowing custom (whether backend or user defined) files in .dist-info, then we have a potential backward compatibility issue whenever we add new files. For example, the proposal here for a licenses directory would be incompatible with a custom file called licenses. This isn’t mentioned in the PEP’s “Backward compatibility” section, by the way… A way round this would be to reserve all top level names (maybe explicitly noting that setuptools adds a number of non-standard names, which will remain for compatibility) and require all non-standard metadata files to go in a custom subdirectory. That means accepting that subdirectories are allowed.

2 Likes

Directories would be fine.

I would like to see new keys in the .data/ directory to mark categories of files.

2 Likes

Hey all, is there any update on this? I’ve now ran into multiple situations where having this PEP active would be helpful.

1 Like

It’s waiting on @CAM-Gerlach to finish his next draft of the PEP for review.

1 Like

I sat down with @CAM-Gerlach at PyCon US sprints yesterday, and we made progress on the draft updates.

8 Likes

Has there been any update on this?

Is there anything I can do to help?

Has there been any update on this?

I think as this point @CAM-Gerlach should either withdraw the PEP so someone else can draft a new one, or find a co-author who can drive this to completion. As PEP delegate, I’m giving CAM a week to decide which he prefers (I’m not giving explicit time for another draft as the amount of time and pinging has been enough to show CAM simply doesn’t have the bandwidth to drive this right now).

2 Likes

Sorry, I’d DMed Brett as well as some interested co-authors a couple weeks ago, but after getting confirmations from each of them, following up with them further on Discord and letting them know I’d post this here, I didn’t realize I’d accidentally left this as a draft and not submitted it. My fault…original message follows:

I’m very sorry to have disappointed everyone yet again with my continued slow progress and delinquency of my responsibilities on PEP 639, and I appreciate all your pings and attempts to get things moving again. While I’m sorry he had to post it, I’m of course very thankful for Brett and others here having been proactive in helping move things along without me given the growing interest in seeing it through.

The good news is that we’ve found not one but two new motivated and very well-qualified authors to help drive this PEP to completion. Please welcome @sethmlarson and @ksurma on board! As I’m sure many of you know, Seth is the new Security Developer in Residence with the PSF, and reached out to Brett and myself to help with this PEP due to its SPDX support helping provide metadata important for SBOMs and in turn, supply chain security. Meanwhile, Karolina works along with Petr Viktorin at Red Hat on Python packaging, and her and her team are already making considerable use of initial implementations of this PEP to more easily ingest Python package license files and metadata, and reached out to me following our discussion at the Python core dev sprint looking to help with this PEP so they can further that goal.

I’ve been following up with them to help get them up to speed on the current status of the PEP and the remaining steps needed to get it over the finish line. While I’d been slowly working through a thorough (if content-preserving) rewrite incorporating editorial improvements, more concise language and a glossary with more consistent, cross-linked terminology and using it throughout the PEP, I’m going to just cut that off at the end of the Specifications section where I am now and just submit what I have, so it doesn’t block further work, and of course update the PEP to formally add them as authors.

I’ll still be as involved as I can be supporting them, contributing my relevant expertise and taking on some of the work as time allows, but I’ll no longer be a blocker to continued progress, and they’ll be able to move the PEP forward however needed.

Here’s the remaining priority work items that I’m aware that we’ll likely be tackling over the next couple months:

  • Review the “tool” definitions and validation requirements from the perspective of ensuring a conforming implementation is practical and not over-burdensome for maintainers, while still covering the most important points.
  • Make the PEP much shorter and more concise (its the longest current PEP, which is a big barrier to it being useful to reviewers and readers)
    • Move the non-core content to separate documents and just linking them
    • Edit down the remaining core PEP text to be less verbose and more focused
  • Make a few further small content tweaks requested by people on the thread since my last big revision:
    • Add mention of license subdir in dist-info being not being technically backward compatible with at least theoretical implementations
    • Add rejected idea (for now) of a single file concatenating license file text
    • Mention that the design is REUSE-compliant and important for SBOM initiatives
    • Link mesonbuild/meson-python#270 and Red Hat’s existing work as additional evidence of existing implementation of the proposed draft

Additionally, Seth and Karolina will each be providing and incorporating additional valuable input and feedback from their organizations, their many contacts and the community, to ensure the PEP delivers on its stated benefits and is practical to implement and make use of in the real world.

Please give a warm welcome and three cheers for Seth and Karolina for helping pick up the torch that I so carelessly dropped much too long ago, and we look forward to sharing more updates and PRs as they get up to speed on the PEP and the remaining work items. And thanks to everyone in the community for keeping the flame alive in the meantime!

17 Likes

This is great news! Please Seth and Karolina let me know if I can be of any assistance.

I have a vested interest in getting this finished finally as I implemented support for this in Hatchling since the very first draft and I get issues opened sometimes about the string version of license not matching existing schema validators :sweat_smile:

3 Likes

Hey, thanks!

So actually, as a tool author who has made the effort to fully implement the PEP in a popular build backend and frontend, it would be particularly valuable if you could share some feedback on what specific nominal requirements and recommendations of the specification you implemented and which you didn’t, and which seemed reasonable and straightforward vs. unnecessary or onerous for backends. We don’t want them to be too burdensome for some backends in practice, and reducing complexity there would also help simplify the specification.

Additionally, as a corollary, it would also be helpful to have some feedback on the distinctions made between different types of tools (backends, publishing, installers, etc), and whether each of the various differing requirements for each of them make sense or could be loosened or harmonized, since it seems that could use a bit of a rethink and simplification pass too.

And of course, any further author or user feedback is always helpful, thanks!

Unfortunately a new draft has not appeared 3 months later. I now have lock files to contend w/ while staring down becoming a parent for the first time next month. As such, this may need to wait until June when I am back from family leave or I need to withdraw from being the PEP delegate. But if I do remain the delegate then I am going to require a new draft before I return in June or else I’m going to have the PEP be withdrawn.

8 Likes

Would someone be so kind as to jot down what needs doing? I am willing to do the work but do not have the time to investigate what needs doing.

Just as a heads-up, Hatchling has supported this since forever and setuptools began outputting the License-File that this PEP introduces 3 years ago.

1 Like

Either PEP 639 needs to be withdrawn due to inactivity and a new PEP that just clearly outlines what the spec is, or PEP 639 needs an update for the exact spec.

If Hatch and Setuptools are already doing something, then just write a PEP of what that looks like.