Version 0.3.4 of distlib has recently been released on PyPI . For newcomers, distlib is a library of packaging functionality which is intended to be usable as the basis for third-party packaging tools.
The main changes in this release are as follows:
Fixed #153: Raise warnings in get_distributions() if bad metadata seen, but keep
going.
Fixed #154: Determine Python versions correctly for Python >= 3.10.
Updated launcher executables with changes to handle duplication logic.
Code relating to support for Python 2.6 was also removed (support for Python 2.6 was
dropped in an earlier release, but supporting code wasn’t removed until now).
Thanks for the update! I was just looking into distlib recently, so good to see this here.
I hope I’m not hijacking this thread, but since we have you here, I’ve been a little unsure of how distlib fits into the modern Python packaging landscape. I seem to understand that the original plan some years ago was to factor all the common packaging functionality into distlib, and have the major packaging tools rely on it as a backend, but looking over the codebase recently, it is unclear how much progress has been made toward that goal or if that is indeed still actively planned, there doesn’t seem to a ton of activity on the project outside of your diligent maintenance and bug fixes, and retaining compat with EOL language versions and standards seems to imply a focus on supporting legacy libraries and applications over new ones—which is perfectly valid, of course, I’m just curious what specifically those are.
Do any current packaging tools use distlib as their basis? How does it intersect with packaging, build and installer? What is the interplay, if any, between setuptools and its maintenance of distutils? Is there any relationship between distlib and the old distutils2 project? I ask both out of curiosity, and in considering its intersection with the potential implementation of PEP 639, potential future PEPs and its use in other tools.
That was the plan when I started distlib. It was at the time of Python 3.3, when there was a plan to integrate distutils2 as the packaging package in the stdlib. At the time, setuptools was outside the fold and distribute was the blessed fork to use instead. The packaging plan was pulled late in the 3.3 alpha cycle when it was realized that it wouldn’t be ready. At the time I had some bandwidth to work in this area, and it seemed a shame to let all the hard work that had been put into distutils2 go to waste. So I pulled in some of the bits from distutils2, and added some functionality I thought would be useful to packaging tools, and released it as distlib. Of course packaging is something of a poisoned chalice, and the graveyard of many an effort, but I must have been feeling especially masochistic at the time
Whether makers of packaging tools would actually use distlib would be up to them. I merely released it with the intention to act on feedback from interested parties to make it work better for them, with the hope that they would find it useful for their needs. There was some feedback, but not much
Well, the codebase reflects a set of functionality which was proposed for use by packaging tools, but if take-up, experimentation and feedback is wanting, then it doesn’t have any basis on which to evolve. Over time, my bandwidth (and inclination, given the apparent lack of interest) has declined a little. What’s the point of adding more features if no one will use the ones that are there?
Well, retaining compatibility with EOL versions (2.7) is still important to some people, I guess, and I don’t do anything special to maintain that compatibility other than not use new features such as f-strings, the match statement etc. It doesn’t mean that the library can’t be used with the newest Python versions and applications - it’s tested with Python 3.10, as well as other recent Python releases and 2.7, on the mainstream platforms.
As far as “diligent maintenance” goes, I like to stand by the software I’ve written, as much as possible. It’s generally not a big time sink (at present), so I can do it.
Well, none “as their basis” as far as I’m aware, but pip does use it for working with setuptools “entry point” scripts (it converts them to single-file executables on Windows using distlib).
There’s no intersection with build, as distlib doesn’t have functionality to build things. As far as packaging and installer go, their authors have chosen to implement functionality that was already in distlib (version management, markers, requirements, tags in packaging, installation from wheels in install). I don’t know why they didn’t use distlib for these things - perhaps they just didn’t like my approach (fair enough). The documentation for these things in distlib is to a reasonable standard, IMO.
In terms of distutils, there are only two references in distlib to it. One is in get_host_platform() on macOS, and that function is copied from Python 3.10. When Python removes that distutils reference, I will too The other is in setup.py. It has been suggested that I import setuptools in there, but it doesn’t make sense to make setuptools a dependency when distlib was intended to provide the same core functionality as setuptools. (Remember, this was at a time when setuptools was something of a black sheep of the packaging family.)
I’ve already mentioned above the connection with distutils2.
The idea behind packaging PEPs was that they would specify a standard, and tool users could implement to that standard, and people could choose the tools they wanted. It doesn’t always work based on any clear technical merit (at least from what I can tell) - for example, packaging and installer seem “blessed” in a way that distlib is not.
In trying to make distlib useful, I tried to be “innovative” (not especially deep in this area) but some comments I had in the early days was that the innovation was somehow too innovative. Well. sometimes things are just a matter of taste, and you can’t do much about that!
Well, I’ve generally aimed to track PEPs in the packaging area where they apply to distlib, but these days I’m perhaps more reactive than proactive. There was a plan for a time to make JSON-based metadata (which had technical merits - see PEP 426 - but was withdrawn). I spent a fair bit of time updating distlib in line with PEP 426 drafts during development, but I sadly don’t have the time now to engage actively in discussions and implementation iterations around metadata formats. I’ll aim to support PEPs as far as is practical, once they’re final.
Wow, thanks for that absolutely fantastic response, @vsajip ! That answered all my questions and more. I’d been able to clean the basics from the readme, but that provided way more detail, context, insight and analysis beyond just describing the basic facts. It does seem like a real shame that distlib hasn’t seen more use given all the effort involved in creating it.
Just a quick suggestion, that might not be realistic—do you think distlib might see more interest, contribution and adoption if it were hosted on a more widely used platform like GitHub or GitLab? I’m not sure I’m aware of any other active major OSS projects that still use BitBucket (I’m sure there are a few) and I wasn’t even aware it still existed until very recently by chance, whereas I would have encountered it quite a while back just browsing the PyPA org, searching GitHub or elsewhere, as for almost all the other major PyPA projects. However, I understand if you have your reasons for keeping it there, aside from the far from trivial migration cost.
Well, it’s possible - I’ll see about it. Certainly BitBucket wouldn’t be my choice now for new projects, because it is now far behind other platforms in functionality, responsiveness and ease of use, though it does the basics. Since distlib is not targeted at a wide user base, but rather the much smaller number of people working on developing packaging tools, “drive-by” contributions are unlikely to result from a switch to e.g. GitHub. The main reason for keeping it on BitBucket has been because of the existing issues and downloads I would have to migrate, for what might be little benefit. I’m pretty sure the developers of packaging, installer etc. know about distlib already, and it is used by pip, which is pretty central in the packaging space. While there is a mirror of the repository on GitHub, it’s not in the PyPA org. I don’t expect things will change much when I do migrate it, though
For the record, build does use packaging to check that build dependency versions are satisfied. I think it’s very unfortunate that we have two different libraries under PyPA fulfilling basically the same function.
FWIW, one thing I always feel distlib could benefit from if it drops support for older Python versions is it could be splitted into multiple namespace packages; one reason I see people sited for choosing packaging, re-implementing a Simple Repository API client, etc. over distlib is that people usually just want some subset of packaging features, and distlib is a giant blob with everything most people would never need, such as the JSON API client.
Current support is for 2.7 and 3.6 - 3.10 - which older versions do you mean? Just 2.7?
As for “a giant blob”, I’d like to understand the issue better - is the problem in your view how much disk space it takes up? How much memory it takes up? These days both of these are so cheap and plentiful, it hasn’t occurred to me that those could be an issue. Has someone hit a problem in practice because of the size? When I use a library package, I don’t worry about the bits in it I don’t use, unless they somehow get in the way. After all, Python’s philosophy has long been “batteries included”, and distlib’s batteries are all focused on packaging concerns even if they are not all of use to some projects some of the time. Some rough sizes on disk, for comparison (excluding binaries and vendored stuff):
Size
Package
128K
packaging
412K
distlib
220K
requests
1.4M
setuptools
Given that distlib does quite a bit more than packaging, I think it offers a reasonable bang for the buck. Thoughts about splitting into namespace packages now:
It might be a problem because of backward compatibility for existing users.
There’s no guarantee of any increased adoption by anyone because that splitting has been done (since they have already committed to packaging or similar).
Numerous of the components depend on each other (as you might expect - while the version part might be standalone, the markers part depends on that, the metadata part depends on both of those, and so on - I would expect the same pattern occurs in packaging, and many other libraries). So the size benefits of splitting aren’t very clear-cut.
Obviously I’m reasonably responsive to feedback from distlib users in general and developers of packaging tools in particular, as you will see from the issues addressed and commit history over the years, but there’s been no approach from anyone who is working in the packaging tools space along the lines of “I’m interested in using distlib, but here are the issues I have with it …” which would have allowed me to at least try and address those issues. Maybe I haven’t been very good at publicizing it - I don’t really use social media, for example - but I certainly post to mailing lists whenever there’s something to talk about. But maybe that ship has sailed …
Per PEP-609 you should have first triggered a vote to accept the project under the PyPA banner. I’d personally would have +1d the motion, but would have been nice to follow the rules we agreed on.
Maybe in spirit and practice but not how PEP-609 is formulated PEP 609 -- PyPA Governance | Python.org, the way I read it the project must live under GitHub to be a PyPA project (so perhaps we should alter the document).
PEP 609 is not entirely clear on that point.The only place GitHub is mentioned in the prose is referring to “the GitHub org/user to transfer the project to”, i.e. out of the PyPA, and one place that the phrase PyPA organization is used links to the GitHub org via a footnote, but it is never explictly stated in the text that this applies solely to the PyPA GitHub organization. Furthermore, the project was already under the PyPA organization on Bitbucket for several years prior to PEP 609’s acceptance, a fact which is acknowledged for existing PyPA projects in the Rationale section:
Currently, the PyPA is an informal and loosely defined organization that lacks true authority, and the inclusion of a given project under the PyPA umbrella or the creation of new projects has been done in an ad-hoc, one-off manner. Similarly, individual membership in the PyPA is not well-defined.
Unless the intent of PEP 609 was to disenfranchise existing projects, while it is not explicitly stated either way, it would logically follow that existing projects that were already members of the PyPA org on other platforms would be grandfathered in and not have to re-apply. I seem to recall that some projects on GitLab were transferred over as well.
Ultimately, given the significant ambiguities in the PEP, what likely matters most here is precedent. Were there any previous projects on Bitbucket, GitLab or other platforms that were already PyPA members and part of a PyPA administered organization or platform get transferred over? If so, what happened in those cases?
All that said, I think we can all agree that there are ambiguities in PEP 609 on this point, which if deemed important should be resolved by a PEP update (which would require a vote), though aside from this project, as far as I’m aware it is a moot point for the immediate future, since there are no remaining projects hosted on Bitbucket or GitLab under a PyPA org, and I’m not aware of any currently hosted on other platforms (though I’m not a PyPA member or expert as yourselves are, so perhaps there are some). It could potentially matter for the future, though, if PyPA were to re-expand to other platforms (though that could be accompanied by a PEP change as well).
To be fair I care less about the legality of this, but more about we update the PEP to be clearer on the topic As I said at the moment is a bit ambiguous because PyPA member is people with commit bit on the GitHub org, which leaves me to believe the project must be under GitHub too, but this is never clearly stated.
I agree with what @CAM-Gerlach said - the above assertion is nowhere in the PEP, from my reading. I’m not sure such an assertion even belongs there - what would be the wisdom of so strongly linking the choice of repository platform with project governance? I realise that it’s nice when everything is under the same umbrella, but mandating it would seem to me to be a step too far.
I think this is fine, and that we don’t need votes for this level of micromanagement of projects – that’s explicitly a non-goal. Let’s fix the PEP to explicitly note that the Bitbucket org is a thing and to note that migrating projects between platforms doesn’t really need any sort of votes.
Are we ok with projects migrating to another platform from our preferred host without discussion? For one, who’s gonna manage the PyPA org on the target platform? I honestly don’t think we need to do anything here, distlib was the last remaining project which wasn’t being hosted on GH. Should a project which is not hosted on GH apply to join the PyPA in the future, we can revisit this discussion.
The other is in 
under the PyPA banner. I’d personally would have +1d the motion, but would have been nice to follow the rules we agreed on.
As I said at the moment is a bit ambiguous because PyPA member is people with commit bit on the GitHub org, which leaves me to believe the project must be under GitHub too, but this is never clearly stated.