Just to be sure, with “newer page” you mean this one?
Maybe I am missing some subtle detail, but I don’t even see how the new page takes any position. It just lists various points to consider and then says “check with your backend, here are a few links”. I don’t see how anyone can substantially disagree with the page, except if they are saying “packages shouldn’t have a version”. As long as the old page is still somewhere in setuptools specific docs in it’s entirety, I don’t see much value in keeping it around.
By implication, it takes the position that the old page is no longer valid. As @python-intermediate said:
There’s also the more subtle point that even having a page that says “single sourcing the project version”, we’re implying that (a) people should make the version available in multiple places, (b) that the versions should be synchronised[1], and (c) that generating them from a single source is unequivocally better than just repeating the information.
At the moment, we don’t have consensus on any of these points, so we’re not (IMO) in a position to update either of the existing pages. All I’m asking is that we get consensus before we act. I’m not promoting a particular answer, nor am I even saying that what we have is wrong. Just that we should agree before putting statements into a document that’s supposed to be the community view!
If you want a different approach, delete both pages, and let’s have nothing until we have a consensus. But of course, we’d need consensus before doing that, so how do we get that?
The point has been made a few times in this thread that the project’s version metadata and the imported __version__ attribute are versioning two fundamentally different things, and it’s not necessarily right that they should be the same in the first place ↩︎
Here are my experiences concerning the Packaging Guide
As someone who attempted to read the Packaging guide front to back, I came across Choosing a build backend before single sourcing version, where the user is primed to use Hatchling because “this tutorial uses Hatchling by default”.
If the tutorial uses Hatchling by default, having an older page dedicated to setuptools is a whiplash for anyone trying to follow along and the user will likely neglect/skip the page even if it has useful material because of the mismatch.
Realistically, as someone new to packaging, if the Packaging website has (seemingly) outdated info regarding single sourcing version, I will do a google search for “python package versioning”, find the first stackoverflow answer and then implement the suggestion to do the "quasi-standard __version__ ".
Therefore the first page inadvertently pushes the user to __version__.
I am not saying take a stand either way on __version__. What I am saying is the existence of the first page promoting setuptools muddies the water on this topic and does more harm than good in its current state for new packagers. Removing it removes some of the mud.
PS: Paul, how do you do the […] formatting in your comment?
It’s done using a footnote^[like this][1]. While editing your post, it will look like a non-inline footnote in the preview, but will render as an inline footnote[2] once you post.
I think that from a process point of view, if adding the second page was a mistake then it should be removed. I’ve seen this kind of thing in other projects and other contexts, a sort of “first merger advantage” :-). If there wasn’t consensus to make a change, it’s unwise to say “well we don’t have consensus to revert it”. If you need consensus, you need consensus, and changes that are somehow slipped in without consensus should be reverted[1]. Otherwise it just encourages more people to try to slip in their changes to “take the high ground” and then let the battle rage while their change is in place.
As I have said before, I think the best solution would be large warnings on all packaging-related pages saying in effect “there is no official way to do anything, please check your tool’s documentation”.
unless of course you can retroactively get consensus, which can be the case for “obvious” changes for which no one got around to actually verifying consensus ↩︎
The PR for that second page went through some pretty intense review from both sides and was was shared here (or at least a predecessor PR since it kept getting restarted) for all to pitch in on with plenty of time to do so. If that’s really not enough then what on earth is?
Well, for a start, it appears that the review didn’t point out that having two pages covering this topic was a bad thing. If the review was sufficient, then why is it ok to reopen that question now, but not other aspects of the discussion?
At the time, it was assumed that the old page was on the way out. The PR to remove it appeared to be one rubber stamp and a rebase away from being merged.
The way packaging.python.org is handled right now does not seem sustainable to me. I often wonder whether it would work better as a wiki. I also believe the guide should strive to stick to facts and avoid making recommendations. Other resources can focus on recommendations.
Some facts:
Distribution package and import package can have different versions.
Think “vendored libraries” for example (whether or not they should be part of the public API is a different question).
__version__ is a somewhat common convention (not a standard, at least not yet).
But what does it apply to? The import module? The import package and all its sub-packages?
importlib.metadata is a thing and it is part of the standard library (as opposed to pkg_resources for example).
Computing/retrieving version string from metadata has a cost, maybe it should not be done at import-time.
There exists tools like setuptools-scm that can help “hard-code” the version string at build-time.
There is also the question of versions for vendored non-Python libraries that are not importable at all but whose versions are very important.
I think it is useful for the guide to provide advice and recommendations. Packaging best practice is confusing for me even though I’ve been following these threads for years. For most aspects of packaging there is a straight-forward non-controversial way of doing things that should probably be recommended for most situations but I find it hard to keep track of what that even is. Most people packaging stuff (including me) are well served by having the best practice clearly listed in a well maintained guide. The recommended way may be different from the way things needed to be done in the past or from what many projects currently do and so it is good to document it somewhere for package authors to see.
The issue in this case is just that there is not a clear best way. I think that those pushing against __version__ perhaps did not appreciate that other things don’t replace it. We’ve had the discussion here that as far as I can tell has resolved that. The guide should now reflect the outcome of this discussion.
You would need very active and strict curation/moderation for that to work properly. Otherwise, and especially given the diversity and controversiality of positions on Python packaging, it would quickly devolve into an unsustainable mess.
It’s a guide, how can it not make recommendations?
I am trying to shake the tree… No, I don’t actually believe we have to make it a wiki. Yes, I do believe these pages could profit from prioritizing indisputable facts over somewhat opinionated recommendations.
My point is that it sometimes feels like we are already stuck in an unsustainable mess. Changes get debated for way longer than necessary (in my opinion), so that current messy pages is what visitors get.
It seems like there is still disagreement on recommendations for this topic, so I suggest taking a step back, removing the current content, and replacing it with indisputable facts about the topic. Once we know what we really have to work with (the facts), maybe it will become clearer to us how we can write recommendations from there.
As far as I know there is no one (team or individual) responsible for the editorial content. So of course it is hard to come up with a set of unified guides and recommendations, this is normal, there is no one to blame for this. Focusing on indisputable facts (and thus avoiding lengthy debates) might help in delivering useful content more rapidly to the users within the constraints that we have (volunteer based and so on).
This is just one opinion. I like the Python Packaging User Guide, I happily contribute as much as I can. : )
There is a group of people with commit rights to the packaging user guide. They are therefore ultimately responsible for the content. However, it’s entirely possible (I’m not one of them, so I’m only speculating here) that they are sufficiently demotivated by the endless arguments and debates over topics like this, that they aren’t willing to get sucked into taking a position on anything other than indisputable facts.
As individuals outside of the editorial team, the best hope we have of getting content added to the guide is by demonstrating that we are able to form a consensus. That may not be possible for universal best practices like “how should people maintain their version metadata”, but it surely is for simpler and more focused questions like:
How should a beginner to Python store their version? Answer: Put it in pyproject.toml. Don’t worry about maintaining it in multiple places, that’s “advanced” usage. If you (or your users) want the version available at runtime, add a copy of the version number in a __version__ attribute of your main importable package. Again, don’t worry about “single sourcing” yet. You have far more important things to work on right now.
Of course, that belongs in a tutorial, not in a “guide” for experienced users or a “discussion”. But the structure of the packaging user guide is a whole other debate that (a) needs its own thread, and (b) absolutely must be driven by the guide editors - the structure and style of the guide needs to be part of the “vision” for the document, and actively enforced by the maintainer team.
Also because we’ve been burned before by listening to the cries of “we need something prescriptive for beginners to use!” and hence making early UX bets that didn’t always pay off. (The absence of clear editorial authority also makes it tempting to advise prospective editors to invest their time in improving the documentation for specific tools rather than encouraging them to get more heavily involved in maintaining PyPUG itself, and the new Docs Editorial Board for CPython has plenty on their plate with just the core reference interpreter docs before we could consider asking them for advice regarding packaging ecosystem docs)
On a more constructive note, “document the controversy” has sometimes been a viable intermediate resolution for improving PyPUG documentation in the absence of clear consensus (hence the emergence of the “discussion” category in the guide). So the lack of consensus isn’t necessarily a hard barrier to improving PyPUG (although it certainly doesn’t help).
The problem is that the sum total of indisputable facts would leave us well short of many basic tasks that people want to do.[1] So we’re back on the horns of the same dilemma that’s been discussed at length, namely that because there is no single comprehensive “official” packaging tool, many tasks can’t be accomplished without making some kind of opinion-infused tool choice, but at the same time, because there is no single comprehensive “official” packaging tool, there is a reluctance to have the official docs wade into expressing opinions about the various unofficial tools.
As an obvious example, there is no indisputable-fact answer to “what build backend should I use”, and yet people somehow have to choose one. ↩︎
Well, of course. Facts are not a guided recommendation. There is no debate about this. Also in my message it is clear that this is not the final step and what the next step would be.
I am trying to get us out of way of thinking that keeps leading us in dead ends. The idea is to assume the whole content of this page is bad, scratch it, as if it did not exist. Then gather all the up-to-date facts we know about the topic, and the use cases we want to cater to.
And from there, with a blank page, our facts, and our use cases, it will become very quickly apparent what we feel comfortable recommending and where we feel like saying “sorry we can not do a recommendation for this use case because of reason X, but we know of tool A and blog post M that have opinionated solutions for this”.
I am confused as to where the debate is. That is a completely normal way of working.
I had started a list of facts. I can start listing use cases:
I want the source of the version string to be in pyproject.toml.
I want the source of the version string to be in the code.
I want the source of the version string to be in the source code versioning metadata.
I want distribution package version to be different from my top-level import package version.
I more or less agree with that. The problem is that during the time that we are gathering those facts and deciding how to present them and what recommendations (if any) to make, the facts will change, because the facts include facts about third-party tooling and that third-party tooling continues to evolve at a relatively rapid pace to do all the things people want to do that they can’t do with the built-in tooling.
There was a time when I believed this as well. How I long for those carefree days, that magical time of possibility and potential. . .
My interpretation of previous discussions is that some people were reluctant to even do that, as it would constitute a recommendation of tool A and blog post M.
Just to be clear, I think we’re mostly in agreement that the packaging docs are deeply flawed and it would be better to pare them back drastically. It’s just that, as I see it, there seems to be little chance of subsequently building them back up to any level approaching what users need, because fundamentally the docs are on packaging.python.org and the tools users need to do things are not controlled by packaging.python.org . For instance all of the use cases you mentioned about the version string could be handled in myriad different ways by different tools, and someone could write a new tool tomorrow that targets one of those use cases.
Unless the actual tools become unified, in the sense that there is a coherent set of tools that all work together to meet all of users’ needs[1], the documentation seems unlikely to be unified.
or at least much closer to 100% than we have today ↩︎
OK, you convinced me, this is hopeless and we should all give up until the one true tool comes to save us all. ; )
No, but clearly we are repeating ourselves in a loop. I will wait and see if others have ideas to unblock the situation. I will try to review more thoroughly the existing PR(s) on GitHub and/or write one of my own. : )
