Thanks for posting this. I agree, that’s a very frustrating journey. And although you apologise for the lack of actionable suggestions, I agree that a lot of what you’re saying self-evidently needs to be better, so I think there’s plenty to work on here.
The transition process for projects which rely on setuptools is far from ideal. I have my views on why that is, but I’m a pip maintainer and it’s entirely possible to point out the problems with pip’s approach in all this (as you did) - so I don’t want to say anything that would give the impression that I think this is someone else’s fault. It’s a community issue, and one we should solve as a community, not by pointing fingers at individual projects.
I do think, though, that the biggest issue we have here is a lack of resources. All of the projects involved are entirely volunteer-maintained, and critically under-resourced. So we have a problem of a lot of people talking, and recognising the issues, but too few able to do anything about them. If anyone is interested in helping in this area, please feel free to dive in! The problems are hard, so don’t expect to be able to provide a quick fix, and you will get pushback as you learn all the problems that left us where we are now, but help would definitely be welcomed.
The packaging survey gave us a strong impression that users wanted better tools - and that triggered some big discussions over where we wanted to end up. But I think that in doing so, we dropped the ball. We need to look at how we handle all the legacy projects, code, and practices that are holding progress back, before we look at where we want to go. A discussion about “what packaging tools should exist 10 years from now” seems a bit optimistic if you consider that after six years, you’re still suggesting that the move to a standardised mechanism for backend options from completely arbitrary user-coded mechanisms is being rushed!
Unfortunately, strategy discussions about legacy support and transition processes simply aren’t very interesting for the community. And tool maintainers don’t have the resources or the authority to force anything to happen, especially not to block people from moving forward simply because they didn’t think enough about transition. So I don’t have any good answers either. The best I, personally, can do is to make sure that any new PEPs have a really good “how do we handle legacy and transition” story. But that’s just keeping things from getting worse. We still need to help setuptools catch up with providing transition mechanisms for people using the old approaches, and help update documentation to the point where web searches pull up current and useful information, and where it’s easy to know what’s out of date.