Thanks for clarifying. I just meant it in the sense that it’s ubiquitous and - I would say - it’s always going to be shipped with Python.
Yes, PDM does that too. The maintainer built a library, called unearth and PDM uses it + resolvelib and friends to perform the installation.
I think the main reason for those tools being written is also the fact that pip does not have a Python API. Neither does rip by the way.
Ah, we’re miscommunicating here, I suspect. I knew about PDM and unearth. The problem is that the collection of resolvelib, unearth, build and installer doesn’t actually make a complete pip replacement, because build needs pip to create isolated build environments (and I note that it doesn’t mention pip in its dependencies… 
)
In reality, an installer that only supported installing from wheels might be enough for a bootstrap installer. Although it’s hard to tell - what if a key workflow tool (that people wanted to bootstrap into) introduced a dependency on a project that didn’t provide wheels (or only provided wheels for some platforms)?
Unfortunately, the pip maintainers really are that random person in Nebraska 
Sorry, I replied to this initially missing the point of you question. Sluething through the Poetry code it appears that indeed Poetry uses build, and so ultimately depends on Pip?
One way to sidestep the “what tool belongs in the stdlib” discussion is PEP 711 or something like it. If the python binary (with stdlib) was distributed on PyPI, I think it’d be simpler for third parties to come up with alternative installers/project management tools. Although maybe this is a distinction without a difference in the end–if most tools are implicitly dependent on pip it doesn’t change much.
Oh, my bad - I did not know about that! Do you know why build needs pip for that? I am afraid I didn’t fully understand PEP517 then.
Yeah. Pradyun made clear the trade-off between bug fixing, maintaining existing code and adding new features. I think just even bug triage would be of help.
I don’t actually think it’s that hard for someone to come up with an alternative installer. More accurately, I do think it’s hard, but not in the way people think it is.
There’s genuinely complex (and extremely interesting!) problems involved in writing a Python installer. But solving complex problems is the normal day to day routine for software. And there’s huge scope for innovation - modular architecture, installer-as-a-library, plugin support, etc. All of these are things that a motivated developer could tackle, and have great fun working on (IMO). Projects like rip are exploring this possibility, and others could certainly do the same. For example, I’d love to see an installer written in Python, which leant into a modular plugin architecture, and left a lot of the non-core features we have in pip to plugins (network stuff like proxy support, authentication and keyring management, constraints, VCS support, …)
One interesting possibility would be if build were enhanced to allow a different installer to be used when it created isolated build environments[1]. This would allow someone to offload the “build a wheel” part of being a full installer onto build, without falling into the trap of implicitly depending on pip.
The problem isn’t writing something like this - it’s finding a good reason for doing so, and working out a good path to getting your work adopted. And with the prevailing sentiment of “we don’t want multiple tools”, you’re probably not going to get enthusiastic community support… There’s no big “itch to scratch” in this area, as far as I can see. No-one loves pip, but no-one hates it enough to work on replacing it, either.
Of course, to actually be a pip replacement, you also need to address the matter of distribution. Do you go for getting into the stdlib, which requires extreme portability, vendoring your dependencies, etc, etc.? Or do you try to publish as a self-contained application, fighting all of the issues around the lack of good solutions for publishing such things? You can’t rely on a tool like pipx, because then you need another installer to install yourself.
IMO, I’d like to see people working on new installers, but without worrying about bootstrapping. Get some innovation in the installer/build frontend area, like PEP 517 did for build backends, and then, once there are alternatives to look at, look again at how best to handle the problem of bootstrapping from core Python to access to PyPI and 3rd party tools.
But I don’t imagine I’ll be doing much to move any of this forward - I’m too burned out these days working on pip to maintain the level of optimism that would be needed…
This is non-trivial, as it would need protection against recursive loops if that installer in turn called build when being asked to install a sdist. ↩︎
I think you outlined the struggle very well, it just strikes me as a self-reinforcing situation. There are fun parts of such a project (and people are out there having fun working on it), but it’s hard to sustain enthusiasm for developing a tool if people aren’t using it, and it’s hard to get people to use it when it can only add to, rather than replace, the tools they currently use.
I brought up the PyBI proposal as a workaround because there are other benefits to that proposal[1], and it would open up this space more by making the bootstrapping/distribution problem simpler (I think). A tool could distribute a standalone binary/script that bootstraps the rest[2]. It could be a zipapp written in Python, or a compiled binary written in some other language.
I don’t think this would be sufficient to justify PEP 711 on its own, but it’s a benefit if something like PyBI gets implemented. And if that workflow caught on, it could someday enable the CPython/pip divorce that @steve.dower alluded to. 
It is implemented, it’s just not a “standard” or endorsed by anyone. That doesn’t mean you can’t use it, can’t build upon it, or can’t promote it - it just means you might be the first person to be doing it.
Yes yes, technically correct is the best kind of correct. And that’s exactly what njsmith is doing with posy. But it’s explicitly described as a hobby project, so it’s less enticing for people to try to build their workflows on top of it (this is not a criticism of posy).
I should have said “if PEP 711 is accepted” but I was trying to avoid the repetition. Standardization there would encourage more experimentation in this space.
Only if we’re incredibly careful about what we standardise. In general, and by design, standardisation prevents experimentation, because it says we’re confident enough in a design to agree to not change it.
I don’t think there’s any belief we’re at a point we could standardise the CPython install layout, and so what would we be standardising?
This is the wrong way to look at standardisation, and is arguably why we’re stuck the way we are. Experimentation should come first and we standardise what’s good. If we standardise before experimenting, we get stuck in a “design by committee” situation, where no-one wins.
I see how it happens (we’ve given “implementation defined behaviour” such a bad name that people shy away from it) but that doesn’t mean it’s a good mentality to have.
I suppose I didn’t communicate my thoughts clearly enough. Standardizing one aspect can encourage more experimentation in another aspect of the ecosystem, by allowing multiple parties to try different approaches that satisfy the same specification. I hope this is not a controversial statement? By standardizing project metadata, PEP 517 allowed for more experimentation in build systems.
I’m not sure what aspect of the install layout you are concerned with here. But I think there must be some belief it is possible, given that conda has done it for their use-cases and PyBI is a proposal for another set of use cases.
I’m not saying it’s a universally solved problem, and I’m not advocating that a specific solution must be adopted immediately. I was just reading the discussions in this thread and brought up the long-term possibility of how the ecosystem would change, if an existing proposal happened to be standardized for unrelated reasons.
Conda hasn’t done any more than our upstream and downstream distros do, none of which have required a standard to be designed prior to them doing it. (And in some ways we suffer from not having fully specified it…)
If you want to become another downstream distributor, you can. Apart from the occasional angst when a distributor actually does try something different from upstream (which I fully support, but not everyone likes), there’s no need to wait for upstream. And that’s fundamentally what PyBI currently is - a downstream distribution that is designed for easy integration into other tools, as opposed to a user-friendly front end (which is posy).
Standardising a distribution format so that downstream distributors don’t have to take responsibility for their own stability is something that we’ve essentially decided we don’t have the volunteer time to handle, especially since it can be handled just as well by the distributor (with feedback to upstream if we change things too much between major versions, which is the extent of our promises).
Ultimately, we all scope our OSS projects to a manageable level of work for the team that we have working on it. And most CPython and PyPA contributors are experienced/jaded enough that we can’t be guilt-tripped into taking on more work anymore
Hopefully it’s not considered off-topic, but I really appreciate that you (Paul) articulated some of the ways that this could be innovated. As someone who understands pip at a high level but not at a low enough level to know what kind of innovation would be possible or of interest, this is really useful and helps me understand why it’s an interesting problem space to innovate and work in.
What I’m interpreting from the group is that pip has a high barrier to entry in terms of innovation due to its install base and role in the space. As a result, innovating outside of pip itself seems the safest way to experiment. If that’s the case, is there be any interest in a compromise where experiments that the group finds beneficial (e.g. greater modularity or an extension / plugin system) could be ported back into pip without fully replacing it?
If this question is too off-topic for this thread, please feel free to ignore it and I’m sorry for derailing the conversation.
The biggest problem you’d hit is that pip is an application, not a library. It has no stable programmatic API, and as a result plugins (for example) are either going to risk being broken on every pip release, or they are going to have no access to any of pip’s internal machinery (in which case, why do they need to be a plugin as opposed to a standalone app?)
Extracting chunks of pip’s functionality into libraries that can then be vendored back into pip, making them reusable in other tools, is (IMO) a better way of breaking down the barriers to innovation that pip imposes.
Improvements to pip are always welcome, but can be really hard to achieve. You may find that if you read through some of the issues on the pip tracker, you’ll find a lot of food for thought. For example:
And having a separate index for metadata only that can be updated (without having to modify the sdists and wheels)… could that help here or not?
This was meant for another thread: Could resolver be more clever about deps that can't be met? - #14 by sinoroc
Both of those positions make total sense. All I am saying here is that there is more to the world than just distributions and there is more to Python.org than files that people download to install distributions. Python.org also has text that is aimed at human readers. If we think that it often may be a good idea for people to consider alternatives to the python.org distributions, then wouldn’t it be a good idea to say that up front on the downloads page instead of burying it under an “alternative implementations”[1] page which people will never even see unless they are intrepid enough to scroll past all the big headings telling them how to immediately download the python.org distribution without thinking about whether that is the best choice?[2]
I don’t understand what you mean by this. Who is ‘people’ in this case?
Any reasonably sophisticated user is going to know to use an alternative python distribution or ideally be curious enough to go explore.
For the brand-new user to Python, I wouldn’t want them seeing a bunch of different distributions for Python and having to make a decision on which to use.
Agreed. In fact, one of the sentiments from the packaging survey that people like to quote is that users don’t want to be given too much choice. So having python.org direct them to a single preferred distribution (with “advanced” documentation offering more options for people who know they want to assess the various choices available) seems to me to be very much in the spirit of that.
And thus we complete the cycle of a packaging discussion. 
Users want a simple way to get started and do things. Nobody wants to endorse a specific choice of tools. Workflow tools have the freedom to support the workflow(s) they find most important, because there are always other options for people if they don’t like it. But python.org shouldn’t link to all these options because users want a simple choice to start with.
Despite all this, things work pretty dang well…if you’re familiar with all of this stuff and make your own choices. But it doesn’t seem to be a good experience for new users.
I don’t think that the hard choices can be made right now, but I hope that a future Packaging Council declares their intent to make these decisions on a reasonable timeline, so that the people interested in that space have something to work toward.