PEP 711: PyBI: a standard format for distributing Python Binaries

[BrenBarn (Brendan Barnwell)]

I have mixed feelings about this.

First, it’s very cool that you’ve been able to create a self-contained binary Python. It’s something that several people mentioned as a desideratum on one of the other threads. It also seems like it could potentially be a step towards a manager-first ecosystem for official Python releases^[1], since a manager could draw on these builds to populate the environments it manages. And it could be great for distributing self-contained applications that don’t need to assume an existing Python install. So in terms of what can potentially be done with it, it seems good.

On the other hand, the given rationale seems focused in a different direction, particularly on doubling down on the existing PyPI model, which I see as largely a hindrance rather than a help to the improvement of the pypackaging world.

So basically I agree with you that PyBI may potentially be very useful, but I think my reasons for thinking that are not the same as yours.

The most concrete question I have in this regard is: right now, as far as I know, everything available on PyPI is meant to be installed by Python (specifically, by pip). How does PyBI fit into this, when it is by nature something that has to be (or at least may need to be) installed before Python can install anything? What tool would someone use to install Python from a PyBI? What is the gain in leveraging the wheel format for this, when its role in the installed “stack” is going to be so different?

My other comments are really about the non-normative sections of the PEP, because they’re more about the conceptual path on which you (or others) see this PEP as a step.

@njs

But, there are lots of python users who aren’t conda users, and they deserve nice things too. This PEP just gives them another option.

To be frank, this strikes me as a very weak rationale. It’s something I’ve heard on these threads before, and I’m still puzzled by it. Again and again it is mentioned that conda solves problems that people have, but the response is “well I don’t want to use conda”, with a generic justification like “I had issues with it”, followed by some attempt to re-implement or re-conceive what conda does. What irks me about this is that there does not seem be corresponding sympathy or uptake for those who said “well I don’t want to use pip because I had issues with it”. I hope by that I make clear that what I’m concerned with is not the use of a tool spelled c-o-n-d-a but the actual functionality provided by each tool and what needs it does or does not serve. As I mentioned above, I can see that having a self-contained Python binary is useful; I do not see how tying that into the existing PyPI ecosystem is more useful than using it to create a better ecosystem.

@njs

The deeper answer is: the maintainers who upload packages to PyPI are the backbone of the Python ecosystem. They’re the first audience for Python packaging tools. And one thing they want is to upload a package once, and have it be accessible across all the different ways Python is deployed: in Debian and Fedora and Homebrew and FreeBSD, in Conda environments, in big companies’ monorepos, in Nix, in Blender plugins, in RenPy games, …… you get the idea.

As I’ve mentioned on other threads, my view is that the main and in some cases the only reasons people use PyPI are:

1. it is the default repository for the default installer that comes with Python
2. it has a lot of packages people want

If a different installer came with Python that used a different repository but still had the packages people want, they would use that instead. So my view is that there is no need to hew closely to the way things have heretofore been done on PyPI, because many people will happily switch to a new system if it is better.

Moreover, as discussed on pypacking-native, the multiple usage of PyPI as a source for end-users pip-installing things as well as for distro maintainers, plugin-architecture programmers, etc., is in some ways actually a problem with PyPI, not an advantage. Probably those functions should be separated.

@njs

So what’s special about PyPI and wheels is that they’re designed to describe dependencies in a standard, abstract way, that all these downstream systems can consume and convert into their local conventions.

I don’t think that is special, or insofar as it is special, some of its specialness is of a negative kind. The PyPI dependency mechanism is, for instance,“special” in that it precludes non-Python dependencies, which I see as a disadvantage. As mentioned occasionally on other threads, perhaps most cogently by @steve.dower here, wheels are not the solution to everything and the PyPI ecosystem has major limitations.

In my view, the main reason PyPI and the wheel format are special, the main reason that people use them, is, like Mount Everest, because they’re there. It is not because of any wonderful qualities they have; they are simply the most convenient vehicle for access to the wonderful qualities of Python and various libraries written in Python. For me the utility of something like PyBI would be to move away from the existing limitations, down a different path.

So, like I said, I think from a technical perspective PyBI is interesting, and actually I think it has the potential to improve the Python packaging world. But I wouldn’t say that crafting a wheel-like format to fit into PyPI is the way to achieve that improvement. Rather it would be to use PyBI as the kernel of an alternative packaging ecosystem, in which a manager installs Python by using a PyBI. This would allow moving all the dependency resolution, environment solving, etc., out of Python-level libraries like pip and into the manager. The main question for me is just whether PyBI would be a more effective starting point for this than conda.

---

1. which as I’ve mentioned repeatedly is what I’d like to eventually see ↩︎