PEP 711: PyBI: a standard format for distributing Python Binaries

Further fragmentation. When you choose which build of Python to use, it’s already incredibly fragmented (version, ABI, distributor, system build/local build, python.org, various containers, etc.). This adds one more, and doesn’t immediately appear to remove the need for any of the others.

The way to resolve it is to have at least some of the various stakeholders agree that they can unify around this approach. If they can’t do that - probably for very legitimate reasons! - then we have to show in some other way that adding a new distribution without removing or replacing any[1] is worth it.


  1. i.e. if all the users said they’d stop using system Python because of this, and so we can stop trying to support anyone who still does, that would be removing fragmentation. ↩︎

I agree. Responding to some specific points:

True, but unless we have a genuine, practical plan for simplifying things using only what we currently have in place then we cannot move forward without adding features. To put this another way, if we don’t have a solution now, then we can’t get one without adding components of the solution. Clearly, there’s a risk that what we add turns out not to be the correct solution. But we have no way of knowing that without trying - we’ve already established that simply talking about the potential issues doesn’t make progress - we’ve expended hundreds of messages and person-hours demonstrating that :slightly_frowning_face:

We should also remember that historically, every change we’ve made to the packaging system has been with the intention of solving issues and providing a better user experience. The current issue is “fragmentation”, but when PEP 517 was developed, for example, the issue was “lock in to a single solution”. And when setuptools and conda was developed (the very first examples of “fragmentation”) they were solving issues distutils didn’t address. We could have had a less fragmented ecosystem if we’d stuck with distutils and conda/setuptools had never been created, but I don’t think anyone would argue that would have been a good outcome.

This argument is essentially that we do nothing until we’ve defined the form of the final solution - a classic “waterfall” approach to design. While I agree that certain problems need a more structured approach to addressing them, it is also true that big up-front design approaches are very difficult to handle with open source, volunteer-driven projects. So we have to accept compromises and work in ways that actually deliver results, even if those results aren’t perfect.

Remember - we’ve already concluded that “one perfect solution” isn’t the end goal here. Linux distros aren’t going away, nor is conda, and I’d be very surprised if Microsoft will drop the Windows Store distribution. At least not in the sort of timescales where we need to address the “fragmentation” problem. So what do we do? Stick with what we have, gridlocked because we’ve made solving a problem that cannot be solved with what we have a pre-requisite for developing anything new? Or accept that we have to add new solutions that have the potential to solve the “fragmentation” issue in order to make progress?

I’m also increasingly frustrated at the characterisation of the user feedback as “we must remove choice”. I don’t think that’s a fair interpretation at all. What the feedback says to me is that users want clear guidance on what to use. They don’t actually care if there are 20 build backends available, they just want us to tell them which one to use. And personally, I don’t think they mind if that advice starts with a sufficiently clear-cut decision - “if your project is pure Python use flit, otherwise use setuptools”. Other backends are experimental, specialised or obsolete, and end users are allowed to ignore them. We could do a similar thing with distribution, if the feedback is that users want that - “you should get Python as follows”, with alternatives like PyBI (or even conda - one of the problems with being opinionated is that not everyone will agree with or like your opinions!!!) being characterised as experimental or specialised.

But we’ve seen some very clear feedback already that none of the existing stakeholders have any intention of doing so - regardless of what new approach is proposed.

Also, I think there’s a certain level of reinterpretation of the survey results going on here. We got a clear message that users think there’s “too much choice”, but we seem to have decided based on very little evidence that this means “too much choice in where to get Python from”. My personal experience is that the most frustration I see from users is around the mixed messages over what build systems, workflow tools and utilities to use. I’ve seen very few people complain that they didn’t know where they should get Python from (the big exception here is people struggling with workflow being told that they should switch to conda, but ultimately that’s still a “you’re using the wrong tools” issue at heart).

Maybe the “workflow tool manages Python” model is the right solution. But if so, then PyBI offers a clear advantage in enabling that approach. So even if we insist that proposals “address fragmentation” somehow, PyPI covers that, as it addresses a key limitation of the design of any “workflow tool that manages Python” solution, where do you get Python from?

Anyway, that’s another thousand words or so and a person-hour or so writing yet more on this subject. If it’s of use in explaining why PyBI is worth pursuing, then great. If it’s not, then maybe it at least demonstrates how simply talking about the issue isn’t actually moving us forward :roll_eyes:

8 Likes

The problem (or one of the problems) users have when picking through all existing installation solutions is that many of them need to be run differently, because all those are packaged very differently. What this format provides is a foundation to unify those Python distributions, so the installation interface can be more unified.

I think one thing we view differently is where the format stands in the stack. The response above implies the format would be a thing users directly interact with like the Windows and Mac Installers. I don’t think that is reasonable. Using the direct analogy to the format inspired this proposal—we don’t expect most users to handle wheels directly, but instead rely on existing solutions (e.g. pip) and swap the thing underneath (eggs to wheels). Python installers don’t even have eggs at this point, every installer is just ad-hoc inventing its own format, so pybi seems to me a unification instead.

7 Likes

My understanding was that end users would never know about PyBI. That PyBI is meant to be consumed by programs not by people, in the sense that no one would download a PyBI file with their browser and then double-click on it to install Python.

In my mind PyBIs would be consumed by already existing installers either directly (Posy, pyenv, asdf) or with their own (hopefully as small as possible) wrapping layer around it: conda & co., NuGet (currently it is the other way around, right?), Homebrew, Windows Store (and other Windows installers .exe, .msi?), Docker & co., GitHub Actions, CI/CD pipelines, snap, Flatpak, Appimage.

Am I understanding this right? PyBI is not aiming to compete with installers, but is aiming to be some kind of distribution format for built Python interpreters that installers can use either directly or with as little wrapping as possible (instead of building Python from source).

If no installer adopts PyBI, then PyBI disappears and there is no added fragmentation. If two installers adopt it then there is less fragmentation. No? I guess it is up to the proponents of PyBI to lobby the maintainers of installers or write the code for it. If the Windows installers on python.org can be based on PyBIs then it is already more than half-way to a win, no?

4 Likes

They are packaged differently for legitimate reasons though, they’re not just different for the sake of it. PyBI itself can certainly provide another option, but it can’t replace those options. But if the aim is to have something that we can promote and explicitly tell users not to use system installers for Python, then it may be a path forward.

I was trying to provide a fairly independent answer to your question about why people are calling it fragmenting, I wasn’t necessarily representing my own position.

Not a perfect equivalence, as for eggs->wheels we always controlled the entire stack. And we are fully aware of the impact of that (e.g. wheels can’t require system-level dependencies, etc.). It mostly works because the runtime is able to pull in system dependencies, and because wheels assume that they can assume a fairly normal set of dependencies.

PyBI could make this situation better for wheels. If we basically require that all PyPI packages must work with a particular CPython build as defined by a PyBI package, we offer a guaranteed set of dependencies, not a “fairly normal” set. But this might also be a worse situation (e.g. it could make many wheels totally incompatible with non-PyBI runtimes).

Most installers are following the system that they install into. MSIX and MSI are the “wheels” of Windows, and similarly for the other formats. So the suggestion sounds like we’d be going into integrated ecosystems and telling them to stop getting Python from their wheels and instead let us install it in our own way (this is exactly the kind of thing that all the app stores try to ban, for the record).

So our new proposal looks like the ad-hoc invention. Some of the platforms aren’t opposed to it, but it does mean sacrificing features that the current installers have. Is it worth removing “Reset” and “Repair” functions, secure system-wide installs, shortcuts, and file associations? Or significantly changing how they work (with all the associated back-compat breakage)? Maybe it is, but I haven’t formed a strong opinion on that point yet.[1]


  1. To expand, I really want to change it all, but I don’t want to cop all the abuse like last time I touched this stuff. So I’m in favour, but want to be sure we’re doing the right thing, not just any thing. ↩︎

Either lobby, or to listen to the requirements of those installers and show that they can be retained or enhanced.

Yes, except right now I don’t believe they can. The next most requested package format for Windows is a simple MSI that can be integrated with existing platform tooling. PyBI would definitely not integrate with this tooling - even executables aren’t always supported - and so this need doesn’t go away just because we provide a PyBI package.

Users also need PEP 514 registration for their tools, and virtual environments need reliable in-place upgrades. They also expect Start Menu entries (particularly for IDLE) and a global py.exe launcher that can find them all (though I think these are probably easier to support than the first two). And there’s no shortage of other examples of “requirements” our users currently have that we’d either have to replicate or convince them that they don’t really need them.

So it’s already not as simple as extracting a ZIP file, though most of the installers are kind of already generated from something like a PyBI. The PC/layout script copies stuff into the right places, and then most of the installer builds basically wrap it with the platform-specific stuff. So nothing fundamentally wrong with using an intermediate PyBI (instead of an intermediate, er, ZIP), but it doesn’t automatically mean we don’t have to provide native installers.

I’m not convinced that would make things worse. A lot of the problems with binary wheel come from from some miss-matched assumption between the build and host system (that we do not have enough meta-data to save the user from at install time).

If binary wheels were only expected to work with PyBIs and the PyBI+wheel ecosystem was documented as one-of-many ways of getting a coherent Python environment I think that would be a big improvement both from the project side and from the user side.

That’s true, but the way I see it, the most natural source of inspiration for such additional components is existing third-party tools. That is, let’s think about what users’ unmet needs are that currently drive them to external tools. If a new tool meets those needs plus more, then they can replace the existing tool with the new one, and[1] that reduces complexity. For users, third-party tools are part of “what’s currently in place”, and part of their confusion has to do with the peculiar slice of functionality that currently happens to be in place with official pypa tools.

Of course, I’ve framed that terms of tool functionality rather than under-the-hood components like pybi. But that’s what I was getting at with my earlier post: when considering such components, we need to try to project forward to how they will actually impact users’ experiences with user-facing tools, and evaluate the components on that basis. Relatedly:

That’s a fair point, but a more moderate version of my position would be "as part of evaluating this proposal, let’s try to forecast how this might solve the fragmentation problem, and also think about how reasonable/robust that forecast is ". Maybe we don’t need to completely define the final design, but at least we need to think how the proposal relates to potential final designs. There has been a little discussion of such projections in this thread, but not much.

So just to take a stab here is one such forecast:

  1. The PEP (or something like it) is accepted and PyBI becomes an official Python/PyPA standard.
  2. Once PyBI is out there, someone builds on it to develop what @njs suggested: “a single workflow manager that automatically handles installation/pinning/etc for both the interpreter and the packages within it”
  3. That tool gradually adds other functionality similar to what poetry et al. do (e.g., “project management” stuff like updating pyproject.toml) and maybe some new useful features as well.
  4. Everyone starts using that tool because it combines so much useful functionality into one tool.
  5. Python.org starts providing that new tool as the main official way to get Python, and other distributors start distributing that tool instead of “directly” distributing Python[2]. We now have a situation where “the way” to get Python is a unified tool that includes most of the functionality that’s currently fragmented. Success! :slight_smile:

Does that sound like a plausible trajectory? It’s sort of what I was envisioning when I said in my first post on this thread that PyBI could be “be a step towards a manager-first ecosystem”. Is that what other people see as the potential benefit from this down the road?[3]

However, even though I see this as plausible, I have some reservations about it. The main one is that, well, I don’t see what is so critical about steps 1 and 2[4]. The meat of what needs to be done is in step 3, namely the unification of existing functionality that currently requires people to use multiple competing tools. A significant amount of that work has already been done by some of those tools, especially (as it seems to me) poetry and conda. Why “start over” with PyBI instead of beginning with what those tools already do? If later it turns out that PyBI is just the right way to provide Python within the context of such a tool, we can think about that then.

The other really important step is step 5, namely some major shift in official guidance on how to get Python (or at least what tool should be “the default” one to use to work with it once you get it, which relates to another point that I’ll get to in a sec). And for that also I remain uncertain as to the specific need for pybi, rather than simply having official guidance that points towards existing tools like poetry or conda (or evolutions from them).

The main advantage of pybi is apparently that it provides a way to install Python that ties into the existing PyPI system. However, I’m unclear on how important or even desirable this is on a trajectory like the one I outlined above. If step 5 occurs and the official way to get Python shifts so now you get a manager tool, it won’t matter whether that manager tool gets its internal Python from PyPI or not, because the user won’t need to care about where that internal Python comes from. And if step 5 doesn’t happen, and we just get a new crop of various tools using pybi (or a multiplicative effect where existing tools may or may not support pybi in addition to everything else), then that would mean fragmentation increased rather than decreasing.

So from my perspective a trajectory like the above rests heavily on the assumption that the existence of pybi will somehow facilitate development of a tool that, like conda, lets you manage Python as part of an environment, but the new tool will leapfrog conda in terms of functionality and/or popularity. I suppose it’s plausible that the tie-in to PyPI could attract more users and/or developers than the separate ecosystem of conda. But I’m still a bit skeptical.

Also, this trajectory itself still would not address a significant pain point with the current PyPI ecosystem, which is the inability to handle non-Python dependencies. There may be ways to resolve that too, but again to me they seem orthogonal to pybi.

That’s an interesting point. You’re right that that inference isn’t directly supported by the survey. The way I fill that logical gap is that I think if people say “the existence of multiple tools is not beneficial to the Python packaging ecosystem” and “I prefer to use one packaging tool rather than several”[5], a natural idea for how to resolve that is to make sure there is “one tool” that meets the needs of a broad segment of the userbase, and make sure that tool comes with Python, or is itself the way to get Python (i.e., a “manager” of some sort).

So it’s not that people have confusion over where to get Python per se, but that they expect to get a full-featured packaging tool when they get it, and they don’t currently get such a tool from the official or most common places to get Python from, so then they have to wade into the confusing world of conflicting information on which tool that might be. That could potentially be resolved by giving them the tool they want along with Python.

I agree that it’s possible that the problem could instead be resolved with better guidance in the documentation. The thing is, though, if that’s the case, I see even less reason to go with something like PyBI rather than simply immediately changing the documentation to recommend some existing third-party tool(s), because the existing tools, well, already exist, and are way more full-featured than PyBI right now. In that situation, the main task would be to evaluate the existing tools and gather more user feedback to see which workflow seems to cover the broadest swath of what people want, and to figure out how to structure the docs to introduce people to that workflow.

I share your frustration to some extent, but in my view there is a risk of confusing movement with progress. It’s not clear to me that Pybi will move us forward any more than this discussion will.


  1. modulo the unsolvable problem of old blog posts and the like showing up in google search results ↩︎

  2. and perhaps are “officially encouraged” to do so :slight_smile: ↩︎

  3. I’m leaving aside some other potential benefits, such as easier distribution of totally self-contained Python apps, to focus on what’s germane to the fragmentation problem. ↩︎

  4. especially since there already is a tool that does step 2, whose name we all probably know by now :slight_smile: ↩︎

  5. which were the actual plurality viewpoints in the survey ↩︎

5 Likes

What sort of timescales are you thinking of for this transition? The above statement makes me think you’re talking on the order of 10 years, as I doubt it’ll be feasible to replace PyPI in anything under that time (unless it’s done by some sort of migration of existing wheels, which sort of defeats the purpose in the context of this discussion). We’re still finding people who are working on transitioning from distutils to setuptools - what makes you think that transitioning from wheels/PyPI to a different system will be any faster? And how will that le less “fragmenting” than incrementally improving the PyPI/wheel system?

If conda-forge could become that system, then that might have a chance. See below for more on that.

I could argue that PyBI gives “wheel based” tools the opportunity to add “manage your Python installation” capabilities that they currently don’t have, so how is this not a step towards your “end goal”? But I won’t, because I’m not actually convinced that the “one tool to manage everything” approach is the right solution. I personally still prefer the poetry/hatch/PDM model of getting your Python installation independently, and then have a workflow manager tool “on top of that”.

Personally, I don’t think that conclusion is warranted. In fact, I think the results of the survey suggest that there’s a really important question that we haven’t asked yet (possibly because it’s likely to cause a certain amount of conflict :slightly_frowning_face:) - namely “So why don’t you use conda?”

That’s a serious question - if the user community is strongly in favour of a single tool, and conda has been around for so long, why aren’t they using it? That suggests that there are important things conda doesn’t get right, at least for a significant part of the user base. And while I have personal views, and reasons for not wanting to use conda, that’s not important here - what we need is facts, not speculation. And building a conda-like system when we don’t know why users aren’t using the existing one strikes me as rather naïve.

I’m reluctant to go any further with this discussion on this thread, though, as I think it’s veering sharply away from the subject of PyBI.

Let’s get back to discussing PEP 711 directly. At some point, @njs is going to submit it for pronouncement, and it would be helpful if we make sure that by the time that happens, any technical issues around the PEP are discussed, rather than having the discussion swamped with (more!) strategy debates.

2 Likes

(First, @BrenBarn I think your last post is fantastic, but I have nothing to add to it or reply to, which is why it looks like I’m ignoring you :smiley: )

I agree, but honestly there doesn’t seem to be anything technical to debate about it? It fulfills the goal it sets out to fill, based on past experience with similar systems, and scopes-out issues that it doesn’t want to address at this time. We could argue endlessly about whether Windows users should just enable symlinks if you want, I suppose, but the proposed approach works in reality.

The proposal is fine, the question is whether it’s necessary. Yes we can add a standard format for distributing Python binaries, but do we need a standard? We already have the format and a tool that uses it, so what does the standard provide? (I have a range of ideas, but only the cynical ones are at all compelling :wink: )

2 Likes

I have a technical question (really rewording what @uranusjr brought up at the beginning of the thread) and two suggestions.

The question is: what are the limitations/issues of the builds with the example tooling (given the non-insignificant issues raised on Behavior Quirks — python-build-standalone documentation)? Following on from that, from what I can see, the PEP itself doesn’t actually specify how PyBIs are built (instead focusing on the transport format and metadata), so there seems some freedom in how PyBIs are built/configured (possibly allowing some of the quirks mentioned in the standalone page to be solved for specific builds).

This brings up the first suggestion: have an additional field in the file name which indicates origin/builder/configuration. Assuming the idea is to have multiple groups produce PyBIs, this would allow for the origin to be filtered on/managed so that specific builds that are known to work for a specific deployment can be used (or conversely, known bad builds can be skipped).

The second suggestion would be to clarify that this PEP does not specify how PyBIs are built, and assuming that python-build-standalone will support PyBI, also link to it as an example also (to emphasise the independence of the format from specific builds).

Also, a more general question: rather than users having to “reinstall” using the Official Installers every release, would it be possible (i.e. what would be the issues with) if the installers rather installed a tool which managed PyBIs (and included whichever version you selected to enable offline installs), and that trying to install with a new installer popped up with a message saying “You don’t need to run this installer, you’ve already got Python, here’s how to manage Python versions using your installation” or similar to try to help with reducing xkcd: Python Environment?

I’ll fully admit my bias towards conda on this, but with that said, there’s an alternative explanation that in my view explains the situation far more than frustration with the capabilities of conda (which exists, and is valid): people have a strong need to follow “official guidance”, especially in the corporate world, because if you do something homegrown and it breaks, it’s your fault.

I’ve seen this happen consistently across many different clients I’ve worked with, and – believe me – in an average IT department, the joke in the A of PyPA really doesn’t come through. So then it’s a bit of a self-fulfilling prophecy, conda is not an “official” recommendation from the perceived language authorities, and so people will look for other ways[1]. This is of course just a high-level oversimplification, but you get the picture.

This leads to a degree of self-selection[2] where – predominantly – only those who have been through enough pain with native dependencies (or cross-platform compatibility, or lack of binaries on windows, …; any of the points that conda unequivocally improves upon) make the tradeoff to veer off the beaten track.

On a personal note as a heavy conda user[3], I am constantly confronted with that dichotomy, and I’d really like there to be a python-universal solution to the problem[4]. But I don’t care if the solution looks like conda or not, as long as it solves at least the same set of problems.


  1. actually, it takes on a kind of life of its own, where people expect (and loudly demand) that every project interoperates with the “official” way, even where wheels as a format are stretched to the breaking point. ↩︎

  2. of course strongly dependent on what kind of packages people want to consume ↩︎

  3. who cannot go back to the way wheels on windows are unavailable, broken or incompatible for many packages I’m interested in ↩︎

  4. add me to the list of people from the survey I guess; I didn’t see it when it was live ↩︎

4 Likes

Maybe I should have explicitly said this, but I consider that as another equally valid reason why people aren’t using conda. My point was that we can speculate, but what we really need to do is ask the people who responded, saying that they wanted a single tool, what prevents them from using one that exists.

And while my wording wasn’t ideal, I still consider this as “something conda doesn’t get right”, in the sense of not satisfying a requirement that those users have. I’ve been trying not to word things in a way that suggests blame, but this one slipped through - sorry. The point here is that if conda isn’t getting adopted because people want an “officially sanctioned” solution, we need to work out if that’s fixable.

Another thing we need to know is who are the “perceived language authorities” in this context. If the PyPA were to recommend conda, but python.org continued shipping binaries, Microsoft continued making Python available from the Windows Store, and Linux distros continued to ship a python command available by default, wouldn’t this just be perceived as still being a mixed message? I don’t know - we need to ask the people telling us it matters to them.

Regarding “authority”, I agree that it’s an important point. But speaking for myself, I consider python.org the authoritative distribution, and after that, my OS vendor (the Windows Store distribution, in my case). The PyPA has no bearing for me on where I get my Python from, only on where I get packages for that Python from. Consider that my response to the survey.

[Sorry, it is gonna be another “strategy post” rather than one that discusses the PEP itself. Maybe someone can split this, I have no idea where to cut though, it might be tricky…]

From my point of view I do not care much whether PEP 711 exactly gets accepted[1], but I wish something in that vein does get standardized. Does conda (& co.) have something akin to this proposal here that could get standardized? Has it been submitted as a proposal already at some point?

I understand feelings such as “we do not want conda in our infrastructure because it is not official[2]. So what about making some of conda’s technical solutions into standards (PEPs) bit by bit so that on one side interoperability is improved altogether and on the other side conda becomes more “official” in some eyes (whatever official means here)? Has it been tried already? Is it something that is happening in the background? Is it impossible? Technically? Or on principles alone?


[1] I do not have the knowledge to judge the real qualities of PEP 711, but I do like the workflows and user experience I believe it would enable (and I do not mean the one packaging tool ™ but interoperable packaging tools, libraries, workflows, processes, user experiences).

[2] I do not agree that it is a sensible decision on the “not official” criteria alone, but I can understand that such reasoning exists.

4 Likes

I agree with this, big time.

Another piece of it is the combination of conda’s “separate universe” approach and its lack of officialness. Because conda isn’t the first thing that comes up when you go looking for stuff like “how to install Python”, it’s often not what people start with; and if they then do learn about it, there’s a bit of a price to pay to switch because, well, you have to switch, you can’t just add conda to your existing setup. Like @h-vetinari I’m biased, but I do think there are a lot of people who don’t use conda but would be happier if they did, and even more people who don’t use conda and might find it annoying to switch now but still would be happier if they had started with it.

In this sense what matters is not even what’s official in the sense of supported vs. homegrown, but rather just what is the messaging people get about what Python is and how to use it. Maybe that messaging can be just “there are several ways to start, here are some of the main ones with a brief comparison of their features, maybe you don’t understand enough to evaluate this right now but at least you know it’s there”. I think that would still be better than “Just download from Python.org and not realize until later that other choices existed and may have been wiser”.

In fact, throughout these various threads, I get the sense that one of the areas of disagreement that’s not been so often brought explicitly to the fore is basically about this issue: it’s about whether the obstacles to “unification” are technical or social. To me, the existence of many tools doing useful things suggests that a lot of the biggest problems are not technical. It’s more that a) functionality is fragmented across multiple tools; and b) in particular that means the official tools have some important functionality missing.

It is also related to a seeming tension between, on the one hand, a desire to separate packaging concerns from official Python things like the stdlib (for which there are legitimate reasons, like concerns about release cycles), and, on the other, a reluctance to have the official documentation explicitly tell people to use third-party tools. It sometimes feels as if we’re a bit trapped because we can’t make big breaking changes to pip or PyPI, but also (as far as I can see) people seem leery of ideas like having the main Python packaging docs page[1] tell people straight up “You should consider using conda or poetry instead because these official tools have serious limitations” (also perhaps for legitimate reasons, like worry about directing people to some external resource that could change over time in unknown ways). But the way I interpret the user feedback is one or the other of those has gotta give; either the official tools need to incorporate significant functionality that currently can only be had from third-party tools, or the official docs need to be more forthcoming about the limitations of the official tools.

Personally I agree with @h-vetinari that the social factors are of great importance. That’s one reason why, to tie things back to this PEP, I don’t see the specific need for something like PyBI. Its approach is interesting from a technical perspective, but it’s not the first approach to solving the problem it solves, and existing tools with other approaches solve a lot more problems, and in any case the technical issues don’t seem like the biggest obstacle to me. The bigger challenge is coming up with a clear message about how to do things and effectively communicating that in some way that has enough official aura to clear the fog that currently befuddles many users trying to work with Python packaging.

I also agree with @pf_moore that it makes sense to get more solid data on this. The survey we have is a good starting point, but it doesn’t delve into the details of what people’s actual pain points are, let alone how to solve them. Of course each of us in these discussions has their own experiences that lead them to view problems and solutions in their own way. :slight_smile: I do think there’s a decent amount of consensus for some of the problems (e.g., it seems clear that lockfile-type functionality is something a lot of people really want), but not all, and there’s much more disagreement about possible solutions.

Gathering such data probably requires some care, though. The nature of these problems is such that a simple multiple-choice survey may not be enough to get a full picture (for instance because different people may have “the same problem” but conceptualize it with slightly different terminology, or have not explored things enough to make it more specific than “I tried installing package x and it broke my environment”).


  1. I mean on docs.python.org, not on packaging.python.org ↩︎

I’m not sure if that’s necessary simply because you will probably download the files from some namespaced location. For instance, we don’t embed who built a wheel in a wheel file name, so why would this situation have to be different?

I will also say that I would love to get distributor information into the sys module ala PEP 514 – Python registration in the Windows registry | peps.python.org . That way the sys module covers the Python version, implementation and version, and who built/distributed the binary.

It’s possible, but one thing at a time. :wink: This PEP isn’t trying to specify how to distribute things on python.org, just a distribution format. Now it may come to pass that some official tool comes out like rustup whose entire job is to help you install and manage Python (see the discussion here about fragmentation for that topic), but that isn’t necessarily known upfront.

No one has written PEPs because whenever the sentiment to standardize what conda has done (more specifically: dealing with native, non-Python dependencies which have an ABI that needs to be held constant across consumers) comes up, people have – understandably! – shied away from it, because it is a huge can of worms.

In other words, it has so far always been declared “out of scope” for Python packaging, and thus no-one wasted their time writing PEPs for something that will not be considered.

Ironically, PyBI might be a (small) step in such a direction, even if unintentionally:

1 Like

Running Distributions — python-build-standalone documentation seems to have a categorisation that is more detailed than the current wheel tags, and it would seem reasonable to me that a single provider would have at least as many options as python-build-standalone, so the extra field would cover that need. Having both @njs’s tooling and python-build-standalone current system being able to fully be expressed by the metadata would show that the PEP hasn’t missed something.

The installer question was more “could the official installers use this, or does this PEP need changes to meet their needs?”, given the caveats python-build-standalone lists. I don’t think there are any, but I figure having a yes or no from those maintainers would be good (and adding a paragraph to the comments section noting this would be useful for future readers of the PEP).

I know it’s confusing because I’m employed there, but the Store distribution is from the PSF, not from Microsoft. All the login accounts are separate, and the builds are literally the same binaries as in the python.org installers (apart from python.exe itself, which needs a different implementation to locate its install location, and since I believe that’s always belonged in python.exe and not the DLL, I put it there). All Microsoft has to do with it is provide the developer account for free, as they always do for charitable orgs.

(The preinstalled python.exe that directs to the Store is entirely Microsoft, though. Just as the winget configuration that points to python.org is also entirely Microsoft, though the resulting binaries are not.)

I deliberately avoided suggesting the “unofficialness” as a problem, but now that other have brought it up I’ll vouch that I see it regularly.

And I like your framing here, because “forthcoming about the limitations of the official tools” is far more palatable than promoting specific 3rd party products. But as long as these are the interactions new users get, there’s far more to be done than simply updating official docs.

It definitely requires a lot of care! At this point, I think it really needs to be interviews with an interviewer who knows how to dig deep, not merely find answers to questions. Expensive and time-consuming, but we’re just not going to capture the nuance we seem to need in any self-categorisation survey.

2 Likes