This discussion thread is to understand community requirements around how Python is distributed.
Currently, official built Python releases are distributed from python.org as installers for Mac and Windows platforms. Is there a pre-existing community requirement that is not currently being filled by these options that would make sense for being distributed via python.org?
For example,
Would pre-built binaries from a tarball be useful?
Would getting rid of installers and relying on the community for such things be problematic?
What would make Python work better with respect to distribution?
Are there any solutions that you would be interested in seeing implemented or any that we should avoid?
The main outcome of this discussion thread would be to hear your thoughts on the existing solution or any new or existing ideas it brings to light. There is no compulsion to identify a solution right away. If there are competing solutions, they would need to be proposed as PEPs for the steering council to decide on how to proceed.
Relocatable Python seems to be a recurring demand. Python distributed as pip-installable wheels from PyPI (or something similar). So that developers can distribute Python applications that contain the interpreter.
I know there are some solutions around (I have never tried them). Maybe we need “blessed” technical solutions (PEPs if necessary).
I need (for monotrail but also in general) python distributions that you can just download, unzip and run for all major operating systems and architectures, something that python-build-standalone currently provides. I know there are some reason why it isn’t that easy, but for me blessing python-build-standalone and putting it on python.org together with a json index would be ideal. For my specific use case (preparing an environment then having python main take over the process) having a separate pythonxy.so (or platform equivalent) is also important.
Additionally, it would be cool if cpython and pypy had the same downloadable format. That does not necessarily mean putting pypy on python.org, but it would be neat if could switch urls and get the same format for both.
From a normal python user perspective classic installers and a platform independent launcher (see e.g. python-launcher) would be important
I agree that relocatable unzip-and-run distributions would be useful (windows more or less has these in the embeddable distributions) but I would caution against using wheels as the distribution mechanism. Wheels were designed for package distribution, not application distribution (even if entry points mean that people use them in that role) and I definitely don’t think they are a good fit for distributing Python (wheel tags are an obvious problem - they don’t capture the variety of builds people are likely to want for Python).
By all means, let’s work out how to do unzip-and-run builds. But not via the wheel format, please.
But the other question - wouldn’t this be something for the core devs to produce? I don’t see the packaging community maintaining patches to allow building these builds. I know there are 3rd party builds, but I get the impression that they would be a lot easier if the tricky bits could be upstreamed into core…
Right, I should not have phrased it like this. My fault. I do not think Python interpreters being distributed on PyPI as wheels should be the goal on a technical level, but on a conceptual level I feel like it could be a good thing to aim for, as in it should be as easy as pip-installing wheels from PyPI. I think this argument was already made in the context of Pybi and Posy.
I am wondering a bit if Packaging is the right place for this thread.
More in the Nuget distributions, which have similar limitations to using wheels but at least don’t require an existing Python install to get a new Python install.[1]
The script wrappers generated by pip are the only actual issue with relocatable installs on Windows. (Users who use python -m pip rather than simply pip have no problems.)
Every problem after that one is user expectation (“why can’t py.exe find it?”, “why do my venvs break when I move it?”, “why can’t I get updates?”, “why doesn’t my IDE recognise it?”, “why isn’t it on PATH?”, “why does/doesn’t it include the test suite?”, etc.), and I am quite concerned about users choosing an unzipped build rather than the installer and then wondering why things that require an actual installer don’t work.
Either the core devs, or some new hypothetical group that takes ownership of “distribution and packaging” rather than treating those two things separately.
Or, of course, any third-party who wants can distribute whatever they like. We allow and encourage that, just as we expect them to provide support for their users when [things from my list above] don’t work.
I think the most valuable thing here would be to distribute Python builds that explicitly identify their wheel tag. So you know you’ve got a win_arm64 build of Python, or a manylinux2014 build, and so you know which prebuilt wheels are going to be compatible.
The actual packaging of the Python build doesn’t matter so much. Being able to easily identify which wheels were built for that build does matter.
At least for the use case I have in mind, these would not matter, the end user would not interact with the Python interpreter directly (or maybe at worst the user would be directly dropped into the REPL). The use case would be for Python developers to build standalone applications that include the interpreter.
[Meta: Should we discuss details in this thread? I would be happy to discuss further in a dedicated thread. We maybe have identified a possible gap, I would assume that is good enough for now. @smm?]
That’s literally what the embeddable distro is for, so it exists on Windows (and the standalone builds handle other platforms).
And I agree that when you properly understand your own use case, you can select the right tool. My concern is that most users who end up on python.org don’t actually understand their own use case (yet), and so will choose incorrectly. (My evidence for this is the relative frequency of users who currently choose the embeddable distro and need to have explained that they chose the wrong tool, and also the number of complaints about having too many download options. Increasing the number of options seems unlikely to help.)
And yes, this will start getting off topic very quick. There’s an open issue somewhere on GitHub about it already (and many many closed ones).
I’ve found it difficult to install specific versions (usually the latest) of Python on various Linux distros. The most egregious is non-LTS versions of Ubuntu, which the deadsnakes PPA repo doesn’t support.
Whether or not I use pyenv to manage the version, it still has to be complied from source (unless there’s another reliable source for pre-compiled distributions that I haven’t heard of), which has two main issues: compilation time, and figuring out the required dependencies for building (and which can be uninstalled after compilation).
I don’t know if this is within the purview of official channels, but distributing a snap/flatpak seems similar to the Microsoft store distribution for windows, assuming it won’t conflict with the system Python (or of it does, manage that, eg with a launcher) and can access all files/devices.
So not a single binary is fine, correct? I.e., if that zip file contained a bunch of files and directories and the location of the interpreter was predetermined/known within that zip file, that would be enough?
Yes, but the SC wanted feedback from folks dealing with packaging to find out how Python is distributed may be impacting how code is being packaged (i.e. this post was done at the request of the SC).
Some of you wanted more core dev participation, now you have a small bit.
If such a thing existed, it might make sense to message that the zip files are for tool usage and not general usage (so maybe list it in a separate place or not at all on the downloads pages)?
I think that’s totally reasonable, it just might push wheel tags more into the stdlib than they currently are (which I am not saying is either a good or bad thing).
Are we talking down to the micro version? Release level?
To briefly restate what I mentioned repeatedly on other threads: I think the “main” form in which Python is released should not be the Python interpreter per se, but rather a manager which can:
manage environments with different collections of packages installed
manage the version of Python in an environment in the same way as other packages in the environment
do both of the above in a manner that is integrated with the installation of packages (i.e., manage environments and install packages into them with a single tool, or at least a set of tools designed as a single cohesive unit)
The official Python distributions should encourage and embody a workflow in which all usage of Python takes place within the context of an explicitly managed environment. As long as there is a one-shot way to run a command in the context of a particular environment without requiring a shell activation step (e.g., pymanager --env someenv --run somecommand), I don’t see why this couldn’t be extended in time even to system-level tools (e.g., Linux distros could create their own system environments, and what we currently know as “the system Python” would be just one more environment managed by a manager).
See above.
Avoid anything that puts the environment inside Python rather than Python inside the environment.
Cool. I personally think that with the store, Python.org, embeddable and nuget distributions, Windows is well covered.
I’d like to see better guidance on the purposes of the distributions, and I’d prefer the nuget distribution to not require a proprietary tool and just be a zipfile (I imagine that’s why people go for the embedded version rather than the nuget one). But that’s just details.
Add conda/activestate for people who want a “managed distribution”. I can’t really speak for Linux/Mac, although when I do use Linux, I miss the ability to use multiple versions of “standard” Python (Ubuntu, I’m looking at your patches…) without having to set up something like pyenv. So I’d like something like the Windows options there too.
Yes. As it happens, I recently compiled and installed 3.11 from source… twice, because I forgot (or naively assumed I wouldn’t need them, or already had them) the dependencies the first time. It wasn’t especially pleasant (and the main Python documentation doesn’t tell you about secrets like configure --enable-optimizations or make -s -j4 that you get from reading the developer’s guide instead). The alternative I guess is to figure out the deadsnakes PPA, which doesn’t seem like a lot of fun (it’s hard to verify that it’s official/safe, doesn’t seem terribly well documented, and just generally the idea of using a third-party PPA is something a lot of Linux users wouldn’t otherwise have to bother with).
Aside from that, Linux distros have gotten quite a bit easier to use in recent years, and I keep hearing more and more complaints about Windows (personally, I made the switch when Windows 10 decided to warn me, completely unprompted, that my computer didn’t meet minimum specs to run 11…). As a consequence, in the long run it would be prudent not to expect Linux users to be any more technically proficient than Windows users on average
Absolutely. I don’t want to trust third parties any more than absolutely necessary, when it comes to installing something as powerful as Python (i.e., an executable explicitly intended to “run” its data, in a Turing-complete sense, rather than simply viewing it). The Windows installer is also a huge boon for Windows users: it meets standard Windows UI expectations and includes important configuration options, especially as regards py (itself crucial because Windows has no concept of shebangs).
