Publish Linux installer on python.org

Ideas
81

No, Alpine and Debian-based distros are the outliers. The rest of the world can deal with multiple Python installations out of the box.

  • Arch supports multiple Python versions via AUR
  • CentOS supports multiple Python versions
  • Gentoo supports multiple Python versions
  • Fedora supports multiple Python versions
  • FreeBSD supports multiple Python versions via ports
  • RHEL and RHEL derivates like AlmaLinux and Rocky Linux supports multiple Python versions
  • OpenSUSE supports multiple Python versions
  • CPython and PyPy upstream are designed to handle parallel installations of multiple versions at the same time. CPythonā€™s macOS and Windows installers support multiple Python versions.
  • Conda supports multiple Python versions

The existence of deadsnakes PPA shows that itā€™s technically possible to support multiple Python versions on a Debian-based system, too. Debian and Ubuntu simply decided against solving the problem and against providing multiple Python versions to their users. They ship multiple versions of other packages, e.g. latest Ubuntu has gcc 9-14 and llvm 14-18.

8 Likes
82

Is that really the case? I have multiple Pythons installed, even on the computers that Iā€™m not building from source on. Thereā€™s a single python3 package but you can install python3.11 to get a specific version. There arenā€™t usually more than two Python versions in any given Debian release, and some of the recent ones have only included one, but thereā€™ve definitely been more versions in others; for example, Debian Trixie (current ā€œTestingā€ distribution) includes both 3.12 and 3.13.

Is the problem that there arenā€™t enough different versions available, or that the system Python is always a single specific version? The former doesnā€™t seem like a fundamental problem, and the latter makes a lot of sense to me (if you want, say, Python 3.9, it shouldnā€™t be a problem that you also have a Python 3.11 for the sake of system scripts).

2 Likes
83

Disclaimer: Iā€™m not a Debian expert.

Yes, sometimes Debian and Ubuntu have two Python versions. For example, Ubuntu 22.04 LTS has Python 3.10 and 3.11. However the 3.11 version is not useful at all. Itā€™s release candidate 3.11.0rc1 from August 2022.

I guess Debian Trixie currently has 3.12 and 3.13-dev in preparation of moving Trixie to Python 3.13 as system Python. Trixie will become the next stable in 2025. I would be surprised if Debian ship 3.12 in next stable.

2 Likes
84

my observation is that Ubuntu continuously reduced the number of available versions per edition. my interpretation is that Python versions can be considered as increasingly coherent since 3.6 and the current release schedule makes the process more projectable by the package maintainers.

i donā€™t understand what would make a Linux ā€œinstallerā€ (i know that such things exist, but i rarely used these in the past 25 years) desirable, but maybe an overview with distribution specific installation instructions (including a hint to the fantastic deadsnakes repo) like you may know it from other software projects would be a help to those who feel so lost that they retrieve https://python.org/download.

1 Like
85

Maybe an option is for someone to write a mostly-universal shell script that can install a local python with a given version.

That kind of sounds like pyenv. Why not just a simple clickable shell script or other wrapper around pyenv to get python with a specific version in a folder?

ā€¦ then link that in the downloads section.

It avoids worries about messing with the system Python, it wouldnā€™t be much work, sounds like a cute little gui could be written around it if desired.

Of course there would be edge cases but probably less than other ways.

86

pyenv builds from source on linux/mac so you both need to install build dependencies and wait for the build process, which can be significant if you want an optimised build[1].

Ideally I would just want the same general experience for installing Python that you get on windows. Go to python.org - open the downloads tab, get something that will just guide the user through a few steps - preferably without needing to enter specific commands into a terminal - and have the latest stable python installed and available. Currently if you click the big download button on Linux you just get the source bundle, which isnā€™t particularly helpful if youā€™re looking for some kind of guided install as you would have on other platforms.

  1. Building an optimised CPython on linux on my aging laptop can take over 10 minutes, which isnā€™t a great experience compared to pyenv-win that uses the windows installers and takes closer to 10 seconds. ā†©ļøŽ

6 Likes
87

Yes, there should be some notice on the download page that Linux users should use their system packager to install (if even necessary; I wonder how many Unixā„¢-like systems donā€™t have Python already installed).

4 Likes
88

Yes. We (Debian) support having multiple Pythons installed concurrently, to ease distribution development. But we donā€™t want to support multiple versions of Python in our stable releases. We aim to support one version per release. Ubuntu now sometimes supports a 2nd version in LTS.

More versions are absolutely possible to be provided outside the distribution, a la deadsnakes, and I think I may have found some people in Debian who are interested in helping to make that happen. But no movement yet. The caveat is that these extra versions will not have C extensions compiled for them, of course. But thatā€™s a non-issue for most of the developer community who will live in virtualenvs.

89

Yeah, the extra runtimes in a Linux environment are almost entirely fodder for tox & nox running test suites and other commands in multiple environments, developers locking specific target Python versions for general development activities, and tools like pipx & uv tool being able to upgrade the Python environments they manage independent of the timing of the OS updating the main system Python package. The fact everything above the Python runtime layer needs to be managed with Python level tooling is a feature for this use case rather than being a problem.

(edit: switched example to python3.11, as I forgot the mechanism for declaring the weak dependencies changed to something easier to maintain for the 3.11 release. For anyone unfamiliar with weak dependency declarations in Linux systems, see Weak Dependencies Policy :: Fedora Docs)

Using python3.11 in Fedora 40 as an example, only the other components built from the same SRPM have a hard dependency on the binary RPM containing the CPython CLI:

~$ sudo dnf repoquery --whatrequires python3.11 2>/dev/null | grep '3.11.9.*x86_64'
python3.11-debug-0:3.11.9-6.fc40.x86_64
python3.11-devel-0:3.11.9-6.fc40.x86_64
python3.11-idle-0:3.11.9-6.fc40.x86_64
python3.11-test-0:3.11.9-6.fc40.x86_64
python3.11-tkinter-0:3.11.9-6.fc40.x86_64

And only the Python 3.11 shared libraries directly recommend it:

~$ sudo dnf repoquery --whatrecommends python3.11 2>/dev/null --installed
python3.11-libs-0:3.11.9-2.fc40.x86_64

However, dnf install tox still defaults to setting up a machine to run against every Python environment that Fedora supports, either because the tox package recommends the runtime:

~$ sudo dnf repoquery --recommends tox 2>/dev/null
pypy2-devel
pypy3-devel
python2-devel
python2.7
python3-devel
python3.10
python3.6
python3.7
python3.8
python3.9

or because the runtimeā€™s devel package indicates that it supplements tox (this declaration is in the devel packages so that it also installs the C/C++ header files needed to build binary extensions from source):

~$ sudo dnf repoquery --whatsupplements tox 2>/dev/null --installed
pypy3.10-devel-0:7.3.15-2.3.10.fc40.x86_64
pypy3.9-devel-0:7.3.16-1.3.9.fc40.x86_64
python3-devel-0:3.12.4-1.fc40.x86_64
python3.10-devel-0:3.10.14-2.fc40.x86_64
python3.11-devel-0:3.11.9-2.fc40.x86_64
python3.13-devel-0:3.13.0~rc1-2.fc40.x86_64

Itā€™s that developer experience of ā€œtox just works for Python cross-version testingā€` that is hard to replicate outside the main repositories for a distribution (while 3rd party repositories and custom Python builds or downloads can make this kind of cross-version testing setup possible, only the main distro project can truly make it easy).

(FWIW, you can put me in the camp of folks that believe the level of demand for pre-built Linux binaries would be much lower if tox worked nicely on Debian by default. Iā€™m generally in favour of python.org offering something comparable to the python-build-standalone builds, since there are some genuine use cases for that, but I see it as a workaround for the Debian ā€œparallel Python installation for cross-version compatibility testingā€ developer UX issue, rather than a genuine fix for it. Most notably, the standalone binary approach means that each runtime bundles its own copy of OpenSSL instead of using the common system provided shared library)

@hroncok I noticed that tox isnā€™t currently recommending python3.11 while writing the above illustrative commands, so I posted a BZ for that (Not a bug, the way the weak dependencies are declared just changed as per @barry-scottā€™s post below)

90

The 3.11 and 3.12 RPMs have this in them:

# tox users are likely to need the devel subpackage
Supplements: tox

And this Using reverse week deps for Python interpreters and tox (Supplements instead of Recommends)? - python-devel - Fedora mailing-lists explains why Recomends: in tox was replaced with the Supplements: in the python spec files. I do not know what dnf search wil show this info as ths obvious dnf repoquery --supplements tox returns nothing.

1 Like
91

Linux is not an operating system. Itā€™s a kernel folks add a bunch of GNU, et. al. stuff to make operating systems. Itā€™s a deep rabbit hole to go down.

As an Ubuntu admin itā€™s easy enough to add the deadsnakes PPA and add additional Pythons. I know Iā€™m going outside the stable system which depends on a stable Python for that release.

Just as the stdlib provides a stable limited set of functionality that can be extended by going to pypi.org, Ubuntu provides a stable limited set of functionality that can be extended by adding PPAs.

92

FWIW I think should core CPython adopt GitHub - indygreg/python-build-standalone: Produce redistributable builds of Python and make it part of the release process would solve this need for 99% of the users :thinking:

6 Likes
93

By the way, besides this excellent project there are others

94

We can use a code for installing Linux through using cmd

95

Iā€™d like to second this. What the uv team is doing to improve the Python bootstrapping/development experience is very exciting but one thing that is still a little sub-optimital is that when uv bootstraps Python itā€™s using python-build-standalone, which isnā€™t officially supported by the CPython team.

I thought I heard on a podcast or talk that Łukasz or someone mentioned that in recent Python releases, they helped out/incorporated the deadsnakes PPA releases at the same time as the CPython releases. Perhaps something similar could happen with python-build-standalone.

4 Likes
96

Yep, and from reading this thread itā€™s also clear that itā€™s not just uv (and rye before it) that is taking that approach but also pdm and hatch, so itā€™s going to be the source of Python for increasing numbers of users.

Official binaries were also noted as something that a better solution for Microsoft Store Python would ideally make use of.

Maybe even conda would switch to using official binaries should they exist (though perhaps thereā€™s some technical differences in their interpreters that I am ignorant to).

It seems like adding official binaries would be a good first step that would enable or improve many of the ways that Python is already currently obtained.

Then a lightweight installer that can be downloaded from python.org that installs said binaries might be really nice for completely new users, but that could be a second step to be considered separately and on its own merits. Or even a more capable official Python management tool, as floated in the MS Store thread. All easier with official binaries available.

Maintenance workload is naturally a factor but there is already developer time being spent on maintaining such binaries, right? ā€“ just not under the official Python umbrella. :slight_smile: Perhaps the folks who work on the indygreg/python-build-standalone builds would be up for helping out/transferring their efforts. The uv devs have also contributed significantly to that project, and would presumably rather be using and contributing to upstream ā€œrealā€ Python.

2 Likes
97

99% is ambitious, unless youā€™ve got some data showing that 99% of the users are using the same version of the same distro.

As weā€™ve said a number of times, weā€™re open to PRs that upstream virtually any of that functionality. Whatā€™s missing are the dedicated people who will maintain it, including handling all the reports of issues that will start coming to the core project (that we could no longer deflect) instead of the teams who are familiar with each OS. We canā€™t make fixes any faster than the release cycle, even if itā€™s totally unrelated to CPython (as it often is).

Thereā€™s a lot more involved in maintaining a CPython distribution for an OS than maintaining a source release. Particularly where the OS is designed around building from source (as opposed to Windows and macOS, which are built around binary distribution).

1 Like
98

Are there technical reasons why Python users of different distros canā€™t use the same build of the Python interpreter to execute their Python code? (I would be interested to learn.)

For the system Python it makes intuitive sense to me that patches can be necessary because so many system packages depend on it and it is therefore heavily intertwined with the runtime behaviour of the distro.

But why canā€™t a single non-system Python binary for Python users, used in isolation from the system via virtual environments, suit everyone on Linux?

Is that a fair characterization of Linux? Most users donā€™t ever build anything from source, they exclusively use distributed binaries.

1 Like
99

Maybe not so much from source but they rarely use statically linked all dependencies included binaries. Dynamic linking is a key performance and security feature of Linux. Throw that away and you donā€™t really have Linux anymore.

3 Likes
100

As @bwoodsend said, dynamic linking to system binaries is a feature of the distribution. But to achieve this, the application has to be compiled with awareness of the environment, which may include awareness of how to determine the environment at runtime (e.g. by inspecting an environment variable). Since the environments vary so much, this is unreliable at best, but more likely just fundamentally broken for most distributions.[1]

If we statically link everything, then yes, it will be portable. But now we arenā€™t just distributing CPython - weā€™re a distributor for everything CPython depends on, including libc and OpenSSL. Whenever a user wants a security fix for any of these, they need to wait for us to make a new release. Thatā€™s a silly amount of work, and while we do it for Windows and macOS, in general itā€™s only for some dependencies (not as many as Linux would need) and we tend to presume that these OSs are not for certain scenarios.[2]

The most efficient way to get a working distro of CPython on the range of platforms that uses the systemā€™s own libraries is for the distribution to build it themselves. Which is where we are today.

An alternative either has to drastically reduce the number of OS configurations (ideally to 1-2), or the number of 3rd party dependencies (ideally to 1-2). Either way, most users lose, and so it hasnā€™t looked like a good idea to actually do either of these.

  1. Iā€™m going to avoid speaking in terms of numbers of users, because I have no idea what the breakdown looks like. But I can confidently say that a build using Ubuntuā€™s system libraries wonā€™t work on Alpine, even though I donā€™t know how many users each distro has. ā†©ļøŽ

  2. For example, our distros for Windows are not suited for internet-facing servers, without a reverse proxy that can properly do the stuff that (e.g.) OpenSSL canā€™t. ā†©ļøŽ

4 Likes