IMO we should eventually have official interpreter builds stashed on pypi somewhere, in a format where you can just download and unzip, using the same platform tags as we use for binary wheels. That’s way easier and more reliable for a tool like this than trying to interface to eleventy different package managers that all have their own quirks. You could spend years trying to make that work reliably.
And the prototype version of that is for someone like David to “just do it” by making their own ad hoc builds and show everyone that the basic principle works.
I guess if you have multiple-environment support, then you probably want to also support (a) a default environment, (b) the ability to define aliases in the config, so you can tell people to pypackage run precommit or whatever and that does black+flake8, etc. And then the alias setup would say which environment to use.
I see. It’s be brilliant if the end goal is reached, but I feel we need to be extremely careful to make it a collaborated effort that covers uses for everybody (or at least most people), and can work well with as most other tools as possible. Otherwise we’re going to risk fragmenting the Python runtime ecosystem even more than it is now. I’m thinking how we have pyenv, adsf, Python-xy, Homebrew, system package managers, etc.; all of them are slightly broken in some way, and does not work with each other at all, so everyone has their favorite one, can offer no solution for anyone else, is unable to fix problems between each other, and leave users wondering why they need to install 3 different Python 3.6 on their machine.
There’s already dozens of different python distributions floating around, so I don’t think adding one more will make much difference to fragmentation. And the ability to grab an arbitrary python version, unzip, and go, is a powerful primitive that’s useful in lots of situations: I bet pyenv and similar tools would use it (not exclusively, but as an option), it would be super handy for CI scripts, and it’s useful for project management tools like pypackage or pipenv.
I imagine that if something like pypackage is successful then it will eventually want to support other system-specific methods of getting python. But for an MVP? Just do the thing that always works and figure out how to handle the more exotic needs later.
I’ve got colleagues who are keen for us (CPython) to do exactly this, and we’d happily put up the CI to do tagged and daily builds (well, it’d probably be included in the Azure Pipelines donation).
The challenge is the Linux build, of course. Given a suitable Docker container (the current manylinux1 container can’t build Python, haven’t tried newer ones) and configure/make invocation, it’s pretty easy to set this up.
The manylinux images ship with a bunch of Python versions that were built inside the container. You can check the container build scripts to see all the details:
As the 3.8 and 3.9 RM, I’d be +1 to making this happen and I’d be happy to provide source tarballs and Linux builds. If @nad publishes how to make macOS installers, I’m also up for providing those
Yes, I know, and I have. But last I tried there were issues with building Python itself inside the container. I don’t remember the details (though I think obsolete zlib was one).
I’m guessing the comment “dependencies for compiling Python that we want to remove from the final image” basically explains why it’s not directly usable for compiling Python. And I didn’t sign up to maintain my own image, since I don’t actually want this that badly myself
We don’t want installers - just ZIP extractable binaries (I’m used to referring to them as “xcopy installable”, but perhaps “cp -r installable” makes more sense for the non-Windows devs in the room.)
We don’t want installers - just ZIP extractable binaries
This sounds like a good goal but unfortunately we don’t really support something like that today in the macOS world. I suppose we could provide a standard *nix-style build (“non-framework”) in its own prefix that could be made position independent. But then you lose the advantages of running as a GUI application when you need it, something the current Python framework-style builds provide today by installing a hidden Python.app bundle within the framework install and having a hack to exec the interpreter under the app bundle when running python from a shell command line (yeah, it’s complicated but it has proved useful over the years). Ideally, we should now be distributing Python directly in an app bundle and providing a transparent way to launch that executable from a shell command line. We’d need to do some non-trivial work to make that all happen. There is also the issue of explicitly defining the mapping of wheel tags with the Python executable download tags to cover the range of versions and architectures we support today on macOS, e.g. for some branches we supply one Python executable that supports macOS versions from 10.6 through 10.14 and native 64-bit and 32-bit archs. These are all solvable issues (I think) and desirable to solve but they will require a concerted design and implementation effort, essentially defining a next-generation replacement for macOS delivery to replace the current framework scheme that has served us for many years (and that most third-party distributors of Python on macOS follow our lead on).
Sure, but not everyone needs that either (e.g. me ). There’s still potential benefit for users if we can say “if you don’t need to launch a GUI app, then this is an easy way to get Python without having to install Homebrew”.
In that case, we should intercept whatever calls someone is likely to make that would require this and give them very specific instructions. Otherwise we’re creating a feedback magnet for people who misunderstand the intended use - we already have people come in wondering why the Windows embeddable distro doesn’t include venv on a semi-regular basis (and asking about MSI installs on a more regular basis).
Being able to explain who and what Python is for in a clear, concise way is something we’re sorely lacking. The more random ways to install it that can’t be easily explained, the harder this gets.
Certainly it would be nicer if these python builds had full functionality, since the whole idea is to provide something that “just works” with a minimum of hassle.
@nad can you say a bit more about why exactly the app bundle stuff makes it hard to relocate python? Do app bundles have to be installed in a specific directory or something?
As an (expected) anecdote, simply following the readme instructions for building appears to produce binaries that, while are standalone, are very OS/distro version-specific.
can you say a bit more about why exactly the app bundle stuff makes it hard to relocate python?
It is certainly possible to make a relocatable Python embedded in an app bundle; that’s exactly what py2app does. But the Python app bundle produced by framework builds isn’t really a GUI app at all: you still need to interface with it exactly like you do a standard (non-framework) Unix build (via stdin et al in a terminal shell, for example) and the current framework build and install steps in the Makefile assume a configure-time fixed install location. So there would be work needed to support invoking Python from the command line (which is 99% of the usage today) for a Python executable in a relocatable app bundle (since the mechanisms today depend on knowing the final install locations at build time), e.g. to install symlinks in /usr/local/bin and/or add the framework bin directory to the $PATH. From an end-user perspective, we don’t do a particularly good job of sanely managing multiple Python versions (especially in the presence of the vendor-supplied system Python); - but that’s a separate issue possibly largely solvable with a py launcher.
The only true Python-based macOS GUI app (i.e. double-clickable icon) we supply today is an IDLE.app bundle in the framework build which, of course, is a (Tk-based) GUI app. It’s also possible to invoke IDLE via the command line; that would also need some hacking to retain the command line functionality with a relocatable Python app.
For completeness sake, I should mention that macOS does provide a mechanism for launching app bundles from a traditional shell: the open command (inherited from NeXT, I believe). It didn’t use to be possible to pass command line args via argv but that problem has been solved. But I’m quite sure we don’t want to be encouraging users to use that mechanism for lots of reasons, not the least of which would be start up speed.
So to check if I understand, it sounds like you’re saying:
There’s a Python app bundle, which is a python program that’s fully-featured and relocatable, but you can’t use python to invoke it directly from the command line
So we have a shim python executable, that just turns around and invokes the real Python that lives inside the app bundle
In order to do this, the shim code needs to know where to find the app bundle
And right now, the way it finds it is simply that there’s a hard-coded absolute path baked into the binary
Is seems like the solution might be for the shim bundle to be adjusted to find the path relative to its location, using something like dirname(sys.argv[0]) + "/../Python.app/python"?
Does that sound about right? Am I missing something?
There’s a Python app bundle, which is a python program that’s fully-featured and relocatable, but you can’t use python to invoke it directly from the command line
Not quite. The Python app bundle today is not relocatable and it is useless by itself, e.g. you can’t double-click on it to do anything, etc. It’s sole purpose today is to convince macOS that python is running inside of an app so that python programs being executed by this python can take advantage of macOS GUI calls,
Is seems like the solution might be for the shim bundle to be adjusted to find the path relative to its location, using something like dirname(sys.argv[0]) + "/../Python.app/python" ?
Well, no, the app bundle already knows how to figure its path dynamically. The problem is that the shim program to fork/exec the interpreter executable under the app bundle is itself in the app bundle. So the issue is how to find it. The traditional solution has been to either modify the user’s shell profile to add to PATH the fixed path to the Python framework bin directory (where the shim and any installed scripts reside). Or to install fixed path symlinks in /usr/local/bin that point to the shim.
We don’t have to solve the problems here, btw. I was just pointing out that we recognize that the current macOS framework installation defaults have some issues that need to be addressed regardless of the outcome of this proposal.
I can’t quite tell if we’re talking past each other or not, so just in case: in this thread we’re talking about wanting to be able to download a zip file, unzip it, and then run <path to unzipped directory>/bin/python and have that work. We’re not worried about adding stuff to $PATH – for the use cases we’re thinking about here, that’s unnecessary. The only tricky thing is that we don’t know which directory it’s going to be unzipped in ahead of time, so we can’t have hard-coded absolute paths baked in.
I’m attempting to tackle this one OS/version at a time, starting with Windows/Python 3.7. Testing this with no system Python installed. If I navigate to the python source directory, pcbuild, and run build.bat -p x64, A subfolder’s created called amd64. Its directory structure’s different from a normal Linux or venv Python intall, ie everything’s in one folder with no subfolders. Running python.exe here directly works as expected.
If I move this amd64 folder elsewhere and attempt to run python.exe, I receive this error: ImportError: No module named 'encodings'. This is a common question on StackOverflow, and its solution is related to an improperly-configured PYTHONPATH or PYTHONHOME env var. Is there a way to remove this dependency on env vars to make the install portable? Note that even if I move it with the whole pcbuild directory, I receive this error. The move could be to a different Windows computer, or to a different directory in the same folder: Same error. As @njs pointed out, the intent here is to make portable, independent python builds.
). There’s still potential benefit for users if we can say “if you don’t need to launch a GUI app, then this is an easy way to get Python without having to install Homebrew”.