I’m new to python, I’m confused about whether python can be compiled to binary in any case ? I mean no matter how my code is written, as long as it runs correctly, it could be compiled to binary in spec platform?
Python code is compiled to a bytecode that’s stored in .pyc files. You can’t compile it to a .exe file, but you can cheat a bit by taking the normal Python executable and combining it with your .pyc files to make a single package. I don’t recommend this, though. It causes unnecessary problems for a lot of people. Instead, distribute your program as a zip file of Python scripts - a zipapp - which can then be run as conveniently as an executable, but isn’t locked to one build of Python. The zipapp can be considered “compiled to binary”.
2 Likes
Everything on the computer, including text and source code, is “binary” because everything is stored in 1 and 0 bits.
You need to explain what you mean by “compiled to binary”. Python is compiled to binary byte-code. That is what you will find in the .pyc files, and what the compile function does:
>>> compile('len([2, 3])', '', 'single').co_code
b'e\x00d\x00d\x01g\x02\x83\x01F\x00d\x02S\x00'
Is that what you are looking for?
I deliver apps written in python as windows .exe, macOS .app and RPMs on linux all the time. Seems very convientent for my users.
2 Likes
Maybe so, but you aren’t compiling your code in the same way that one compiles C to machine code. (At least, I don’t know for sure you aren’t, but I would be very surprised if you are.) That’s what I said about “cheating” - by combining the regular Python executable with your code, you can make what LOOKS like a compiled executable. But that’s not actually compiled Python code. It will always be run using the exact interpreter that you package with it, and even a basic print("hello, world") has to, by necessity, include the entire language interpreter.
So the question is: are you trying to compile your code, or are you trying to distribute a one-click runnable? They are quite different goals, and “one-click runnable” can be achieved in a number of ways. That’s why I mentioned zipapp, which is specifically designed to create runnable .pyz archives that work perfectly on all platforms.
I’m responding to your “use zipapp” recommendation. In the use cases of
my users zipapp is not a good way to deliver my code.
But I agree with you that the OP needs to be clear what is being queried.
Q1. Can .py files be compiled into native CPU instruction? No.
Q2. Can .py files be delivered as a natice app? Yes there are tools to help do this.
1 Like
That’s fine. Not every solution works for every problem. It’s a great way of distributing Python code for the situations where it works.
Yes, but it’s also worth noting that “native app” is often the wrong term, and “one-click runnable” (while a bit of a mouthful) is really what people are looking for.
I could take anything and package it up as a shell script. That doesn’t mean that you want a shell script. Far too many people think they want an “exe file” when that’s actually quite irrelevant to their use-case.
If Python is installed on Windows, the “.pyz[w]” filetype is registered with the shell API for use with shell functions such as ShellExecuteW() (e.g. os.startfile()). However, it isn’t seen as an application by the system and can’t be directly executed by CreateProcessW() (e.g. subprocess.Popen).
On Windows, a zipapp can be appended to an executable launcher, as explained in the documentation. The application needs Python to be installed on the system and added to PATH, or it needs to include required files from the embedded distribution (e.g. “python311._pth”, “python311.dll”, “python3.dll”, “vcruntime140.dll”, etc). Extension modules and DLLs have to be distributed alongside the launcher, so it’s not a bundled app, but it can be bundled up in an installer.
As to the subject of compiling Python code to a native binary image, don’t forget about Cython. A pure Python application can be ‘cythonized’ into C files that can be compiled into an executable and extension modules.
That’s because Windows is unnecessarily limited compared to Unix-like platforms. Put a shebang on a script and it truly becomes executable. But, question: how many people actually call CreateProcessW on something that they claim to need to be executable, and how many people are really wanting “a thing that sits on the desktop and can be run”?
How many Python programs run an application via subprocess.Popen with shell=False? subprocess only calls CreateProcessW(). OTOH, shells such as CMD and PowerShell try CreateProcessW() and fall back on ShellExecuteExW().
But it’s not just a matter of execution. A filetype isn’t an application in the eyes of the shell API. It’s not an application in regard to registering file associations (i.e. opening or editing some filetype with a zipapp) and integration with the start menu and taskbar (e.g. jumplists). To the system it’s just a “.pyz” file, and the application is “python.exe” or “py.exe”.
Again, I ask: how many people need to register file extensions with a Python app? This is FAR less common than the “need to click the thing” use-case. Of course you can point to differences. I never tried to say that a pyz is an executable in every possible sense. It just happens to solve a specific set of problems in a way that doesn’t introduce the restrictions that executables do.
And the problem is that people reach for executables when they really shouldn’t, with the result that an unfortunate number of apps have been distributed in a way that is USELESS for everyone outside the narrow target audience they originally picked.
I was responding to qualify the phrase “work perfectly”. I don’t think zipapps without a full launcher and its DLL depenencies work perfectly on Windows. I’m not saying that the limitations make them useless. If the standard Python distribution is already installed on the system, using a zipapp works well enough. If Python isn’t installed, the extra step of having to download and install it makes using a zipapp a bit more involved and susceptible to user error and abnormal system configurations.
It’s worth mentioning that Python’s app distribution on the Microsoft Store doesn’t register an association for “.pyz[w]” files. Due to complications with apps being locked down, the file association can’t be added using the GUI. In this case, the zipapp has to be run via python <pathname> from the command line or the Win+R run dialog.
Python is a language. CPython is the reference implementation of that language, and they are often confused for one another (which worries me a bit, but that’s a topic for another time).
CPython compiles to byte code. You can wrap that up as a binary executable, but some dependencies have problems with this. Sometimes there are workarounds for those problems, sometimes there are not.
There are many alternative implementations of Python, and some of those can compile to a binary executable. Nuitka seems to have this as its reason for existence, and seems to be pretty compatible. Cython is a dialect of Python that allows you to mix Python and C datatypes using a mostly-python-like syntax; it too allows you to transpile that to C, or sometimes C++, and from there to a binary executable. There are several more projects in various states of development that will transpile from Python to C++, which naturally can be compiled from there to a binary executable; I gather Shedskin may make the leap to Python 3.x after all, which is kind of exciting. Shedskin and others take as input an implicitly static dialect of Python, and get very satisfying performance improvements.
Cython can also cythonize a pure Python script or module if all you want is to translate it to the C API and compile a native binary.
Curious. Two of the most popular Python apps, youtube-dl and its fork yt-dlp, are delivered as zipapp. What’s missing for your users?
My users do not need to know that python is needed.
I also register file types with the OS so that things work well.
The python code uses included extensions as .DLLs or .dynlibs.
scm-workbench has pysvn with subversion inside it.
barrys-emacs has the editor core as a python extension.
thanks for the answer and zipapp is a good way for distribute python program. In mycase, my goal is
- Compile once and run everywhere
- Protect my source code
- Several independent Python programs will be bundled and delivered together . for example, deliver several machine learning models in one docker image. However, i want to share dynamic link libraries, such as torch and opencv for all the models, because the so files needed by torch and cv are huge.
Well, your code isn’t very protected. Most of it is right there in the bytecode. Anyone can decompile it and get a pretty good idea of what your code is doing.
It really sounds like a better option would be a package. That makes it trivially easy to share dependencies, and to allow your code to be updated without redownloading those huge dependencies.
As a package way like wheel or zipapp. i got some confuse about how to share dependencies. for example, i create a virtual env for program A named A-env , a B-env for program B. both program A and B need some dependencies which are huge. i just want install the huge dependencies one time, maybe i install them in C-env. then the A-env and B-env could using the same huge dependencies in C-env.
in this way, do you need to manually compile for each platform in advance ？