I’m looking for some info on what’s currently the recommended way to deploy a python app. Let’s say I have an app that:
has lots of dependencies, including indirect ones (some from private indexes)
has extensions that need to be compiled (like Cython or Thrift)
I care about fully reproducible deployments (and making sure the deployed version has been tested with exactly the same dependencies), so I’d like to use requirements.txt (or pylock.toml/uv.lock if that brings any benefits). Let’s also assume that the person doing the actual deployment might be different from the one who prepared and built the code.
Right now, my approach for these kinds of apps looks like this:
I build a wheel for my app
I have a separate repo with deployment scripts, configs, etc. – deploying the code itself is done via pip install
For locking dependencies, I use pip-compile along with dynamic dependencies in setuptools
Unfortunately, this functionality in setuptools has been marked as BETA for years, and it only supports a limited subset of what you can do with requirements.txt (for example, you can’t use hashes). On top of that, pip-compile can only generate dependencies for a specific Python version and architecture. uv has a --universal option, so that partially helps with this issue.
Is there currently any standard or recommended way to deploy apps with a lock file? I’m looking for solutions that don’t rely on containers and are as independent as possible from the build backend and package manager.
I’d suggest that you only slightly care about reproducibility if this is a solution. If you really care, then you should vendor all your dependencies and distribute a complete package (even going as far as including CPython and whatever native dependencies you need). You’ll likely need separate packages for different operating systems, but if you’re prioritising reproducibility over portability, this is a small price to pay (if it’s too big a price to pay, you’re prioritising portability over reproducibility).
Requirements files (and lock files) are really for consumers of your app, not for deployments. Docker works because it essentially vendors everything before you deploy - you can do the same without Docker, you just need to get familiar with how your target operating systems resolve paths.
Python-standalone-builds is a helpful project for non-Windows OS here, as they can give you pre-built CPython packages that are relocatable[1]. Add in your app and your dependencies and you should be pretty close to a redistributable, reproducible package for your app.
Windows is relocatable by default, but I believe their builds statically link more components, so there’s fewer files to redistribute to get the full functionality. Personally I like being able to omit unnecessary modules when I do this, but I also know the internals pretty well. ↩︎
I don’t particularly see the benefits of using vendor packages instead of fetching them from a private package index, pinning all transitive dependencies and saving the hashes.
I’m familiar with that project, but that’s not where the problem I’m trying to solve lies.
Who exactly do you mean by “consumers” then? I’m building an application and distributing it as a wheel to internal end users. They install the application (deploy it on their servers), and I want to ensure that they install it with exactly the same dependencies (i.e., the same versions of Python packages) that I used during testing. Based on another discussion about lock files, it seems that lock files are designed specifically for this purpose.
The closest to what I need is probably Pex. I was hoping it would now be possible to avoid creating an artifact with all dependencies already installed, and instead use a lock file to install the required dependencies at the time of application installation/deployment.
By installing individual Python packages, you’re acting as a system integrator, not merely a user. So you never just install an app, you have much more responsibility to ensure that all dependencies are satisfied. We use tools to assist with parts of this, such as pip[1], but ultimately you’re consuming the source packages and producing your own distribution (and potentially not distributing it any further).
There’s a relative shortage of app distribution tools in the Python ecosystem, but if you want your users to just have an app that happens to be written in Python, you’re looking at PyOxidizer, PyInstaller, cx_freeze, and tools like that, or an OS-integrated package manager (depending on your OS).
That’s the hope, but they’re still closer to “for use by a sysadmin” than “for use by users”. The main scenario we all had in mind was for a single deployment to a controlled environment, such as your own web server. Other people have been thinking about app deployment (see e.g. some of the pyz file format discussions, though it’s not that important as they probably won’t solve your concerns yet, I just want to point out that you’re in a different space from what lockfiles were thinking about, but it’s not an unheard/unexplored space).
Which can help with Python packages, but doesn’t help with Python itself or many native, OS-level, dependencies. ↩︎
OK, so let’s assume such a scenario. Let’s keep the other assumptions though — the application requires prior building. I think it’s reasonable to assume that compiling extensions with every deployment is not a good idea. Is the recommended solution here a monorepo (dependencies built as packages + deployment from the repository), or is there another solution to this problem?
Yes, that’s exactly what I’m doing now using setuptools. The only problem with this approach is that I can’t take advantage of the other benefits of a lock file (especially the hashes, which improve security - unfortunately, Requires-Dist does not support hashes).
I’m looking for a solution for how to distribute the lock file together with the built application (in my case today, in the form of a wheel).
Use a build process that copies all the files required into a single directory, zip it up, and distribute a zip file.
You seem to be assuming that your end users are going to clone the sources and run from that? I haven’t been assuming that, since most end users are not doing this, and would prefer a platform-native installer for their platform. But if your end users are ~developers (in that they prefer to use tools like Git/etc. over installers/system package managers), then having a lockfile in your repository is fine.
Otherwise, you will have a build step that produces some kind of output. That output is what you make available for your users to install and use. (My apologies if this is obvious and you’ve already moved past it to talk about something deeper, but I can’t tell if we’ve both assumed this part of the software distribution process or if we’ve started from different ideas of how to get ready-to-use apps onto user machines. Specifying which operating systems you are targeting may help, since they largely have different conventions here - if you are only targeting Gentoo, then a lot of the discussion will make far more sense )
So, going back to the beginning of this discussion – is the wheel format a good way to distribute applications today? Is it possible to use a lock file with it so that everyone installing the package gets exactly the same dependencies as I do? I’m aware that I could use a different package manager than pip, but up until now I thought it was the best solution. My users are developers – people who deploy the application to servers, prepare the configuration, etc.
I was hoping that the wheel is the kind of archive that makes sense to distribute.
I’m targeting Ubuntu, assuming that the person deploying the application is not a system administrator (they deploy the application as a regular user and, for example, run it using systemd). Although that might change over time, that’s why I’d prefer to use tools that are relatively system-independent (compatible with popular Linux distributions and macOS). That’s the reason why I initially chose wheel and pip/uv.
It’s meant for Python libraries, and reasonably extends to “apps that I use in the context of a Python environment”, which are mostly the tools that apply to your Python project (e.g. linters, formatters, test runners). They’re not intended as an alternative to apt install <my-app> for cases where that is more natural.
In this case, just distribute the lockfile. The format doesn’t matter, but if your instructions can be “create a virtual environment and install this exact list of packages” then the lockfile is what you need. That way you can release your app with unlocked dependencies as a wheel and people who want to integrate it into their own projects can do it freely (or don’t release it if it’s private, but the point is to not lock dependencies in your package, but to provide the locked list to users who aren’t going to figure it out themselves).
This sounds like an unusual configuration already? I thought systemd was meant to run things as root? But I’m not a Linux expert, so maybe there’s a case here I’m not aware of.
Regardless, I’m pretty sure if you were to distribute as a native app (e.g. apt install ...) then you could integrate the systemd setup into that package. If you distribute as a Python library (i.e. a wheel), then you need to provide manual instructions to your users on what to do.
Again, the separation isn’t “what technology is best” but “what are my users expected to do manually”. If you’re clear on the latter, then the answer to the first question is generally much easier.
So, going back to the beginning of this discussion – is the wheel format a good way to distribute applications today?
Just my opinion (though I think it’s shared by many in the Python packaging community), no it’s not really designed for that.
Wheels are great for distributing Python libraries, but applications often need more flexibility. If your “application” is essentially a pure-Python library with a command-line wrapper that can be implemented by an entrypoint script then wheels can be a convenient distribution format. As soon as you need pre-compiled architecture-dependent binaries or vendored (especially non-Python) dependencies it gets more complicated. If you need to ship data then more complicated still. Configuration files are tough (you can have your application write out some embedded defaults at runtime, maybe). Automated post-install scripting is right out.
I’m targeting Ubuntu, assuming that the person deploying the application is not a system administrator (they deploy the application as a regular user and, for example, run it using systemd).
If it weren’t for the lack of sysadmin privs I’d say the preferred format for distributing applications to Ubuntu systems is typically DEB, either working to get your software included in Ubuntu (which is automatic if you add it to Debian). Though I think default permissions for Ubuntu Desktop these days gives users a means to do system package installation by elevating privileges in the background.
Common alternatives are Flatpak packages or Docker-style container images. Conda is also popular in this scenario from what I understand.
the preferred format for distributing applications to Ubuntu systems is typically DEB, either working to get your software included in Ubuntu (which is automatic if you add it to Debian).
Sorry, I failed to finish that sentence. I meant to add “…or making it available through a separate package repository such as an Ubuntu PPA.”