Context
PEP 751 introduced a standardized format for reproducible package installation when working on a Python project.
At the same time, tools like pipenv
, poetry
, uv
, and pipx
made it much easier to install a Python project in an isolated virtual environment (venv).
So there is mostly a reproducible set of dependencies (= locked) and Python interpreters a Python project is tested against, before it is released as a Python package.
However, there is currently no defined way to reproduce this environment when installing the Python package for use with a Python project.
Motivation
Python projects are released[1] with dependencies as abstract as possible in order to prevent dependency hell. This especially means that upper version bounds of dependencies are discouraged.
With a growing number of packages and dependencies, the likelihood that a new release of a (sub-)dependency introduces a conflicting or, in the worst case, malicious change increases significantly.
This proposal aims to keep the abstract nature of Python dependencies while allowing users to install a verified set of dependencies for one Python package into their venv.
In other words: Add the possibility to install one Python package into a userās venv with dependencies that were verified to be valid.
Use cases:
- A data science Python project that wants to provide reproducible results[2], while still being usable by others for their own projects.
- Authors of Python libraries with many dependencies providing a way for users to set up a venv in a state that was verified to work.
- Companies that need to ensure a specific, validated[3] set of external dependencies is used.
- Maintainers of Python infrastructure who want to give less-technical users a cross-platform way to install a reproducible set of packages without requiring them to learn new tools.
- Authors of tools that want to allow Python users to use them as reliably as possible.
- Making deployments of Python applications into venvs reproducible while keeping flexibility[4].
- Allow backporting to a working version of a Python package and/or its dependencies.
Non-Goals
-
Install more than one Python package as locked.
- Due to mismatched pinning, it is nearly impossible to install two or more packages using their individual lock files.
- Still, authors of package
B
could use pinning provided by packageA
to generate their own lock file with additional or changed dependencies[5].
-
Create applications / End-user distribution.
- This proposal is only intended for installing one package into a venv in a reproducible way.
- While many Python projects serve both as a library and as an application, and can be used as applications from a venv, this proposal does not cover end-user application deployment.
- It touches only a small subset of the end-user application problem space. That space requires solving interpreter deployment and platform-specific distribution. See BeeWare or PyInstaller for tools that address that space.
Proposal: Including and handling pylock.toml
in Wheels
To allow for reproducible installation of Python packages, it is proposed to include the pylock.toml
(or named lock files like pylock.<name>.toml
, as defined in PEP 751) inside a wheelās *.dist-info/pylock/
folder.
As explained in PEP 770, adding new files or folders to a wheel does not require a new metadata version.
The existence of a lock file SHALL NOT be required.
The content of the lock file SHALL reflect a verified state of dependencies[6] for the given Python package release.
Package Manager Handling
Python package installers SHALL NOT consider this file unless explicitly requested by the user.
Special-purpose tools (like pipx
, uv tool
, etc.) that focus on installing main entry-point packages MAY differ in behavior.
A package installer SHALL require user confirmation if any requirement in the lock file is to be installed from a source different from the one used for the original wheel[7].
How to Include Lock Files in Wheels
Including lock files in wheels would be the responsibility of the build system.
The build system could either copy an existing pylock.toml
file or generate one dynamically during the build process[8].
Changes to requires-python
The strict lower-bound restriction for requires-python
in the lock file specification (PEP 751) shall be relaxed to support full version specifiers, like those used in packages.requires-python
.
This allows packages to document the full range of Python interpreter versions they were tested with[9].
FAQ
Why link the lock file to the wheel?
- Wheels are releases of Python projects and typically represent a tested state of the package.
- So itās natural to link the lock file used during testing with the final wheel.
Why not distribute the lock file separately via URL?
- Adds a dependency on a second service being available at install time.
- Requires the user to trust an additional installation source.
- Makes the lock file mutable (since it can be changed after release). In contrast, a wheel is already safely stored with hashes in the index. If you trust the index, you can trust the lock file bundled within the wheel.
Alternatives
Here are possible alternatives ā some are currently used as workarounds:
-
Custom package index with only a subset of validated packages[10]
No real locking
Hard to maintain
-
Dynamically add locked dependencies as
Requires-Dist
during buildWorks today without standard changes
Loss of abstract dependency model
Locked dependencies are hidden from tools
-
Distribute lock files alongside wheels, similar to
.METADATA
(warehouse issue #8254)No standards change needed
Manual coordination needed to upload lock files with releases
Requires extra HTTP requests to detect and fetch
Difficult to support named lock files
Requires changes to wheel-hosting services like PyPI or
devpi
-
Add lock file URL
Requires additional services to be trusted and available
Would require a metadata version change (unlike the current proposal)
For Later
There are still some open questions if this proposal moves forward. Since the discussion isnāt there yet, letās keep these as placeholders:
- What should tools do if a user installs a package with a lock file and later adds another dependency that conflicts with the lock? Should behavior be defined?
- Will this place additional strain on PyPI? Who should be contacted to coordinate?
Disclosure: I used an LLM to proofread/improve (NOT generate) this post as I am not a native English speaker and a bit dyslexic.
This is the continuation of: Pre-PEP: Include pylock.toml files inside wheels
A word about terminology: A released Python project is a Python package ā©ļø
Which is very important, e.g. when releasing a scientific paper ā©ļø
e.g. regarding licenses (which may change between project releases), CVEs, compatibility with CPU flags, etc. ā©ļø
In an organization I work for, services are deployed within a venv, and it is common for the people responsible for deployment to dynamically install extra packages/debuggers during daily business ā©ļø
A standard would allow reusable tooling to be developed, while still supporting very project-specific needs ā©ļø
This is most likely achieved by requiring the wheel to be built as the result of a successful CI/CD pipeline that executed tests with the given lock file ā©ļø
This is a security requirement. If a user installs a package from a trusted index, they should not unknowingly install dependencies from unexpected sources ā©ļø
This makes the process fully customizable per project, while still enabling standardized tools ā©ļø
As discussed in the thread on Locking a PEP 723 single-file script, special attention must be given to Python version compatibility. Wheels are typically built once for a specific Python interpreter. With newer Python versions, users are often forced to build from sdist ā which may be difficult or even impossible in some environments ā©ļø
This is used in one of my organizations and creates many problems. ā©ļø