Help me shake out pip-lock 🔒

Hi! :waving_hand: I’ve been building a new top-level command for pip-tools called pip-lock. It produces a PEP 751 pylock.toml from the inputs pip-tools already understands (pyproject.toml, requirements.in, raw arguments) and adds PEP 735 [dependency-groups] parsing for the new command. The output is a cross-platform lockfile that any PEP 751 installer can consume.

The work lives on PR #2380 and is ready for real-world testing. pip 26.1+ ships pip lock and a pip install -r pylock.toml / --lockfile reader, both labeled experimental. uv ships uv export --format pylock.toml on the writer side and uv pip install --lockfile on the consumer side. With several independent implementations in the field at different points on the maturity curve, comparing outputs across them is a cheap way to find the spec’s corners; that’s what I’m asking for help with.

What I’d love help with

Try pip-lock on a project you already maintain. Any datapoint helps; you do not need a project that matches a particular shape. The shapes below stress code paths I want to exercise, and a small requirements.in is as useful as a monorepo with conflict groups. What matters is that the inputs come from a project I did not test against, instead of from the few projects I happened to pick.

  • A library or application with several extras and dependency groups. Bonus points for projects that already declare [tool.uv].conflicts for mutually exclusive variants (multiple lint pins, GPU/CPU builds, alternative TLS backends, …); the same shape works in [tool.pip-tools].conflicts.
  • A monorepo that resolves for several platforms and Python versions at once. The marker-driven cohort scan does the most work when the matrix is non-trivial.
  • A project with VCS dependencies pinned to commit SHAs.
  • A project with a non-trivial [build-system].requires you want captured in the lock alongside the runtime graph.

How to try it

$ pip install --pre 'pip-tools @ git+https://github.com/jazzband/pip-tools.git@refs/pull/2380/head'
$ pip-lock                                # universal lock for the project
$ pip-lock --check                        # CI: fail when the lock drifts
$ pip-lock --dry-run                      # preview without writing
$ pip-lock --all-extras --all-groups      # the maximal lock
$ pip-lock --no-universal                 # current platform only, fastest

The full README walkthrough lives in the PR description and the Example usage for pip-lock section of README.md on the PR’s branch.

How pip-lock relates to the other PEP 751 producers

Three other tools write pylock.toml from a Python project today: pip lock, uv export --format pylock.toml, and pdm lock --format pylock (also pdm export -f pylock). They make different trade-offs, and pip-lock makes a fourth set. Here is the feature surface at a glance:

pip lock (pip 26.1) uv export pdm lock pip-lock (this PR)
universal lock default no yes yes yes
PEP 735 groups no yes yes yes
declared conflicts no yes yes yes
multi-variant emit no PR #14728, unmerged no yes
resolver runtime Python Rust Python Python
status experimental stable stable proposed
reads pip-tools inputs yes no no yes

pip lock is an experimental subcommand in pip 26.1+. It uses pip’s resolver, reads requirements.in-style inputs plus pyproject.toml, and writes a single-environment pylock.toml for the interpreter pip runs under. The command has no native cross-platform mode and no universal lock across multiple Python versions in one invocation. PEP 735 dependency groups and declared conflicts are not on its surface yet. The reader side (pip install --lockfile pylock.toml) is also experimental in the same release.

uv export --format pylock.toml produces pylock.toml from a uv project. The export is single-resolution: every package picks one version, and markers describe which environment each candidate fires on. The “multi-use” PEP 751 export that emits multiple version variants for one package under disjoint markers is in motion upstream as astral-sh/uv#14728 and has not landed in a public release. On the consumer side, uv pip install --pylock and uv pip sync --pylock read PEP 751 today; they emit an experimental-feature warning unless the user passes --preview-features pylock.

pdm lock --format pylock and pdm export -f pylock ship in PDM. PDM had cross-platform locking before PEP 751 existed (its team co-authored the spec), so the producer is one of the more mature ones; the tracking issue for full PEP 751 support closed in mid-2025 and bug fixes through 2026 have closed most of the corner cases. Choose this if PDM is the project’s workflow root.

pip-lock (this PR) is the new third command in pip-tools, alongside pip-compile and pip-sync. The default lock is universal across the project’s requires-python and the built-in platform matrix; a marker-driven cohort partitioner collapses target environments that share a dependency closure into one resolver invocation, then dispatches the cohorts to worker processes for parallelism. PEP 735 dependency groups are read for this command (the existing pip-compile and pip-sync siblings do not parse [dependency-groups] yet; that work is tracked separately in pip-tools issue #2062). [tool.pip-tools].conflicts mirrors uv’s shape; the difference is that pip-lock --all-groups accepts the request and emits one [[packages]] entry per conflicting variant under disjoint markers instead of refusing, so a single pylock.toml carries every variant a downstream installer can pick from with --group <name>. Resolution runs in Python on top of pip’s resolver, so wall-clock times will lag uv; the design goal is to produce a correct PEP 751 lockfile from the same authoring inputs pip-compile already understands, including the multi-variant case. Choose this if the project already uses pip-tools, if the multi-variant emission is what you want, or if you want a producer that lives inside the pip-tools workflow.

Cross-checking with pip and uv

Once you have a pylock.toml, install it the same way every other PEP 751 lockfile gets installed:

$ pip install --lockfile pylock.toml .       # pip 26.1+
$ uv pip install --lockfile pylock.toml .    # uv

A divergence between any two of pip-lock, pip lock, and uv export --format pylock.toml is interesting evidence: sometimes a real bug, sometimes a spec ambiguity that needs sorting out between the implementations, sometimes a deliberate design choice on one side. The bench is triangulation, not an oracle. When you find one, please reach out to me with whatever you have to hand: the pyproject.toml (or a reduction if you can produce one easily), the pylock.toml files produced by each tool, and the exact command line you ran on each side. The size of the reproduction does not matter; the existence of the reproduction does.

What I’m looking for feedback on

The first bucket is correctness bugs. Anything that produces a lockfile one of the implementations rejects, picks a different version of a package than the others given the same inputs, or fails the disjointness check on a project where uv is happy. A divergence on a real project’s lockfile that survives the design-difference filter (project-as-package, conflict variants under markers, no-universal scoping) and reproduces against the latest commit on the PR is a strong signal. If something looks wrong, it probably is; please reach out to me.

The second bucket is usability: error messages, defaults, and the intuitiveness of the CLI surface. When a same-name marker collision raises, the witness environment plus the conflicts hint should tell you what to change in pyproject.toml. When --uploaded-prior-to rejects a stale pip, the error should say what version is needed. When --platform some-typo-arch falls through to the synthetic-os-arch fallback, the marker emitted should match what you wanted. If any of those messages leave you guessing, the text is a doc bug; flag it. The same goes for the README itself: the deeper documentation under Example usage for pip-lock on the PR’s branch covers the realistic flag combinations, and I would love feedback on which scenarios still feel ambiguous.

The third bucket is performance. pip-lock will not be uv-fast: uv resolves in Rust against a tightly engineered native solver, while pip-lock runs in Python on top of pip’s resolver, so an order-of-magnitude gap is the floor. The cohort partitioner is what keeps the universal mode tractable; it collapses target environments that share a dependency closure into a single resolver invocation:

   17 platforms                                                ┌─ resolve once ─┐
   ──────────────         ┌─ cohort 1: 3.10 envs ─────────────►│   (parallel)   │
   ×                      │                                    └────────────────┘
   5 python versions      │
   ──────────────    partition  cohort 2: 3.11 envs ──────────►   resolve once
   = 85 target envs    by marker  ...                              ...
                       equivalence
                          └─ cohort N: distinct dep closure ──►   resolve once
                                                                  ─────────────
                                                                  N invocations
                                                                  (N << 85)

As a rough datapoint, a recent run on datamodel-code-generator main (124 packages, conflicts declared, --no-default-groups baseline, single laptop sample) measured uv lock at around 460 ms; pip-lock --no-universal with a warm cache at around 1.2 s; pip-lock --no-universal --rebuild (cold cache) at around 3.4 s; and pip-lock universal across 17 platforms and 5 Python versions (85 target environments) at around 10.3 s with the cohort partitioner doing the work. Those are single-sample numbers on one machine and will vary with project shape, package server latency, and matrix size. The interesting numbers are the comparative ones from your project: the matrix dimensions, the Locking for N platforms x M python versions line printed at the start of a run, the wall-clock time, and the cohort count after partitioning. Datapoints from larger or more conflict-heavy projects are the input I need to prioritise the next round of optimisation.

Where to send feedback

Two channels: this thread (replies here, or via direct message here) and the PR itself at https://github.com/jazzband/pip-tools/pull/2380 for review comments, line-level questions, and anything that pairs naturally with a diff. For background on the installer side, pip’s PEP 751 support is documented at https://pip.pypa.io/en/stable/news/ and uv’s writer/reader pair lives at https://docs.astral.sh/uv/concepts/projects/sync/#exporting-the-lockfile.

Thanks ahead for any help you can provide getting this right from the get-go. :folded_hands:

PS. Why pip-lock --all-groups does not refuse a conflicting request

If you set [tool.pip-tools].conflicts on a project that declares mutually exclusive groups (a CPU build vs. a GPU build, several alternative TLS backends, several pinned linter variants like black22 / black23 / black24) and then run pip-lock --all-groups, pip-lock accepts the request and emits one [[packages]] entry per conflicting variant, each carrying a marker that fires only when the user requests that group and not the others. uv refuses the same request with “you asked for a contradictory combination”. Both behaviors are PEP 751 conformant; they reflect different defaults about who decides at lock time.

uv’s refusal is better when the lockfile’s audience is one install scenario. A team locking for “the production deployment” or for a single CI job passes --all-groups thinking “include everything”, forgets about the declared conflicts, and uv’s error catches the mistake before it becomes a confusing install-time failure on whichever variant happens to win. The same applies to a tox or build-matrix repo where each job has its own pinned variant: the author already knows which group each job wants, runs uv export --group <variant> per job, and the refusal reinforces “lock per variant” as the right shape. Audit-first teams that want pylock.toml to be a 1:1 bill of materials for a specific install scenario also benefit from uv’s strictness, because three [[packages]] entries for black with disjoint markers in one file, even when formally correct, take longer to read at a glance than three separate lockfiles each holding one version.

pip-lock’s accept-and-emit-variants is better when the lockfile’s audience is several install scenarios served from one artifact. A library or framework publisher who wants downstream consumers to pick a variant at install time without re-running the resolver gets that with one pylock.toml and three different --group flags on the consumer side; the uv approach forces the publisher to ship N separate lockfiles or pushes the resolver work onto every consumer. A monorepo whose deploy pipeline supports --group cpu and --group gpu (or --group postgres and --group sqlite) from one canonical lockfile committed by SRE benefits from “lock once, install many ways”; generating N lockfiles per variant means N commit churns whenever any unrelated dependency moves. Dockerfiles with multiple stages benefit too: the build stage runs pip install --lockfile pylock.toml --group dev while the runtime stage runs pip install --lockfile pylock.toml --group prod, both reading one authoritative file instead of juggling pylock.dev.toml and pylock.prod.toml.

The simplest heuristic: if the lockfile’s audience is one environment (a service deployment, a CI job that already knows what it wants), prefer uv’s strictness; failing loud on contradictions catches forgetfulness early. If the audience is several environments served from the same artifact (a publisher, a monorepo with several deploy targets, a Dockerfile with several stages), prefer pip-lock’s variant-per-marker approach. The underlying resolver math is the same in both tools; the surface contract for --all-groups plus declared conflicts is what differs, and the right contract depends on who consumes the lock.

7 Likes

Thanks a lot, I’ll be checking this out tonight.

I’m about to make an annoucment in the same space, but coming at the problem from a different direction. I don’t suppose you’ll be at PyCon US to discuss?

1 Like

I’ll be at PyCon US 2026.

3 Likes

I will try to test this out over the weekend. I am sure we can also catch up at PyCon US on anything. We are working on hatch producing pylock.toml per environment and preparing to release that soon so this falls into I very much want to test this before we ship for hatch.

For your cross-checking, if you need another comparison point I wanted to add that GitHub - pex-tool/pex: A tool for generating .pex (Python EXecutable) files, lock files and venvs. · GitHub supports both producing and consuming PEP 751 lock files.