Pre-PEP: Add ability to install a package with reproducible dependencies

Packaging
23

I just stumbled upon this thread - a bit late yes, but I am not sure what’s the current status - maybe someone can enlighten me?

For the record - I am Airflow maintainer who created, developed and perfected our constraint mechanism for the last few years. We needed it badly. We could not get standard support for it so we went with our own ways. And it saved us myriad of times.

Our users who follow "installation with constraints” are automatically saved from their installations falling because package a released a breaking version even in patchlevel release (last time it happend was 10 days ago, but it is already maybe 20-30 times over the last five years). And we have a super easy advice to our users when it happens when they don’t use constraints. Just go and use constraints Installation from PyPI — Airflow 3.1.1 Documentation .

At the same time (and more importantly) this is a fantastic mechanism for the open-source project like Airflow to get prepared for regulation in security space (like the CRA Page not found | Shaping Europe’s digital future ). We do not “pin” our dependencies - we pretty much exclusively never upper-bind any dependencies, PRECISELY becasue we have the escape hatch that constraints give us - we can always tell our users “use constraints if this new release of package b broke things”. And - at the same time - the lack of upper-binding of our dependencies makes it possible by our users (i.e. manufacturers in terms of the CRA) to upgrade problematic dependencies of ours that have vulnerabilities on their own - without waiting for us to relase a new version (whcih might never happen - as open source steward, as opposed to manufacturers we are not forced to release new versions in this case). Or without them to upgrade to latest version of Airflow that already bumped that dependency in it’s constraints.

This is very powerful mechanism that we use for years and it works . Beutifully. I cannot count the number of times we got the request “the security scan we run in our BIG ENTERPRISE have found those vulnerabilities in those 100 of dependencies of yours in this old 2.3.0 release - FIX IT ASAP !!!”. And our response always is “here is link to instructions how you can do it yourself and BTW you should update to latest versions of airlfow and those dependencies you care about security”. And we never hear back (not unsurprisingly).

So if you ask “Do you know other projects than Airflow that do it?” - I think this is a wrong question. I honestly think “How many projects in about a year will SCREAM to get similar solution in place because they won’t be able to cope with manufacturers banging on their doors to upgrade their dependencies ?” is a better one.

I would love to be able to use PEP-751 insted of our “poor man’s constraints solution” - and if we want similar approach to spread among others - noone- would be able to affort build what we’ve build for airflow - but should rather use a “Standard” solution.

I am super happy to collaborate on it. The moment pip and uv both support PEP-751 installations, I drop everything else and in a week or so this will be the preferred way we tell our users to install airflow (with constraints for some time as backwards compatibility for older pip versions). So let’s make it happen BEFORE other open source projects will start begging for a similar solution.

1 Like
24

pylock.toml is a great improvement for dependency management in python, but I don’t believe it will be more appropriate than the constraints file approach currently recommended.

airflow is a bit of an outlier because it’s usable in the following contexts:

  • standalone application runtime / platform
  • library extensions / plugins / ecosystem
  • userland airflow workflows

A lockfile installation with pinned dependencies will only be reproducible and usable for the first context.

Subsequent installation of additional dependencies ontop of an airflow installer could lead to broken installations and won’t remove the need for the constraints files.

Lockfiles can help airflow plugins with deeper integration tests potentially, but it shouldn’t encourage them to update their dependency specifiers to the same pins as an upstream lockfile.

In my opinion, what is being discussed here is related to Should you use upper bound version constraints and the constraints files mechanism of airflow is already working well to ensure the most flexibility for usage in multiple contexts. Looking to standardize constraints files or a distribution mechanism for constraints files sounds interesting, but it’s not a lockfile. I’m also not sure how applicable it would be outside airflow, which is a bit of an outlier (but a popular one)

25

I disagree with the statement that Airflow is an outlier for both being an end user application written in Python, and being able to interact directly with it by writing Python code, leading to the need for both a mix of constrained and unconstrained dependencies depending the use case.

It holds for any application that lets you configure, plugin, extend, or write apps via Python, for example Dash, Jupyter, Stable Diffusion web UI, Sypder, etc.

In my experience the limitation of Python’s dependency management leads most applications to come up with alternative distribution mechanisms outside packaging standards, such as bespoke shell scripts, binary packaging like pyinstaller, or conda. With then a completely alternative pip or Python package manager system used for users who want to code against those applications.

5 Likes
26

Yes. This is deliberate and part of the setup. As maintainers we don’t guarantee that installting stuff “on top” will not break installation. But also we do not block the upgrade. At this time - this makes a relationship between our users and our dependencies and THEY have to agree on non-breaking and security-solving feature. It’s deliberate and very carefully designed as a “feature” of the system. We do not want to be part of it and this is our way to make sure that users are talking to the right parties. It’s on them to fix it.

If they want to wait for Airflow to release a new version and are willing to take the cost of upgrading to the LATEST version of Airflow - they can still do it. And as maintainers - we can suppot that. Anything else is between the users and the dependencies, We have no means, money, time to do more.

What you describe is a very well designed property of the setup

2 Likes
27

It’s really very clear boundaries where “volunteer maintainers” can spend their time vs. where “corporate and business users” who have money (as opposed to time of volunteers) to invest in return of the free software they get from volunteers.

if commercial users expect volunteers to do stuff they are tasked with by the law (this is what CRA does) - getting a response “here is how you can do it and BTW it’s on you to do testing” is quite far of a deal, I think. We - open source people - at most what we can do is to describe the commercial users how they can spend money they harvest from their customers to actually serve them secure software. That’s the least, and the most we can do.

2 Likes
28

I also don’t think it is. We are an “amplifier” of problems with dependencies - because we have so many of them and we had to do some automation around that.

But once your users will start asking questions about YOUR security and will start involve YOUR DEPENDENCIES in the discussion. The whole problem will stop being “Airflow as outlier” - pretty much all python app will have the same issue.

1 Like
29

So if you ask “Do you know other projects than Airflow that do it?” - I think this is a wrong question. I honestly think “How many projects in about a year will SCREAM to get similar solution in place because they won’t be able to cope with manufacturers banging on their doors to upgrade their dependencies ?” is a better one.

To be clear, the OpenStack community has been doing this for a decade or more. An OpenStack contributor added the constraints feature to pip in the first place, specifically because at the time the dep solver in pip was far worse than today and this was a good first step (you could run whatever solver you wanted outside pip, turn that into a constraints file, and then force your solution at install time).

We use it more or less the same way today: We have an automated system that proposes updates to our constraints list in an attempt to use latest versions of our dependencies, and then run a full battery of tests to find out what breaks (if anything) before approving dependency version increases. It still works remarkably well and saves us from constant regressions in our thousand-order transitive dependency set.

3 Likes
30

Good point! I was trying to jog my memory on Spyder, but my searches failed. Jupyter is an obvious one and not an outlier.

I think these are all ecosystems where it may not be possible to standardise on the extension / plugin runtimes. Worth exploring though. In the case of Jupyter, kernels can be written in other languages and some extensions are UI / browser extensions and some are kernels. So the fact they’ve built their own plugin and distribution ecosystem makes sense to me and provides a pleasant experience.

If an application and ecosystem grows to be large like VS Code or Jupyter, where its both a host and runtime for plugins, to ensure the host is not impacted by shared plugins and extensions, often the plugins run in isolated environments and processes so they can then reuse something like package.json or pylock for distribution. So I think this is quite a deep “it depends” topic that is quite large to standardize in my opinion. Because it very much depends on the extension architecture of the application in question.

Edit: Im conscious we may be drifting off-topic or need to create a new topic about standardising dependency management for Python applications that host Python plugins/extensions. This would be a large topic in terms of scope and would also need to cover “entrypoints” which are also a style of in-process plugin discovery. Im not looking to drive that conversation though.

31

Similar in airflow - we got an automated CI that updates constraints (3 types) automaticaly based on all CI passing for our “main” canary builds - and same in our release branch. I wish we would not have to discover it separately in parallel - but that it would become a “de-facto” standard.

1 Like
32

I think you’re mostly caught up by now, but I want to note that my feeling was that this thread is about real/legitimate use cases that are widely recognized as such. However, because the proposal itself opens a number of questions, we’re not going to be able to come up with a viable proposal without a much stronger shared grasp of the use cases. The thread lost momentum because we’re unclear about what’s needed.

What I still don’t understand here is how this application locking approach is supposed to scale to more than one package. I think a firm idea of “what’s supposed to happen” with two packages is necessary in order to consider pushing for a standard.

From what @potiuk has said, it seems that the airflow way is to say that “if you have any conflicting requirements, it is up to you, as the user, to decide how to reconcile them with the airflow constraints files. This is an advanced workflow.” I think that’s a good approach because it’s very realistic about the tradeoffs. But what would a standard do to improve upon this situation?

Let’s suppose we standardize some method of publishing a lockfile and that pip install is extended to support using that lock data. The simplest possible interface would be a new option, i.e. pip install --locked airflow.
What would I then expect from pip install --locked pkg1 --locked pkg2?

My expectation would be that this is flat out rejected. Not even with an error about how “the pkg1 and pkg2 locks are incompatible” (which they may or may not be) but rather with “you can only install one locked package at a time”. The reason being that compatible uncoordinated locks would be rare.

Following from that, what exactly are we standardizing here? A registry for lockfiles?
Putting a lock inside of a wheel seems wrong, but publishing a lock seems reasonable. Can we see if projects make their lockfiles available via https and git repos how much benefit we get? I think letting the tool ecosystem experiment a bit will help us understand the texture of this problem space.

Standardized lockfiles are still pretty new. It’s natural that we should think about “next steps”, but personally I don’t think we’re ready for that next step standard until we start trying out the practice a bit more.

3 Likes
33

I agree we need to narrow down the use cases and indeed, come up with a stronger case.

Yes. My understanding is that it should be rejected. How I see it we should only allow to have EXACTLY ONE package to “drive it”.

I actually think now that this is more of use case for pipx rather than for pip - because this is really much more what “airflow” users think about when they are installing Airtflow - they think of it as an “application” with additional dependencies.

Also, the “power user” story of airflow is a bit different than “let the user solve the conflict”. The story is to let pip (or pipx ) to resolve the “additional” installation as a second resolution step with additional packages after the constraints have been used to install “base” application with “selected standard options”.

So my ideal “user story” is:

pipx install --locked apache-airflow[amazon,google,sentry]==3.1.1 my-custom-package==3.5.6,my-other-custom-package==10.4.2

How I imagine it could work - mimicking what we suggest our users - is to run 2-step installation

  • step 1: pipx installs airflow 3.1.1 + all optional dependencies specified using the lockfile stored in the registry
  • step 2: pipx installs all the other packages on top of the original installation WITHOUT lockfile (i.e. for example it would allow to upgrade some of deps airflow uses to a different version than in the constraints, if the custom packages need it). One important part of it such installation step should have also apache-airflow==3.1.1 as additional dependency, in order to prevent resolver to downgrade or upgrade airflow, because the intention is that “airflow 3.1.1” is result of such pipx install step - that’s the intention of the user. Also “–locked” is a good name in this case because it effectively “locks” airflow 3.1.1 (and initially other deps) - but then it allows to upgrade those deps (but not airflow). Also there should be a possibilty to update that environment

pipx update "apache-airflow" new-dep==3.2.3

That would only perform the 2nd step (without constraints, still pinning “apache-airfflow” to 3.1.1.

Following from that, what exactly are we standardizing here? A registry for lockfiles?

What I would like to standardise it is a mechanism built in any registry (including PyPI) how package maintainer might want to upload a “golden” set of dependencies that they knew worked. I would love if I would not have to tell the user two things:

  • where to get constraints from
  • that they should pin their “driving” project to the original version when they are updating the environment

That is, I think, what I would seek from standardisation efforts.

Currently, there is significant friction with the constraints mechanism Airflow uses:

  • users should discover that this is the “recommended” way - this is not a standard, this is specific to airflow and many of our users don’t really realise this is “THE” way we recommend them
  • they have to construct the URL to get constraints from - and that URL contains not only version of airflow but also Python version they use - which often leads to confusion as they might not even realise which python version they are using. With standardized lockfile this is gone.
  • they should pin the “driving” dependency next time they update dependencies - this is an error many of our users do - they add new dependencies and pip resolver finds out that the easiest way to installl those is to downgrade or upgrade airflow - which is against the intention of the user
  • we rely on GitHub - this is something we would like to get rid of. Especially for cases where GitHub has control over it. Recently they decreased rate limits for raw URLs - and I can easily imagine some of our users behind NATs might get throttled - because this is unauthenticated request - also one day they might change ways how such raw URLs are constructed. We would prefer that our users only rely on the registry where they are already installing airflow from (public PyPI, or private one they are using internally). If we add it to standard registry APIs - this problem is solved.
  • our constraints generation is “poor mans” version. We miss a lot of potential variations people might have - architecture, operating system, etc. etc. with standardized lockfile, this problem is gone.

Standardized lockfiles are still pretty new. It’s natural that we should think about “next steps”, but personally I don’t think we’re ready for that next step standard until we start trying out the practice a bit more.

I think currently the problem is that standardized lockfiles can be produced, yes, but their use is I guess almost 0. I don’t understand what would be the use case for standardized lock files “using” currently and why someone would like to produce them at all (but I might be wrong of course - maybe I am missing something). So I have hard time understanding how we are even “testing” it in the community. Is there someone who uses them for something?

I quite agree that maybe it’s “early” for standardizing it, but the thing is that the biggest benefit we can achieve from those files is when we have some standard, low-friction way of using them. Other than that, they seem to be pretty useless if you ask me. So yes - it’s early, but I don’t think it is “too early” - especially that cases like Airflow and use-cases for them are very well tested and tried using custom mechanism (we run Airflow for > 5 years now with constraints and it has proven to be immensely useful). Yeah maybe outlier, but maybe it’s actually a valid use case that will be unlocked for others when it will get standardized? I guess we won’t find out until we try and no amount of “waiting” will change it. “Doing” is the only way to find out as I see it.

34

I agree. There are a number of important points to address here.

First of all, if you’re adding requirements to an existing environment and you’re not using a tool that manages the full environment for you, then there are significant risks. For example pip (which doesn’t manage the environment) can fairly easily produce a broken environment. It will tell you that the environment is broken, but that’s not much consolation if you now have to recreate it from scratch. For simple management of a development environment, this is often fine, but for a complex application like Airflow, it’s far from ideal.

For many applications, the correct solution is to create an environment that is dedicated solely to the application, and only contains the application and its dependencies. This is the model that pipx, uv tool, and similar utilities follow. For those, installing from a lockfile seems like a sensible approach, and I don’t doubt that they will gain the ability to install applications from a lockfile in due course. I fully expect pip to also get “install from a lockfile” functionality, but in this context pip is a low-level tool - you need to create the environment yourself, and pip won’t manage the environment for you (although it gives you the commands to do so manually).

For applications with “plugins”, I think there is scope for standardisation on how plugins are managed. It’s not unreasonable, for example, to require plugins to work with the the same versions of packages that are required by the main application, and for plugins to be checked to ensure that any other dependencies they have are consistent. To do this would require the developers of projects that use plugins to get together and propose some standards, though. But right now, I’m not aware that plugin-based systems have significant management issues, so (IMO) there’s no point going looking for problems where none exist.

For applications that allow users to override the official dependency versions (provided by a published lockfile/constraint file, or similar), that’s very much a specialist situation, and I think that careful use of pip install and a statement “do this at your own risk” is about the best we can offer.

For applications that are both installed in an environment, and allow the user to interact with that environment (Jupyter, for example, and I believe Airflow falls into this category), this is a complex issue, and I don’t believe that we’re even close to having a “one size fits all” solution. I don’t think there’s much scope for standardisation here yet. As with plugins, if we were hoping to develop standards, it would need the affected projects to work together to find some common ground first, so that any proposal started from a position of being acceptable to the major players.

And as for installing more than one application in a single environment, I think we should just say that deliberately isn’t covered by the standards. It can work just fine, of course - many people have black, mypy, pylint, nox, etc., all installed in their development environment with no problems at all. But it’s not suitable for standardisation, and if there are issues, the standards aren’t the place to address them.

I can see extending the index API to allow publishing of lockfiles[1]. But I think it’s premature to standardise this yet. We don’t even have a working mechanism for shipping applications as lockfiles at the moment - at a minimum, we need to wait until pipx, pip and uv tool support standard lockfiles. Once that’s in place, we can start thinking about standardising the distribution of lockfiles. For example, we can determine based on experience whether there’s value in allowing projects to version their lockfiles independently of their main application wheel. For example, if there’s a need to re-lock to pick up newer dependencies with security fixes, why would that need a new application wheel release?

What I’d like to see here is:

  1. Tools start supporting installation from lockfiles. Applications like Airflow can help here by requesting that functionality. It’s easy to assume that it’s “obvious” that people want this, when in reality it might not be.
  2. Applications like Airflow start publishing a lockfile as one of their official installation methods.
  3. Based on user experience, applications give feedback to installation tools on what functionality is needed for use cases like “hot patching” dependencies, and managing plugins.

  1. It should be lockfiles rather than, say, constraints, because that’s the agreed standard for this type of data. ↩︎

4 Likes
35

100% agree with that line of thinking @pf_moore and with proposed approach how to tackle it.

I am going to request (and see maybe already someone did it so I will join the effort from Airflow side) to add support for pipx and uv tool as they are both prime candidates for having it. I am also absolutely fine with us publishing the lockfiles (we can also solve the problem of GitHub reliance this way as we might simply publish it in “airflow.apache.org” similarly as we publish SBOM files CycloneDX SBOMs for Apache Airflow 3.1.1 currently (which hopefully eventually will - be published in the .whl files PEP 770 – Improving measurability of Python packages with Software Bill-of-Materials | peps.python.org → though I also hope - for the reason of potentially regenerating the SBOM with more/updated information eventually those SBOMS will also stop being “immutable” in .whl file.

So except support for installation in pipx and uv tool - we are not blocking to try it out and report it here or in further discussion on our learnings and experiences.

Also BTW. I am not too happy with the way how SBOMS in PEP-770 are going to be embedded in the .whl file. Simlarly for lock files - the fact that both SBOMS and package lock should not (IMHO) be treated as immmutable, this is a bit of problematic choice for me.

Maybe this can be solved by wider acceptance of “post” releases to update such meta-data. I don’t think (and I would love to be corrected on that), that “post” releases are not expected to modify such “sbom” (and maybe futture lock) metadata?

Since “post” is very rarely used (and PEP-440 is not too detailed on what metadata really could be updated in post release (only release notes are explicitly specified). I have a bit of feeling that “post” releases are treated a bit as “ugly child” and their behaviour, scope and usage for post releases. But maybe clarifying that you can update SBOM/LOCK metadata in post releases and (ideally) we could do it easily without recreating the original packages (maybe even just post “empty” post packages with only the metadata changed?) - this would practically allow us to remove the need of publishing the lock files separately - they could be part of the .whl and could be updated in .post releases. From my experience, the constraint files in airflow are MOSTLY IMMUTABLE, we change them extremely rarely in very specific circumstances, it requires us for example to rebuild our “convenience” production images that we also publish, so doing it via “.post” release does not seem to far off.

36

I have no experience with, or understanding of, SBOMs, but I think that should be a separate topic.

What little I do know of SBOMs suggests that they should only cover what’s in the wheel, so if you don’t change the wheel, why would the SBOM change? In particular, dependency information shouldn’t be there unless you’re bundling your dependencies. So maybe the SBOM for “Airflow as an application” really should be distinct from the SBOM for “Airflow as a wheel”?

Getting back on topic, this is why I suggested we need to think separately about how to publish lockfiles (if we do want to publish them in indexes). They aren’t necessarily versioned the same as the application wheel. I think published lockfiles should be immutable, but it should be possible to publish a new version of the lockfile without needing to publish a new wheel.

2 Likes
37

Yep. I only tangentially mentioned SBOMS as similar concept and inspiration.

I think published lockfiles should be immutable, but it should be possible to publish a new version of the lockfile without needing to publish a new wheel.

Yeah. I think that how potentially “.post” could work (that would be my preference) if we agree to store sboms in .whl and we can make .post releases only contain that meta-data (I am not sure if that’s even a remote possibility of .post working this way).

BTW. There are several variants of SBOMS (build time, release time, deployment time etc.) - and general consensus on what exactly should be in them is not yet reached. But I think industry recently converges on SBOMS containing transitively all dependencies as the best practice.

If you are interested - here is the source: this is request for comment / proposal by CISA from August 2025 where they asked industry for comments and this is the part that I am referring to:

Coverage (Major Update) An SBOM should include information for all components that make up the target software, including transitive dependencies.

38

So all in all - package.lock and SBOM (assuming that the “Coverage” scope will become an industry practice - which is highly likely) are essentially covering the same information - complete list of all (even potential/optional) dependencies (including versions) of a software that you install.

But they serve slightly different purpose (and SBOM allows to have even multiple versions of such dependencies) to be present (which is also possible for vendored-in-dependencies in Python).

I think the “behavioural patterns” of package lock and SBOMs are very similar and there is very close relationship with each other.

It is for Airflow - we are actually using our constraints to generate the SBOMs - and whenever we update our constraints, our SBOMS needs to be also updated - we alraeady have that pattern in-place, and it basically means that SBOMS and constraints for us are always synced and updated together.

That’s why I think it’s important to mention it here, even if this is slightly tangential.

39

(moderators: feel welcome to split this out into a separate topic, if you deem that appropriate)

Yes, and IMO a wheel SBOM should only encode the information it definitely knows about. That is, it should reflect what was used to build the wheel.

In case an analogy is useful… We don’t list the other components of a dish, on the box for one of the ingredients – your jam jar doesn’t need to list the ingredients for bread, just because it’ll likely end up in a jam sandwich (it would be wrong to!). You would need to include them both on the sandwich’s box though (possibly transitively).

Here, the jam jar would be analogous to the airflow wheel and the downstream application/install of airflow is the sandwich. Airflow’s wheel should not have the same SBOM as the install of Airflow using your provided constraints.

If you conflate “what was this built with” with “what is this likely deployed with”, there’s a risk of divergence resulting in a mismatch here.

2 Likes
40

(moderators: feel welcome to split this out into a separate topic, if you deem that appropriate)

Yep.

If you conflate “what was this built with” with “what is this likely deployed with”, there’s a risk of divergence resulting in a mismatch here.

Correct. That’s what the split between differnt SBOM types (“source”, “build”, “deployment” time). The current consensus that CISA proposal is about is to - independently from the type of SBOM - to include all the “turtles-down” that were used to do this stuff - what was needed at build time, release time and deployment time. Indeed if we choose the .whl file to be “build time sbom” - then it should generally only contain all the “turtles down” needed to release a package - i.e. essentialy versions of isolated build environment coming from “build-dependencies” - but including all transitive dependencies for that. Quite agree here that we should not contain all the 760+ dependencies that airflow needs at build time (including all the dependencies neede to run tests in this case). It’s really the quetsion of the right “type” of dependencies used - in this case this is not really needed to update the SBOM because something changed in those “build” dependencies. I think more about the SBOM updates as “tooling to build SBOMS improved” and we can get “better SBOMS for the same”.

I hope that clarifies my way of thinking

UPDATE: I corrected the types of SBOM I referred to - which re also more standardised now https://www.cisa.gov/sites/default/files/2023-04/sbom-types-document-508c.pdf

41

I would, but you all keep interspersing your SBOM posts with non-SBOM stuff. So if you want a separate conversation then please explicitly start one and refer it back here.

1 Like