PEP 650: Specifying Installer Requirements for Python Projects

[brettcannon]

@bernatgabor

but if you would introduce a standard way to specify your dependency groups and a tool-agnostic way of discovering/invoking that tool (which this PEP proposes), you’d likely be able to do so.

We tried that in Structured, Exchangeable lock file format (requirements.txt 2.0?), and 107 comments later we still no closer to solving that problem. My experience from that thread is I don’t have the mental energy to try and drive consensus on that idea, hence this one. If someone else feels up for trying to make that happen again I would happily take a standardized lock file format over this.

@bernatgabor

Let’s not merge dev dependencies, please. Linting and formatting and testing, can all go into their own dedicated group/env.

People can do whatever they want. This PEP makes no decision on that front.

@bernatgabor

Am I right to assume you assume the installer must be a python package and live in the same virtual environment like the one you’re provisioning?

Yes.

@bernatgabor

Would this PEP close the door for someone writing a python package installer in Rust, or not polluting the target environment with themselves

I think “closing the door” is a bit strong of a statement, but it isn’t a specific workflow this PEP is trying to support. I mean if someone wanted to download and run a Rust tool they can, but once again there’s an issue here of how the heck do you support all of these potential workflows? Editors and cloud providers do not want to keep guessing at what people want, so having an API that simply gets called to do whatever the user wants to install stuff solves an actual problem that currently exists.

Or put another way: how do you get Amazon, Google, and Azure to all install the packages for your project on deployment while not asking them all to constantly be adding support for whatever tools you would prefer to use? Even today, it’s only by historical convention they support requirements.txt files via pip, not by some standard (otherwise you have to deploy all your dependencies with your code which sucks when you are deploying from a git repo on push). This is why pipenv and Poetry and not universally supported by cloud providers (and editors are in the same situation).

This also means we are already in this situation where other tools can’t be used. So an imaginary Rust tool wouldn’t be able to get traction unless it was just pip install -r requirements.txt-but-faster since that’s what the common denominator has become.

@bernatgabor

I think part of the confusion in reading PEP is this seems to be PEP-517+PEP-518 together. It first introduces a way to specify your package manager for your project (and imposes limitations on how you must/can install that). In parallel, it presents a programmatic API for a machine invoking that. Considering these two problems are orthogonal, would it make sense to put them in separate sections?

I personally don’t view them as orthogonal. From the use-cases motivating this PEP, installing installer tools on a machine but knowing how to use them isn’t useful. If a user is going to have to manually run a command to use the tools then they can be in charge of also doing the installation to begin with (and yes, to me that means PEP 517+518 are basically a unit with PEP 518 introducing pyproject.toml on top of it).

@bernatgabor

So this PEP side-steps standardizing the requirements file format, by making it a package installers problem. And instead of trying to make a universal file format, we now make projects provide what they expect to install packages so that this tool can have its interpretation of a requirements file format.

Yep, as long as “this tool” means “whatever installation tool you choose”.

@pf_moore

As a user, I manage my project dependencies using requirements.txt and pip install -r , and I’d be annoyed if that wasn’t a supported way of working for VS Code. But if pip doesn’t support this API, then it sounds like you’re saying this might be the case

Nope, I think you’re making a leap here from “VS Code support” that I wasn’t trying to imply. If this PEP was accepted, then the Python extension of VS Code could use it to automatically install dependencies for you. Otherwise you can still install things manually in the terminal like you do now (i.e. there isn’t any loss of support that we currently have, but this would open more possibilities for us to help users get set up).

@pf_moore

What if I want my project to use an externally installed dependency? For example, I have globally installed copies of black, tox, sphinx, etc, and I want to use those, and not a per-project install?

That’s between you and your tooling.

@pf_moore

That very definitely implies that pip needs to support this, as I can’t imaging that requiring projects to use a “higher level” install tool, just to be able to work on Azure Functions, would be very popular…

Steve answered this, but wrapping a pip install -r requirements.txt in a Python script should be trivial, so I don’t view that as much of a blocker, just like having setuptools call wheel isn’t something that’s burdensome either.

@bernatgabor

then drop the installer name totally. Go instead for something like python environment providers

Can’t because we are not providing the environment, we are just trying to help install packages into an environment.

@bernatgabor

Because of the way I understand they can provide a working python environment? Or are they only seeding an existing python environment with some packages?

An earlier version of this PEP tried to also tie in environment creation, but was left out as it was muddling everything. So your latter view of seeding/installing some packages into an existing environment is the accurate one.

Now to me, it flows naturally to then potentially define an API for creating a fresh environment after which it can be populated using this environment.

@pradyunsg

this needs buy in from the installers you’d like to see implementing this, so it’d be good to state who those are and make sure everyone relevant is involved in this discussion.

@sdispater and @techalchemy can join in if they want, but I also would not be shocked if people create wrappers around any tools they may want to use, so buy-in from tools would be very nice but isn’t required for this to work (e.g. I would probably create a wrapper around pip-tools if they didn’t want to support this).

To be abundantly clear, I care about solutions to the scenarios this PEP has outlined, not the solution itself. So if people can come up with a different solution that everyone can agree to I’m all in on that. But without a solution I feel like we are then telling the world that they should just standardize on requirements.txt and pip which seems unnecessarily restrictive, under-specified, and not great for e.g. pipenv and Poetry.