PEP 650: Specifying Installer Requirements for Python Projects

Yeah, I’d strongly recommend re-reading the PEP, because this is the opposite of what it’s proposing :slight_smile:

Basically, the PEP is saying “we know you all use different workflows, and currently that information only goes into your README.md or devguide, so here’s a place in a machine-readable file you can also put it so we know how your project should be bootstrapped”.

As I mentioned in my reply to Brett, there are some places in the PEP that ought to call out that pip is not a “universal installer” in this context - it’s just a backend, and all it needs is the callable API that it can translate into an implied pip install -e . (or whatever command makes the most sense - the actual behaviour of it is pip-specific, it’s just the API to trigger the “default dev install” action is standardised).

3 Likes

(I’m still only skimming at this point, sorry, I’ve got very limited time right now)

So I feel like the PEP could benefit from some more explicit examples. For example, if I have a project that has a requirements.txt that contains all of my project’s dependencies, and I install that using pip install -r requirements.txt, how would I specify that?

3 Likes

So pip isn’t a “universal installer” as it’s a “real installer”? Or is it both? Or neither? I can’t tell, personally, so I don’t know what this PEP is requiring pip to implement.

And if pip doesn’t implement any of this PEP, then who does? The reality is that there currently isn’t anything that implements “full” install capabilities (by which I mean from both wheels and source) apart from pip. Sure, we want that to change, but that’s a fair way in the future.

Note that my main technical concern here is that pip doesn’t support being called in-process, so we can’t¹ implement the installer interface. Maybe the idea is that someone writes an in-process wrapper that runs pip in a subprocess, and that wrapper becomes the PEP 650 “installer backend”?

¹ I can’t see it being acceptable for pip to close any PEP 650 related bug reports with the comment “we don’t support being called in-process” :slightly_smiling_face:

1 Like

That’s the idea, yeah. Hopefully that wrapper ships with pip, but if not, it just means users will put this in their pyproject.toml:

[install-system]
requires = ["pip", "piplauncher"]
install-backend = "piplauncher:main"

And then piplauncher will implement invoke_install as a subprocess call to [sys.executable, "-m", "pip", "install" ...] (where the “…” is “sensible default options for building a dev environment” or “configuration read from some well-known file relative to the current directory”).

It’s a “real installer” (referred to as a backend by the PEP). And yeah, if you can’t tell, the PEP needs to make sure this is crystal clear.

Universal installers are not actually installers - they are tools that might trigger installers. Which is why I suggested using a different name for them.

1 Like

Eh, then drop the installer name totally. Go instead for something like python environment providers. Because of the way I understand they can provide a working python environment? Or are they only seeding an existing python environment with some packages? (in which case they are more like project dependency provisioners - being able to talk to or contain the actual install logic is part of their responsibility, but they also manage how those packages/their dependencies are grouped, not?).

We tried that in Structured, Exchangeable lock file format (requirements.txt 2.0?), and 107 comments later we still no closer to solving that problem. My experience from that thread is I don’t have the mental energy to try and drive consensus on that idea, hence this one. If someone else feels up for trying to make that happen again I would happily take a standardized lock file format over this.

People can do whatever they want. This PEP makes no decision on that front.

Yes.

I think “closing the door” is a bit strong of a statement, but it isn’t a specific workflow this PEP is trying to support. I mean if someone wanted to download and run a Rust tool they can, but once again there’s an issue here of how the heck do you support all of these potential workflows? Editors and cloud providers do not want to keep guessing at what people want, so having an API that simply gets called to do whatever the user wants to install stuff solves an actual problem that currently exists.

Or put another way: how do you get Amazon, Google, and Azure to all install the packages for your project on deployment while not asking them all to constantly be adding support for whatever tools you would prefer to use? Even today, it’s only by historical convention they support requirements.txt files via pip, not by some standard (otherwise you have to deploy all your dependencies with your code which sucks when you are deploying from a git repo on push). This is why pipenv and Poetry and not universally supported by cloud providers (and editors are in the same situation).

This also means we are already in this situation where other tools can’t be used. So an imaginary Rust tool wouldn’t be able to get traction unless it was just pip install -r requirements.txt-but-faster since that’s what the common denominator has become.

I personally don’t view them as orthogonal. From the use-cases motivating this PEP, installing installer tools on a machine but knowing how to use them isn’t useful. If a user is going to have to manually run a command to use the tools then they can be in charge of also doing the installation to begin with (and yes, to me that means PEP 517+518 are basically a unit with PEP 518 introducing pyproject.toml on top of it).

Yep, as long as “this tool” means “whatever installation tool you choose”.

Nope, I think you’re making a leap here from “VS Code support” that I wasn’t trying to imply. If this PEP was accepted, then the Python extension of VS Code could use it to automatically install dependencies for you. Otherwise you can still install things manually in the terminal like you do now (i.e. there isn’t any loss of support that we currently have, but this would open more possibilities for us to help users get set up).

That’s between you and your tooling.

Steve answered this, but wrapping a pip install -r requirements.txt in a Python script should be trivial, so I don’t view that as much of a blocker, just like having setuptools call wheel isn’t something that’s burdensome either.

Can’t because we are not providing the environment, we are just trying to help install packages into an environment.

An earlier version of this PEP tried to also tie in environment creation, but was left out as it was muddling everything. So your latter view of seeding/installing some packages into an existing environment is the accurate one.

Now to me, it flows naturally to then potentially define an API for creating a fresh environment after which it can be populated using this environment.

@sdispater and @techalchemy can join in if they want, but I also would not be shocked if people create wrappers around any tools they may want to use, so buy-in from tools would be very nice but isn’t required for this to work (e.g. I would probably create a wrapper around pip-tools if they didn’t want to support this).

To be abundantly clear, I care about solutions to the scenarios this PEP has outlined, not the solution itself. So if people can come up with a different solution that everyone can agree to I’m all in on that. But without a solution I feel like we are then telling the world that they should just standardize on requirements.txt and pip which seems unnecessarily restrictive, under-specified, and not great for e.g. pipenv and Poetry.

2 Likes

The leap I was making was from the PEP enabling VS Code support for installers, to installers needing to have support for the API in order to have VS Code (and other consumers of the PEP) support them. Which is basically just “editors and cloud services will implement the PEP”, so not a huge leap…

As pip install -r requirements.txt is such a common workflow, I therefore find it jarring that the PEP focuses on pip envy and poetry, and never discusses how users of the pip workflow will be supported in a PEP 650 world…

That’s probably the typo of the year. I’ve never seen a better description for pipenv. :laughing:

9 Likes

I think the general idea is a good one, but I also think it needs a concrete example of an installer backend to help clarify the intent (specifically, I’d suggest a sketch of a pip shim that calls pip in a subprocess and parses the output, with a “default” group mapping to “pip install -r requirements.txt” and a “dev” group mapping to “pip install -r dev-requirements.txt”, and the repo itself being usable as is). That should be short enough (sans error handling) to fit in an appendix, but even if it isn’t, the PEP could link to a GitHub repo containing the example.

For the names, I think “project installer backend” and “universal project installer”, with “[project-install-system]” as the table name, would help distinguish what this PEP is about (installing dependencies for a project given that project’s source repo) from what pip does (installing Python packages). Adding a link to Managing Application Dependencies — Python Packaging User Guide may also help to clarify the distinction.

At a more detailed level, I don’t follow the rationale for requiring the hooks to accept arbitrary keywords. These APIs are new, so the backwards compatibility argument doesn’t hold, and tool specific functionality should be exposed through tool specific APIs. Extra config settings for tools should be in the source repo, without needing to be passed in by the front end. If we want to add new optional keywords later, then a clean TypeError for unhandled keywords is easier to detect than having the backend silently ignore the option.

I think the PEP also needs to consider the option of allowing self-hosted project installer shims, where the adapter file lives directly in the repo rather than being an installable package.

5 Likes

Yes, but you/pip are not “them” as you are already supported by everyone. The key point here is that being asked to support every tool out that is a real burden due to no lock file standard or some other mechanism like this PEP to standardize on how to install what people want (unless we are prepared to tell the world that Python only supports containers as the universally supported way to distribute dependencies like that, which is currently how cloud providers tell Python users to do things if their specific installer tool isn’t supported).

That’s because you’re already have your workflow supported around the world. :wink: As well, pip is absolutely not obligated to implement this API (see how short the example below is to understand why it’s not a burden to expect others to implement something).

Please don’t get too attached to the term “installer” in the PEP. If you want to think of this API for “installation coordinators” then that also fits what this is for and probably doesn’t make people think of pip as needing to implement this.

Or put another way, we didn’t need to worry about PEP 517 for setuptools’ benefit because everyone already supported setup.py bdist_wheel; the PEP was for making e.g. flit possible and other future tools that supported the API. Similar to that, this PEP isn’t for everyone who uses requirements.txt files w/ pip, this is for everyone else.

How about an example right here (which can go into the PEP)? :wink:

import os
import pathlib
import subprocess
import sys


def invoke_install(path, *, dependency_group=None, **kwargs):
    file_name = "requirements.txt"
    if dependency_group:
        file_name = f"{dependency_group}-{file_name}"
    requirements_path = pathlib.Path(path) / file_name
    return subprocess.call(
        [sys.executable, "-m", "pip", "install", "-r", os.fspath(requirements_path)]
    )

This is pretty darn close to what I assume a pip-supporting implementation would do. This is also why I don’t think saying other tools that handle installation have to worry about supporting this PEP as there isn’t much to it already.

Now I can’t speak for @dustin , but I personally still think a lock file is a better solution (and that’s how other ecosystems seem to have solved this issue). But the impression I got from the last lock file discussion we all had was people didn’t agree. Now if I’m wrong and people are willing to work towards a lock file solutionthen I’m willing to help drive that PEP much like PEP 621, but I don’t want to do that and abandon this PEP if people are not on board with the idea that this is a problem worth solving and that a lock file is then the best solution for it.

But if all of that fails then it seems to me that we are stuck on pip’s docs on requirements files, requirements.txt as a file name, and pip as an installer based on convention alone. I mean this is why Poetry exports to requirements.txt and so does pipenv as this is the closest thing we have right now.

The funny thing about this is one of the objections/questions about this PEP was whether it excluded other installers not being in Python. And yet without any standard to begin with people are already being told to use pip since you can’t assume a requirements.txt file has every dependency listed and people will expect pip’s resolver to be what solves everything (or to use a container which, to me, is quite the leap from a just a file in your repository as a solution).

1 Like

Point taken. Although I don’t like that as a principle - I really don’t want pip to be “the next setuptools” that everyone has to special case, and no-one can avoid. If this PEP is accepted, new cloud providers should be able to use it and not do anything special for pip.

But I get your point, and agree that the wrapper code is sufficiently trivial that it’s not worth worrying about.

1 Like

This brings up the topic, how is the hook going to be called? Will it be called in-process, so the backend should use subprocesses to call the actual installer? Or will it always be called in a clean process (like PEP 517 currently guarantees) so the installer can be invoked as a Python function directly inside the hook? The PEP does not seem to clarify this at the current time (or I missed it).

(Personal opinion) I think the frontend should call the hooks in-process. This makes the playground most level for installers not implemented in Python (e.g. pyflow). If the hooks are always called in a clean process, non-Python installers will need one extra level of subprocessing.

Overall I’m +0.3 on this PEP. I’m not in love with it, but see where it can help.

FWIW big +1 on this. The topic opens the door for endless bikeshedding of details though and disagreements. IMHO if we look for universal approval we’ll never get there, and in this case lacking a solution hurts more than having a not perfect solution for this. Now this might be controversial, but I think as long as the pip maintainers are happy to implement the new lock file spec I’d be happy for the steering council to push it through (even if it’s a solution I don’t agree with). I’m certain other tools/projects will follow in line, even if their maintainers object to parts of it. If no one does it until then, in a post tox 4 world (read Q3) I’d pick up that thread again.

1 Like

I don’t know @dustin 's view, but currently I’m assuming in-process as there isn’t any other good way to specify what environment to install into ATM in general. Now the API might be able to grow to support such a thing, but once again that’s getting under-specified as I’m not sure what to pass in without making this very virtual environment-specific.

I wrote the above before reading this, so glad we seem to be on the same page. :slight_smile:

Thanks for the feedback!

I have chatted with some people privately about this and I think scoping is another concern. E.g. if I dev on macOS but deploy to Linux, do I need two lock files or just one that’s “good enough” that still has some potential resolution at install time? Doing a fancier pip freeze for the current system versus a hypothetical freeze as if I'm on this other type of system are very different (and become harder in the face of pre-PEP 643/dynamic in sdists and thus actually knowing what a project wants on various platforms).

I will be updating the PEP with the feedback that everyone has left so far, but I am taking into consideration whether the effort of standardizing a lock file is worth it instead or of or in parallel to this.

My impression is here you’d have a header within that requirement file that encodes the target - similar to how the wheel has a platform tag. Or even it can be within the requirements file name, similar to how we encode the platform tag within the wheel files names. Some projects might be able to generate universal requirement files, similar to how it’s possible to generate a wheel that’s universally installable. The tool generating the requirement file can probably determine the type of the requirement file generateed similar to how wheel project can determine at build time the wheel tag. This would follow precedent on how we handle wheel generation, the requirements file generation would follow suit to lessons learned from there. Tools generating requirement files could do a two phase operation, in the first parse the dependency tree to determine the requirement file tag (perhaps can be universal, perhaps it’s Linux only because finding out what’s Windows/macOS requires a Windows/macOS machine), and then based on the tag might generate or omit some platform-specific dependencies (by using the environment markers already introduce in PEP 496 -- Environment Markers | Python.org). You get the idea. E.g. : requirements-dev-py3.txt, requirements-dev-cp36-cp36m-manylinux2014_aarch64.txt.

But this is off-topic and probably more relevant in another thread…

1 Like

That was what I was thinking as well.

Yep. :slight_smile:

1 Like

I have updated the PEP based on the feedback in this topic. Mostly clarification, a loosening of the return types, and an “Open Issues” section.

Hi,

It would be very useful if this PEP stated explicitly that it is not meant to be used by widely used Python packages to specify runtime dependencies for their users. I had to read this PEP twice to be sure, the title and mentions of PEP 517/518 made me think otherwise at first.

Right now we have in a project like, for example, SciPy:

  • install_requires in setup.py for runtime dependencies
  • [build-requires] in pyproject.toml for build dependencies
  • various [dev|doc|test]_requirements.txt files for specifying development dependencies

Plus CI config files, a Dockerfile and an environment.yml (for conda/mamba) where those lists of dependencies may get repeated. Given that we don’t want to prescribe use of a certain installer to either users or contributors/maintainers, the only thing that’s possibly relevant is perhaps some streamlining of CI config files?

It looks to me like the comparison with PEP 517 is misleading too. PEP 517 made sense mainly for Python packages (and not for most of the stakeholders of this PEP), because a package invariable needs one particular build system. For installers on the other hand, a package author doesn’t want to know - they want both their users and contributors to use whatever they like.

Apologies if I’m misunderstanding something.

Cheers,
Ralf

1 Like

@brettcannon pointed out earlier that in his experience this isn’t true, for example from what you described you’re using requirements.txt files that imply pip nowadays. What other installers would people be able to use? conda?

1 Like