PEP 650: Specifying Installer Requirements for Python Projects

[brettcannon]

This is a PEP @dustin and I have been working on for a while now and we think it’s ready (enough) to discuss publicly.

Basically what we are trying to do is abstract out like PEP 517 an interface for installing dependencies. This comes up in cloud deployment scenarios where people ask cloud providers to support requirements.txt, then pipenv, then Poetry, then … you get the idea. Having a unified API for people use whatever tool they want does away with having to wait for your cloud provider to add support or having to always export out to a requirements.txt file (which we all know isn’t standardized).

This also comes up in editors where you want to help people get set up with a new environment but you don’t know how/what they want to install. You can do the same guessing game as a cloud provider, but that’s just as tiring/error-prone in an editor scenario. This is also potentially more user-friendly because if someone follows a tutorial for Poetry but an editor doesn’t support it yet ( VS Code is guilty of this specific case ATM), then you’re just out of luck. But if Poetry’s instructions/tooling included this API then the user wouldn’t even need to know/care about how to get things to work with the editor.

This could even apply to tools like tox where you only have to say what sort of dependency group you care about (or rely on the default one that tox will use). That way your dependencies are only managed in one place, and how to run the installation process is managed in one place.

This also helps abstract out lock files and potentially does away with having to standardize them. Poetry can do it their way, Pipenv can go another way, pip-tools as well. In the end it’s up to the tool to decide how they want to manage dependencies and decide how/what to install.

Rendered version at PEP 650 -- Specifying Installer Requirements for Python Projects | Python.org.

---

PEP: 650
Title: Specifying Installer Requirements for Python Projects
Author: Vikram Jayanthi vikramjayanthi@google.com,
Dustin Ingram di@python.org,
Brett Cannon brett@python.org
Discussions-To: PEP 650: Specifying Installer Requirements for Python Projects
Status: Draft
Type: Process
Content-Type: text/x-rst
Created: 16-Jul-2020
Post-History: 2021-01-14

Abstract

Python package installers are not completely interoperable with each
other. While pip is the most widely used installer and a de-facto
standard, other installers such as Poetry_ or Pipenv_ are popular as
well due to offering unique features which are optimal for certain
workflows and not directly in line with how pip operates.

While the abundance of installer options is good for end-users with
specific needs, the lack of interoperability between them makes it
hard to support all potential installers. Specifically, the lack of a
standard requirements file for declaring dependencies means that each
tool must be explicitly used in order to install dependencies
specified with their respective format. Otherwise tools must emit a
requirements file which leads to potential information loss for the
installer as well as an added export step as part of a developer’s
workflow.

By providing a standardized API that can be used to invoke a
compatible installer, we can solve this problem without needing to
resolve individual concerns, unique requirements, and
incompatibilities between different installers and their lock files.

Installers that implement the specification can be invoked in a
uniform way, allowing users to use their installer of choice as if
they were invoking it directly.

Terminology

Installer interface
The interface by which an installer backend and a
universal installer interact.

Universal installer
An installer that can invoke an installer backend by calling the
optional invocation methods of the installer interface. This can
also be thought of as the installer frontend, ala the build_
project for :pep:517.

Installer backend
An installer that implements the installer interface, allowing
it to be invoked by a universal installer. An
installer backend may also be a universal installer as well,
but it is not required. In comparison to :pep:517, this would
be Flit_. Installer backends may be wrapper packages around
a backing installer, e.g. Poetry could choose to not support this
API, but a package could act as a wrapper to invoke Poetry as
appropriate to use Poetry to perform an installation.

Dependency group
A set of dependencies that are related and required to be
installed simultaneously for some purpose. For example, a
“test” dependency group could include the dependencies required to
run the test suite. How dependency groups are specified is up to
the installer backend.

Motivation

This specification allows anyone to invoke and interact with
installer backends that implement the specified interface, allowing
for a universally supported layer on top of existing tool-specific
installation processes.

This in turn would enable the use of all installers that implement the
specified interface to be used in environments that support a single
universal installer, as long as that installer implements this
specification as well.

Below, we identify various use-cases applicable to stakeholders in the
Python community and anyone who interacts with Python package
installers. For developers or companies, this PEP would allow for
increased functionality and flexibility with Python package
installers.

Providers

Providers are the parties (organization, person, community, etc.) that
supply a service or software tool which interacts with Python
packaging and consequently Python package installers. Two different
types of providers are considered:

Platform/Infrastructure Providers
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Platform providers (cloud environments, application hosting, etc.) and
infrastructure service providers need to support package installers
for their users to install Python dependencies. Most only support pip,
however there is user demand for other Python installers. Most
providers do not want to maintain support for more than one installer
because of the complexity it adds to their software or service and the
resources it takes to do so.

Via this specification, we can enable a provider-supported
universal installer to invoke the user-desired installer backend
without the provider’s platform needing to have specific knowledge of
said backend. What this means is if Poetry implemented the installer
backend API proposed by this PEP (or some other package wrapped Poetry
to provide the API), then platform providers would support Poetry
implicitly.

IDE Providers
^^^^^^^^^^^^^

Integrated development environments may interact with Python package
installation and management. Most only support pip as a Python package
installer, and users are required to find work arounds to install
their dependencies using other package installers. Similar to the
situation with PaaS & IaaS providers, IDE providers do not want to
maintain support for N different Python installers. Instead,
implementers of the installer interface (installer backends) could
be invoked by the IDE by it acting as a universal installer.

Developers

Developers are teams, people, or communities that code and use Python
package installers and Python packages. Three different types of
developers are considered:

Developers using PaaS & IaaS providers
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Most PaaS and IaaS providers only support one Python package
installer: pip_. (Some exceptions include Heroku’s Python buildpack_,
which supports pip and Pipenv_). This dictates the installers that
developers can use while working with these providers, which might not
be optimal for their application or workflow.

Installers adopting this PEP to become installer backends would allow
users to use third party platforms/infrastructure without having to
worry about which Python package installer they are required to use as
long as the provider uses a universal installer.

Developers using IDEs
^^^^^^^^^^^^^^^^^^^^^

Most IDEs only support pip or a few Python package installers.
Consequently, developers must use workarounds or hacky methods to
install their dependencies if they use an unsupported package
installer.

If the IDE uses/provides a universal installer it would allow for
any installer backend that the developer wanted to be used to
install dependencies, freeing them of any extra work to install their
dependencies in order to integrate into the IDE’s workflow more
closely.

Developers working with other developers
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Developers want to be able to use the installer of their choice while
working with other developers, but currently have to synchronize their
installer choice for compatibility of dependency installation. If all
preferred installers instead implemented the specified interface, it
would allow for cross use of installers, allowing developers to choose
an installer regardless of their collaborator’s preference.

Upgraders & Package Infrastructure Providers

Package upgraders and package infrastructure in CI/CD such as
Dependabot_, PyUP_, etc. currently support a few installers. They work
by parsing and editing the installer-specific dependency files
directly (such as requirements.txt or poetry.lock) with
relevant package information such as upgrades, downgrades, or new
hashes. Similar to Platform and IDE providers, most of these providers
do not want to support N different Python package installers as that
would require supporting N different file types.

Currently, these services/bots have to implement support for each
package installer individually. Inevitably, the most popular
installers are supported first, and less popular tools are often never
supported. By implementing this specification, these services/bots can
support any (compliant) installer, allowing users to select the tool
of their choice. This will allow for more innovation in the space, as
platforms and IDEs are no longer forced to prematurely select a
“winner”.

Open Source Community

Specifying installer requirements and adopting this PEP will reduce
the friction between Python package installers and people’s workflows.
Consequently it will reduce the friction between Python package
installers and 3rd party infrastructure/technologies such as PaaS or
IDEs. Overall, it will allow for easier development, deployment and
maintenance of Python projects as Python package installation becomes
simpler and more interoperable.

Specifying requirements and creating an interface for installers can
also increase the pace of innovation around installers. This would
allow for installers to experiment and add unique functionality
without requiring the rest of the ecosystem to do the same. Support
becomes easier and more likely for a new installer regardless of the
functionality it adds and the format in which it writes dependencies,
while reducing the developer time and resources needed to do so.

Specification

Similar to how :pep:517 specifies build systems, the install system
information will live in the pyproject.toml file under the
install-system table.

[install-system]

The install-system table is used to store install-system relevant data
and information. There are multiple required keys for this table:
requires and install-backend. The requires key holds the
minimum requirements for the installer backend to execute and which
will be installed by the universal installer. The install-backend
key holds the name of the install backend’s entry point. This will
allow the universal installer to install the requirements for the
installer backend itself to execute (not the requirements that the
installer backend itself will install) as well as invoke the
installer backend.

If either of the required keys are missing or empty then the
universal installer SHOULD raise an error.

All package names interacting with this interface are assumed to
follow :pep:508's “Dependency specification for Python Software
Packages” format.

An example install-system table::

#pyproject.toml
[install-system]
#Eg : pipenv
requires = ["pipenv"]
install-backend = "pipenv.api:main"

Installer Requirements:
^^^^^^^^^^^^^^^^^^^^^^^
The requirements specified by the requires key must be within the
constraints specified by :pep:517. Specifically, that dependency
cycles are not permitted and the universal installer SHOULD refuse
to install the dependencies if a cycle is detected.

Additional parameters or tool specific data
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Additional parameters or tool (installer backend) data may also be
stored in the pyproject.toml file. This would be in the “tool.*”
table as specified by :pep:518. For example if the
installer backend is Poetry and you wanted to specify multiple
dependency groups, the tool.poetry tables could look like this:

::

[tool.poetry.dev-dependencies]
dependencies = "dev"

[tool.poetry.deploy]
dependencies = "deploy"

Data may also be stored in other ways as the installer backend sees
fit (e.g. separate configuration file).

Installer interface:

The installer interface contains mandatory and optional hooks.
Compliant installer backends MUST implement the mandatory hooks and
MAY implement the optional hooks. A universal installer MAY
implement any of the installer backend hooks itself, to act as both
a universal installer and installer backend, but this is not
required.

All hooks take **kwargs arbitrary parameters that a
installer backend may require that are not already specified,
allowing for backwards compatibility. If unexpected parameters are
passed to the installer backend, it should ignore them.

The following information is akin to the corresponding section in
:pep:517. The hooks may be called with keyword arguments, so
installer backends implementing them should be careful to make sure
that their signatures match both the order and the names of the
arguments above.

All hooks MAY print arbitrary informational text to stdout and
stderr. They MUST NOT read from stdin, and the
universal installer MAY close stdin before invoking the hooks.

The universal installer may capture stdout and/or stderr
from the backend. If the backend detects that an output stream is not
a terminal/console (e.g. not sys.stdout.isatty()), it SHOULD
ensure that any output it writes to that stream is UTF-8 encoded.
The universal installer MUST NOT fail if captured output is not
valid UTF-8, but it MAY not preserve all the information in that case
(e.g. it may decode using the replace error handler in Python). If the
output stream is a terminal, the installer backend is responsible
for presenting its output accurately, as for any program running in a
terminal.

If a hook raises an exception, or causes the process to terminate,
then this indicates an error.

Mandatory hooks:

invoke_install
^^^^^^^^^^^^^^
Installs the dependencies::

def invoke_install(
    path: Union[str, bytes, PathLike[str]],
    *,
    dependency_group: str = None,
    **kwargs
) -> int:
    ...

• path : An absolute path where the installer backend should be
  invoked from (e.g. the directory where pyproject.toml is
  located).

• dependency_group : An optional flag specifying a dependency
  group that the installer backend should install. The install will
  error if the dependency group doesn’t exist. A user can find all
  dependency groups by calling
  get_dependencies_groups() if dependency groups are
  supported by the installer backend.

• **kwargs : Arbitrary parameters that a installer backend may
  require that are not already specified, allows for backwards
  compatibility.

• Returns : An exit code (int). 0 if successful, any positive integer
  if unsuccessful.

The universal installer will use the exit code to determine if the
installation is successful and SHOULD return the exit code itself.

Optional hooks:

invoke_uninstall
^^^^^^^^^^^^^^^^
Uninstall the specified dependencies::

def invoke_uninstall(
    path: Union[str, bytes, PathLike[str]],
    *,
    dependency_group: str = None,
    **kwargs
) -> int:
    ...

• path : An absolute path where the installer backend should be
  invoked from (e.g. the directory where pyproject.toml is
  located).

• dependency_group : An optional flag specifying a dependency
  group that the installer backend should uninstall.

• **kwargs : Arbitrary parameters that a installer backend may
  require that are not already specified, allows for backwards
  compatibility.

• Returns : An exit code (int). 0 if successful, any positive integer
  if unsuccessful.

The universal installer MUST invoke the installer backend at the
same path that the universal installer itself was invoked.

The universal installer will use the exit code to determine if the
uninstall is successful and SHOULD return the exit code itself.

get_dependencies_to_install
^^^^^^^^^^^^^^^^^^^^^^^^^^^
Returns the dependencies that would be installed by
invoke_install(...). This allows package upgraders
(e.g., Dependabot) to retrieve the dependencies attempting to be
installed without parsing the dependency file::

def get_dependencies_to_install(
    path: Union[str, bytes, PathLike[str]],
    *,
    dependency_group: str = None,
    **kwargs
) -> Sequence[str]:
    ...

• path : An absolute path where the installer backend should be
  invoked from (e.g. the directory where pyproject.toml is
  located).

• dependency_group : Specify a dependency group to get the
  dependencies invoke_install(...) would install for that
  dependency group.

• **kwargs : Arbitrary parameters that a installer backend may
  require that are not already specified, allows for backwards
  compatibility.

• Returns: A list of dependencies (:pep:508 strings) to install.

If the group is specified, the installer backend MUST return the
dependencies corresponding to the provided dependency group. If the
specified group doesn’t exist, or dependency groups are not supported
by the installer backend, the installer backend MUST raise an
error.

If the group is not specified, and the installer backend provides
the concept of a default/unspecified group, the installer backend
MAY return the dependencies for the default/unspecified group, but
otherwise MUST raise an error.

get_dependency_groups
^^^^^^^^^^^^^^^^^^^^^
Returns the dependency groups available to be installed. This allows
universal installers to enumerate all dependency groups the
installer backend is aware of::

def get_dependency_groups(
    path: Union[str, bytes, PathLike[str]],
    **kwargs
) -> AbstractSet[str]:
    ...

• path : An absolute path where the installer backend should be
  invoked from (e.g. the directory where pyproject.toml is
  located).

• **kwargs : Arbitrary parameters that a installer backend may
  require that are not already specified, allows for backwards
  compatibility.

• Returns: A set of known dependency groups, as strings The empty set
  represents no dependency groups.

update_dependencies
^^^^^^^^^^^^^^^^^^^
Outputs a dependency file based off of inputted package list::

def update_dependencies(
    path: Union[str, bytes, PathLike[str]],
    dependency_specifiers: Iterable[str],
    *,
    dependency_group=None,
    **kwargs
) -> int:
    ...

• path : An absolute path where the installer backend should be
  invoked from (e.g. the directory where pyproject.toml is
  located).

• dependency_specifiers : An iterable of dependencies as
  :pep:508 strings that are being updated, for example :
  ["requests==2.8.1", ...]. Optionally for a specific dependency
  group.

• dependency_group : The dependency group that the list of
  packages is for.

• **kwargs : Arbitrary parameters that a installer backend may
  require that are not already specified, allows for backwards
  compatibility.

• Returns : An exit code (int). 0 if successful, any positive integer
  if unsuccessful.

Example

Let’s consider implementing an installer backend that uses pip and
its requirements files for dependency groups. An implementation may
(very roughly) look like the following::

import os
import pathlib
import subprocess
import sys

def invoke_install(path, *, dependency_group=None, **kwargs):
file_name = “requirements.txt”
if dependency_group:
file_name = f"{dependency_group}-{file_name}"
requirements_path = pathlib.Path(path) / file_name
return subprocess.call(
[sys.executable, “-m”, “pip”, “install”, “-r”, os.fspath(requirements_path)]
)

If we named this package pep650pip, then we could specify in
pyproject.toml::

[install-system]
#Eg : pipenv
requires = [“pep650pip”, “pip”]
install-backend = “pep650pip:main”

Rationale

All hooks take **kwargs to allow for backwards compatibility and
allow for tool specific installer backend functionality which
requires a user to provide additional information not required by the
hook.

While installer backends must be Python packages, what they do when
invoked is an implementation detail of that tool. For example, an
installer backend could act as a wrapper for a platform package
manager (e.g., apt).

The interface does not in any way try to specify how
installer backends should function. This is on purpose so that
installer backends can be allowed to innovate and solve problem in
their own way. This also means this PEP takes no stance on OS
packaging as that would be an installer backend’s domain.

Defining the API in Python does mean that some Python code will
eventually need to be executed. That does not preclude non-Python
installer backends from being used, though (e.g. mamba_), as they
could be executed as a subprocess from Python code.

Backwards Compatibility

This PEP would have no impact on pre-existing code and functionality
as it only adds new functionality to a universal installer. Any
existing installer should maintain its existing functionality and use
cases, therefore having no backwards compatibility issues. Only code
aiming to take advantage of this new functionality will have
motivation to make changes to their pre existing code.

Security Implications

A malicious user has no increased ability or easier access to anything
with the addition of standardized installer specifications. The
installer that could be invoked by a universal installer via the
interface specified in this PEP would be explicitly declared by the
user. If the user has chosen a malicious installer, then invoking it
with a universal installer is no different than the user invoking
the installer directly. A malicious installer being an
installer backend doesn’t give it additional permissions or
abilities.

Rejected Ideas

A standardized lock file

A standardized lock file would solve a lot of the same problems that
specifying installer requirements would. For example, it would allow
for PaaS/IaaS to just support one installer that could read the
standardized lock file regardless of the installer that created it.
The problem with a standardized lock file is the difference in needs
between Python package installers as well as a fundamental issue with
creating reproducible environments via the lockfile (one of the main
benefits).

Needs and information stored in dependency files between installers
differ significantly and are dependent on installer functionality. For
example, a Python package installer such as Poetry requires
information for all Python versions and platforms and calculates
appropriate hashes while pip doesn’t. Additionally, pip would not be
able to guarantee recreating the same environment (install the exact
same dependencies) as it is outside the scope of its functionality.
This makes a standardized lock file harder to implement and makes it
seem more appropriate to make lock files tool specific.

Have installer backends support creating virtual environments

Because installer backends will very likely have a concept of virtual
environments and how to install into them, it was briefly considered
to have them also support creating virtual environments. In the end,
though, it was considered an orthogonal idea.

Open Issues

Should the dependency_group argument take an iterable?

This would allow for specifying non-overlapping dependency groups in
a single call, e.g. “docs” and “test” groups which have independent
dependencies but which a developer may want to install simultaneously
while doing development.

Is the installer backend executed in-process?

If the installer backend is executed in-process then it greatly
simplifies knowing what environment to install for/into, as the live
Python environment can be queried for appropriate information.

Executing out-of-process allows for minimizing potential issues of
clashes between the environment being installed into and the
installer backend (and potentially universal installer).

Enforce that results from the proposed interface feed into other parts?

E.g. the results from get_dependencies_to_install() and
get_dependency_groups() can be passed into invoke_install().
This would prevent drift between the results of various parts of the
proposed interface, but it makes more of the interface required
instead of optional.

Raising exceptions instead of exit codes for failure conditions

It has been suggested that instead of returning an exit code the API
should raise exceptions. If you view this PEP as helping to translate
current installers into installer backends, then relying on exit
codes makes sense. There’s is also the point that the APIs have no
specific return value, so passing along an exit code does not
interfere with what the functions return.

Compare that to raising exceptions in case of an error. That could
potentially provide a more structured approach to error raising,
although to be able to capture errors it would require specifying
exception types as part of the interface.

References

… _build: GitHub - pypa/build: A simple, correct PEP517 package builder
… _Buildpack: heroku/heroku-buildpack-python - Buildpacks - Heroku Elements
… _Dependabot: https://dependabot.com/
… _Flit: https://flit.readthedocs.io
… _mamba: GitHub - mamba-org/mamba: The Fast Cross-Platform Package Manager
… _pip: https://pip.pypa.io
… _Pipenv: Pipenv: Python Dev Workflow for Humans — pipenv 2020.6.2.dev0 documentation
… _Poetry: https://python-poetry.org/
… _PyUP: https://pyup.io/

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.

…
Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
coding: utf-8
End:

[dustin (Dustin Ingram)]

Thanks Brett!

For those following along, this is roughly my idea from Structured, Exchangeable lock file format (requirements.txt 2.0?) - #83 by dustin, and that thread adds some additional context.

[bernatgabor (Bernát Gábor)]

At though level looks ok, but I have quite a few concerns/questions for this.

Moving past pipenv and pip (where pipenv is just an abstraction on top of pip), I see little mention, e.g., of how this would work for OS installers? How would PEP-508 specifiers from the table transform to OS packages?

I also find it hard to follow in places the document because there’s no clear explanation of who and where and how specifies the dependency groups. E.g., how would this translate to pip? I see poetry has the dev/deploy group, but how does this translate to pip/docker/conda package manager? I would expect some more research and explanation in this domain.

Finally, why would we hardcode the install system in the project metadata? Isn’t that plug and replace-able often? One could use pip, pipenv, conda, or httpx+https://github.com/pradyunsg/installer and achieve the same, not? The installer is often more a choice of the development environment than a trait of the package, so specifying it in the project metadata feels odd. At best could be suggested/default installer rather than the hardcoded must be seen in the proposal. Something like below to denote it’s just a suggestion, not a hard requirement:

[install-system]
#Eg : pipenv
suggested-requires = ["pipenv"]
suggested-install-backend = "pipenv.api:main"

[brettcannon]

@bernatgabor

I see little mention, e.g., of how this would work for OS installers? How would PEP-508 specifiers from the table transform to OS packages?

It wouldn’t just like we don’t have that sort of support for any other part of the packaging ecosystem ATM.

@bernatgabor

I also find it hard to follow in places the document because there’s no clear explanation of who and where and how specifies the dependency groups.

That’s not up to us, it’s up to the tool, so we purposefully don’t state anything around this. It’s just a common enough concept to offer support for it. We are very purposefully not dictating how the tools want to store anything so we don’t get bogged down in the same issues we had when we discussed standardizing the lock files.

@bernatgabor

E.g., how would this translate to pip?

It quite possibly wouldn’t as I don’t know if the pip team to want to get into the business of supporting this API. Think more pip-tools than pip as the target tool.

@bernatgabor

how does this translate to pip/docker/conda package manager?

I don’t see why Docker has to be involved with this beyond you installing a tool in your Docker container to use this API to install your dependencies. As for conda, I don’t think they do any form of grouping so I don’t expect them to support dependency groups.

@bernatgabor

why would we hardcode the install system in the project metadata?

If you’re using Poetry do manage your dependencies then why wouldn’t you specify that fact in your project’s metadata? Where else would it go?

@bernatgabor

Isn’t that plug and replace-able often?

Not in my experience. I can’t use Poetry to on my dev machine and then use pipenv in production to install dependencies in production w/o putting a bunch of work in.

Think of this PEP as clearly stating what tool you expect to run to install the dependencies for your project. Packages may use this for their dev dependencies (e.g. what to install to run their linting, formatting, testing, etc. like how some projects keep a dev-requirements.txt file around for such things). A Django project would use this to make sure their dependencies can be installed on depoyment without having to make sure production supports their chosen Python dependency management tool.

A very concrete example is Azure Funcions. If it supported this (and I’m not saying they will), then they could install the dependencies for a project that uses this PEP. No more reading the docs to see if they support e.g. Poetry or having to specifying how to name your requirements.txt file.

Another concrete example is setting up your environment in VS Code. With this we could detect this PEP and ask if you want help setting up a new virtual environment and automatically install your “dev” dependency group as part of that creation.

[uranusjr (Tzu-ping Chung)]

@brettcannon

def get_dependencies_to_install(
    dependency_group : string = None,
    **kwargs
) -> List[str]:
    ...

IMO dependency_group should be a list of groups, like PEP 508 allows multiple extras to be specified and installed together. Both pipenv and poetry have two groups (default and develop), but this has been a pretty popular feature request, and it is very possible some other tool would allow defining more groups that are not supersets to another (unlike pipenv and poetry, where dev-packages is always a superset to the default group).

[bernatgabor (Bernát Gábor)]

@brettcannon

Not in my experience. I can’t use Poetry to on my dev machine and then use pipenv in production to install dependencies in production w/o putting a bunch of work in.

Well, not today, but if you would introduce a standard way to specify your dependency groups and a tool-agnostic way of discovering/invoking that tool (which this PEP proposes), you’d likely be able to do so.

@brettcannon

Packages may use this for their dev dependencies (e.g., what to install to run their linting, formatting, testing, etc., like how some projects keep a dev-requirements.txt file around for such things).

Let’s not merge dev dependencies, please. Linting and formatting and testing, can all go into their own dedicated group/env.

@brettcannon

Another concrete example is setting up your environment in VS Code. With this, we could detect this PEP and ask if you want help setting up a new virtual environment and automatically install your “dev” dependency group as part of that creation.

Am I right to assume you assume the installer must be a python package and live in the same virtual environment like the one you’re provisioning? Would this PEP close the door for someone writing a python package installer in Rust, or not polluting the target environment with themselves (e.g., pip allows installing into other virtual environments, e.g., build/src/build/env.py at main · pypa/build · GitHub)?

@brettcannon

A Django project would use this to make sure their dependencies can be installed on deployment without having to make sure production supports their chosen Python dependency management tool.

I think part of the confusion in reading PEP is this seems to be PEP-517+PEP-518 together. It first introduces a way to specify your package manager for your project (and imposes limitations on how you must/can install that). In parallel, it presents a programmatic API for a machine invoking that. Considering these two problems are orthogonal, would it make sense to put them in separate sections?

@brettcannon

A very concrete example is Azure Functions. If it supported this (and I’m not saying they will), then they could install the dependencies for a project that uses this PEP. No more reading the docs to see if they support e.g. Poetry or having to specifying how to name your requirements.txt file.

So this PEP side-steps standardizing the requirements file format, by making it a package installers problem. And instead of trying to make a universal file format, we now make projects provide what they expect to install packages so that this tool can have its interpretation of a requirements file format.

@brettcannon

Not in my experience. I can’t use Poetry to on my dev machine and then use pipenv in production to install dependencies in production w/o putting a bunch of work in.

Those two tools are so far away from each other that you are right. However, in a more prolific world of package installers I can see them converging in similar groups that are compatible with each other.

[pf_moore (Paul Moore)]

@brettcannon

It quite possibly wouldn’t as I don’t know if the pip team to want to get into the business of supporting this API. Think more pip-tools than pip as the target tool.

As a pip developer, I can’t tell from this PEP whether it’s something I should be wanting to support in pip. As a user, I manage my project dependencies using requirements.txt and pip install -r, and I’d be annoyed if that wasn’t a supported way of working for VS Code. But if pip doesn’t support this API, then it sounds like you’re saying this might be the case

@brettcannon

Think of this PEP as clearly stating what tool you expect to run to install the dependencies for your project. Packages may use this for their dev dependencies (e.g. what to install to run their linting, formatting, testing, etc. like how some projects keep a dev-requirements.txt file around for such things).

What if I want my project to use an externally installed dependency? For example, I have globally installed copies of black, tox, sphinx, etc, and I want to use those, and not a per-project install? (But for a different project, I want specific versions, so I do want a local install for that project).

@bernatgabor

Am I right to assume you assume the installer must be a python package and live in the same virtual environment like the one you’re provisioning?

A long term interest of mine is definitely to move to a single “global” pip that can install packages in environments other than the one it’s installed in. So I’d be a strong -1 on any proposal that prohibits that possibility.

@brettcannon

A very concrete example is Azure Funcions. If it supported this (and I’m not saying they will), then they could install the dependencies for a project that uses this PEP. No more reading the docs to see if they support e.g. Poetry or having to specifying how to name your requirements.txt file.

That very definitely implies that pip needs to support this, as I can’t imaging that requiring projects to use a “higher level” install tool, just to be able to work on Azure Functions, would be very popular…

I need to read the PEP in more detail (I’ve only skimmed it so far) but it feels like it’s making some very specific assumptions about how people manage their projects and structure their workflow - assumptions that in my experience don’t match reality. So far, Python’s packaging community has carefully avoided blessing any particular workflow (that’s why we can have tools like pip, pip-tools, poetry, pipenv, etc, all coexisting and competing on an even basis). If we intend to focus on one particular workflow, or even a particular “style” of workflow (tool-managed, using a high level tool like poetry or pipenv) then that’s a fairly major change, and one I’d rather see happen explicitly rather than being presumed by any one PEP. As a specific suggestion, the Packaging User Guide should describe the key points of any “recommended workflow” before we start building machinery that assumes you’re using particular workflows.

(Disclaimer - I’ve tried both pipenv and poetry for my personal projects, and I don’t like either of them. That’s a personal thing, but it means I’m basically against any proposal that assumes either of them is the norm for project development…)

[pradyunsg (Pradyun Gedam)]

I haven’t thought about the implications of this yet, but I do have a couple of suggestions:

• Q: how about enforcing that the result of get_dependencies matches the result from invoke_install, by forcing the result of get_dependencies to be passed into invoke_install and other processes? That’ll also more closely model how most of these tools work anyway (resolve / get resolution result from disk -> do the install), and avoid the class of bugs due to the results from those drifting apart. (we’re seeing bugs like this with prepare_metadata_for_build_wheel and build_wheel in PEP 517, but it’s too late to change that)
• this needs buy in from the installers you’d like to see implementing this, so it’d be good to state who those are and make sure everyone relevant is involved in this discussion.

[steve.dower]

I support this (I’ve chatted with Brett in the past about it, but haven’t seen the actual proposal until now).

As a general comment, I think the wording can be simplified throughout (in a “try and delete at least one word from every sentence” sense), but otherwise is easy enough to follow. Specific comments below.

@brettcannon

Universal installer
An installer that can invoke an installer backend by calling the
optional invocation methods of the installer interface .

Personally, I’d have gone with “installer launcher” instead of “universal installer”, but I can see how you got here. Maybe just a clarifying note that the universal installer is actually just going to launch a real installer, and not try to supplant its interactive interface and configuration?

@brettcannon

Dependency group
A set of dependencies that are related. For example, a development
environment dependency group might include linting and formatting
modules while a production dependency group contains dependencies
required for deployment.

I’d clarify “are related” to something like “are required simultaneously for some user purposes but not others”. The example can probably also refer to “the ‘development environment’ dependency group” and “the ‘production’ dependency group” to emphasise that these are example names of groups, and not a binding definition of what the groups mean (which some people seem to be worried about above).

@brettcannon

The current system relies on these services/bots to keep up support as
new file formats and types are created and existing ones are changed.
By implementing this specification we can allow these services/bots to
interface through the spec and parse/write changes to dependencies
consistently, regardless of which installer is being used. Additionally
it would allow for more innovation in the space as it becomes easier
to support different installers and gives developers a standardized
way of interacting with them.

Proposal: “Currently, these services/bots have to implement support for each package installer individually. Inevitably, the most popular installers are supported first, and less popular tools are often never supported. By implementing this specification, these services/bots can support any (compliant) installer, allowing users to select the tool of their choice. This will allow for more innovation in the space, as platforms and IDEs are no longer forced to prematurely select a “winner”.”

@brettcannon

[install-system]

The install-system table is used to store install-system relevant data
and information

I think you want to clarify again here that this is purely for universal installers that are triggering dependency installation from the project source and not for install backends or this project in the form of a package. Maybe even explicitly say that pip and similar tools are not expected to read or process this table at all? I know it’s implied elsewhere, but having it clearly stated at the start of a section will make sure that people actually get it

@brettcannon

Additional parameters or tool ( installer backend ) data may also be
stored in the pyproject.toml file. This would be in the “tool.*”
table as specified by :pep: 518 .

Proposal: “Additional parameters for the installer backend will be read from the backend’s default locations. This may include pyproject.toml if the tool supports it, provided the data is stored in a “tool.*” table as specified by :pep:518.”

@brettcannon

A user can find all
dependency groups by calling
get_dependencies_to_install().keys() if dependency groups are
supported by the installer backend .

Looks like the get_dependencies_to_install spec was updated to return a list without updating this part. I think you want to call get_dependency_groups here.

@brettcannon

List[str]

@brettcannon

FrozenSet[str]

Oddly specific return types… can we just specify Iterable[str] here?

@brettcannon

Returns : An exit code (int). 0 if successful, any positive integer
if unsuccessful.

Why not require an exception here? The conversion through an exit code is always going to be lossy, and it’s no worse for anyone if the universal installer just handles all exceptions and makes up its own exit codes. But it’s better if it’s able to capture additional context.

Fundamentally, a failure to specify things correctly and a failure to install are the same to a user.

@brettcannon

The universal installer will use the exit code to determine if the
uninstall is successful and SHOULD return the exit code itself.

Why assume that the universal installer is about to exit? Why assume that a failed install means that the universal installer failed? Where does Azure Function’s build process return a failed exit code to? This seems like it’s presuming too much about a particular universal installer that I assume does not yet exist

---

One thing that might be helpful to also add into here is a concept of getting “activation” information. I’m thinking an API that returns the leading argv elements and env variables needed to launch Python with access to one/more dependency groups. I can see that being very valuable for, well, basically everyone except pip (particularly Poetry, Conda, and Tox).

I would definitely rule out any concept of a list of commands to run. We are not a terminal and are not responsible for interpreting shell commands and transferring ambient state between steps. Either a backend can specify an open-ended command line (where any arguments added are as-if they were passed directly to python3) and environment, or it doesn’t support the API.

[steve.dower]

@pf_moore

I need to read the PEP in more detail (I’ve only skimmed it so far) but it feels like it’s making some very specific assumptions about how people manage their projects and structure their workflow - assumptions that in my experience don’t match reality. So far, Python’s packaging community has carefully avoided blessing any particular workflow (that’s why we can have tools like pip, pip-tools, poetry, pipenv, etc, all coexisting and competing on an even basis). If we intend to focus on one particular workflow, or even a particular “style” of workflow (tool-managed, using a high level tool like poetry or pipenv) then that’s a fairly major change, and one I’d rather see happen explicitly rather than being presumed by any one PEP.

Yeah, I’d strongly recommend re-reading the PEP, because this is the opposite of what it’s proposing

Basically, the PEP is saying “we know you all use different workflows, and currently that information only goes into your README.md or devguide, so here’s a place in a machine-readable file you can also put it so we know how your project should be bootstrapped”.

As I mentioned in my reply to Brett, there are some places in the PEP that ought to call out that pip is not a “universal installer” in this context - it’s just a backend, and all it needs is the callable API that it can translate into an implied pip install -e . (or whatever command makes the most sense - the actual behaviour of it is pip-specific, it’s just the API to trigger the “default dev install” action is standardised).

[pf_moore (Paul Moore)]

@steve.dower

Basically, the PEP is saying “we know you all use different workflows, and currently that information only goes into your README.md or devguide, so here’s a place in a machine-readable file you can also put it so we know how your project should be bootstrapped”.

(I’m still only skimming at this point, sorry, I’ve got very limited time right now)

So I feel like the PEP could benefit from some more explicit examples. For example, if I have a project that has a requirements.txt that contains all of my project’s dependencies, and I install that using pip install -r requirements.txt, how would I specify that?

[pf_moore (Paul Moore)]

@steve.dower

Maybe just a clarifying note that the universal installer is actually just going to launch a real installer, and not try to supplant its interactive interface and configuration?

So pip isn’t a “universal installer” as it’s a “real installer”? Or is it both? Or neither? I can’t tell, personally, so I don’t know what this PEP is requiring pip to implement.

And if pip doesn’t implement any of this PEP, then who does? The reality is that there currently isn’t anything that implements “full” install capabilities (by which I mean from both wheels and source) apart from pip. Sure, we want that to change, but that’s a fair way in the future.

Note that my main technical concern here is that pip doesn’t support being called in-process, so we can’t¹ implement the installer interface. Maybe the idea is that someone writes an in-process wrapper that runs pip in a subprocess, and that wrapper becomes the PEP 650 “installer backend”?

¹ I can’t see it being acceptable for pip to close any PEP 650 related bug reports with the comment “we don’t support being called in-process”

[steve.dower]

@pf_moore

Maybe the idea is that someone writes an in-process wrapper that runs pip in a subprocess, and that wrapper becomes the PEP 650 “installer backend”?

That’s the idea, yeah. Hopefully that wrapper ships with pip, but if not, it just means users will put this in their pyproject.toml:

[install-system]
requires = ["pip", "piplauncher"]
install-backend = "piplauncher:main"

And then piplauncher will implement invoke_install as a subprocess call to [sys.executable, "-m", "pip", "install" ...] (where the “…” is “sensible default options for building a dev environment” or “configuration read from some well-known file relative to the current directory”).

@pf_moore

So pip isn’t a “universal installer” as it’s a “real installer”? Or is it both? Or neither? I can’t tell, personally, so I don’t know what this PEP is requiring pip to implement.

It’s a “real installer” (referred to as a backend by the PEP). And yeah, if you can’t tell, the PEP needs to make sure this is crystal clear.

Universal installers are not actually installers - they are tools that might trigger installers. Which is why I suggested using a different name for them.

[bernatgabor (Bernát Gábor)]

@steve.dower

Universal installers are not actually installers - they are tools that might trigger installers

Eh, then drop the installer name totally. Go instead for something like python environment providers. Because of the way I understand they can provide a working python environment? Or are they only seeding an existing python environment with some packages? (in which case they are more like project dependency provisioners - being able to talk to or contain the actual install logic is part of their responsibility, but they also manage how those packages/their dependencies are grouped, not?).

[brettcannon]

@bernatgabor

but if you would introduce a standard way to specify your dependency groups and a tool-agnostic way of discovering/invoking that tool (which this PEP proposes), you’d likely be able to do so.

We tried that in Structured, Exchangeable lock file format (requirements.txt 2.0?), and 107 comments later we still no closer to solving that problem. My experience from that thread is I don’t have the mental energy to try and drive consensus on that idea, hence this one. If someone else feels up for trying to make that happen again I would happily take a standardized lock file format over this.

@bernatgabor

Let’s not merge dev dependencies, please. Linting and formatting and testing, can all go into their own dedicated group/env.

People can do whatever they want. This PEP makes no decision on that front.

@bernatgabor

Am I right to assume you assume the installer must be a python package and live in the same virtual environment like the one you’re provisioning?

Yes.

@bernatgabor

Would this PEP close the door for someone writing a python package installer in Rust, or not polluting the target environment with themselves

I think “closing the door” is a bit strong of a statement, but it isn’t a specific workflow this PEP is trying to support. I mean if someone wanted to download and run a Rust tool they can, but once again there’s an issue here of how the heck do you support all of these potential workflows? Editors and cloud providers do not want to keep guessing at what people want, so having an API that simply gets called to do whatever the user wants to install stuff solves an actual problem that currently exists.

Or put another way: how do you get Amazon, Google, and Azure to all install the packages for your project on deployment while not asking them all to constantly be adding support for whatever tools you would prefer to use? Even today, it’s only by historical convention they support requirements.txt files via pip, not by some standard (otherwise you have to deploy all your dependencies with your code which sucks when you are deploying from a git repo on push). This is why pipenv and Poetry and not universally supported by cloud providers (and editors are in the same situation).

This also means we are already in this situation where other tools can’t be used. So an imaginary Rust tool wouldn’t be able to get traction unless it was just pip install -r requirements.txt-but-faster since that’s what the common denominator has become.

@bernatgabor

I think part of the confusion in reading PEP is this seems to be PEP-517+PEP-518 together. It first introduces a way to specify your package manager for your project (and imposes limitations on how you must/can install that). In parallel, it presents a programmatic API for a machine invoking that. Considering these two problems are orthogonal, would it make sense to put them in separate sections?

I personally don’t view them as orthogonal. From the use-cases motivating this PEP, installing installer tools on a machine but knowing how to use them isn’t useful. If a user is going to have to manually run a command to use the tools then they can be in charge of also doing the installation to begin with (and yes, to me that means PEP 517+518 are basically a unit with PEP 518 introducing pyproject.toml on top of it).

@bernatgabor

So this PEP side-steps standardizing the requirements file format, by making it a package installers problem. And instead of trying to make a universal file format, we now make projects provide what they expect to install packages so that this tool can have its interpretation of a requirements file format.

Yep, as long as “this tool” means “whatever installation tool you choose”.

@pf_moore

As a user, I manage my project dependencies using requirements.txt and pip install -r , and I’d be annoyed if that wasn’t a supported way of working for VS Code. But if pip doesn’t support this API, then it sounds like you’re saying this might be the case

Nope, I think you’re making a leap here from “VS Code support” that I wasn’t trying to imply. If this PEP was accepted, then the Python extension of VS Code could use it to automatically install dependencies for you. Otherwise you can still install things manually in the terminal like you do now (i.e. there isn’t any loss of support that we currently have, but this would open more possibilities for us to help users get set up).

@pf_moore

What if I want my project to use an externally installed dependency? For example, I have globally installed copies of black, tox, sphinx, etc, and I want to use those, and not a per-project install?

That’s between you and your tooling.

@pf_moore

That very definitely implies that pip needs to support this, as I can’t imaging that requiring projects to use a “higher level” install tool, just to be able to work on Azure Functions, would be very popular…

Steve answered this, but wrapping a pip install -r requirements.txt in a Python script should be trivial, so I don’t view that as much of a blocker, just like having setuptools call wheel isn’t something that’s burdensome either.

@bernatgabor

then drop the installer name totally. Go instead for something like python environment providers

Can’t because we are not providing the environment, we are just trying to help install packages into an environment.

@bernatgabor

Because of the way I understand they can provide a working python environment? Or are they only seeding an existing python environment with some packages?

An earlier version of this PEP tried to also tie in environment creation, but was left out as it was muddling everything. So your latter view of seeding/installing some packages into an existing environment is the accurate one.

Now to me, it flows naturally to then potentially define an API for creating a fresh environment after which it can be populated using this environment.

@pradyunsg

this needs buy in from the installers you’d like to see implementing this, so it’d be good to state who those are and make sure everyone relevant is involved in this discussion.

@sdispater and @techalchemy can join in if they want, but I also would not be shocked if people create wrappers around any tools they may want to use, so buy-in from tools would be very nice but isn’t required for this to work (e.g. I would probably create a wrapper around pip-tools if they didn’t want to support this).

To be abundantly clear, I care about solutions to the scenarios this PEP has outlined, not the solution itself. So if people can come up with a different solution that everyone can agree to I’m all in on that. But without a solution I feel like we are then telling the world that they should just standardize on requirements.txt and pip which seems unnecessarily restrictive, under-specified, and not great for e.g. pipenv and Poetry.

[pf_moore (Paul Moore)]

@brettcannon

Nope, I think you’re making a leap here from “VS Code support” that I wasn’t trying to imply. If this PEP was accepted, then the Python extension of VS Code could use it to automatically install dependencies for you.

The leap I was making was from the PEP enabling VS Code support for installers, to installers needing to have support for the API in order to have VS Code (and other consumers of the PEP) support them. Which is basically just “editors and cloud services will implement the PEP”, so not a huge leap…

As pip install -r requirements.txt is such a common workflow, I therefore find it jarring that the PEP focuses on pip envy and poetry, and never discusses how users of the pip workflow will be supported in a PEP 650 world…

[bernatgabor (Bernát Gábor)]

@pf_moore

pip envy

That’s probably the typo of the year. I’ve never seen a better description for pipenv.

[ncoghlan (Alyssa Coghlan)]

I think the general idea is a good one, but I also think it needs a concrete example of an installer backend to help clarify the intent (specifically, I’d suggest a sketch of a pip shim that calls pip in a subprocess and parses the output, with a “default” group mapping to “pip install -r requirements.txt” and a “dev” group mapping to “pip install -r dev-requirements.txt”, and the repo itself being usable as is). That should be short enough (sans error handling) to fit in an appendix, but even if it isn’t, the PEP could link to a GitHub repo containing the example.

For the names, I think “project installer backend” and “universal project installer”, with “[project-install-system]” as the table name, would help distinguish what this PEP is about (installing dependencies for a project given that project’s source repo) from what pip does (installing Python packages). Adding a link to Managing Application Dependencies — Python Packaging User Guide may also help to clarify the distinction.

At a more detailed level, I don’t follow the rationale for requiring the hooks to accept arbitrary keywords. These APIs are new, so the backwards compatibility argument doesn’t hold, and tool specific functionality should be exposed through tool specific APIs. Extra config settings for tools should be in the source repo, without needing to be passed in by the front end. If we want to add new optional keywords later, then a clean TypeError for unhandled keywords is easier to detect than having the backend silently ignore the option.

I think the PEP also needs to consider the option of allowing self-hosted project installer shims, where the adapter file lives directly in the repo rather than being an installable package.

[brettcannon]

@pf_moore

The leap I was making was from the PEP enabling VS Code support for installers, to installers needing to have support for the API in order to have VS Code (and other consumers of the PEP) support them.

Yes, but you/pip are not “them” as you are already supported by everyone. The key point here is that being asked to support every tool out that is a real burden due to no lock file standard or some other mechanism like this PEP to standardize on how to install what people want (unless we are prepared to tell the world that Python only supports containers as the universally supported way to distribute dependencies like that, which is currently how cloud providers tell Python users to do things if their specific installer tool isn’t supported).

@pf_moore

I therefore find it jarring that the PEP focuses on pip envy and poetry, and never discusses how users of the pip workflow will be supported in a PEP 650 world…

That’s because you’re already have your workflow supported around the world. As well, pip is absolutely not obligated to implement this API (see how short the example below is to understand why it’s not a burden to expect others to implement something).

Please don’t get too attached to the term “installer” in the PEP. If you want to think of this API for “installation coordinators” then that also fits what this is for and probably doesn’t make people think of pip as needing to implement this.

Or put another way, we didn’t need to worry about PEP 517 for setuptools’ benefit because everyone already supported setup.py bdist_wheel; the PEP was for making e.g. flit possible and other future tools that supported the API. Similar to that, this PEP isn’t for everyone who uses requirements.txt files w/ pip, this is for everyone else.

@ncoghlan

I also think it needs a concrete example of an installer backend to help clarify the intent (specifically, I’d suggest a sketch of a pip shim that calls pip in a subprocess and parses the output, with a “default” group mapping to “pip install -r requirements.txt” and a “dev” group mapping to “pip install -r dev-requirements.txt”, and the repo itself being usable as is). That should be short enough (sans error handling) to fit in an appendix, but even if it isn’t, the PEP could link to a GitHub repo containing the example.

How about an example right here (which can go into the PEP)?

import os
import pathlib
import subprocess
import sys


def invoke_install(path, *, dependency_group=None, **kwargs):
    file_name = "requirements.txt"
    if dependency_group:
        file_name = f"{dependency_group}-{file_name}"
    requirements_path = pathlib.Path(path) / file_name
    return subprocess.call(
        [sys.executable, "-m", "pip", "install", "-r", os.fspath(requirements_path)]
    )

This is pretty darn close to what I assume a pip-supporting implementation would do. This is also why I don’t think saying other tools that handle installation have to worry about supporting this PEP as there isn’t much to it already.

Now I can’t speak for @dustin , but I personally still think a lock file is a better solution (and that’s how other ecosystems seem to have solved this issue). But the impression I got from the last lock file discussion we all had was people didn’t agree. Now if I’m wrong and people are willing to work towards a lock file solutionthen I’m willing to help drive that PEP much like PEP 621, but I don’t want to do that and abandon this PEP if people are not on board with the idea that this is a problem worth solving and that a lock file is then the best solution for it.

But if all of that fails then it seems to me that we are stuck on pip’s docs on requirements files, requirements.txt as a file name, and pip as an installer based on convention alone. I mean this is why Poetry exports to requirements.txt and so does pipenv as this is the closest thing we have right now.

The funny thing about this is one of the objections/questions about this PEP was whether it excluded other installers not being in Python. And yet without any standard to begin with people are already being told to use pip since you can’t assume a requirements.txt file has every dependency listed and people will expect pip’s resolver to be what solves everything (or to use a container which, to me, is quite the leap from a just a file in your repository as a solution).

[pf_moore (Paul Moore)]

@brettcannon

Yes, but you/pip are not “them” as you are already supported by everyone.

Point taken. Although I don’t like that as a principle - I really don’t want pip to be “the next setuptools” that everyone has to special case, and no-one can avoid. If this PEP is accepted, new cloud providers should be able to use it and not do anything special for pip.

But I get your point, and agree that the wrapper code is sufficiently trivial that it’s not worth worrying about.