PEP 629: Versioning PyPI's Simple API

A tool like pip currently has 3 real external inputs, Wheels, Sdists, and Repositories.

Currently wheels have internal versioning that allow evolving it and having clients be able to tell if they can accurately understand that wheel file. The relevant sdist threads appear like sdists are possibly going to gain the same thing. That leaves the repository API, which this PEP solves.

A major benefit of this is that it means that, combined with wheel and sdist versioning, that something like pip could stop ā€œphoning homeā€ every so often to check for new versions, and can instead recommend upgrading when an end user actually needs to upgrade (e.g. when they attempt to install something or interact with a repository that uses some new feature they donā€™t understand).

You can find the rendered view of this online at https://www.python.org/dev/peps/pep-0629/

PEP: 629
Title: Versioning PyPI's Simple API
Author: Donald Stufft <donald@stufft.io>
Discussions-To: https://discuss.python.org/
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 16-Jul-2020
Post-History: 16-Jul-2020


Abstract
========

This PEP proposes adding a method for versioning the simple API so
that clients can determine which features of the simple API that a
specific repository supports.


Rationale
=========

When evolving the simple API, clients wish to be able to determine
which features the repository supports. Currently there is no
mechanism to do this, except by attempting to detect new features
by looking at the data in the reponses and see if it appears like
a particular feature is in use.

This works reasonably well for a modern version of a client to determine
if the repository supports all of the features it wants to implement,
however it does not do anything to tell an older version the client that
the repository supports features that it might not understand and to
allow messaging to indicate that it might not be correctly understanding
the output of the repository.

An example of a scenario where this happened was the phasing in of
python-requires metadata, while existing clients could still successfully
use the repository, they were lacking the ability to understand this new
piece of data which would have informed their behavior to select a better
file for end users.


Overview
========

This PEP proposes the inclusion of a meta tag on the responses of every
sucessful request to a simple API page, which contains a name attribute
of "pypi:repository-version", and a content that is a PEP 440 compatible
version number, which is further constrained to ONLY be Major.Minor, and
none of the additional features supported by PEP 440.

This would end up looking like::

  <meta name="pypi:repository-version" content="1.0">

Interpretation of this version is that incrementing the major version
is considereed a backwards incompatible change such that existing
clients would no longer be expected to be able to meaningfully use the
new API, and incrementing the minor version is a backwards compatible
change that existing clients would no longer be able to meaningfully
use the API.

It is expectation of this PEP that the major version will never be
incremented, and any future major API evolutions would utilize a
different mechanism for API evolution. However the major version
is included to disambiguate with future versions (e.g. a hypothetical
simple api v2 that lived at /v2/, but which would be confusing if the
repository-version was set to a version >= 2).

This PEP sets the current API version to "1.0", and expects that
future PEPs that further evolve the simple API will increment the
minor version number.


Clients
-------

Clients interacting with the simple API **SHOULD** introspect each
response for the repository version, and if that data does not exist
**MUST** assumee that it is version 1.0.

When encountering a major version greater than expected, clients
**MUST** hard fail with an appropiate error message for the user.

When encountering a minor version greater than expected, clients
**SHOULD** warn users with an appropiate message.

Clients **MAY** still continue to use feature detection in order to
determine what features a repository uses.


Rejected Ideas
==============

Using a Header
--------------

Instead of baking this information into the actual HTML, an
alternative would be to use a HTTP header. This idea was
considered and ultimately was rejected because it would make
mirrors have to start modifying headers instead of being able
to operate as a "dumb" HTTP server of files.


Using an URL
------------

Another traditional mechanism for versioning APIs is to bake it
into the URL, something like ``/1.0/simplee/`` or so. This works
well for major version changes where olders clients are not
expected to be capable of continuing to use it, but it is not
well suited to minor version bumps, particularly when the version
numbers can be viewed as largely advisory for end users.




Copyright
=========

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.



..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End:
1 Like

Technically pip has four inputs, the last being the unspecified --find-links :stuck_out_tongue:

Adding a version metadata is the right thing; it wouldā€™ve prevented the data-requires-python confusion we had a while ago.

2 Likes

Sounds good, and the PEP text looks great to me!

If this were GitHub, Iā€™d have clicked the approve button with this comment. :slight_smile:

I forgot to mention, since I expect this PEP to be uncontroversial, I plan to just be the BDFL-Delegate for it myself. If anyone would rather I not, feel free to say so and Iā€™ll ask the SC to handle it.

I think something went weird here in that last bit?

Otherwise looks good to me.

Fine with me. If anyone wants another BDFL for process reasons but we donā€™t want to bother the SC with it, I can cover it, too.

Yup, just mixed up the wording.

Fixed here PEP 629: Fix and clarify wording, add BDFL-Delegate header by dstufft Ā· Pull Request #1530 Ā· python/peps Ā· GitHub

I can be the PEP delegate to make it a little less sketchy having the author approve their own thing. :slight_smile:

2 Likes

If either of your or Paul wants to do it, feel free to just take it over with a PR to the PEP :slight_smile:

I expected this PEP to be pretty uncontroversial, and it appears that everyone either doesnā€™t have an opinion on it, or are in favor of it.

Given that, Iā€™m asking @brettcannon to go ahead and make a decision on it, unless he feels thereā€™s a benefit to leaving it open longer.

3 Likes

Iā€™m only leaving it open because this is a bad week for me to do one more pass on the PEP before (very likely) accepting it. Can you ping me next week to make sure I have one more look at it?

1 Like

@brettcannon Will do, thanks!

Just following up on this @brettcannon for the upcoming week!

1 Like

And Iā€™m back from vacation! I will aim to go over the PEP this week and either provide feedback or accept.

LGTM! Feel free to update the PEP to final, @dstufft.

I guess there also needs to be an update to packaging.python.org to correspond with the PR that accepts the PEP?

I just noticed that neither @dstufft or myself updated the PEP to final. That also means that the Simple repository API spec page does not list the PEP.

1 Like

I accepted the PEP via Accept PEP 629 by brettcannon Ā· Pull Request #1782 Ā· python/peps Ā· GitHub. Waiting until a PyPA library supports the tag or Warehouse starts using it to mark the PEP as final.

2 Likes

Add PEP 629 to the Simple repository API spec page by brettcannon Ā· Pull Request #815 Ā· pypa/packaging.python.org Ā· GitHub adds the PEP to the spec page on packaging.python.org.

And the PEP is now marked as ā€œFinalā€!