Semantics of PEP 508 URL Requirements

I’m designing a set of helper utilities for PEP 723 script dependency metadata. One of the things I want to be able to support is asking the question “does this environment support this requirement?” However, in order to do that, I need to know what it even means for an environment to match a URL requirement.

It seems to me there are 3 possible approaches here:

  1. If there is an installed package with the given name, it matches.
  2. If there is an installed package with the given name, and the URL in direct_url.json is the same, it matches.
  3. It never matches.

Option (1) is the most lenient, but also the most likely to be wrong. Presumablt there’s a reason to specify a URL, and “any old version of the package from PyPI” is unlikely to satisfy that reason, whatever it is.

Option (2) is not as simple as it first seems, because the URLs must match exactly. It would be very easy to accidentally use an equivalent, but textually different, URL and get a false negative. (Also, direct_url.text specifies that authentication data must be stripped, but not “well known, non security-sensitive” authentication data. So it’s not easy to be sure clear what will end up in direct_url.json for a given URL (it’s left to the installer to decide).

Option (3) is safest, but will mean it’s impossible to re-use environments if direct URL requirements are involved.

As far as I am aware, there is no standard interpretation of what the answer should be here, mainly because it’s never come up before (pip’s rules are not based on a standard, and only answer the question “do we need to reinstall?”, not “does this environment satisfy the requirement?”).

For my purposes, I’m inclined towards (3), because I don’t really care that much about supporting re-use of environments for URL requirements - so I’d rather take the safer option. But given that I’m thinking of this as a library function, not as an application choice, I’d be interested in knowing what others think. Also, I think that with PEP 703, we’re likely to see more interest in what the answer is, and I think it would be easier to respond to that if we have some form of community consensus on what we’d like to see.

@ofek - what do you expect to do for hatch?

For context, my pipx implementation[1] only reused environments based on an exact textual match of all requirements, so it’s even more cautious than option (3). I’m not sure if pip-run re-uses environments at all.


  1. I think the new one works the same ↩︎

PEP 723? [1]


  1. A good illustration of why I hate concepts referred to by PEP numbers and why I put effort into importing PEPs in the PUG. Why not ‘dependency specifiers’ and ‘inline script metadata’ instead of ‘PEP 508 requirements’ and ‘PEP 723 script dependency metadata’? ↩︎

2 Likes

Oops, fixed - thanks!

I did use the term “script dependency metadata”, so I consider the PEP number as basically a link to the specification (I forgot to make it a hyperlink, that’s also fixed now).

As for “PEP 508 requirements” that’s harder because a URL requirement is a requirement, not a dependency specifier, and in this situation the difference is important[1], but the term “requirement” is way too overloaded to use unqualified. Also, PEP 508 was written before we agreed that having a usable name in “how to teach this” was important, so it doesn’t clearly define usable names for the terminology it introduces. I’d be happy to see that fixed, but that’s a whole separate discussion (and not one I have the energy for).


  1. e.g., the packaging library names the 2 concepts separately, and what pip allows as a URL requirement is different than what PEP 508 allows… ↩︎

Ok. On the other hand, the canonical specification is not PEP 723, it’s Inline script metadata - Python Packaging User Guide :wink:

For the record, I don’t know the difference between dependency specifiers and requirements, and I don’t see where PEP 508 defines it. But I’ll do my homework and look into packaging before asking more questions (and potentially submitting PRs to the PUG).

Fair. I wish there was an easier way to know, remember and/or link these specs (PEPs are easy to link to). But I accept your point.

It’s all a bit of a mess. That’s my point. In fact, what’s defined in PEP 508 (and the “dependency specifiers” document) is a “specification”, that’s either a “name_req” or a “url_req”. These are the names of terminals in the grammar, there’s no plain text names given (that I could find).

But in packaging a “specifier” is the >= 3.2 bit (i.e., missing the package name) which is defined in the version specifiers spec as opposed to the dependency specifiers spec. What’s in the latter is called a “requirement” in packaging (and which in turn is different from the requirements in a pip requirements.txt file).

:slightly_frowning_face:

Anyway, this is rather off-topic, and I’d still like an answer to my original question, which is what are the intended semantics of URL requirements when checking if an environment already satisfies that requirement?

1 Like

Hatch does this https://github.com/pypa/hatch/blob/4de0f5f5f8209bc669906db715b5dcc2eaa77a35/backend/src/hatchling/dep/core.py#L82

2 Likes