# What is the correct interpretation of path-based PEP 508 URI\_reference?

**URL:** <https://discuss.python.org/t/what-is-the-correct-interpretation-of-path-based-pep-508-uri-reference/2815>\
**Category:** Packaging\
**Created:** [December 7, 2019, 2:54am UTC](https://discuss.python.org/t/what-is-the-correct-interpretation-of-path-based-pep-508-uri-reference/2815 "2019-12-07T02:54:27Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![chrahunt](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/chrahunt/32/947_2.png) [@chrahunt](https://discuss.python.org/u/chrahunt)\
**Post date:** [December 7, 2019, 2:54am UTC](https://discuss.python.org/t/what-is-the-correct-interpretation-of-path-based-pep-508-uri-reference/2815/1 "2019-12-07T02:54:27Z")

</div>

In PEP 508, the grammar allows for the `URI_reference` part of a `url_req` to be a path, like

```auto
pip @ localbuilds/pip-1.3.1-py33-none-any.whl
pip @ /localbuilds/pip-1.3.1-py33-none-any.whl
pip @ ./localbuilds/pip-1.3.1-py33-none-any.whl
pip @ ./localbuilds/pip

```

Currently pip doesn’t support this (see [pypa/pip#6658](https://github.com/pypa/pip/issues/6658)), possibly because packaging doesn’t support this (see [pypa/packaging#120](https://github.com/pypa/packaging/issues/120)). In order for pip to support this, we need to decide how to interpret a path given in the URL reference.

I see two possibilities:

1. We assume this is always a filesystem path and interpret it relative to the parent directory of the unpacked filesystem resource we consider as “containing” the direct reference
2. We interpret this relative to the input for the package which has the given requirement

Some examples:

1. `pip install ./pkg`, where `./pkg` contains a `setup.py` and which has an `install_requires=['dep @ ./dep']`
  1. pip installs `./pkg/dep` (a directory containing a `setup.py`)
  2. same as above

2. `pip install ./pkg-0.1.0.tar.gz`, where `./pkg-0.1.0.tar.gz` is an sdist containing a `setup.py` which has an `install_requires=['dep @ ./dep-0.1.0.tar.gz']`
  1. pip installs `<req-build-tmpdir>/dep-0.1.0.tar.gz`, where `<req-build-tmpdir>` is the temporary path into which `pkg-0.1.0.tar.gz` was extracted
  2. pip installs `./dep-0.1.0.tar.gz`

3. `pip install "pkg @ git+https://github.com/user/pkg.git@abcd1234"`, which refers to a git repository containing a top-level `setup.py`, which has an `install_requires=['dep @ ./dep']`
  1. pip installs `<req-build-tmpdir>/dep`, where `<req-build-dir>` is the temporary path into which the git repository was cloned
  2. pip installs `git+https://github.com/user/dep`

Another possibility is that we pick what “makes sense” for the use case, but that may not apply across the board and leads to packages that may only work with tools that interpret these references the same way.

I didn’t see anything that clarified this on the distutils-sig threads I could find on the subject [here](https://mail.python.org/archives/list/distutils-sig@python.org/thread/BRFWRKHIR5EX7LJXX4PKJYSEPT6HRMBE/#DXAC3NWV6ZLJXK76WBCQB3MDHTQHYQMY) and [here](https://mail.python.org/archives/list/distutils-sig@python.org/thread/5YJG2GYGD7TOW7JV26G2YX7SGWVX3ZP4/#HV3HQZHAMKRO4BOURTDBPVIXRKXVJNU5). [PEP 440](https://www.python.org/dev/peps/pep-0440/#direct-references) seems to leave a lot of decisions for tool developers.

---

<div class="post-metadata">

**Author:** ![uranusjr](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/uranusjr/32/103_2.png) [@uranusjr](https://discuss.python.org/u/uranusjr)\
**Post date:** [December 7, 2019, 7:19am UTC](https://discuss.python.org/t/what-is-the-correct-interpretation-of-path-based-pep-508-uri-reference/2815/2 "2019-12-07T07:19:04Z")

</div>

I vague recall there was a discussion somewhere (in Poetry’s tracker?) about how tools should infer version from a URL specifier, but can’t find it right now. @ncoghlan do you remember?

---

<div class="post-metadata">

**Author:** ![chrahunt](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/chrahunt/32/947_2.png) [@chrahunt](https://discuss.python.org/u/chrahunt)\
**Post date:** [December 7, 2019, 2:01pm UTC](https://discuss.python.org/t/what-is-the-correct-interpretation-of-path-based-pep-508-uri-reference/2815/3 "2019-12-07T14:01:14Z")

</div>

I’ve updated the examples in case they were not clear. The issue is not with version (which I would assume could be given like `dep==1.0 @ ...`), but with finding dependencies. Sorry if I’m misunderstanding!

---

<div class="post-metadata">

**Author:** ![uranusjr](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/uranusjr/32/103_2.png) [@uranusjr](https://discuss.python.org/u/uranusjr)\
**Post date:** [December 8, 2019, 5:14am UTC](https://discuss.python.org/t/what-is-the-correct-interpretation-of-path-based-pep-508-uri-reference/2815/4 "2019-12-08T05:14:16Z")

</div>

> [@chrahunt](#):
>
> which I would assume could be given like `dep==1.0 @ ...`

AFAIK this is not possible. PEP 508 gives the following rules:

```auto
name = identifier
extras_list = identifier (wsp* ',' wsp* identifier)*
extras = '[' wsp* extras_list? wsp* ']'
urlspec = '@' wsp* <URI_reference>
url_req = name wsp* extras? wsp* urlspec wsp+ quoted_marker?

```

There’s no place for version specifiers, which is what started that previous discussion.

* * *

Indeed this is different from the problem you mentioned. I should have read the post more carefully, sorry.

IMO the natural choice would be to interpret it as relative to the URL of the parent _artifact_, i.e. your option 2. This would make deploying artifacts without an index more natural.

---

<div class="post-metadata">

**Author:** ![chrahunt](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/chrahunt/32/947_2.png) [@chrahunt](https://discuss.python.org/u/chrahunt)\
**Post date:** [December 9, 2019, 4:39pm UTC](https://discuss.python.org/t/what-is-the-correct-interpretation-of-path-based-pep-508-uri-reference/2815/5 "2019-12-09T16:39:40Z")

</div>

> [@uranusjr](#):
>
> There’s no place for version specifiers, which is what started that previous discussion.

Thanks for pointing this out!

> [@uranusjr](#):
>
> IMO the natural choice would be to interpret it as relative to the URL of the parent _artifact_ , i.e. your option 2. This would make deploying artifacts without an index more natural.

I agree for the index and local artifact use case this makes the most sense. Any build system that generates (and uploads) one package containing multiple archives should be able to do just as well do it for multiple archives.

For the VCS-originating use case the right answer is less obvious to me.

With option 1 we would be able to express dependencies within a single repository succinctly in a way that should work whether the VCS repo is local or remote. The alternative is that we specify the full VCS URL with the relevant subdirectory in the fragment, but this would not work locally.

With option 2 we would be able to refer to sibling repositories independent of the actual remote VCS host.

As an alternative we can just say there’s no natural way to resolve relative refs from VCS URL-originating distributions and reject them if encountered, leaving the tying-together of multiple projects to higher-level tools until there’s a clear winner.

---

<div class="post-metadata">

**Author:** ![sbidoul](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/sbidoul/32/244_2.png) [@sbidoul](https://discuss.python.org/u/sbidoul)\
**Post date:** [December 9, 2019, 5:10pm UTC](https://discuss.python.org/t/what-is-the-correct-interpretation-of-path-based-pep-508-uri-reference/2815/6 "2019-12-09T17:10:03Z")

</div>

> [@chrahunt](#):
>
> With option 2 we would be able to refer to sibling repositories independent of the actual remote VCS host.

Are there security implications if we allow relative URLs to navigate up the root of the parent artifact?

Another interesting topic will be the behavior of `pip list` and `pip freeze` with relative URLs. I suppose it should make them absolute first?

---

<div class="post-metadata">

**Author:** ![chrahunt](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/chrahunt/32/947_2.png) [@chrahunt](https://discuss.python.org/u/chrahunt)\
**Post date:** [December 9, 2019, 5:47pm UTC](https://discuss.python.org/t/what-is-the-correct-interpretation-of-path-based-pep-508-uri-reference/2815/7 "2019-12-09T17:47:52Z")

</div>

> [@sbidoul](#):
>
> Are there security implications if we allow relative URLs to navigate up the root of the parent artifact?

If locking dependencies with specific hashes, which I assume security-conscious deployments are doing, then only artifacts matching the hashes would be installed. Does that mitigate the concern?

> [@sbidoul](#):
>
> Another interesting topic will be the behavior of `pip list` and `pip freeze` with relative URLs. I suppose it should make them absolute first?

Not just the behavior of `pip list` and `pip freeze`, but what should be stored in the `direct_url.json` in the first place. I made a comment [here](https://discuss.python.org/t/draft-pep-recording-the-origin-of-distributions-installed-from-direct-url-references/1535/34) with specific related questions.

---

<div class="post-metadata">

**Author:** ![uranusjr](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/uranusjr/32/103_2.png) [@uranusjr](https://discuss.python.org/u/uranusjr)\
**Post date:** [December 9, 2019, 6:03pm UTC](https://discuss.python.org/t/what-is-the-correct-interpretation-of-path-based-pep-508-uri-reference/2815/8 "2019-12-09T18:03:24Z")

</div>

> [@chrahunt](#):
>
> or the VCS-originating use case the right answer is less obvious to me.
> 
> With option 1 we would be able to express dependencies within a single repository succinctly in a way that should work whether the VCS repo is local or remote. The alternative is that we specify the full VCS URL with the relevant subdirectory in the fragment, but this would not work locally.
> 
> With option 2 we would be able to refer to sibling repositories independent of the actual remote VCS host.
> 
> As an alternative we can just say there’s no natural way to resolve relative refs from VCS URL-originating distributions and reject them if encountered, leaving the tying-together of multiple projects to higher-level tools until there’s a clear winner.

TBH, the whole idea of installing from VCS never makes any sense to me, personally 😛 If I were to design pip from scratch today, I would never involve it in VCS interpolation, and instead require the user to manually checkout the code and install from _that_ instead. This is not an option now, alas. In the same sense though, I would have zero problems if we go with the “alternative” approach, and simply reject relative paths in a VCS package.

If we _must_ accept relative paths for VCS schemes, I would still go for option 2 since it is more consistent. And again, users can always checkout the repo themselves if they want to refer to packages in the same repo with relative paths.

---

<div class="post-metadata">

**Author:** ![sbidoul](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/sbidoul/32/244_2.png) [@sbidoul](https://discuss.python.org/u/sbidoul)\
**Post date:** [December 9, 2019, 6:40pm UTC](https://discuss.python.org/t/what-is-the-correct-interpretation-of-path-based-pep-508-uri-reference/2815/9 "2019-12-09T18:40:08Z")

</div>

> [@chrahunt](#):
>
> `pip install ./pkg-0.1.0.tar.gz` , where `./pkg-0.1.0.tar.gz` is an sdist containing a `setup.py` which has an `install_requires=['dep @ ./dep-0.1.0.tar.gz']`

The manual equivalent is to download `pkg-0.1.0.tar.gz`, unpack it, change to the directory and run `pip install .` In that case (back to example 1.) `install_requires=['dep @ ./dep-0.1.0.tar.gz']` refers to `dep-0.1.0.tar.gz` in the current directory, i.e. inside `pkg-0.1.0.tar.gz`.

So that would indicate that option 1. is actually more natural, and would produce the same behavior when downloading and unpacking the artifact manually or when providing the artifact URL to `pip install`.

To have option 2., it would need dependencies such as `dep @ ../dep-0.1.0.tar.gz` (if allowed at all).

---

<div class="post-metadata">

**Author:** ![sbidoul](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/sbidoul/32/244_2.png) [@sbidoul](https://discuss.python.org/u/sbidoul)\
**Post date:** [December 9, 2019, 8:46pm UTC](https://discuss.python.org/t/what-is-the-correct-interpretation-of-path-based-pep-508-uri-reference/2815/10 "2019-12-09T20:46:04Z")

</div>

Similarly when doing `pip install ./pkg-0.1.0.tar.gz#subdirectory=subdir`, relative paths in `install_requires` would be relative to `subdir`. So it would mean that (if allowed) relative paths must be considered relative to the location of `setup.py` / `pyproject.toml`. And I would disallow escaping from the artifact / VCS root.

---

<div class="post-metadata">

**Author:** ![uranusjr](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/uranusjr/32/103_2.png) [@uranusjr](https://discuss.python.org/u/uranusjr)\
**Post date:** [December 11, 2019, 5:23am UTC](https://discuss.python.org/t/what-is-the-correct-interpretation-of-path-based-pep-508-uri-reference/2815/11 "2019-12-11T05:23:42Z")

</div>

> [@sbidoul](#):
>
> The manual equivalent is to download `pkg-0.1.0.tar.gz` , unpack it, change to the directory and run `pip install .`

Not really. Relative URIs are resolved based on directories, not path components (unlike Python’s `os.path.abspath`), so by extracting the file into a directory you’re changing the semantic meaning of the URI.

Given a relative path `./bar`, on a page `/foo/` it is resolved into `/foo/bar`, but on page `/foo` (notice the lack of trailing slash) it is resolved into `/bar`. So `./dep-0.1.0.tar.gz` should resolve to a file _besides_ `pkg-0.1.tar.gz`, not _inside_ it. Which is why option 2 is correct IMO.

---

<div class="post-metadata">

**Author:** ![sbidoul](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/sbidoul/32/244_2.png) [@sbidoul](https://discuss.python.org/u/sbidoul)\
**Post date:** [December 11, 2019, 7:18am UTC](https://discuss.python.org/t/what-is-the-correct-interpretation-of-path-based-pep-508-uri-reference/2815/12 "2019-12-11T07:18:28Z")

</div>

> [@uranusjr](#):
>
> you’re changing the semantic meaning of the URI.

It depends what you consider the base URI is in the first place. Let’s talk about `install_requires` first:

- option 1 considers the base URI is the file where the requirement is written, e.g. `setup.py`, `setup.cfg` or `pyproject.toml`
- option 2 considers the base URI is the artifact in which the project is packaged

My point is that option 1 is more natural for developers. I argue it is also more predictable, by considering the relative URI points to something local in the project. It will then work the same with different ways to install the project:

- `pip install .` from within the project directory
- `pip install git+https://.../pkg.git`
- `pip install https://.../pkg-1.0.0.tgz`

With option 2, the exact meaning of `install_requires` depends on where the artifact has been located, and the three above ways to install will likely break or give different results.

The other way relative URLs may be used, is as top level requirements, i.e.

- provided on the pip command line, in which case the base URI is the current working directory
- provided in a `requirements.txt`, in which case the base URI is the `requirements.txt` file

Using relative URLs in top level requirements is necessary and the behavior does not seem controversial.

Are there compelling use cases to allow relative URLs in `install_requires`?

---

<div class="post-metadata">

**Author:** ![pradyunsg](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pradyunsg/32/206_2.png) [@pradyunsg](https://discuss.python.org/u/pradyunsg)\
**Post date:** [December 11, 2019, 7:30am UTC](https://discuss.python.org/t/what-is-the-correct-interpretation-of-path-based-pep-508-uri-reference/2815/13 "2019-12-11T07:30:39Z")

</div>

I think we should go down the route of 1, since it avoids relative references to out-of-tree dependencies, which makes it easier to reason about wrt behavior of a package.

IMO an sdist should not be rendered not-installable if I change the directory it is kept in, or move a different file.

---

<div class="post-metadata">

**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\
**Post date:** [December 11, 2019, 8:20am UTC](https://discuss.python.org/t/what-is-the-correct-interpretation-of-path-based-pep-508-uri-reference/2815/14 "2019-12-11T08:20:48Z")

</div>

What is the use case for this type of URI reference? Unless I’m missing something, this discussion is purely theoretical. It would be easier to judge the options if we had at least some real world examples where this would be used…

---

<div class="post-metadata">

**Author:** ![uranusjr](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/uranusjr/32/103_2.png) [@uranusjr](https://discuss.python.org/u/uranusjr)\
**Post date:** [December 11, 2019, 8:50am UTC](https://discuss.python.org/t/what-is-the-correct-interpretation-of-path-based-pep-508-uri-reference/2815/15 "2019-12-11T08:50:32Z")

</div>

Judging on issues inquiring this enhancement, most people are looking for a way to put multiple packages (either as a source tree or vendored wheel/sdist) in a project, and be able to run a single `pip install .` in the project root to set them all up.

---

<div class="post-metadata">

**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\
**Post date:** [December 11, 2019, 9:22am UTC](https://discuss.python.org/t/what-is-the-correct-interpretation-of-path-based-pep-508-uri-reference/2815/16 "2019-12-11T09:22:55Z")

</div>

OK. I’m not actually sure that’s something that I even think we should support in pip, TBH. Why not just use `--find-links` or something? But as long as we do have some real-world use cases, that’s the main thing, so thanks for the information.

---

<div class="post-metadata">

**Author:** ![finswimmer](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/finswimmer/32/2030_2.png) [@finswimmer](https://discuss.python.org/u/finswimmer)\
**Post date:** [December 27, 2019, 1:40pm UTC](https://discuss.python.org/t/what-is-the-correct-interpretation-of-path-based-pep-508-uri-reference/2815/17 "2019-12-27T13:40:39Z")

</div>

I would support option 1 as well.

At the moment I’m very interested in this topic as we face a problem with `poetry` if a dependency, that should be installed, has a directory dependency ([https://github.com/python-poetry/poetry/issues/1689](https://github.com/python-poetry/poetry/issues/1689)). To fix this, we need to know what `pip` expect from `get_requires_for_build_wheel` / `get_requires_for_build_sdist` as a return value.

---

<div class="post-metadata">

**Author:** ![sbidoul](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/sbidoul/32/244_2.png) [@sbidoul](https://discuss.python.org/u/sbidoul)\
**Post date:** [January 17, 2020, 10:55am UTC](https://discuss.python.org/t/what-is-the-correct-interpretation-of-path-based-pep-508-uri-reference/2815/18 "2020-01-17T10:55:04Z")

</div>

Further thinking about this, my opinion now lends toward disallowing relative paths in `Install-Requires` metadata.

The main reason is that such relative paths would only make sense when installing from sdist (with sdists inside sdists) or VCS. Building a wheel with correct `Install-Requires` would not be feasible in such case.

From what I understand of the use cases, there are viable alternatives (such as `--find-links` or vendoring).

---

<div class="post-metadata">

**Author:** ![pradyunsg](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pradyunsg/32/206_2.png) [@pradyunsg](https://discuss.python.org/u/pradyunsg)\
**Post date:** [January 18, 2020, 7:11pm UTC](https://discuss.python.org/t/what-is-the-correct-interpretation-of-path-based-pep-508-uri-reference/2815/19 "2020-01-18T19:11:46Z")

</div>

> [@sbidoul](#):
>
> Further thinking about this, my opinion now lends toward disallowing relative paths in `Install-Requires` metadata.

Ditto. [more-characters-to-make-discourse-happy]

---

<div class="post-metadata">

**Author:** ![ofek](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/ofek/32/1033_2.png) [@ofek](https://discuss.python.org/u/ofek)\
**Post date:** [May 23, 2022, 3:31am UTC](https://discuss.python.org/t/what-is-the-correct-interpretation-of-path-based-pep-508-uri-reference/2815/20 "2022-05-23T03:31:41Z")

</div>

> [@pradyunsg](#):
>
> > [@](#):
> >
> > Further thinking about this, my opinion now lends toward disallowing relative paths in `Install-Requires` metadata.
> 
> Ditto.

Agreed. FYI Hatch now supports relative paths using the new context formatting capability [Dependencies - Hatch](https://hatch.pypa.io/latest/config/dependency/#local)