Pre-PEP discussion: Relative Path Dependencies

Packaging
1

Hi all! I’m gearing up to start seriously working on a standard for expanding dependency specifiers to support relative paths.[1]

There a bunch of topics which need to be addressed. I’m looking for some initial reactions to the idea (anyone interested in sponsoring it?) and areas of concern which I might be missing.

Here’s my thought process so far, as a rough sketch…

Motivation

There are a number of project contexts in which you want to be able to refer to installation of the current project (implicitly a source tree, not a pre built wheel). Additionally, in monorepos multiple source trees may want to refer to one another.

A few workflow tools already have syntax for this (e.g., pdm), exemplifying the need.

Is there other stuff that’s worth thinking about as motivation? As far as I know, that’s about the long and short of it.

Editable Support

The most basic question is: should relative path dependencies support specifying editable installs?
Several tools support this, especially for the “current project”, so there’s some level of demand for this.
I’d be inclined to include it unless someone explains why it should not be supported.

Here’s a fun, but somewhat confusing idea about syntax: -e path/to/pkg. As in, steal the -e from CLI pip syntax and make it part of the dependency string. It has some interesting implications, like pip install "-e ." being supported. I like how it would play out in dependent groups. Is this idea too wacky?

Publication of Packages with Relative Path Dependencies

Can you publish a package which depends on a relative path? What does that mean for installers?

I recall looking into the npm ecosystem support for relative paths, and they allow publication of such dependencies. I don’t remember seeing any special notes on installer behavior for npm.

I’d be inclined to allow this, but to explicitly allow installers to do what they like, including warning or erroring on such dependencies when installing. I think the only sensible non-error behavior is to resolve relative paths vis-a-vis the CWD or an explicitly specified directory.

Unix-style Paths

How naive am I being if I suggest that we use unix style paths and consider cross platform support a non-issue?

How much of the spec needs to be dedicated to spelling out this behavior?

Named Package @ Path

Is there significant benefit in requiring the package found at a path to be named, similar to what’s supported for URI dependencies?

For example, a syntax like foolib @ ./libs/foolib.

This makes the package name available just from parsing, so it seems mildly advantageous. But I’m only seeing mild benefits to this, so unless someone can help explain why it would be worthwhile I’d be inclined not to require it.

So, yeah, very much interested in hearing some initial thoughts on this! Especially “you need to think about Topic X” which I could be missing.

  1. Naturally, I was lying in wait so the lockfile discussion could get everyone’s full attention. Now I’m ready to monopolize all of your time again! Muahahahaha! ↩︎

3 Likes
2

I will throw this out there:

path/to/pkg[extra-one,extra=extra-two,install-mode=editable]

Otherwise, I’ll repeat what I said last time: I think it is important to keep the separation between abstract and concrete dependencies (see here, and also I guess my suggestion above would go against this principle but whatever we are brainstorming).

3

Is it actually supported, or is there just no validation to prevent it? Especially considering that in npm,

directory dependencies were poorly designed to begin with (hint: they weren’t designed at all, they were added as a “sure why not”, off-the-cuff, many years ago).

4

Given that URIs are supported, how does that relate? Do you think URIs are fine for some specific reasons which would be important properties to preserve here, or that they are problematic because they are concrete?

Relatedly, do you think that requiring the package name in name @ path syntax be satisfactory for this purpose? The name part is abstract, the path part is concrete.

I hadn’t seen commentary from the npm devs to that effect before, thanks for the reference.

I presume that support in npm is explicit, but I don’t really know that. It’s probably moot anyway: if the system is considered poorly designed by the maintainers, that undermines any reliance on it as precedent.

I’ll look more into what some other language communities have.

5

For reference, crates.io (PyPI equivalent of the Rust Ecosystem) does not allow packages with local path dependencies.

Crates that use dependencies specified with only a path are not permitted on crates.io.

2 Likes
6

Editables are entrenched and useful. But, they mix the concerns of “what to install” (the Specifiers), with “how to install” (the configuration that an installer should install via symlinks in-tree). So for this reason, I don’t believe there should be syntax for editables in specifiers and it should be considered an installer tool UX / configuration concern. Yes, I’m aware there are numerous PEP to the contrary. My opinion is just that we shouldn’t add more. :blush:

My opinion here is that this only makes sense in monorepo. And what we’re really looking for here isn’t support for physical file-paths, but a first-class logical treatment of monorepo in general. Examples would include “workspaces” in Cargo or Bazel that allow overrides on the installer side.

5 Likes
7

@sirosen this is what is meant by “abstract” vs. “concrete”. “Abstract” is the “what”. “Concrete” is the “how”. A typical example of this issue that I recall was when in a Poetry’s pyproject.toml the source index for a dependency was hard-coded to an index that was not available in certain countries where some of the contributors were working from. I guess it would have been fine if Poetry allowed to override this, but it did not. I am quite sure that I and others (including a Poetry developer) expanded a bit more on this issue in (one of) the previous discussion(s). This has quite a lot of ramifications. I have not followed the lockfile discussion very closely, but I can imagine that if source indexes are in the lockfiles, then it might be an issue. If the tools offer overrides, then all might be well, I do not know. I only know that mixing “abstract” and “concrete” is a red flag to me.

1 Like
8

I guess both URIs and name @ path are in the “problematic” box, unless the specification allows and encourages tools to offer practical ways to override them one way or another.

9

I have a reasonable (I won’t say perfect) grasp of the abstract/concrete divide as a practical reality in how dependencies are written and discovered.
What I’m less knowledgeable about is: what are the essential properties of dependency specification which are preserved via that distinction?

URI dependencies are concrete already, but offer some syntax (@) which allows us to abstract out the dependency names vs the URI itself.

I’m unclear on why this is insufficient as an accommodation or can’t be strengthened in the case of a new spec to be made sufficient. e.g., If we make the name portion required, then I can write

click @ ./vendor/pallets/click

And an installer may or may not accept options which alter resolution for this abstract dependency (click) with some attached – let’s call it “default” – concrete resolution data.

Have I missed something fundamental about how the abstract/concrete divide has to work that makes this entire idea non-viable? I’ve been operating under the assumption that this is a problem that I’ll need to understand better and solve, but perhaps I’ve been missing important details.

I agree with the premise here mostly[1], but I disagree with the conclusion. I think first class treatment of monorepos (e.g., workspaces) is a question of tooling, and that not being able to express relative paths is a consistent issue for users who are trying to leverage and “talk to their tools” via standardized metadata.

I would personally like to see packaging capabilities grow incrementally more friendly towards application (rather than library) use cases. If there’s a non-incremental path here, someone needs to articulate the plan for making that change – I think that’s the harder path vs the incremental one but it is, of course, possible.

  1. I would instead say “is the most useful for monorepos”. I have some non-monorepo use cases. ↩︎

10

To be absolutely clear here in case people don’t read far enough down that page, specifying only paths is not allowed; if you specify both a path and a version then it’s fine. Doing that is how cargo supports both local development of all of your crates – via the path – together along with them being dependencies when they are published to crates.io – via the version – so you don’t have to have separate requirements for local and install time.

2 Likes