Pre-PEP: Dependency sources in pyproject.toml

Note: I recognize that there have been a few different similar threads close to this topic that have appeared to stall.

I would like to ask for feedback on the concept of Dependency Sources as a top level table in pyproject.toml to become a part of the spec. This concept is implemented in different ways across different tools including uv, hatch, poetry, and PDM.

Dependency Sources are a way to redirect dependencies during development while leaving the metadata untouched, these redirects can point to a local Path, a URL, a version control repository URL such as git URL, a separate index which must be the index used to resolve this specific dependency such as torch GPU wheels from the pytorch index.

The motivation behind this PEP is to have a single spec that will improve the experience using them along with providing better monorepo support in the packaging space.

12 Likes

For a minute there I was scared you actually want to change how dependencies are declared and normalize fetching dependencies from URL while building wheels. But after reading through it, I see it’s just a somewhat confusing name. I think that’s called dependency “patching” in the Rust world, right?

Overall, I think the proposal’s pretty solid. Some early editorial feedback:

This PEP defines a new table in pyproject.toml named sources. The table MAY appear in a pyproject.toml that has no [project] table, in which case it governs that file’s own dependency groups and does not reach the projects in any subdirectory.

I think this would benefit from explicitly saying that it can occur irrespective of whether [project] table exists, as right now you’re only covering the case it doesn’t.

Each key of [sources] is the name of a project whose requirements are to be redirected.

Right now this sounds like you’re referring to the project.name key of the project containing dependencies. I think it’d be better to be somewhat more verbose here: “specifies a package name, and all dependencies matching that package name …”

By the way, I personally would lean towards saying “dependency” rather than “requirement”, as this is term used in PEP 621.

Keys MUST be valid non-normalized names […]

Nit: it sounds like using a name that’s already normalized isn’t valid :-). Also the URL references non-existing anchor.

The value of each key is either a source entry or a non-empty array of source entries.

I think it would make sense to at least hint where the use of arrays is explained, since it’s some distance from this.

subdirectory

Optional, string. The path, relative to path, of the directory containing the project’s pyproject.toml. Used when a repository or archive contains a project below its root.

What’s the difference between specifying path + subdirectory and the subdirectory directly into path? I’m guessing this exists merely for consistency with other source kinds, but probably worth giving an explicit rationale.

rev, tag, branch

Optional, string. These keys are mutually exclusive, and tools MUST report an error if more than one is present. When none is present, the repository’s default branch is used.

Nit: you’re not explaining what these keys mean, and more importantly, what are the limitations on using them. I suppose tools are supposed to make sure that rev can only specify a revision and not a tag (nor use a tag that has the same name as the revision hash), etc., but worth making that explicit.

Because a branch is not a fixed revision, tools SHOULD record the resolved commit in any lock file they produce.

Strictly speaking, tag is not guaranteed to be fixed either.

When the value of a key in [sources] is an array, entries are considered in order and the first entry whose selectors all match is applied.

I suppose this somewhat answers my question, but you aren’t really specified whether all selectors have to match when it’s not an array. Also, I was wondering if extra and group should be mutually exclusive, given that (per the cited logic) they could never both match, and with a single entry someone could wrongly use both as a shorthand.

2 Likes

Yes this is closer to what the Rust world would call “patching” which is an overloaded term. I went with sources as this is what multiple tools mainly hatch and uv have implemented as their name for it.

Spot on this was more about consistency with other source kinds.

I think they should but I am open to feedback or ideas on why they should not be.

I also really like the idea, but I wonder if [sources] is too generic for the top-level section name. The other question is whether we’ll end up wanting future configurations for development apart from runtime dependencies and metadata.

Which leads me to think we might want to add a top-level [develop] key, under which [develop.sources] would be as you specific.

2 Likes

Just a nit:

Required, string, non-empty. The URL of a PEP 503 simple repository.

This should probably link to the Simple Repository API living spec instead, since PEP 503 is (1) frozen and (2) not the only index representation anymore, now that we have JSON too :slightly_smiling_face:


In terms of security: should this PEP contain some language about how indices (e.g. PyPI) are expected to validate this metadata (if at all?). There has historically been a “closure” assumption with PyPI, i.e. that any distribution published to PyPI only contains dependency metadata that references PyPI itself, so I think the PEP should probably dedicate a bit of space (maybe under security?) to how it does (or doesn’t) affect that assumption.

2 Likes

This breaks the barrier between “abstract dependencies” and “concrete dependencies”, doesn’t it?

I have seen cases in the past where having concrete dependencies hard-coded in the project (i.e. in pyproject.toml) caused issues. Different developers have different expectations regarding where dependencies should be installed from.

Thanks for the proposal, Cary!

One thing I would ask is you explicitly say you’re supporting everything Direct URL Data Structure - Python Packaging User Guide does (which pylock.toml also supports). That way it’s obvious/upfront there are no gaps.

1 Like

I will make sure to get that more explicit in the next changes as the intention is as you likely guessed to support everything in the Direct URL spec. I will link the spec and call it out that explicitly.

I am open to this and can see [develop] having potentially more use cases. Sources was just what had been chosen both by hatch and uv and seemed like a good place to start.

1 Like

There’s also [dependency-sources] which has nice symmetry with [dependency-groups]. I think even if we can never standardize a way to specify source files for sdists, [sources] at the top level is too easy to get confused with that idea.

How much does this cover workspaces? I think it would cover at least what I’ve used them for (path= would, I think).

3 Likes

It does cover a lot of the workspace use cases, the only thing that something like sources doesn’t cover for monorepos would be coordinated versioning across sub packages of the monorepo during releases

1 Like

I have committed changes that contain a lot of the feedback so far.

I think the table naming is still up in the air but for now I have it as [develop] with a sources key.

2 Likes

I made this more explicit in the draft now, but the intention is that there is a flag --no-sources to skip using whatever is in the concrete dependencies hard-coded into the project.

1 Like

Fixed this. Thanks for pointing it out!

I added a section in security, the intention is that it should never impact that assumption.

1 Like

It is not something I am confronted with personally, so I can not really judge the suggested solution here, but it seems fine. If you are interested I think I had collected links to such concrete reports over there at Proposal: overrides for installers (an old idea with overlapping scope).

1 Like

The main drawback of the proposed sources approach is that we split the dependency declaration into two parts responsible for different parts of the declaration, and in the process, create two different formats for declaring dependencies (PEP 508 and TOML sources table, not counting requirements.txt which extend PEP 508 with some of those features such as -e), requiring users to use both formats at different locations in their pyproject.toml. Instead of declaring your dependencies once per section, you declare one half in an unrelated section and then have to use tricks like extra and group to reference backs. This syntactically also allow a number of strange states such as both project.dependencies and develop.sources declaring a path.

Before introducing uv’s tool.uv.sources, we considered various other approaches that would avoid splitting uv’s dependency declaration, but ultimately settled on the sources table as the only feasible way to be compliant with PEP 621. If we’re doing a PEP fixing the big limitations of PEP 621 (relative paths, editables, index pinning), I’d strongly recommend exploding the PEP 508 strings from PEP 621 into inline tables and - if required - having build backends to rewrite [project]. We can take inspiration from Poetry and Cargo with their inline table formats and Cargo’s rewriting features.

The motivating example on the document is multi-package repositories. For this case, wouldn’t it be simpler to have a single foo/<TBD>.toml that declares which packages belong to a workspace and automatically sets up the right paths? This avoids editing every single pyproject.toml, allows moving projects and only changing at most one path and avoids accidentally installing a package from PyPI by forgetting a sources declaration.


On the PEP text:

Optional, boolean, default false. When true, the project at path MUST be installed as an editable install per PEP 660, and path MUST refer to a directory containing a project. Tools MUST report an error if it refers to a wheel or source archive.

The MUST formulation would break --no-editable in uv, and the editable controls in uv’s workspaces. I’d follow Dependency specifiers - Python Packaging User Guide which only mandates format and meaning, but does not mandate behavior.

This PEP defines one key in [develop], sources. Other keys are reserved for future specifications. Tools MUST NOT define keys of their own in [develop], and MUST report an error for keys they do not recognize.

What are tools supposed to do for features not covered in develop? Would we have project.dependencies, develop.sources and then also still tool.uv.sources for features not in the standard?

In URL sources:

Required, string, non-empty. The URL of a wheel or source archive. Tools MAY refuse schemes other than https.

Are there requirements on the filename in the URL, to distinguish what file type to expect? Is it intentional that it say wheel and source archive, but not source distribution?

Outside Python, Cargo’s [patch] table redirects dependencies in a similar way. Cargo reads [patch] only from the manifest at the root of a workspace, applies it to every member, and ignores it in the manifests of dependencies.

Cargo’s patch and replace table work more like uv’s override-dependencies. For declaring source such as workspace dependencies, Cargo instead uses an inline table that is rewritten as the crate is published.

PDM expresses local and VCS dependencies through direct references together with a separate index configuration.

There’s also the extensions to PEP 621 by evaluating environment variables inside PEP 508 strings, working

uv names indexes rather than taking URLs, does not accept the string shorthand, applies sources to build-system.requires, and applies tool.uv.sources from a workspace root to every member of the workspace.

Those are important features in uv! It would be bad to regress this support with develop.sources.

Sources MUST NOT be applied to build-system.requires. Build requirements decide which backend builds a distribution, and which version of it. If sources could redirect them, the same source tree could produce a different distribution depending on whether [develop.sources] was honored, which Sources and built distributions does not allow. uv currently applies sources to build requirements by default; see Existing implementations.

I’m not following this limitation, if you can override the runtime dependencies, why can’t you override the build dependencies too? For the motivating use case, multi-package repositories, it is common to have in-tree build backends, either vendored, patched or entirely custom, that you’d similarly want to target.

Tools MUST NOT read [develop.sources] from any other file. In particular:

  • A [develop.sources] table in a source tree or source distribution obtained as a dependency MUST NOT be honored, even when that dependency is itself redirected to a path below the invocation root.
  • Tools MUST NOT search ancestor directories, or directories between the invocation root and the target, for further tables.

What happens if a project has a git dependency where the git repo uses relative dependencies internally, i.e. a dependency on a remote multi-package repository?

4 Likes

Thanks for sharing this draft @cjames23! It would be great to see progress on this set of use cases.

It would be quite useful to have a set of links to previous discussions and proposals on this topic. Would you mind posting the ones you considered or found most relevant?

Yes, I called this out in the precedence but I can make it more clear that tools still can have features not covered in develop exist in their [tool.*.sources] The precedence should be that anything in [tool.uv.sources] overrides [develop.sources]. Which I think also addresses some concerns below. In fact it would be a requirement for hatch as well given that we apply sources per environment and allow overrides per environment, so that you can have an environment that does not apply sources and one that does.

I am going to spend some time thinking about adding named indexes to the spec for the Develop Sources PEP. I purposely left out workspace members as out of scope for this PEP. I think trying to include them here can make things more confusing. If you think it would make sense to have something like develop.workspace.members that might be worth a conversation. I had stayed away from this just based on the reaction from uv folks on standardization workspace functionality across tools. (Which I believe is only uv and hatch that currently offer workspace support?)

Yes, it follows the Direct URL spec, which allows any source archive (a GitHub tarball, for example). I’ll add that a `.whl` filename means a wheel and must follow the wheel filename spec, and anything else is treated as a source archive.

Good callout. I will fix this to only describe the request and not the behavior.

This one was just one I had not thought all the way through but after thinking about it now, I think this does not need to be a limitation. I am adding in this in the spec that it is allowed.

So my thought here and maybe I need to rework this a bit is more around avoiding breaking the assumption on “closure” with PyPI. What sources should not do is be applied when installing from PyPI. If you have thoughts on better ways to approach this I would welcome any input here!

I will try to get that list together for you!

@rgommers does the PPC have a process in mind for PEP sponsors yet or just sticking with the current process? Just would help to know as this gets closer to being ready to be put into a PEP draft state officially. I think within the next few weeks it could be ready for the first PR.

No special process for now. We have talked to the SC about getting more folks in the packaging community able to sponsor, but nothing to announce ATM.

1 Like