Proposal for dynamic metadata plugins

[CAM-Gerlach]

@rgommers

@CAM-Gerlach

optional-dependencies.test.file = “test-requirements.in”

This should not be optimized for imho. The better solution is to have pip and installers be able to install from

[project.optional-dependencies]
test = [....]

We have a proper section in pyproject.toml, so let’s use it.

Just to note, since I was quoted, I wasn’t necessarily endorsing that aspect of the proposal, just pointing out a more concise alternative syntax for it. FWIW, I don’t use or favor this pattern myself, though it’s already fully possible to do this without this proposal using the current dynamic.

@rgommers

There’s discussion on the pip issue tracker about gaining the ability to install the various groups of dependencies listed in pyproject.toml.

Err, maybe getting a little OT, but isn’t that perfectly possible right now with pip install -e .[test], etc. (except you get the package built and installed too)? Or are you referring to a --deps-only flag which would enable installing dependency groups (and package deps in general) without installing the package itself, which would replace that use case for requirements.txt (which is certainly something I’ve missed from using it in Conda, and would very much like to see in pip)?

@rgommers

My impression so far is that I quite like the core argument @henryiii is making about allowing specifying metadata statically and adding them to dynamic.

To note, this would partially, but not completely address what you’re asking for in the other thread; your backend could add additional requirements, but not alter the specifiersets for existing ones…unless either there was a special case allowing additional specifiers for when SpecifierSet(<modified>) in SpecifierSet(<original>), i.e. the modified version specifierset was a subset (i.e. contained within, stricter than) the original—or multiple specifiersets were allowed for the same distribution name key, in which case the intersection was taken (which could potentially result in impossible to satisfy dependencies if the sets are disjoint).

[dstufft (Donald Stufft)]

I think that we should not explicitly version pyproject.toml, I think that ends up making things much more complex than they need to be for end users.

I also don’t think that pyproject.toml should be frozen in time.

We’re going to make changes to it, and that means that older tools may not understand parts of it, and possibly may even break with newer standards. I think that’s perfectly fine. Likewise tools will need to continue to understand older styles for some period of time (perhaps indefinitely). This is how every new feature in basically all of packaging works. It generally works out fine.

To that end, I don’t see a problem with adding something like this (or for the license PEP either). Each PEP that wants to change the spec will need to determine what the backwards compatibility impact is, and whether the benefits outweigh the impact or not.

---

On allowing “partial” values for dynamic fields. I think that makes a lot of sense to allow, as long as we only allow extending and not replacing/removing/etc.

---

On the topic of the idea itself. I think the overall idea is sound. I think where it likely to run into problems is the discussions around the UX (as these things tend to do). I personally am +0.5 on the inline configuration option, but one thing that would be interesting to do maybe is to figure out if there is a sane way to layer this proposal over multiple PEPs, such that we can make smaller more focused changes, and use the experience from that to inform our choices in the future.

One way I could see doing this is to break this up something like:

First, a PEP that defines the API that backends can call the plugins with, but leaves exposing those plugins to the build backend. This could look something like:

[project]
dynamic = ["version"]

[tool.setuptools]
metadata.providers.version = {...} # the dict from the current PEP

Each build backend would be free to decide how to expose those settings, they might even require the use of some particular plugin, or choose not to make it generic or whatever.

This minimizes the changes to pyproject.toml, and lets us get a PEP that focuses on the actual interface and unlocks the bulk of the value. It also makes it easier to deprecate it or even remove it if it ends up not being super useful.

Then, assuming people use it, we can later add another PEP that modifies the project table to try and standardize exactly how these plugins are called and configured across build backends. Except we can do that with the experience we’ve gained from having various build backends implementing this themselves. Or we can decide that we ultimately don’t need it, and just leave it outside of the project table completely.

[pf_moore (Paul Moore)]

@dstufft

To that end, I don’t see a problem with adding something like this (or for the license PEP either). Each PEP that wants to change the spec will need to determine what the backwards compatibility impact is, and whether the benefits outweigh the impact or not.

I’m OK with this - I didn’t mean to imply that pyproject.toml had to be versioned, just that unless it’s versioned, new proposals have to consider and discuss how backward compatibility and transition will be handled, and have to take into account the reality that that old tools will still want to read new projects, and old projects (with old-style pyproject.toml files) will need to use newer versions of tools.

[domdfcoding (Dominic Davis-Foster)]

As the maintainer of whey I’m supportive of this proposal as originally presented in this thread. In particular, allowing values in [project] to be amended if they’re listed in dynamic would lead to a clearer configuration (especially regarding classifiers)

Besides whey, this PEP would also be beneficial for hatch-requirements-txt (which I also maintain) to allow its use with e.g. flit or maturin. It’s currently hatch specific, and allows dependencies and optional-dependencies to be defined in requirements.txt-style files (like setuptools allows)

---

Re changing the [project] table between the source and the sdist (as it was brought up earlier in the thread), whey does this only for dependencies, classifiers and requires-python, and only if they’re marked as dynamic. My rationale was that it makes the sdist’s pyproject.toml more useful for tools (or humans) inspecting it, as those fields are now populated with something that won’t change, and it matches the data in PKG-INFO. It doesn’t touch README or LICENSE, even if those point to a file.

[abravalheri (Anderson Bravalheri)]

Thank you very much for the thoughtful response Paul.

Maybe this is a subject we should discuss separately.
I do like Donald’s proposal, it seems like a good start, and meanwhile we can analyse an build consensus around improving the UI of pyproject.toml (not lightly, but after a careful discussion about what are the objective implications, what are the benefits and downsides, what is the conceptual shift, etc …).

[henryiii (Henry Schreiner)]

I’m going to need to rework the proposal quite a bit; IMO this is a key issue we must address:

@pf_moore

Furthermore, the spec might need some words to discuss how this interacts with the core metadata Dynamic field.

I’ve not really looked at metadata 2.2+, due to missing PyPI support, so completely missed this issue.

Especially if we break it up a bit further, maybe we could even suggest or require a backend when computing a non-Dynamic dynamic metadata field (okay, not sure I like the two concepts being slightly different but having the same name!) to pull directly from the SDist instead of running the plugin when doing a build from SDist? Don’t know if that’s a good idea or not, but I know third-party packagers would love not requiring the plugins be present - packagers that build from SDist like conda-forge could just skip adding the plugin to their build dependency lists in that case.

@pf_moore

My concern is that the easy answer is to declare everything a plugin manages as dynamic

I think both of the key initial use cases, version and readme, are probably always not Dynamic. And we should really be avoiding Dynamic whenever possible. An upcoming use case, pinning a dependency at build-time (whatever version of NumPy you use when building must be equal or newer to be used after you build the wheel) would have to be Dynamic, as it occurs when you do the build. So I think we need the flexibility to specify this and the current API return value doesn’t allow that. I think this should be specified by the plugin. This might be as simple as adding an optional hook function dynamic_metadata_is_dynamic that is assumed false if not present. Not sure.

I think the “additive metadata” portion could possible be proposed sooner. We’ll need to redo the specification a bit (and update the example implementation in scikit-build-core; I think I could also set up an example implemention based on flit and/or a helper for pyproject-metadata.

@dstufft

First, a PEP that defines the API that backends can call the plugins with, but leaves exposing those plugins to the build backend. This could look something like:

This is exactly what we have in scikit-build-core (in main):

[project]
name = "mypackage"
dynamic = ["version"]

[tool.scikit-build]
metadata.version = "scikit_build_core.metadata.setuptools_scm"

It’s based on the original form of the proposal (and was contributed by someone who hadn’t worked with scikit-build-core before!), so it’s still using a simple string, and there are no inline config options, and the API is the original version. I’ve got a branch that adds those, but it’s somewhat on hold until we iron out what’s going to be in the proposal. I’lll probably require the tool.scikit-build.experimental=true flag to be set to use it, at least with non-included plugins. Once we have a stable proposal I could propose it for meson-python, at least.

Though, really, we don’t have to have a PEP for the API - this doesn’t change the pyproject.toml at all at this point, it’s only a standardization agreement between plugins and backends. So perhaps we could propose both things (the API and the pyproject.toml changes), and then backends and plugins could adopt it via custom config, and then accept the PEP if adoption works well. I know I’ll at least have a tool-specific way to enable plugins for a while, since a user might need to add a plugin but work with a tool that doesn’t support the updated specification (just like any other update, including the TOML 1.1 update that includes trailing commas and newlines in inline tables! Very excited for that one.) So far the tools I’ve checked support the change to an inline table, though. I’ll be sure to have a discussion of the possible implications when we work on that part! FYI, the inline a.b syntax isn’t supported by older pytoml based versions of Pip - there is always some backward incompatibilities to worry about, but they disappear over time.

Having a single interface would be much nicer, but this is a good starting point if it’s better to go in steps.

Also, I don’t expect this to be added to beginner tutorials, or to cause packages that have static metadata to switch. I expect it to unify the existing usage of dynamic metadata and simplify converting custom setup.py’s that do README processing, optional-dependency merging, and version computation - which covers almost every non-binary build custom setup.py I’ve had to maintain. I think it will also help with unifying backends, since this currently is one reason to select one backend over another - with this proposal, you could select your favorite dynamic metadata plugin and your favorite backend independently.

This discussion has been immensely helpful, thank you to all that have participated! Quick question: If I make a major revision and split it, should we continue here, or should I start a new topic (possibly two for each side of the split)? Now might be a good time to bike shed on the names and API details for the next draft, as well.

[pf_moore (Paul Moore)]

@henryiii

Especially if we break it up a bit further, maybe we could even suggest or require a backend when computing a non-Dynamic dynamic metadata field (okay, not sure I like the two concepts being slightly different but having the same name!) to pull directly from the SDist instead of running the plugin when doing a build from SDist?

That was always the case with metadata 2.2. When building from a sdist (i.e., a PKG-INFO file is present) backends are required to ensure that the wheel contains the same non-dynamic data as in the sdist. So there’s no need to even read pyproject.toml for those values. (In the case of a sdist where the values in pyproject.toml, that sdist should probably be considered broken and the backend warn about the discrepancy, but that’s a backend QOL question - but it’s definitely not allowed to prefer pyproject.toml over PKG-INFO in that case).

A good example of this is getting version information from VCS or from a project file. The pyproject.toml will have the version marked as dynamic, and will have tool-specific config telling the backend (or a backend support tool like setuptools-scm) where to get the version metadata from. But the backend will write the version to the sdist as not Dynamic (version isn’t allowed to be dynamic in a sdist) and won’t invoke the dynamic detection code when building from a sdist. Doing anything else risks the possibility that building a wheel from foo-1.0.tar.gz could produce foo-1.1-py3-none-any.whl, which would break installers like pip (pip breaks already if it gets something like this, the spec just enforces that at the metadata level, rather than leaving it as installer-specific behaviour).

@henryiii

packagers that build from SDist like conda-forge could just skip adding the plugin to their build dependency lists in that case.

That’s precisely the use case metadata 2.2 was designed to support. Specifically, I had in mind the case of pip reading dependency and version metadata without doing a build, but it’s no surprise that other tools benefit as well.

@henryiii

And we should really be avoiding Dynamic whenever possible.

In the Metadata 2.2 sense, then absolutely, yes. In pyproject.toml, I’d personally prefer it if dynamic fields were the (rare) exception rather than the rule^[1]. But people seem awfully attached to their dynamic build-time logic, so I’m not optimistic. On the other hand, I don’t really care about source trees, as long as built artifacts (sdists and wheels) are statically introspectable.

@henryiii

Though, really, we don’t have to have a PEP for the API - this doesn’t change the pyproject.toml at all at this point, it’s only a standardization agreement between plugins and backends.

Standardisation between plugins and backends are just as much interoperability standards as between frontends and backends, so IMO it’s perfectly fine to write an interoperability PEP for that interface. The reason you’d make it a PEP is the same as any other interop - if you want to be able to make plugins that work with “any backend” and backends that can call “any plugin” then a standard avoids the “NxM interfaces” problem.

However, the backend-plugin programmatic interface is an entirely different specification than the pyproject.toml UI for defining what plugins a backend should call, and what parameters should be passed to the plugin. And that latter UI is very much about “how backends want users to provide settings for the build process”, which is why I’m much less sure it should be the subject of a PEP. It’s not like PEP 621 where the configuration is mapped to already-standardised metadata in a way that’s defined by the PEP, so there’s nothing apart from user convenience that warrants it being under [project] rather than under [tool.<something>]. The only problem is that it’s not clear whether <something> should be the backend, or the plugin, or a bit in one and a bit in the other. And that’s a pure UI question, not a standardisation one.

@henryiii

Quick question: If I make a major revision and split it, should we continue here, or should I start a new topic (possibly two for each side of the split)?

I think two new topics, one for the proposal to allow static “base” values for a pyproject.toml field marked as dynamic and one for the plugin API. I’d personally prefer that the proposal to put plugin config into the [project] section gets dropped, but if you want to pursue it, that should be a third topic (and PEP). My reasoning is that the plugin API is of interest solely to plugin writers and backend maintainers, whereas the config proposal is of little interest to plugin developers, but of significant interest to package developers as well as to backend maintainers. So three topics/PEPs, reflecting three very different user audiences.

---

1. Single-sourcing the version is the only case where I find the idea of dynamic pyproject.toml and static metadata persuasive, TBH. ↩︎

[dholth (Daniel Holth)]

I’m skeptical of declaring dynamic metadata. Are there a bunch of tools that work when pyproject.toml has certain non-dynamic fields but fails when it doesn’t? If you want static metadata it’s in the wheel.

[pradyunsg (Pradyun Gedam)]

Since the conclusion seems to be to split the proposal and that it’s not going to become a PEP in the current form, I’ve used my still-shiny-new powers on the forum to retitle this topic to say “Proposal” instead of PEP, to make it clearer that we’ve not written something that’s in peps.python.org.

Holler if that doesn’t seem right or if I’m overreaching in any way.

[brettcannon]

@pf_moore

@henryiii

packagers that build from SDist like conda-forge could just skip adding the plugin to their build dependency lists in that case.

That’s precisely the use case metadata 2.2 was designed to support. Specifically, I had in mind the case of pip reading dependency and version metadata without doing a build, but it’s no surprise that other tools benefit as well.

I feel like this plug-in discussion is heading towards having metadata for preparing the sdist instead of the wheel (a preparation system for lack of a better term). If you look at the suggested plug-ins, they would all be expected to create data for the sdist (version, README, classifiers, etc.).

[takluyver (Thomas Kluyver)]

For Flit, I have to say: I don’t like the proposal. I haven’t written this down, but no plugins is basically a design goal for Flit. Plugins inevitably add complexity, bugs, and confusion. What’s the oldest version of $backend compatible with $plugin version X? When a command hangs, is that due to the backend, one of the plugins (and which) or an interaction between them? Etc.

If this becomes a standard, I’ll either have to accept it in Flit, or spend the rest of forever grumpily shutting down well meaning issues and PRs asking to implement the standard metadata plugin interface. Unless, I suppose, I’m so grumpy that everyone abandons Flit entirely.

However, I assume that most of the other backends will be happy with it, so it will probably go ahead in some form.

I assume there’s no specific mechanism for finding and installing plugins - they should just be listed in the build-system.requires alongside the backend? I don’t think there’s any need for anything extra, I just didn’t see any info on this, and was a bit distracted by get_requires_for_dynamic_metadata (which allows the plugins to request further dependencies, IIUC).

[henryiii (Henry Schreiner)]

FYI, there are already forks of flit to add plugin-like features - see flit-scm · PyPI - forced in by users desperate to add at least SVN based versioning to Flit. And as you know, there are also consistent requests for at least this feature (and possibly others? But I know about this one) that Flit constantly has to refuse. A plugin system solves this - Flit never has to support SVN versioning or processing READMEs, someone can just use a plugin to do that if they need it. And it’s completely opt-in, and no one ever has to use it if they don’t want to.

Also, I’ll be writing the reference implementation using Flit, so there will be at least one PR.

@takluyver

What’s the oldest version of $backend compatible with $plugin version X?

The point of the the standard would be to avoid this - Either a backend supports this PEP, or it doesn’t, and if it does, it therefore supports every plugin written with this interface. There’s always the possibility of later updates via PEPs, but in general, this solves the X backends vs. Y plugin problem. If Flit had its own plugin ecosystem, sure, but this is proposing a standard interface, and supporting it isn’t that hard. Someone unfamiliar with scikit-build-core’s internals added it with relatively little code, and I’m going to work on making it as simple to implement as possible - I’m going to experiment with a few ideas before finalizing the interface.

@takluyver

I assume there’s no specific mechanism for finding and installing plugins - they should just be listed in the build-system.requires alongside the backend?

Yes, exactly, and also exactly correct about get_requires_for_dynamic_metadata. You just add them to build-system.requires, and the extra hook is a very rarely needed way for the plugin to request dependencies specifically only when it’s being called as a plugin. The main use for that is it’s useful for “built-in” plugins that a backend is providing, and a secondary use could be for wrapper plugins; a single package providing multiple plugins that each have different requirements. I’m not sure either of these is important enough to add it to the standard (and it could be a backend-specific optional addition), but no one has really complained about that hook yet (other than it’s name looking like a PEP 517 hook, which from an plugin implementers perspective, it is supposed to be similar).

This is also why I believe supporting the interface in a standardized way in the pyproject.toml is a critical part - a plugin needs to be able to instruct users on how to enable and use it, and if every backend has it’s own syntax for adding plugins, in it’s own tool sections, there will be no way to write generic docs for the plugins. Plugins must be opt in and explicit, not “automatic if present” or anything like that (setuptools and SCM file finding comes to mind…).

I’m going to try to bring this up at the Packaging Summit at PyCon, and hopefully work on the first PEP (which has nothing to do with plugins, it would be the additive dynamic metadata one).

[pf_moore (Paul Moore)]

Note that not supporting plugins has to remain a valid option for backends. Not every backend author might want to handle the extra complexity.

[pradyunsg (Pradyun Gedam)]

I’ll pile on that, from what I’ve seen so far, I don’t have much interest in a plugin mechanism like this for the build backends that I maintain (sphinx-theme-builder and, allegedly, Flit).

Not opting into this should be possible and OK, and is required for backwards compatibility anyway.

[henryiii (Henry Schreiner)]

I’m not aware of a mechanism to force compliance to a packaging PEP (certainly not one like this). Not every backend supports PEP 621, for example, and that’s more general. Though I’d say if users are actively asking for a backend to support this plugin system, I’d consider that a success and I’d strongly recommend that backend to consider it. A backend could simply state something like “no dynamic metadata allowed” as part of its design (by the way, Flit does allow some dynamic metadata), and that’s fine. I’d recommend against that, personally - the plugins make it opt-in by the user, and a user knows what they need. Also supporting plugins allows you to offload the complexity to the plugin authors - most backends today (including Flit!) have some dynamic metadata support already, and I bet there will be more code supporting that that than required to support this plugin system. But yes, it’s not “required”, even for Flit, though as I said I expect I’ll have a reference implementation using Flit and would like any final decisions to be made after looking how much is required to support it. Like PEP 621, it’s up to the backends to decide; I think many of them will want to. We can specify that in the pep, but I think it’s implied as well.

[konstin (konsti)]

I don’t really see “just pick the standards you like” as an option, at least for maturin. This is not a question of “enforcement” or such^[1] but one impact and inertia of standards. If this proposal gets accepted, users will come to rely on its mechanisms and we will have to implement it in maturin, with all the implications that has for configuration/cli, error handling, documentation, etc.

As a more tangible example in pyproject.toml, take the decision on PEP 508 in PEP 621: I could decide that maturin primarily now uses poetry’s toml tables (maybe even explicitly [tool.poetry.dependencies]) instead of PEP 508, but then users would be confused because all the documentation everywhere else tells you about PEP 508 and IDE support breaks as do all the other tools that rely on the codified convention. Obviously, we don’t want that, so maturin follows the PEP.

---

1. I think that in general we don’t do ourselves a favor by framing PEPs with legal terms - standards are interoperability between software, documentation and a mechanism for change, and i’d like to frame them as grease that lets different parts interact smoothly rather than weapons we wield against each other by legalistic battle ↩︎

[encukou (Petr Viktorin)]

It’s not “just pick the standards you like”. The question is whether a
plugin system in general is a good fit for some backends…

You should pick the behaviors you like and use cases you want to
support. Or let users pick them for you. But for something you want to
support, if there’s a standard for it then the right thing to do is to
follow that standard, rather than use your own configuration/cli, error
handling, documentation, etc.

[pf_moore (Paul Moore)]

The important thing for me is that the standard isn’t framed in terms of “backends must…” or even “backends should…”. A standard that says “backends which want to support plugins for metadata discovery should do so like this” is better because it leaves the choice of whether to support plugins to the backend.

I also think we should give backends control over what plugins they support. A backend may be happy to allow plugin-based determination of the version, but not want to allow dynamic dependencies. And that should be the backend’s choice.

[ofek (Ofek Lev)]

@konstin

we will have to implement it in maturin

Friendly reminder that if you implement this then you will no longer need to be a backend Add option to build just the compiled extension modules · Issue #1419 · PyO3/maturin · GitHub

[konstin (konsti)]

That sounds good, but there too many unanswered questions currently that i see this happening soon: How would maturin be told which crate to build with which options (the current [tool.maturin] section), how would it say what the wheel tags are, how is maturin installed and how does it expose hooks? How do we do error handling, can we chain errors to python (having stack traces is a non-starter for me)? Is there isolation between backend and maturin and if so, how much overhead is it? How does it tells what to include in source distributions, how do editable installations work and how do we cross-compile, including building universal2 wheels? (i think it’s best not do derail this thread, i’ve written more details in PyO3/maturin#1419 but you could also start a new topic about that specifically)

The other aspect of this is that maturin is a stable backend and we’ll soon even publish maturin 1.0, so in the future the only major changes will be adding new features that people depend on from a backend, e.g. new platforms or this provider syntax.