Proposal for dynamic metadata plugins

Packaging
21

I am trying to read this proposal, but I am out of my depth (I never wrote a build back-end or build front-end). I have one question nonetheless, and I hope that it is related enough to the topic because I do not want to derail the discussion. So if it is out of scope, feel free to tell me to ask this in a new discussion thread.

Will this proposal make it so that (or help with) a project can have fields that are dynamic in its source code repository but static in the source distribution? To rephrase and clarify, I have in mind that the build back-end rewrites the pyproject.toml where version is dynamic in the git repository so that version is static in the sdist. Is it something that is already possible today without this proposal? If yes, is there any build back-end that does this already? Or is it something that is plainly against the [project] specification, pyproject.toml files can not be changed between source code repository and source distribution.

22

In summary, it seems the community consensus (and certainly my own opinion) is to strongly discourage this pattern this for multiple sets of reasons, as I and many other folks here describe on

As that’s getting rather OT for the thread, if you have further feedback/points to consider on that, I suggest replying other there (or creating a new thread).

What’s seemed to work well (IMO) for PEPs in the recent past has been adding the pre-draft-PEP in a branch on your fork of the PEPs repo, with a PR of that branch against main for textual comments. I suggest keeping substantive/conceptual discussion here to be easy to follow and consider later, whereas comments/suggestions on the PR work well for textual clarifications and the like.

As @henryiii mentioned and as implied in my post you quoted originally, the potential advantage here is if a metadata key is going to be specified as dynamic anyway, defining strict, clear and explicit semantics for also including it in [project] still allows many tools to extract considerable value from it if they can rely on what can and can’t change, or at least provide a constrained best-guess. Its certainly not ideal, but it’s better than having the key be completely opaque to tools and require building the project/running arbitrary code for all use cases instead of merely some of them.

NB, I don’t necessarily disagree with your conclusions, but I can see where @pf_moore 's confusion comes from. IMO, it would make the most sense (and be most conservative) to show a comparison with the status quo that we have today (i.e. no new top-level pyproject.toml key) as well as pointing out where that falls short when it comes to the motivation of this proposal.

IMO, but if the pyproject.toml already exists checked-in to the project source tree, a backend shouldn’t rewrite it (at least outside the [tool] table) in the sdist, or this makes the [project] table (and [build-system] table, etc) non-static and thus not reliable, defeating the design goal. (If it isn’t present and only dynamically generated by the backend and added to the sdist, then this doesn’t apply).

Yes, just in the PKG-INFO in the sdist rather than the pyproject.toml (and it requires the keys are marked dynamic in the original pyproject.toml, of course). A backend can of course dynamically generate/process pyproject metadata keys marked as dynamic (or any key, if there’s no [project] table at all) and insert them into the PKG-INFO in the sdist, and then not list them in the Dynamic core metadata field for sdists added in PEP 643 for Core Metadata 2.2, meaning they are static and thus reliable for all wheels generated from the sdist.

Unfortunately, no, because AFAIK like several other packaging PEPs this is ultimately blocked on PyPI adding support for Core Metadata >2.1, which has unfortunately been the case for years now.

23

Hi @sinoroc , I think rewriting the [project] table in pyproject.toml is not supported by PEP 621. The following parts of the PEP are relevant:

  • A build back-end MUST honour statically-specified metadata (which means the metadata did not list the field in dynamic).

  • Build back-ends MUST raise an error if the metadata specifies a field statically as well as being listed in dynamic.

  • If the metadata does not list a field in dynamic, then a build back-end CANNOT fill in the requisite metadata on behalf of the user (i.e. dynamic is the only way to allow a tool to fill in metadata and the user must opt into the filling in).

This PEP (in my interpretation) is not exactly changing things in terms of “rewriting pyproject”.

It does however (so far) relaxes the requirement about the same field being specified both statically and dynamic (the justification/proposed interpretation is that the static part stablishes a baseline that can be dynamically augmented during tthe build).

1 Like
24

When I get around to it, pymsbuild will do it this way. It already generates a brand new pyproject.toml for going into an sdist, and when there’s a consumer for pyproject.toml in sdists then I guess I’ll start writing the metadata into that generated file as well.

I don’t personally use any tools that would benefit from having their configuration in the pyproject.toml in an sdist, so preserving the original contents hasn’t come up for me. I guess eventually that might matter, particularly if plugins become worth supporting.

25

Please refrain from calling people’s work as “sloppy”; I didn’t purposefully cut any corners with that PEP that would lead to my work being “sloppy”. It would have been fine to have said, “ambiguity was found after PEP 621 was accepted due to how the terms were used, so please be cautious with your terminology”, i.e. it was an accident instead of due to lack of effort.

I would say so. If we are saying that you can’t mutate metadata in pyproject.toml when going from sdist to wheel, then I would say if you’re mutating what that metadata is pointing is effectively the same thing (i.e. it’s as if you embedded the data from the external source in pyproject.toml, but we gave you a shortcut to not have to).

2 Likes
26

The example confused me at first as well, since I think the discussion was entirely about inline settings, not about making dynamic a table - I haven’t seen any push back on that yet, just (reasonable) warnings about possible implications.

Example one: setuptools

If you remove the backend specific behavior when provider= is omitted, and remove inline configuration, then setuptools would have had to write:

[project.dynamic]
version = {provider = "setuptools.dynamic.verison"}
dependencies = {provider = "setuptools.dynamic.dependencies"}
optional-dependencies = {provider = "setuptools.dynamic.optional_dependencies"}

[tool.setuptools.dynamic]
version = {attr="mymod.__version__"}
dependencies = {file="requeriments.in"}
optional-dependencies.dev = {file="dev-requeriments.in"}
optional-dependencies.test = {file="test-requeriments.in"}

Which is exactly why backend specific behavior and inline configuration were added, actually. This requires keeping multiple tables in sync, and everything is basically specified twice.

Here’s what setuptools wants to write:

[project.dynamic]
version = {attr="mymod.__version__"}
dependencies = {file="requeriments.in"}
optional-dependencies.dev = {file="dev-requeriments.in"}
optional-dependencies.test = {file="test-requeriments.in"}

This is a nicer user experience than the new version or the current version, which requires both tool.setuptools and project.dynamic to be kept in sync.

Both of these were not in the original version of the proposal, and either could be removed if it was really a problem, though I think it’s a great benefit for existing backends wanting to adopt this, as many of them have existing dynamic support and have been actively trying to do exactly this sort of thing - one backend even supported invalid things like project.version = {file = "..."} for a while during the initial PEP 621 adoption! The primary benefit is it places all the configuration for a plugin inline right were the plugin is used, rather than having to split it up (and force the plugin to parse the toml file itself).

SCM version tools could certainly be configured inline, and “(hatch-)fancy-pypi-readme” could too; they currently are only configured via tool sections because they had to be. If these packages get new, plugin-focused releases, I could see them being configured entirely via the project.dynamic.X. The main reason you’d want any other config would be if it needed to be shared.

Example two: regex

Anther example is a regex based version finder:

[project.dynamic.version]
provider="myregexplugin"
file="src/mypackage/__init__.py"
regex="__version__ = '(.*)'"

[project.dynamic.description]
provider="myregexplugin"
file="src/mypackage/__init__.py"
regex="INFO = '(.*)'"

This passes a file to search for and a regex to search.

2 Likes
27
28

Thanks. I’ve approved the change, I’ll leave it for a day or so before merging in case anyone wants to argue that this is a substantive change rather than a clarification (although if you do, be aware that all that’s likely to happen is that we’ll need to write a PEP which I’m predisposed to accept, so you’d better have a really good argument for not making this change :slightly_smiling_face:)

Edit: Someone else merged it, because I forgot to add a similar note on the PR. That’s my mistake, but honestly I don’t think it’s a problem. If anyone wants to argue that this needs a formal discussion, we can just as easily discuss reverting the change as merging it.

2 Likes
29

Thanks for the examples. They help clarify the intention here. I’ll repeat that as a tool maintainer I’m largely indifferent to this proposal. But as a user I dislike this approach.

The problem is that currently, everything under [project] is backend independent. I can safely set up the whole [project] section knowing that it doesn’t matter what backend I choose, that section will do exactly what I want. Then, I can configure my backend under [tool.backend]. I know that section is specific to my build backend and will need to be changed if I switch backend, but that’s a well-defined block of configuration, that I can easily review and change as part of the migration process.

Under this proposal, though, I now need to review [project.dynamic] as well as [tool.backend], and there’s no explicit indication of which parts of the configuration relate to the backend, and which relate to a plugin (which may work with multiple backends, and so not need changing). The migration process is thus a lot harder.

I accept that switching backends is typically a very unusual situation, and we shouldn’t optimise for it over user convenience. But I think that the mental model of “standard” settings vs “backend specific” settings is a useful and natural distinction to make, even if you don’t intend to switch backends - you know that “standard” settings are documented in the standards, and “backend specific” settings are covered in the backend docs, for example. With project.dynamic.description.regex, for example, where should I look for documentation? I have no idea without referring to project.dynamic.description.provider, or if that’s not present, to build-system.backend. That’s much less user friendly than knowing that anything in tool.xxx is explained in xxx’s documentation.

IMO, this proposal is more for the convenience of experts than for users who view packaging as a necessary inconvenience. It assumes that people know all of the configuration details, rather than that they found an example pyproject.toml on Stack Overflow or somewhere, and are trying to modify it for their use case. It’s this latter sort of user who will be immensely frustrated by the fact that they don’t know where to find the docs on the various weird keys and settings.

More convenient if you already know what all the settings mean. Much less convenient if you need to find the relevant documentation among multiple tools (standards, backends, plugins).

I fear that we don’t yet have enough information to know what’s the best answer here, which is why I think we should omit this from the proposal for now.

That’s fair (and useful) feedback, but I think this proposal just exchanges those problems with different, equally problematic, issues.

OK, thanks. That clarifies things a little to me. In that case, I consider making the hook name similar to the PEP 517 hook names to be a problem, rather than a benefit - it makes it too easy to confuse who is responsible for providing and calling that hook. This part of the proposal maybe needs revision to clarify that the protocol is between backends and plugins, and not between build frontends and backends. And at that point, maybe we shouldn’t be looking at just providing metadata, but rather at a full “backend plugin” specification? I appreciate that’s a much bigger task, but if we just standardise one part of it now, it seems almost inevitable that we’ll have to change that once we do tackle the bigger “build backend plugins” issue.

At a minimum, can the proposal acknowledge that metadata is only one interaction that may need to happen between backends and plugins, and address the question of versioning the API now? We’ve had a few PEPs that define an API with no versioning, and it’s never worked out well…

30

(I originally had this in the same post as the quote, but I wanted to pull it out as it’s an important standalone point and I didn’t want it to get lost in my previous lengthy comment).

Actually, PEP 621 is one such API. How will a project specify whether it’s using PEP 621 or this “extended” specification? There may be tools in existence that don’t know how to handle anything other than a list of fields in dynamic - how do we avoid breaking those tools if this extension gets accepted and people start using it? We can’t simply expect all PEP 621 consumers to be updated in a timely manner.

31

The core of this proposal is allowing users to select any metadata plugin they wish and also use any backend they wish. Switching between two backends is easier, because your favorite plugins can still be used. This requires a protocol we can agree on (currently dynamic_metadata, though we could call it compute_dynamic_metadata if the name is confusing - I’m not sure how “different” we could make that from backend hook’s names), and some standard way to select the plugins. The ideal place is in the existing project.dynamic, as it avoids duplicating code either for users writing pyproject.toml’s (if the it’s the user’s job to list things in two places), or for readers processing pyproject.toml’s (if it’s the tools job to read and combine from two places).

I would consider dynamic metadata a bit more of an advanced feature; users who either are comfortable enough with the basics that they want to start streamlining their workflow and reducing the potential for bugs like forgetting to update the version, or users who need it due to the way their project is set up (for example, if the version must be stored in C++, or if their README doesn’t render well on PyPI, etc). Not that it should be hard to use, but erring just a bit on the side of flexibility for advanced users is probably okay. We would document provider=, just like build-system.build-backend= is documented today.

Passing arguments still keeps the project section completely free of backend-specifics, and you can still switch backends. The part that breaks that is allowing backend specific behavior, which reduces the verbosity for the user (not having to look up and type provider= for something built into the backend), but keeping the verbosity might be fine - the main use case at the moment would be setuptools, and that could include a provider. Since the provider knows what field(s) it’s computing, you could probably make this pretty simple for the user’s side, ="setuptools.dynamic" every time, for example.

I’m planning on working on a “extensionlib” proposal in the future, which would focus on plugins providing extensions. This is a much bigger issue, though, and a bit up in the air as to how well it would work. There’s a lot of things the build backend still needs to know how to do, like computing the correct wheel tag (even when cross-compiling), that likely will make that tricky.

I’ve also wanted the ability to extend get_requires_for_build_*, and even tried to work it into this proposal originally. But that’s probably not worth a plugin proposal, and would necessarily be part of the extensionlib idea.

I’m not sure what other sorts of plugins would be useful, or if there would be a way to specify something more general - a plugin does a specific task and is fit into a specific part of the build process (like preparing metadata in this case) - I don’t see that being something you can generalize to generic plugins.

This is already an issue with every PEP that touches pyproject.toml after PEP 621, like PEP 639. This could either be turned into a “version”, like the metadata files (in which case do we expect it to get stuck at some version like metadata 2.1?), or tools could just mention what PEPs they support - that’s how most build backends work today. Given the push to get away from mentioning packaging PEP numbers quite so often, adding a version to the Packaging specification page and bumping it when PEPs change it seems reasonable.

PS: If we decide against turning tool.dynamic into a table, the “new array section” alternative proposal could be revisited.

1 Like
32

OK, so we now have interchangeable frontends, backends, and plugins? Where does that leave existing tools like setuptools_scm? How will they work? Or can plugins say that they only work with some backends? Which is another axis of complexity. Honestly, I don’t like where this is going - it seems like there’s a huge rabbit hole that we could end up going down. But if we don’t, then there’s a bunch of unanswered questions.

As a user, I’m going with “I don’t like this and I hope I never need to use it and can stick to using [tool.backend] for this purpose”. And as (I assume for now) PEP delegate, I’m going with “you’d better get all of this clarified or I’m going to reject the PEP”…

Yes. As I say, we missed this problem in PEP 621, and I don’t want to keep hitting it. Basically, PEP 639 and/or this PEP (whichever gets completed first) need to solve the problem now, because we didn’t do it then.

33

I’m confused, isn’t allowing plugins that work with any build backend exactly the point of writing a PEP about “dynamic metadata plugins”? If a backed supports this proposal, it will work with any plugin written for it. So setuptools-scm (ideally with a different name!) adds setuptools_scm.dynamic_metadata(...), then you can use:

[build-system]
requires = ["any-backend", "setuptools_scm"]
build-backend = "any-backend"

[project.dynamic]
provider = "setuptools_scm"

What backend you selected doesn’t matter - any backend supporting this proposal will call the proposed hook and get the version from setuptools_scm. Assuming backends do support it, you’d be able to select a frontend, a backend, and a metadata plugin, yes. (And the proposal doesn’t actually touch frontends at all).

The current situation is that there are a hodge podge of different adaptors, tools, and built-in features for each backend for dynamic metadata - setuptools_scm has been wrapped into hatch-vcs, but with different configuration, Poetry and PDM both have similar functionality, but being a different implementation so they use different environment variables (if they support that basic feature at all) for version overrides, hatch-fancy-pypi-readme only works with hatch, etc. 9 of the 11 backends I list in the proposal support some form of dynamic metadata, and nothing at all is shared (except for the rewrapping of setuptools_scm into hatch-vcs, if you count that as shared). New backends, like scikit-build-core, meson-python, and maturin, are forced to come up with their own ways to compute dynamic metadata.

What about we simplify the proposal back down to the original proposal, without the features setuptools requested (plugin inline config & a clause allowing backend-specific behavior if provider= is missing), and restart from there?

34

I guess so. So this is a PEP for (a subset of) a backend plugin mechanism? If so, then let’s start with a basic question - how do we tell the backend to use a given plugin? Putting the information in dynamic works for metadata plugins, but what about other types of plugin? Will we need different mechanisms for other types of plugin? I think what I’m struggling with is the lack of a plugin framework that this ties into.

But the backend needs to call the plugin. So if I switch backends to flit, who tells me that flit doesn’t support setuptools_scm (or doesn’t support metadata plugins at all)? Or is it that setuptools_scm needs support for flit? Either way, my config no longer does what I think it does and it’s not clear who is going to report that to me.

I think the answer here is back to versioning - if an older flit sees this, it fails because it doesn’t recognise this form of dynamic, and an updated flit needs to see the plugin, try to call it, and report the error if it fails. Is that right?

That might be helpful to those of us who weren’t party to the preliminary discussions. Or alternatively you could split the proposal more clearly into a “core plugin mechanism” section, and a “extended features” section. That would also allow for backends to only implement the core.

It would also be worth the PEP discussing how this affects non-backend consumers of pyproject.toml. Tools like dependabot don’t have a way to support this plugin mechanism, so how should they proceed? Treat all of the new config as just meaning “this field is dynamic” and ignore the details, I guess. But we should be explicit.

Furthermore, the spec might need some words to discuss how this interacts with the core metadata Dynamic field. If a plugin generates the metadata for a sdist, presumably the plugin is then responsible for respecting the contract that a value that is not declared dynamic in the sdist will have the same value in the wheel? How will that be communicated? Will the sdist metadata be available to the plugin? Is the calling backend required to check? My concern is that the easy answer is to declare everything a plugin manages as dynamic, but (a) that defeats the purpose of encouraging as much static metadata as possible, and (b) you’re not allowed to declare version as dynamic in metadata, so we still have to address the issue for that field at least.

35

Is there a [project] table in the original pyproject.toml? If so, are all the keys you’re modifying (at least in a way that would change the output METADATA) listed in dynamic?

Sorry, that was a very poor and inconsiderate choice of words on my part, particularly due to the uncalled-for negative connotations of “sloppy”, and I apologize. I rewrote the paragraph in question to focus on my suggestion for the future PEP instead, with the mention of PEP 621 being reframed in the more appropriate context of lessons learned for the future as opposed to critiquing the past. To clear the record, while its various ambiguities did cause me many hours of frustration and rewriting when trying to interpret it precisely, as I’ve seen firsthand it can be nearly impossible for me when I’m in the shoes of the writer to notice many of those same kinds of ambiguities that someone else coming across the document for the first time would, even if the writer tries hard to avoid that.

Sidenote, but is there a reason for the inline table, instead of just doing the simpler and cleaner (and fixing a few typos):

[project.dynamic]
version.provider = "setuptools.dynamic.version"
# ...
[tool.setuptools.dynamic]
optional-dependencies.test.file = "test-requirements.in"

[project.dynamic]
version.attr = "mymod.__version__"
# ...
optional-dependencies.test.file = "test-requirements.in"

:+1: , that’s one of the things I’m (still, sorry) working on adding in the next major revision of PEP 639 (as well as overhauling it to clarify the terminology, and addressing your and others’ further helpful feedback). Assuming PEP 639 isn’t accepted before @henryiii 's PEP is merged, we can coordinate to ensure our respective language is synced up on that point so there isn’t a race condition.

ISTM that the concerns here with these features (especially those you’ve articulated) is not the complexity of the backend implementation, but rather of the end-user UX, especially when switching between backends or adopting a new one. If some backends supported them but others didn’t, and some pyproject.toml files relied upon them but some didn’t, it seems like this would make things a lot more complicated for users in this department rather than less, no?

Great point; this is something I’m interested in seeing clarified as well.

2 Likes
36

Hi Paul, thank you very much for the feedback regarding my comments.
I am trying to understand your view when you say it would be less convenient, to me it is not immediately clear what do you mean by that.
I will try to explain the steps in my thinking below:

  • If you have no need for dynamic metadata, the proposal will not change anything for you. So in this sense, there is no change in convenience.
  • If you do have need for dynamic metadata, you need to know if the backend supports it, or if you can use a plugin. Then you need to find the relevant documentation among multiple tools about their settings. This is already what happens today, and the proposal don’t change the need for looking the settings up.
    When writing the pyproject.toml, the proposal will however (in my view), make it more convinent to write and maintain (all the information is concentrated in a single place, so there is no risk of forgotting to change a different section further apart in the document; moreover the interdependencies between sections describing how the dynamic behavior work became more evident because they are centralized in a single location).

As an anecdote on the “perceived convenience” of the proposed UI, I can mention PEP 621 support for dynamic metadata · Issue #1537 · PyO3/maturin · GitHub. In this feature request we can see a maturin user with interest on the implementation of a UI for specifying dynamic metadata similar to the UI setuptools has today. When the user gets to know the proposal, its feedback mentions the user finds it simpler.

Do you have a suggestion of how we could collect feedback about this? What would be the type of information that we would need to choose the best answer more assertively?

My concern is that doing multiple consecutive changes spaced in time can be detrimental to “user satisfaction”[1] and implementation effort.

Could you please share with us what would be these different issues? Maybe if we think about those we could devise how we can propose something that solves both problems…

I understand the reasoning behind this comment, but I am afraid that is not what I verify in practice. Depending which backend you choose you can use a subset of fields in project.dynamic.
If you have projectl.dynamic set in you pyproject.toml, then changing the backend is not a safe operation. You will have to look the documentation up (regardless if the proposal is accepted or not).

Let’s consider the following (toy) example, that builds correctly if you use whey:

> docker run --rm -it python:3.10 /bin/bash
mkdir -p /tmp/myproj
cd /tmp/myproj
mkdir myproj/
touch myproj/__init__.py
cat <<EOF > pyproject.toml
[build-system]
requires = ["whey"]
build-backend = "whey"
[project]
name = "myproj"
version = "0.42"
description = "hello world"
dynamic = ["classifiers"]
EOF
python -m venv .venv
.venv/bin/python -m pip install -U build
.venv/bin/python -m build
# ...
# Successfully built myproj-0.42.tar.gz and myproj-0.42-py3-none-any.whl

If you change the backend to flit_core, we can see the build failing:

sed -i 's~requires = \["whey"\]~requires = ["flit_core"]~' pyproject.toml
sed -i 's~build-backend = "whey"~build-backend = "flit_core.buildapi"~' pyproject.toml
.venv/bin/python -m build
# ...
# flit_core.config.ConfigError: flit only supports dynamic metadata for 'version' & 'description'
# ERROR Backend subprocess exited when trying to invoke get_requires_for_build_sdist

I think the situation is even more complicated than that… In general it is not safe to just replace the backend without knowing what is supported. The users need to look information up about about the tools they use during build time.

PEP 621 does makes it easier to swap backends, but don’t make them fully interoperable, even if only the project table is used, with no project.dynamic. We can see a demonstration of that in the example below:

> docker run --rm -it python:3.10 /bin/bash
mkdir -p /tmp/myproj
cd /tmp/myproj
mkdir -p src/myproj/
touch src/myproj/__init__.py
cat <<EOF > pyproject.toml
[build-system]
requires = ["flit_core"]
build-backend = "flit_core.buildapi"
[project]
name = "myproj"
version = "0.42"
description = "hello world"
EOF
python -m venv .venv
.venv/bin/python -m pip install -U build
.venv/bin/python -m build
# ...
# Successfully built myproj-0.42.tar.gz and myproj-0.42-py2.py3-none-any.whl
sed -i 's~requires = \["flit_core"\]~requires = ["whey"]~' pyproject.toml
sed -i 's~build-backend = "flit_core.buildapi"~build-backend = "whey"~' pyproject.toml
.venv/bin/python -m build
# ...
# File Not Found: Package directory 'myproj' not found in '.'.
# ERROR Backend subprocess exited when trying to invoke build_sdist

Even now, pyproject.dynamic is just another “non-fully interoperable” part of the spec.

With regards to this specific aspect[2], I believe that the proposal will not change much what already happens and is already recommended by PEP 621: the backend should fail. We could clarify that better in the proposal.

In my interpretation, the proposal contributions are as follows:

  • It stablishes a generic hook mechanism for backends to offload dynamic metadata resolution to plugins.
    This mechanism can be tapped in by any backend. All plugins that follow this specification will be compatible with all backends that follow this specification. There is no need to write a compatibility layer.
  • It provides a more convenient UI that unifies the specification of dynamic metadata (regardless if it is being provided by a plugin or the backend itself).
    The convenience here happens in 3 levels from the package developer point of view:
    • It is convenient to write
      • effectively, you need to write less
    • It is convenient to maintain
      • if you need to change something, e.g. making previously dynamic field static, you have to modify the file in less places, and the risk of forgetting to modify other related tables further away in the document is mitigated.
    • It is easier to explain to a user that don’t care about the intricacies of the Python packaging and just want to get things done.
      • For example, we can think about a common question that is hard to explain for a person that is not interested in knowing the relationship betwee frontends/backends/other tools: “why do I have to write dynamic = ["X"] if I already have another place in pyproject.toml that specifically tells how to dynamically obtain X?”.
        If everything is centralized in a single place, writing down the configuration for a dynamic field fells more natural and less doubts arise (potentially)[3].

I believe that the following aspects would not differ significantly from what already happens (status quo):

  • If the backend does not support the proposed hook, the backend should raise an exception.
  • If the a specific dynamic field is given without provider and the backend does not know how to obtain, the backend should raise an exception.
  • The relationship between pyproject.toml dynamic and METADATA/PKG-INFO dynamic will not change.
    As far as I am aware, most of the tools resolve dynamic nowadays before generating the sdist, so when it is time to write PKG-INFO, the values are already determined, and it would not change how Dynamic is used in core-metadata.
    (I confess that I don’t have full visibility about this point, so please correct me if I am wrong).

The part that might be backward incompatible is the following:

  • The proposal includes chaging the data structure associated with project.dynamic from a list to a dict (in Python terms).
    This is “partially” backward compatible in tools written Python (operations like for ... in ... or if ... in ... will work, but others like isinstance will not)[4]. That will depend on how tools process or validate project.dynamic.
    The justification for this change (in my opnion) is convenience and UI.

  1. People already complain that the pace in packaging PEPs is too big. ↩︎

  2. i.e.: “what happens if the backend does not support the proposed dynamic_metadata hook, or the backend does support the hook, but a specific field is given without provider and it does not know how to obtain that information?” ↩︎

  3. The doubts specific tool configuration details will always exist and will need to be looked up in the docs, being the proposal accepted or not. ↩︎

  4. This is language dependent and might be true for other languages (e.g. in Ruby x.include?(y) work for both arrays and hash tables, as well as for y in x) ↩︎

37

I personally find this suggestion less clear…
If I write optional-dependencies.test = {file = "test-requirements.in"} I know that I am specifying the test group in optional-dependencies, and it should be obtained by parsing the test-requirements.in file… The way you are suggesting takes longer in my brain to process :sweat_smile:.

38

This should not be optimized for imho. The better solution is to have pip and installers be able to install from

[project.optional-dependencies]
test = [....]

We have a proper section in pyproject.toml, so let’s use it. There’s discussion on the pip issue tracker about gaining the ability to install the various groups of dependencies listed in pyproject.toml.

+1 for this, at least in the alternatives section. I am having a hard time wrapping my head around this discussion, it got quite complex pretty quickly.

As another alternative to cover in the PEP, it would be useful to at least mention using a regular package for backends to depend on. If what we’re talking about here is primarily the ability for backends to share niceties like VCS version parsing, README prettification and the like, it seems like having frontends and PEP 517 like hooks involved is fairly complicated when compared to just having a single package that provides a decent set of bells and whistles. And then the very fancy or more specific stuff can be left in tool-specific sections of pyproject.toml.

My impression so far is that I quite like the core argument @henryiii is making about allowing specifying metadata statically and adding them to dynamic. However, the extra complexity and elevating plugins to this level of prominence I’m less sure about.

1 Like
39

One of the questions that I have is whether this needs to be a “standard” at the pyproject.toml level, or is this something that could be implemented with a shared implementation/package that all build-backends use while relying the existing mechanisms for dynamic behaviours?

I’m having trouble imagining that all backends would want to have the exact same interface here and the number of behaviours described thus far seem to suggest that there’s a need for a shared implementation for this anyway.

2 Likes
40

Hi Anderson. I haven’t had the time yet to fully digest your response, but I have been thinking some more about what my reservations are, and I think my conclusions link to this point. So I’ll post my current thinking here, but defer responding to your other points until I’ve had more time to think about them. I hope that’s OK.

A key part of my reservations here is that we’ve had a number of threads recently which have demostrated pretty clearly that we have no community consensus that PEPs are appropriate for defining user interface for tools.

PEP 621 is pretty close to being precisely that, and this proposal (particularly the part that specifies tool specific parameters) is even more so. I think that, in order to resepect the (lack of) community consensus, we need to look at PEP 621 and this proposal more in terms of being an interoperability standard.

In that context, PEP 621 is a way for projects to define, in a standardised form, project metadata that is present in the source tree and guaranteed to be in all artifacts built from that source tree. Static metadata defined in pyproject.toml can be read by any tool simply by reading the standard elements of that file.

The dynamic list isn’t specifying metadata, as such, it’s simply distinguishing between “explicitly not present” and “not specified”. Maybe this wasn’t the best way of doing this (there was an active debate over whether static or dynamic should be the default) but it’s how we ended up specifying this distinction.

Importantly, everything that PEP 621 specifies can be consumed by any tool, even tools not written in Python. The consumer may be told “there’s no static value available for this metadata item, it potentially varies depending on what built artifact you select”, but that’s still a valid answer for “what is the value of this field?”

To address your point about dynamic making pyproject.toml backend specific, that’s valid up to a point. The PEP 621 metadata (in effect) says that it’s the responsibility of the build system to fill in any required dynamic fields as part of the build. The user then has to specify a build backend that satisfies that requirement, and if they don’t do so, the build won’t work. But that’s no different from a mis-spelled build backend - for tools that don’t actually need to do a build, it’s not important. I concede that this is a fairly “lawyerish” way of explaining away this point, but importantly (to me, at least) it demonstrates that we can still preserve the tool-independence of the standard under this view.

Supporting this, we have a clear namespacing in pyproject.toml. The root namespace is explicitly reserved for tool-independent standard information, and individual tools are assigned their own namespace to use as they see fit. This acknowledged the fact that tools might want to use pyproject.toml for their own data, without requiring standards, or tools that relied solely on those standards, to have to handle unexpected or arbitrary additional data in standard-controlled areas.

The provider=xxx part of the new proposal extends this relatively naturally, following PEP 517 in allowing tools to determine “what software do I need to generate this value?” It remains possible for any consumer to use this information - even non-Python tools could potentially start up a Python interpreter and call the named plugin using the defined plugin API.

The “provider-specific settings” part of the proposal, however, violates the principle established back in PEP 518 of clear separation between standards-defined data and tool-specific data. And I don’t think that is something we should do lightly. Maybe the tool.xxx approach has turned out in practice to be somewhat clumsy and unsatisfying for real-world use. We can look at reserving other namespaces for tool use, or otherwise refining the UI aspects of pyproject.toml.

I have some sympathy for the view that defining behaviour in pyproject.toml can get over-complicated when faced with an array of backends, backend plugins, supporting tools, etc. But that seems to me to be a separate discussion, and one that leads right back into the contentious matter of standards defining UI. So I’d rather that it be kept separate, to avoid impacting the much simpler, core proposal that @henryiii suggests was the original intention here.

Agreed. I don’t think that putting all the various aspects into a single proposal is helping at all here :slightly_frowning_face:

Precisely - this all feels more like a series of UI and implementation issues, rather than a standardisation matter. The only standardisation point I can see is “having separate [tool.xxx] sections for different tools isn’t sufficient - can we redesign the tool-specific config provisions in the standard?” And we’d need a concrete proposal for that which had support from the majority of tools using pyproject.toml for configuration (which is more than just build backends). So it should be an independent PEP.

1 Like