Partly dynamic project metadata (eg. adding dependency constraints based on the build environment)

Packaging
21

There are many users for the static metadata, and the existing static metadata is not strictly wrong for sdists, but when it comes to binary artifacts, there is indeed a need to tighten dependencies depending to reflect binary dependencies.

Specially now that we are moving more towards shipping native dependencies in wheels, it becomes incredibly more important.

I’ll draft up a PEP to try to iron out the exact definition, but the idea is relatively simple — introduce a way for binary artifacts to be match only a subset of specific sdists dependencies.

I think maybe the Requires-Dist-Dynamic idea would be the best, as backends could specify exactly which dependencies are expected to result in different binary artifacts.

Or maybe we wait for the wheel 2.0 proposal and its custom tags feature? Though, I’d like to be able to fix this in the current wheel version, allowing for much faster adoption.

22

Yeah, but the current Dynamic definition allows simply ignoring Requires-Dist if Dynamic: Requires-Dist is specified.

We could update the definition to require existing entries to be kept, and new ones to be added. Though, that allows introducing new dependencies, which I don’t think is a good idea. I can see a future where that Dynamic behavior may be desired (eg. for python version trove classifiers), so I am hesitant in changing the Dynamic definition to work around it. So, I feel maybe the best solution is to introduce a new core metadata key that serves this purpose (eg. Dynamic, or Requires-Dist-Dynamic).

PEP 621 doesn’t necessarily prevent any sort of dynamic behavior in Requires-Dist, it simply requires a direct map between project.dependencies and Requires-Dist. We could simply allow adding an extra Requires-Dist-Dynamic, or extend the PEP 621 mapping to allow both Requires-Dist and Requires-Dist-Dynamic.

23

Yes, but by whom? A tool that’s trying to statically analyse the metadata probably still has to ignore the non-updated Requires-Dist anyway, and all tools are going to have to be updated for new metadata just as much as if the existing metadata changes. Any tool’s existing behaviour will be just as correct after the change as it is now, although slightly less efficient.

The only reason we’d even need a version number change is for tools that want to provide an error for having the wrong version number (or alternatively, for having incorrect values in Requires-Dist, which is equally impossible to determine). Nobody else even needs a version number change - this can be a very simple “we now allow Requires-Dist to be non-empty when marked as dynamic, and tools can assume that any items listed will also be present after building a wheel, though they may have additional constraints and entirely new items may be added”.

24

We seem to have changed the topic here. Are we now talking about the core metadata field Requires-Dist, rather than the pyproject.toml keyword dependencies? Because while the former is derived from the latter, there are subtle but important differences in semantics (particularly around static/dynamic distinctions).

As I’ve already said, for pip, changing the rules around pyproject.toml would make little or no difference, but changing how the core metadata works for dependencies would potentially be highly disruptive. For build backends, I guess the balance would be the other way.

Can I request that people are very precise when discussing this topic, to distinguish between the two cases?

2 Likes
25

Indeed, the motivating example given is tools that read pyproject.toml, so I don’t understand how the discussion morphed into core metadata or how their use case applies if only the core metadata spec is updated.

26

Good point, I was just following the terms Filipe used. (Though personally I don’t really care about what goes in the pyproject.toml at all, but I do care about PKG-INFO :man_shrugging: )

27

At least, from where I came from when starting the discussion, the topic has always been the pyproject.toml to core metadata relationship.

pip changing the rules around pyproject.toml would make little to no difference, correct. That’s because pip shouldn’t really care at all about the project table, but other tools do.
Changing the semantics of project.dynamic to allow a dynamic field to have a value, which is what was proposed if I understood correctly, would break a lot of stuff. For eg. any project using pyproject-metadata that isn’t a build backend would now get an error. [1]

Any changes that we make need to retain the static validation requirements for the project table mandated by PEP 621. When it comes to core metadata, though, we can make changes as it’s versioned.

I don’t think changing the core metadata would be that disruptive, backends would only use the new version when the keys are needed.

Anyway, I don’t love the fact that Requires-Dist could change drastically between the sdist and wheel, if it is specified as Dynamic, but doing that is a possible solution. I re-read the specs and relevant PEPs and don’t see anything preventing build backends from specifying Dynamic: Requires-Dist in the sdist, regardless of the value of project.dynamic, so I guess no changes are needed, right? Or did I miss something?

  1. Perhaps I misunderstood, and the proposal was to actually add Dynamic: Requires-Dist to the sdist, if so, I guess that works, as I mention below. ↩︎

28

If the dependencies can differ between different wheels built from the same sdist, then build backends are required to specify Requires-Dist: Dynamic. That causes a performance hit for installers, as they can’t trust the sdist metadata but have to call the build backend. But that’s unavoidable, so :person_shrugging:

But when you say “no changes are needed”, does that mean you’re withdrawing your proposal? Or am I still missing something?

One of us is missing something, certainly :slightly_frowning_face: What exactly is the proposal here?

  1. Nothing, any more, because you said “no changes are needed”?
  2. A change to pyproject.toml - [project.dependencies]?
  3. A change to the core metadata - Requires-Dist, or some new Requires-Dist-Dynamic field?
  4. Both (2) and (3)?

As I said, being clear about whether we’re talking about pyproject.toml or core metadata is important here.

29

I am a bit confused. Don’t you mean Dynamic: Requires-Dist?

When I opened the topic, I think I had a slightly inaccurate understanding of the semantics of the Dynamic core metadata field.

Before we continue, I want to make sure I have a correct understanding of things, so that we are on the same page.

As I interpret it, the current specification allows for the sdist to include a Dynamic: Requires-Dist PKG-INFO entry, regardless of 'depencencies' being included in the project.dynamic key of pyproject.toml, right?

Eg.

  • pyproject.toml
[project]
name = 'example'
version = '1.0.0'
dependencies = ['something >= 3.0']
dynamic = []
  • sdist PKG-INFO
Metadata-Version: 2.2
Name: example
Version: 1.0.0
Requires-Dist: something >= 3.0
Dynamic: Requires-Dist
  • Example wheel METADATA
Metadata-Version: 2.2
Name: example
Version: 1.0.0
Requires-Dist: something == 3.4.*

If this example is indeed valid, then no, I don’t think we need any changes.

Though, I don’t like that Requires-Dist can be dynamic in the sdist, even though no metadata is marked as dynamic in pyproject.toml.

30

dynamic: [] means “here is an empty list of all the dynamic things”, i.e. nothing. I’m not sure if tools will validate that but I imagine they should (it’s just redundant).

Requires-Dist is the metadata that corresponds to dependencies. For Requires-Dist to be listed in the metadata then dependencies should be listed as a member of dynamic.

How would the tool know that Requires-Dist was dynamic if it’s not listed in the pyproject?

edit: maybe I was just rehashing what’s already been discussed. I’ll just cop to also being confused about what’s being proposed. It seems like it’ll be tool-specific, and so tools can just keep it until their [tool.*] table.

31

Yes. That was a typo on my part, sorry.

Technically, this is correct in terms of what the core metadata spec says. But the definition of the dynamic field in pyproject.toml says

If the metadata does not list a key in dynamic , then a build back-end CANNOT fill in the requisite metadata on behalf of the user

This means that the if a field isn’t marked as dynamic in pyproject.toml, then it should not be marked as dynamic in the core metadata, because it’s not allowed to change in the way that the core metadata Dynamic field is designed to allow.

So your example is in violation of the PEP 621 definition of the semantics of dynamic. Because when building the wheel, the backend did fill in the metadata on behalf of the user.

IMO, we’re getting into unnecessary rulebook lawyering here, though. The intent is clear to me - a field not marked as dynamic in pyproject.toml should be static in the metadata and the value in sdist and wheel metadata should be the same as the value in pyproject.toml. If you don’t think the specs say that, we can clarify the specs. If you don’t want the specs to say that, you’ll need to write a PEP proposing a (backward incompatible) change.

1 Like
32

Sorry, I just thought we should be as clear as possible when it comes to the specification terms, so that we all have the same understanding.

With everyone being on the same page, my initial proposal is to modify the pyproject.toml project.dynamic definition so that it would look something like:

  • If the metadata does not list a key in dynamic, then a build back-end CANNOT fill in the requisite metadata on behalf of the user (i.e. dynamic is the only way to allow a tool to fill in metadata and the user must opt into the filling in), with the following exception:
    • When generating binary artifacts (wheels), the build-backend MAY add additional dependency requirements, as long as they FURTHER CONSTRAINT existing ones — the set of package versions that meet the binary artifact requirements MUST be a subset of the versions that meet the statically defined requirements

But this is tricky, since there are two specifications at play here — pyproject.toml’s project table, and the core metadata.

Adding an exemption like the one above, would require us to add Dynamic: Requires-Dist to the sdist’s PKG-INFO, which I think is within spec, right?
The build-backend would just need to adhere to all the requirements imposed by the pyproject.toml’s project table specification. Essentially, the only metadata that could change is what is covered by the new exemption.

I am not really a fan of the implications this has on PKG-INFO, so I think it would make sense to introduce new core metadata to better handle these scenarios, but I if I understand everything correctly, I don’t think that is strictly required.

33

It seems like adding the exemption itself (to the pyproject.dynamic spec) requires a PEP, because it’s not backwards-compatible. A backend that follows the exemption logic could create a wheel that doesn’t install where it used to–which I think should be considered backward incompatible?

34

Yes, your original proposal was only to change pyproject.toml. This can be handled in the core metadata as you say by simply setting Dynamic: Requires-Dist. I imagine it might be hard for a build backend to be certain in advance whether it would need to alter the dependencies at wheel build time, so I fear there would be a tendency to declare Requires-Dist dynamic “just in case” - which would be bad, as it would have a significant performance impact whenever resolvers needed to try a sdist as part of a resolution.

I’m also not a fan of the implications on PKG-INFO, but I don’t think new metadata would help. The big problem is that we’d be making it less likely for build backends to declare dependencies as static, and that has a long-term impact on the ecosystem.

Definitely - I don’t think that’s ever been in question. But I’d been thinking more in terms of the simple fact that it changes the semantics of the dependencies field (and specifically its relationship to dynamic).

However, you’re right - if I have a project that declares its dependencies in pyproject.toml, and I install that project from source, along with some other packages, then it’s possible that this would install correctly right now, but under this new proposal, with no other change to anything, the install would fail.

For a specific example:

  • My project declares dependencies = ["B <= 2.0"]
  • I do pip install . C
  • C depends on B > 1.0.
  • Currently, this works and builds my project and installs it, C, and B 2.0.
  • Under the new proposal, the built wheel for my project could depend on B < 1.0, and the install would fail.

To be honest, I think that’s a fatal issue with the proposal. I don’t think a silent change like this, which the developer can’t opt out of, is acceptable. Even if we say “build backends shouldn’t do this”, we can’t guarantee that (because the backend is completely unaware of C in my example above).

35

Does pip currently rely on reading the project table from pyproject.toml to determine the dependencies? Shouldn’t it build the wheel to install instead and resolve from there?

Well, I think it’s either this or adding new core metadata. I think that, for eg., adding a Requires-Dist-Dynamic field specifying a Requires-Dist that could be further constrained when building a wheel, would be a reasonable approach.

Eg.

  • pyproject.toml
[project]
name = 'example'
version = '1.0.0'
dependencies = [
  'foo >= 1.0',
  'bar >= 2.0',
  'something >= 3.0',
]
dynamic = []
  • sdist PKG-INFO
Metadata-Version: 2.2
Name: example
Version: 1.0.0
Requires-Dist: foo >= 1.0
Requires-Dist: bar >= 2.0
Requires-Dist-Dynamic: something >= 3.0
  • Example wheel METADATA
Metadata-Version: 2.2
Name: example
Version: 1.0.0
Requires-Dist: foo >= 1.0
Requires-Dist: bar >= 2.0
Requires-Dist: something == 3.4.*
36

If we were going to add new core metadata (which may be the best approach), I’d advocate for Install-Constraints or similar, that basically acts like a constraints file rather than a requirements file. And specify it as project.constraints in pyproject.toml.

So installers are not obliged to add packages listed there into the set of things to install, but are obliged to exclude versions of packages that are not included in each constraint.

Metadata-Version: 3.0
Name: example
Version: 1.0.0
Dynamic: Install-Constraint
Requires-Dist: foo >= 1.0
Requires-Dist: bar >= 2.0
Requires-Dist: something
Install-Constraint: something == 3.4.*

(And yes, this might make resolution horribly slow. That is entirely unavoidable with the situations where this is needed, because the alternative is to make the final install broken.)

3 Likes
37

No, it doesn’t. It will, as you say, build the wheel. But under the proposed rules (and assuming an updated build backend that implements the new rules), the wheel will change, and that’s the problem. The user did nothing to opt into the new rules, and now the build fails. That’s what I think is unacceptable.

It’s not about pip, it’s about the fact that you’re silently changing the meaning of something the user has written in their project code (in pyproject.toml). And worse, not only did the user not opt in, you don’t provide any opt-out.

Let me repeat what I just said. It’s not about core metadata. It’s about the fact that you cannot silently change the meaning of something that users are currently using extensively.

Currently a user can say one of two things:

  1. These are my project’s dependencies. They will be the same in every artifact built from this source code. Specified by giving dependencies an explicit value, and not incuding it in dynamic.
  2. My project’s dependencies are calculated by the build backend. Specified by including dependencies in dynamic and not providing a value.

You want to add something new that a user can say, and that cannot be done by using either of the existing spellings.

The only possible way of spelling this new intention without adding new keys to the [project] table would be to assign a meaning to giving dependencies a value and including it in dynamic. That would mean modifying this statement in the pyproject.toml spec:

Build back-ends MUST raise an error if the metadata specifies a key statically as well as being listed in dynamic.

The change would need to explicitly call out dependencies as being different (and that will likely cause some debate - why not allow it for other fields too, if we’re doing this? For example, optional-dependencies…) You’d also need to define the particular rules that apply to dependencies (the “you can only tighten restrictions on existing dependencies” rule).

To be honest, this seems like an awful lot of effort to standardise something that’s perfectly possible already:

[project]
dynamic = ["dependencies"]

[tool.my_build_backend]
base_dependencies = [...]

Seriously - if a build backend needs this, what’s wrong with just Dynamic: Requires-Dist? I don’t like making dependencies dynamic, because of the performance implications, but the performance will be just as bad with this proposal (and probably worse, as you seem to be suggesting that wheels can have Install-Constraint, and that will slow down getting the dependencies of a wheel, rather than limiting the impact to sdists).

38

My understanding was that if you put dynamic: dependencies then you can’t have any dependencies listed, even if the majority are static. At least with a new field, you can list most of your dependencies in pyproject.toml (and tooling like Dependabot will still function normally).

My preferred alternative was to allow specifying static dependencies while also allowing the field to be added to if it’s marked as dynamic. That was rejected earlier.

Only for installation, not for other tasks that may require analysing dependencies from metadata. If I want to find all the libraries on PyPI that depend on mine, I can still do it, and I’ll find more of them if they can specify mine statically even if they’re specifying someone else’s dynamically.

It just overall provides more info for people doing things that don’t require a fully resolved dependency graph, which I understood to be the idea behind making most of the metadata static. Otherwise, having the dependency list be “fully” dynamic provides zero information about dependencies, even if some information could be available.

1 Like
39

How does the build fail? I still don’t get that, sorry :confused:.

The install in your example would fail due to incompatible dependencies, but the build shouldn’t.

The opt-in/opt-out is provided by the build backend.

I guess this is subjective, but I don’t really view allowing build backends to register binary compatibility constraints in wheels as changing the project dependencies.

That will break a lot of stuff, as PEP 621 explicitly calls to raise an error when encountering such situation, so it’s a non-starter for me.

I think the Requires-Dist-Dynamic proposal addresses the performance implications as much as possible.

The only approach that could improve on it is specifying the binary constraints statically.

Eg.

[project]
name = 'example'
version = '1.0.0'
dependencies = ['something >= 3.0']
dependency-constraints = ['something >= 3.@']

Where @ would be expanded to match the build version (eg. 3.4).

40

You’re confusing pyproject.toml and core metadata again. What you say is true, but it’s unrelated to whether you can use Dynamic: Requires-Dist.

I couldn’t find anyone rejecting it. I’m pretty sure I didn’t (although I can’t say I particularly like it, but that’s at least in part because I don’t think the semantics of the whole “restrict existing values only” constraint are clear).

Hang on, we’re mixing things up again. In a wheel, there’s no need for Install-Constraint - the dependencies are always fully specified in a wheel. In a sdist, it’s allowed (see PEP 643 for the details) for the build backend to mark the field as dynamic, but include the value “as a hint”. So Install-Constraint is somewhat useful, but not necessary (and the definition of Dynamic in core metadata means that I don’t think it would actually help in this case). That only leaves pyproject.toml where it’s not already possible to record this data. I can’t say why the authors of PEP 621 chose not to allow dynamic-with-a-value, but I do know that for most use cases where I’d expect to be processing project metadata, I’d be looking at core metadata, not reading pyproject.toml.

Sorry, the install fails, not the build. The install will fail because the constraints are unsatisfiable. Under the current rules, it’s not possible for the dependency constraints to be unresolvable. It’s only the fact that the new rules allow build backends to create a wheel that has tighter constraints than pyproject.toml that allows a failing install to happen.

How? By a non-standard build backend option? That’s not (in my opinion) a sufficient mitigation for a backward compatibility break.

Yeah, it’s subjective. I very much do, because I’m looking at it from the point of view of a resolver. I won’t go into the gory details of pip’s resolver, but please accept that from pip’s point of view, this is definitely backward incompatible.

OK. But allowing build backends to alter the value given in dependencies without an explicit marker to give them permission will also break a lot of stuff, so I don’t think you can reasonably claim that’s an acceptable approach either.

But that’s core metadata. What’s the pyproject.toml change that goes with it? You’ve excluded using dynamic with a list of values, and I’ve excluded your original proposal of letting build backends modify non-dynamic dependencies. So at this point Requires-Dist-Dynamic is an implementation mechanism without a feature for it to implement :slightly_frowning_face:

1 Like