Enforcing consistent metadata for packages

[abravalheri (Anderson Bravalheri)]

@pf_moore

The impression I get is that the only backend that can produce dynamic metadata in this second (metadata 2.2) sense is setuptools, and then only if the value is marked as dynamic in pyproject.toml, or there’s no [project] section in pyproject.toml.

Just to clarify: if the metadata version is not marked as 2.2+ the assumption that core metadata cannot differ from PKG-INFO and METADATA does not hold, right?

I am asking this because I am not confident that setuptools is in a position right now that we can perform enforcement/consistency checks and just start producing 2.2/2.3 metadata. (So that is expectation management).

[pf_moore (Paul Moore)]

@abravalheri

Just to clarify: if the metadata version is not marked as 2.2+ the assumption that core metadata cannot differ from PKG-INFO and METADATA does not hold, right?

Correct. Metadata versions earlier than 2.2 are in effect presumed to mark every field as dynamic.

I expected setuptools to have difficulty guaranteeing static metadata. If a project has no setup.py (reading metadata from pyproject.toml) does that make things easier, or do plugins still cause issues in that case?

[abravalheri (Anderson Bravalheri)]

I think plugins can still cause some issues.

We did implement some (reasonable) check on PEP 621 dynamic, but I suppose plugins can yield different values when build from sdist or wheel or unpackaged sdist.

[brettcannon]

After reading through the thread, it seems like the summary is:

• The vast majority of metadata is already consistent on PyPI among
• Lots of packaging projects already make that assumption
• Having sdists and wheels is probably not desirable long-term due to not wanting sdists to list e.g. OpenBLAS as a dependency as it really depends on the state of the system you are building on/for

Is that accurate?

[pradyunsg (Pradyun Gedam)]

@brettcannon

Lots of packaging projects already make that assumption

Unless I missed something, all installers except pip.

@brettcannon

Having sdists and wheels is probably not desirable long-term due to not wanting sdists to list e.g. OpenBLAS as a dependency as it really depends on the state of the system you are building on/for

It’s not quite system state but, yea, external-to-Python-packages stuff basically.

@pf_moore

Unless I’m missing something, there’s nothing in PEP 725 that would, for example, help me (or a tool I’m using) find a library with a compatible ABI.

Indeed, that’s intended to be a follow up (IIRC, this was mentioned in the PEP or in the topic for discussion of that PEP).

[barry (Barry Warsaw)]

@brettcannon

Having sdists and wheels is probably not desirable long-term due to not wanting sdists to list e.g. OpenBLAS as a dependency as it really depends on the state of the system you are building on/for

Can you clarify what you mean here? Are you thinking along the lines of eventually deprecating sdists on PyPI ^[1]?

---

1. as opposed perhaps to options like making --only-binary :all: the default for pip, or somehow marking sdists as “not locally buildable” ↩︎

[brettcannon]

@barry

@brettcannon

Having sdists and wheels is probably not desirable long-term due to not wanting sdists to list e.g. OpenBLAS as a dependency as it really depends on the state of the system you are building on/for

Can you clarify what you mean here?

Yes, I had a long day at work reading a lot of messages here and I missed typing out “having sdists and wheels agree on metadata”. My brain probably thought I made some generic statement about the entire bullet list being about metadata agreeing.

Although I have no qualms if installers stopped supporting sdists directly and made building them an out-of-band thing (hence why my first attempt at lock files left sdists out entirely).

[barry (Barry Warsaw)]

@brettcannon

Although I have no qualms if installers stopped supporting sdists directly and made building them an out-of-band thing (hence why my first attempt at lock files left sdists out entirely).

Yeah, and I was sort of against that at the time, but it does make a lot more sense to me now. That said, sdists for pure Python libraries are trivial to build locally so I wonder if it would be helpful to make that distinction? Like we could say that absent any wheels, pip install could still choose pure Python sdists, but only choose wheels for binary / native code libraries?

[steve.dower]

@barry

Like we could say that absent any wheels, pip install could still choose pure Python sdists, but only choose wheels for binary / native code libraries?

There’s nothing consistent in all sdists to identify this - you don’t know what kind of wheel you’re going to get until you install the backend and run it.

[barry (Barry Warsaw)]

Could it be some metadata that the package author controls? Probably, but the complexity ^[1] isn’t worth it.

---

1. it’s okay to build wheels for this sdist but not that sdist ↩︎

[brettcannon]

@barry

Could it be some metadata that the package author controls?

That’s an idea that has come up in other discussions around what exactly an sdist is for?

@barry

Probably, but the complexity [1] isn’t worth it.

The other thing is if you’re using a build back-end to make an sdist (which you would be doing most likely to create PKG-INFO), then the build back-end can very likely make the wheel.

[tacaswell (Thomas A Caswell)]

If I recall the history here correctly, conda tends to ensure that it includes the pip/wheel metadata to reduce the number of things that can go wrong when user use pip on top of a conda env.

I have vague memories of issues where you would conda install a package, then pip install a package that depends on it, pip would not be aware of the conda installed package, so pip would re-install that package from pypi with a possible different version and (for packages with extension) almost certainly different build settings which could result in “chimera installs” if modules were added / removed ^[1]. By including the pip/wheel metadata and subsequent use of pip finds what it would have put there for the version of the package conda installed and things mostly work.

---

1. These can be wild to debug because they tend to manifest as “impossible” things. ↩︎

[sinoroc]

@tacaswell

I have vague memories of issues where you would conda install a package, then pip install a package that depends on it, pip would not be aware of the conda installed package

This reminds me of this: Poetry overwrites existing conda packages even if versions match · Issue #6408 · python-poetry/poetry · GitHub

A whole bunch of conda packages contain a direct_url.json file that conda happily copies as-is (when it obviously shouldn’t), and then fun things happen.