Status or funding for Metadata 2.2+ in PyPI

Packaging
21

I’m pretty sure pip will happily install it, so there’s no issue with that. But what we need to do is add logic to use it to avoid doing a build just to calculate dependency data for a sdist. PRs to that effect would be welcomed :slightly_smiling_face:

4 Likes
22

Huge relief! :blush:

23

I’ve confirmed that pip, poetry and uv all install my metadata 2.3 test package :slight_smile:

10 Likes
24

More updates :tada:

Poetry will soon be publishing Metadata 2.3

Hatchling will soon be publishing Metadata 2.2

4 Likes
25

Oh, I’ll bump to 2.3 then, I didn’t realize PyPI supports that number also now. Thank you!

3 Likes
26

Awesome!

I think if Flit starts publishing 2.3 then the combination of poetry, hatchling, maturin and flit would have defaults that could bring Metadata 2.3 adoption very close to 50% for newly published projects.

1 Like
27

I’ve put together a PR for for Metadata 2.3 support in flit. I’m normalizing Provides-Extra and Requires-Dist. Would appreciate some eyes on it.

28

One pathological case that this (and likely some other implementations of metadata 2.2) gets wrong is the following:

pyproject.toml:

[project]
dynamic = ["version"]

__init__.py

from random import choice
__version__ = f"{choice(['0.1.0', '0.2.0', '0.3.0'])}"

The problem is that the spec for Metadata 2.3 says:

If a field is not marked as Dynamic, then the value of the field in any wheel built from the sdist MUST match the value in the sdist.

That means that the build backend, when provided with sources that include the PKG-INFO file (i.e., you’re building from a sdist) must prioritise that data, and not re-generate any “dynamic” fields from pyproject.toml. (Unless, of course, it is certain that doing so cannot cause any discrepancies).

In the above case, if you build a sdist from the project and you get version 0.2.0 (which is recorded in the sdist as static), building a wheel from that sdist must generate version 0.2.0, and not one of the other possibilities. This was a deliberate feature of Metadata 2.2, and is crucial to installers being able to extract sdist metadata without doing a fresh build.

In practice, this is clearly a pathological case designed to break the implementation, so it’s hard to get too worked up about it - but it is technically a spec violation. I don’t recommend marking any field that is dynamic in pyproject.toml as dynamic in the sdist, though - that would be unnecessary in the vast majority of cases and would defeat the purpose of having metadata 2.2. Instead, I would honestly have expected flit to require that fields calculated at build time like this should be constant literal values from the source, and not allow arbitrary code execution at build time. But if the code execution feature is deliberate, then checking for a PKG-INFO file when code execution could happen is the right approach.

(When I wrote the PEP for Metadata 2.2, I thought setuptools was the only build backend that allowed arbitrary code execution when calculating metadata, and we knew that setuptools had some unique difficulties as a result of this. But I didn’t expect it to be a common issue with other backends, and no-one spoke up at the time pointing out any other cases).

1 Like
29

I’m not really sure where the design of dynamic pyproject.toml metadata comes from in flit. Presumably because it has a light UX and no duplication.

If flit eventually drops support for python < 3.8 then it could use a static version in pyproject.toml and importlib.metadata to fish out any “version” at runtime if required.

I don’t really know who the maintainers of flit are if they’d like to comment or review.

I’ve also just realised I’ve got a bug in my handling of Requires-Dist :person_facepalming:

30

TBH, a pathologically dynamic version like this is broken anyway, as tools almost certainly determine the version from the filename, not from the metadata. The only actual case where this would be important in real-world use is in dependency data, and I don’t think flit allows that to be dynamic (but it’s worth checking). So in practice, I think this is a purely theoretical issue for flit.

I mainly wanted to mention this because “Supporting Metadata 2.2” isn’t quite as simple as just bumping the default version number - it does need a bit of thought, even if the conclusion is usually “we’re OK, we don’t generate any dynamic data in a way that is problematic”.

@takluyver is the main flit maintainer, I believe.

1 Like
31

flit only permits dynamic version and description

Both could theoretically become static, but it would impact the UX. Making version static is easy enough in newer pythons.

Like you said though, it doesnt seem like it will be problematic in practice for flit.

I took a look at the rest of the 2.2 and 2.3 implementation for hatch. There’s a lot more going on now to handle the dynamic metadata correctly.

I dont know if poetry supports dynamic.

1 Like
32

It seems to be reading PKG-INFO which suggests it’s doing the right thing.

Edit: Sorry, my comment was about hatch, which you linked. I don’t know what Poetry does.

1 Like
33

The code execution thing is a misfeature that I wish I had left out, but I think people will yell at me if I remove it.

I’ve worked on some projects that do things like this (paraphrased):

version_info = (1, 2, 3)
__version__ = ".".join(str(v) for v in version_info)

I wanted to allow that pattern, which either requires execution, or some config to let Flit understand the version_info line. And in the early days of Flit, running code seemed like an OK idea. The shifting landscape soon made it a bad choice, and people used the flexibility to do ‘get the version from git’ tricks. :slightly_frowning_face:

I’ve always been against looking at PKG-INFO when building, because I think sdists should be treated as a regular source tree, not as a special case. But maybe we can get the version number as usual and error out if it doesn’t match PKG-INFO.

1 Like
34

I agree - I consider reading PKG-INFO to be a fallback if you can’t find any other way to provide the guarantee that declaring a metadata item static makes. For everything other than version and description, flit can do that simply because it doesn’t have a way to make anything else dynamic. For description, we’re OK because a docstring has to be static (f-strings aren’t allowed). And for version, so many things would break if a sdist version 1.0 generated a wheel for version 2.0 that I’m inclined to say that’s simply a case of “not supported - don’t do that”.

35

Unfortunately assigning to __doc__ inside a module does work, so this isn’t 100% guaranteed. But I don’t see much reason to do that, and even if you do, the human readable description doesn’t really matter to tools, so it’s unlikely to be a problem in practice.

:+1:

1 Like
36

While I’ve got you both, can you please clarify something for me. Maybe provide some test cases for my handling of Requires-Dist? I’ll be honest, I didn’t fully follow the details of the justification for Metadata 2.3 in PEP 685, but I understood enough to understand that it’s to tighten up some specs in relation to how “extras” are named in various pieces of metadata.

I’m trying to implement this part for flit:

For tools writing core metadata, they MUST write out extra names in their normalized form. This applies to the Provides-Extra field and the extra marker when used in the Requires-Dist field.

Which now that I think about it, means that I really need to be normalising Requires-Dist of the shape:
Requires-Dist: reportlab; extra == 'pdf'

I’m currently normalising Requires-Dist of the shape:
Requires-Dist: libexample[test, doc]

Am I correct in thinking that I need to normalize both before writing them to PKG-INFO? I really wish I could reach for packaging at this point!

37

Be sure to upgrade your publishing action or else you will think PyPI does not actually support 2.3 like I just did :sweat_smile: Release v1.8.13 · pypa/gh-action-pypi-publish · GitHub

(also the first part of that error message is incorrect)

image
image1366×183 7.35 KB

3 Likes
38

I got the same confusing message from twine:

/tmp/baipp/bin/python -m twine check --strict /tmp/baipp/dist/*
Checking 
/tmp/baipp/dist/tap_clinicaltrials-0.1.2.dev3+g1a80aef-py3-none-any.whl: ERROR    InvalidDistribution: Metadata is missing required fields: Name,        
         Version.                                                               
         Make sure the distribution includes the files where those fields are   
         specified, and is using a supported Metadata-Version: 1.0, 1.1, 1.2,   
         2.0, 2.1, 2.2.

before the 2.0.2 release of hynek/build-and-inspect-python-package. So folks should also be sure to upgrade that too.

3 Likes
39

I think both PEP 685 and the core metadata spec only explicitly say that the [extra,names] you require of the dependency have to be normalised. But I’d guess that’s an oversight; if we’re normalising extra names, it seems best to normalise them everywhere they’re used, including in environment markers. So I’d be +1 on making that explicit in the spec as well, and +1 on tools normalising them even if it’s not formally required.

1 Like
40

I’m also +1.

1 Like