PEP 739: Static description file for build details of Python installations

We now have a working implementation of the main use-case of this PEP — Prototype support for PEP 739 by FFY00 · Pull Request #13945 · mesonbuild/meson · GitHub.

@pf_moore, I believe the PEP is now ready for review.

2 Likes

I’ll take a look in the next week or so. However, one thing struck me immediately on skimming the PEP - it’s not clear what a “Python installation” is in this context. If I’m looking at a virtual environment, is that an “installation”, or am I expected to look at the pyvenv.cfg file to find the base Python interpreter, and then that is the “installation”? The existence of the base_interpreter field suggests the former, but I couldn’t find anything saying that explicitly.

Also, I’m still uncomfortable with the fact that very little is said about how to find the file. Putting it in <stdlib>/build-details.json is not required, and installations are allowed to put the file elsewhere if they want. The Meson implementation seems to suggest it’s up to the user to find the file and pass it to meson - is that the intended approach? If so, it should probably be described in the PEP.

Sorry if this is covered in the discussion here. It’s been a while since I read it and I haven’t re-read it yet, so if it is answered there, that’s fine. But I do think the PEP should include it as well, as it should be possible to read the PEP without reference to the discussion here.

2 Likes

Good point, I will clarify that. A better description might be “Python distribution”.

No, implementations should place the file in <stdlib>/build-details.json, unless there are technical limitations that prevent it. When such limitations exist, it is up to the implementations to decide if and how to make this file available. Maybe I should make that clearer.

The part about the file possibly being provided in another location is stated to make it clear that in the future we may also provide this file in a different location with better discoverability.

The stdlib was chosen as the install location as it is reasonably easy to find, however if we are trying to discover the Python installation present on the system without running the interpreter, it isn’t a great location to have the config file. To tackle this use-case, I envisioned that in the future we may do something like standardize locations where config files can be installed (e.g. /usr/share/python/installations. ~/.local/python/installations), allowing tools to quickly scan them and discover the Python installations on the system. This is out of scope for the PEP, but I wanted to make it clear that the file config might be provided in extra locations.

Yeah, I agree, I will try to clarify the PEP’s intentions.

1 Like

FWIW, I’m currently working on this to be able to replace the installer on Windows, and while I tried to follow this PEP it turned out to not be of any benefit. So I ended up creating an __install__.json in sys.prefix that contains the info I need (which is basically “what tags should launch this and which executable; what Start menu shortcuts and registry keys should be created; etc.”). This is an internal implementation detail though, so it could merge into build-details.json later, but I wanted to mention that I found there wasn’t enough useful info in there that I felt the need to make use of it. (There’s a long post in Ideas right now if you want more details, or I’ll hopefully be posting the PEP properly later today.)

2 Likes

Which other information do you need?

Including a list of supported wheel tags in build-details.json is something we can consider, but I don’t want to do it in this first format version.

1 Like

Am I still waiting for an update to the PEP here? There’s no rush if you’re busy, but I don’t want to be the one delaying things because I missed that an update had been done…

No, I pushed a large-ish update last week.

3 Likes

There were two recent changes in PEP 739: some (hopefully final) updates by FFY00 · Pull Request #4216 · python/peps · GitHub and PEP 739: some more (hopefully) final updates by FFY00 · Pull Request #4220 · python/peps · GitHub, perhaps?

A

Yes, exactly. I hope those changes were enough to address Paul’s worries, and am open to more direct/quicker feedback rounds to get it in shape. I’d like to have this resolved sooner rather than later, so that we can ship it ASAP to start testing and integrating it into tooling, ensuring that whenever 3.14 gets a final release, things are already at least somewhat mature.

2 Likes

Thanks. Apologies for not spotting the new version, I don’t typically see notifications from the PEPs repo. The revision has addressed my concerns, thanks for that.

With this latest update, I’m pleased to say that the PEP is approved. Thanks to everyone who participated in the discussion, and congratulations to @FFY00.

One note - the “human readable” version of the JSON schema used in the PEP is very much appreciated. It’s far easier to read than the raw schema definitions I’ve occasionally seen elsewhere. I assume a tool was used to produce it (if you did it by hand, my appreciation is even greater!) but the results were definitely worthwhile.

One minor editorial comment that I hadn’t spotted earlier:

  • abi.extension_suffix - states that the entry will be missing if the implementation doesn’t support extensions, but says required=True. I assume required should be False.

I’m happy for the position on this field to be clarified without affecting my approval of the PEP.

6 Likes

No worries, I will explicitly ping you next time.

:tada:

Yes, it’s rendered with a script I wrote. It is available in this gist (archived it just in case), which is mentioned as a comment in the rST source. I intended to commit it to the repo, but the PEP editors told me we should leave it out, so I uploaded it as a gist and added a comment with a link to it.

Oh, yes! Thanks for noticing. It is a mistake, I’ll correct it when updating the PEP status.

1 Like

Even better - my thanks to you for taking the time to write such a script :slightly_smiling_face:

1 Like

Is there a plan to make this information also available through a Python API?

2 Likes

Yes, likely in sysconfig, but I am not sure yet how exactly the API would look like. Probably, the API would either be containers with the same information as the file, or — and this was my initial plan for the new syconfig API — a richer, more Pythonic API, that could load any arbitrary build-details.json file. The goal with the fancier API was to be able to have packaging build backends use the new API for Python introspection, which would then make it seamless to support cross-compilation — they just use the build-details.json file from the target.

One thing I would like to expose regardless, is a way to ask the Python installation for its build-details.json file, if available.

3 Likes

Any possibility of getting this usable for docs builds? I just went through a bit of work trying to figure out “How do I get the Docs make doctest to use my newly-built Python”, be really nice if it could rely on the static definition to find it. (ex. Add a new method to a Python builtin which contains a doctest, doctest will fail unless built with the new Python build). Today that information has to be hand-passed via make variables. Would also be nice dogfooding / “does this provide the benefit expected” without exposing externally APIs.

1 Like

Yes, starting to move away from relying on make variables is precisely one of my motivations behind this PEP and the new sysconfig API work.

3 Likes

Congratulations and thanks for all the hard work! Very exciting!

1 Like