PEP 739: Static description file for build details of Python installations

Packaging Standards
61

It’s defined in:

And it’s only used at:

and:

Basically it’s an imp hold-over.

62

It’s also used internally through the native variable that contains it. The Python one is just a copy of that variable, so changing the value will literally change the files that might be imported.

It’s also used extensively throughout the ecosystem, but that’s a good thing in this case. If we want to change the filename spec for extension modules, most users will automatically get the change.

63

Correct, but Ralf’s comment was about why didn’t the list contain the complete list of suffixes including the platform-specific bits. So my point is the data exposed via Python code is only used by the lines I linked to, hence why the suffix list can get away w/ not being thorough.

64

But isn’t it the complete list of suffixes for an extension module? (Not a wheel - that’s a totally separate list.) What other suffixes exist besides those here?

const char *_PyImport_DynLoadFiletab[] = {
#ifdef __CYGWIN__
    ".dll",
#else  /* !__CYGWIN__ */
    "." SOABI ".so",
#ifdef ALT_SOABI
    "." ALT_SOABI ".so",
#endif
    ".abi" PYTHON_ABI_STRING ".so",
    ".so",
#endif  /* __CYGWIN__ */
    NULL,
};
65

I thought Ralf suggested ".abi" PYTHON_ABI_STRING ".so" wasn’t included or quite right.

Regardless, I think I have derailed this enough.

66

I have updated the PEP according to the feedback in this thread, and also added a JSON Schema definition. I still want to do another pass on the PEP text to ensure better consistency, but this is good enough for further feedback.

2 Likes
67

Thanks for the update @FFY00!

I went through and checked all my previous comments - most are indeed addressed, a couple weren’t yet. For the schema, here is what I noticed:

  • There are several items that were removed from the spec for implementation but are still present in the example: hexversion, cache_tag, _multiarch.
  • For the version info:
    • implementation.version.releaselevel: it doesn’t include a dev/devel option, please consider adding that.
    • implementation.version.serial: unclear to me what that means, since it’s not a normal part of the version number. If it needs to be kept, please expand the description.
  • abi.flags: consider describing what the tags mean (d: debug, t: free-threading), or otherwise linking to the canonical place for it.
  • abi.stable_abi_suffix: please specify that if the stable ABI is supported by the Python install, this field must be present.
    • This comment applies more broadly. Optional fields are hard to rely on, so for the fields that are optional please turn around the rationale and say that they must be present unless (reason). And limit it to what actually does have a reason to be missing - for example, I don’t see a reason for suffixes to go missing. And for libpython the description can be turned around (“must be present unless libpython isn’t provided”).

This wasn’t addressed yet.

1 Like
68

The specification allows for additional items in this field, with the idea that the field (see in “Additional properties: Allowed” PEP 739 – Static description file for build details of Python installations | peps.python.org).

However, I think the specification left it underspecified. To keep it simple, I am just gonna use the same specification as PEP 421, meaning the name, version, hexversion, and cache_tag keys will be required, and any additional implementation-specific keys will need to be prefix with an underscore. The goal is to simply encode sys.implementation as a JSON object.

implementation.version is a JSON object representation of sys.version_info, and I don’t think adding a new field on the static description file only is the best path forward. I opened Add `devel` field to `sys.version_info` · Issue #122123 · python/cpython · GitHub, which proposes adding a devel field to sys.version_info.

It is, see sys — System-specific parameters and functions — Python 3.12.4 documentation. It is mainly used for pre-releases, on final releases it will generally always be 0.

In Python Release Python 3.13.0b4 | Python.org, for example, serial is 4.

I considered that, but since it is implementation-specific, I thought it would put a larger burden on the file generation. Additionally, since there’s no way to get this information from the standard library, there’s no canonical source for the tag name (eg. free-threading vs freethreading vs free threading vs nogil, etc).

I will have another look to try to catch such cases.

Thanks, I have added a location specification to my local WIP.

2 Likes
69

I updated the PEP based on the feedback. It is mostly just adding requirements for fields/sections to be present if available in the Python implementation.

70

This looks fairly promising, I have a tool for searching for Python installs that is progressively getting slower as I add support for additional implementations and install locations as I have to directly query each install (or make assumptions about other details remaining consistent).

However unless I’ve missed something these latest drafts no longer include the full python version information equivalent to what you get from sys.version_info and only include a 2 part language.version alongside the implementation specific implementation.version which is only the same as sys.version_info for CPython.

While I recognise that the two part version is considered the “language version”, there are situations where a full “Python version” would be helpful to know statically. For example without the full version number I don’t think there’s any way to use this file to know if a PyPy version targeting 3.X satisfies requires-python = ">=3.X.1" for installers[1].

  1. Without maintaining some kind of external mapping. ↩︎

1 Like
71

Thank you for the feedback!

You’re right. I’ll add a language.version_info field, which should be equivalent to sys.version_info.

Is there anything else you are missing?

72

The language should only have a two part version, we never change it during a release series. The third field is exclusively about the CPython release.

So a requires-python with three parts must be referring to CPython’s version (and probably shouldn’t be allowed in that context…). Since PyPy doesn’t follow that directly, we need some other way to restrict versions using requires-python.

73

You could consider it to be referring to the standard library ‘version’, which isn’t something that can really be separated from a CPython release. This does appear to be how PyPy decides on its equivalent version.

From the PyPy release notes:

  • PyPy3.10, which is an interpreter supporting the syntax and the features of Python 3.10, including the stdlib for CPython 3.10.14.

From a practical point of view the issue is that tools (like pip) use this value to decide whether or not they will install packages. For example if a package excludes a specific version of Python in its requires-python due to a stdlib bug that was in a specific patch release it’s not possible to know beforehand if this means the package will also refuse to install on a PyPy release without knowing this value.

I understand the full value is not considered the “language version” but it is still useful information to know about an interpreter and it would be nice if it was statically available, even if it needs to be under a different key.

74

“Useful information to know” is fine (PyPy could easily add custom fields labelling itself “compatible-with-CPython”, though non-standard fields are a pain for tooling that’s trying to be transparent about this).

I will push back on things that are going to confuse people in the future. The language version does not have a third field, and if we were to put the third field into something labelled “language version” then we’d also have to explain to each and every person that it’s not really the language version. I’ve seen too many incorrect decisions being made because of things like this - eventually, people forget the history and will say “but the static description file says so” as evidence, even though we’d have set it up knowing that it wasn’t meant to be used as evidence!

If it’s really needed, then perhaps we need either a stdlib version or a CPython-equivalent version field. I assume Filipe will push back on extra fields, but since I’m pushing back on combining fields incorrectly we’ll have to find somewhere in the middle. (And I’d still argue that requires-python should not be used to specify micro-versions. They are meant to be information, not constraints.)

3 Likes
75

I’d note that an example demonstrating rejecting specific micro versions is in the building and publishing guide for dropping support for older Python versions.

I get why you don’t want to put it as language version. I just want the value for compatibility and yes, I’d prefer not to deal with non-standard fields[1].

Edit: Interestingly, uv’s installs for PyPy don’t even list the PyPy version and only give the CPython equivalent.

  1. and I’d really like a solution that could mean eventually not needing to query GraalPy directly. ↩︎

76

Those docs are not rarely reviewed by the core team - it’s certainly the first time I’m seeing it. If someone had previously asked me whether we should suggest filtering by the micro version, I’d have said not to.

If they have a reason for it, that’d be interesting. For core development stuff like this PEP, we can use existing things as rationale, but have to be really careful using them as precedent.

Since sys.version.micro exists, and is presumably what Requires-Python is intended to compare against, it’s probably easiest to just switch the field back to being “sys version” rather than “language version” (and having to attribute new semantics to it).

2 Likes
77

A couple updates:

I have pushed a change to the PEP adding a libpython.dynamic_stableabi field, as it’s a relevant piece of information I identified was missing when trying to use the static description file in a real use-case.

This field should contain a path to the dynamic library that should be link against when targeting stable ABI.

On Linux, for example, libpython.dynamic_stableabi should point to libpython3.so, while libpython.dynamic should point to libpython3.14.so.1.0.

I am currently considering changing the semantics to require unavailable fields (for eg. libpython.dynamic in an installation that does not provide a dynamic libpython) to be specified as null (deserialized as None in Python), instead of allowing the section to be missing.

This makes the code utilizing the config data a bit less cumbersome. In addition, it also makes the feature unavailability more explicit, giving us a better framework to update the schema in a backwards-compatible manner.

I am planning to add some semantics to schema_version allowing for backwards-compatible versions. This would be done by introducing to the specification the possibility of schema_version containing a minor version, allowing for new versions to introduce keys, while keeping compatibility on all versions of the same major number.

1 Like
78

Surely the naming of this would suggest libpython3.dynamic for the stable ABI library?

More importantly though, we should probably refer to it as abi3 (which using libpython3... may achieve). I could easily see us running multiple stable ABIs in parallel for a while if/when we decide to start a new one.

Do we need a version for this? We can’t practically make keys mandatory in any way that a version number helps (are we going to force the reader to read it?), so unless we’re changing the interpretation of an existing key (which wouldn’t be compatible), the user can just ignore anything it doesn’t understand or need.

1 Like
79

I’d like to avoid using the major version in the name.

If we were to start a new ABI, it should very likely be treated as a different implementation, rather than an implementation variant[1]. In which case, it would be in a different Python installation, with its own description file.

I think so, since we will likely add new keys in the future, I think it would be beneficial to version that in a sane way.

The schema_version specification needs to be updated anyway, as it currently requires its value to always be updated, regardless of backwards compatibility. I think we have two options here, keep schema_version as an arbitrary value that never changes as long backwards compatibility is preserved, or make it a partitioned version.

Having a standardized partitioned version only introduces a very small amount of complexity in code parsing the file, and provides extra information, which can be helpful to users interacting with the file.

That said, I don’t feel strongly on this, so if anyone has a strong opinion, we can change it.

  1. I can elaborate on this if you want, I actually started but thought it was maybe unnecessarily going a bit out of topic. ↩︎

80

I think this is less likely than creating a new stable ABI that corrects some of the issues in the existing one but doesn’t drastically change the ABI itself. e.g. it’s thought we may need a new stable ABI for freethreading to work, which wouldn’t necessarily need a separate install (using abi3 in this context might just get you more locks around your extension).

But new keys don’t have any impact, unless they are somehow mandatory. It’s easier to check that they’re present/absent and use a fallback default if you need whatever value it provides.

Requiring any new fields to specify what their default value should be when absent is a reasonable requirement (and it might be specified in terms of other fields, or conditional on other fields, like I did in PEP 514). Code written before the field exists can’t be using it anyway, so it doesn’t matter that they don’t know what the default value is.

This is all true, apart from the amount of complexity in code parsing the file. Because there’s a significant amount of complexity involved in deciding whether a version is “compatible enough” or not (as we’ve been seeing in the Wheel 2.0 threads).

Basically, I’m fine with an extra version field/partition if tools can just ignore it, and it’s only intended for humans to read, and the primary version field means “if you don’t know this version, you shouldn’t read this file”. There really doesn’t seem to be any benefit in being more complex than this for a static file.