PEP 793 – PyModExport: A new entry point for C extension modules

PEPs
1

We’ve just published PEP 793 – PyModExport: A new entry point for C extension modules. Thanks @AA-Turner’s for the reviews!

Abstract

In this PEP, we propose a new entry point for C extension modules, by which one can define a module using an array of PyModuleDef_Slot structures without an enclosing PyModuleDef structure. This allows extension authors to avoid using a statically allocated PyObject, lifting the most common obstacle to making one compiled library file usable with both regular and free-threaded builds of CPython.

To make this viable, we also specify new module slot types to replace PyModuleDef’s fields, and to allow adding a token similar to the Py_tp_token used for type objects.

We also add an API for defining modules from slots dynamically.

Code example

The PEP includes a full example. If you compile it (with non-free-threaded Python headers) and remove extension file’s the version suffix (e.g. on Linux, rename to examplemodule.so), it will work with both regular and t Python.

To get some code on this page, here’s the definition from the example:

static PyModuleDef_Slot examplemodule_slots[] = {
    {Py_mod_name, "examplemodule"},
    {Py_mod_doc, (char*)examplemodule_doc},
    {Py_mod_exec, (void*)examplemodule_exec},
    {Py_mod_methods, examplemodule_methods},
    {Py_mod_state_size, (void*)sizeof(examplemodule_state)},
    {0}
};

PyMODEXPORT_FUNC
PyModExport_examplemodule(PyObject *spec)
{
    return examplemodule_slots;
}

See this topic for the larger plan for stable ABI for free-threaded builds.

What do y’all think?

6 Likes
2

My first impression about this PEP is great!
For now I’m not deeply in nuances but I can read example section and understand what’s going on without reading any docs. And, of course, removing “the most common obstacle to making one compiled library file usable with both regular and free-threaded builds of CPython” sounds nice.

3

Can’t this already be avoided? Or we can add more changes by adding more slots. The “common obstacle” is just single-phase init, right?

This strikes me as primarily a path towards removing single-phase init entirely, which I fully support, but I’d like to see that acknowledged. I didn’t spot in the PEP text any clear answer to what the “common obstacle” actually is - a statically allocated PyObject in theory is perfectly fine provided that you meet a ridiculous set of conditions (which we wouldn’t expect anyone to meet).

But using multi-phase init gets you around that particular issue already, and the problem is just that single-phase is still supported at all and so we can’t assume that multi-phase is being used when the module is first loaded.

It may still be that adding another API (that could take 10+ years for people to actually start using) is the best approach, but I’d like to see more clarity around whether “just kill off single-phase init” would also achieve the same goals.

It’s a mild aside, but I don’t see a problem with adding new slots that also cover static fields and using them in preference to the static fields, if that’s the main value here. No doubt there are backwards compatibility limitations here (I know we don’t love the use of void * for a range of different value types), but if these are actual blockers to improving the range of slots then that needs to be explained better. The text really leads towards single-phase init being the problem being solved here, without actually saying it.

1 Like
4

This PEP is pretty much orthogonal to removing single-phase init, that’s why you don’t see it mentioned :‍)

The obstacle is the static PyModuleDef.

It is not. The layout of PyObject differs between the regular and free-threaded builds, so if you bake a static PyObject into the compiled extension, it will not work for both builds.

It would not, because current multi-phase init expects the entry point to return a statically allocated PyModuleDef. And PyModuleDef is, for reasons lost to time, a PyObject subclass.

Hmm, technically, yes, you can avoid PyModuleDef by using the low-level API (PyModule_NewObject or PyModule_New) with single-phase init, and filling in methods & __doc__ manually.
The PEP should mention that.

5

Ah okay, this is the point that I missed.

Any possibility of making PyModuleDef be shaped as if it were always using non-freethreaded PyObject as the base? Statically allocated, it should be immortal anyway, and so it’s not being refcounted. A lot of the other changes still make sense then (like not returning it from _Get), but I think we can avoid the problem of having three potential APIs to support where everyone is still using the first one…

6

Personally, I’d rather support three well-defined APIs[1] than this kind of fragile trickery.
PyModuleDef layout is public API. Refcounting is a valid on it, though it doesn’t make much sense.
More importantly, the import system (and any 3rd party reimplementation of that) will need to call PyTYPE on it to separate it from single-phase extension that returns PyModule. (BTW, this is what allowed the current multi-phase init to reuse the import hook.)

  1. well, two well defined ones plus a really weird one ↩︎

1 Like
7

Makes sense. The two things I’d really like to see added (and discussed, since I expect controversy):

  • a copy-pasteable implementation of the current entrypoint so that code can skip right to the new API and offer an equivalent entrypoint that works for 3.14 and earlier (copy-pasteable so that Cython and friends can copy-paste it - separately distributed is less useful here, IMHO)
  • a hard deprecation/disablement deadline for the existing API. The biggest problem with the existing two tier API is that people have never moved, so let’s not make that mistake - this design allows for noisy compile-time and import-time warnings, so let’s be noisy and actually remove the old one (at least single-phase, but ideally both it sounds like) as quickly as we can.
2 Likes
8

Let’s discuss all that, but I consider it out of scope here.
This is not the “entrypoint API changes I’d like in 3.15” PEP. It’s compatible with my planned next steps; I believe it’s compatible enough with yours; and I believe it’ll be a net benefit even if ($DEITY forbid) none of the next steps make it.

Trouble is, I think Cython is better off generating both versions directly, rather than using a generic C macro that covers all the bases.
(I do have ideas for a more autogeneration-friendly format, but that’s for a whole different PEP.)

Deprecating only single-phase init feels off-topic for this PEP, and has its own discussion thread.
Deprecating the current multi-phase can’t quite begin for 5 years at the very least – when the new API will be supported by non-EOL versions of Python. And after those years, any deadline we set now is likely to just be discussed all over again. I guess we can set a timer now to start that discussion.
I’d rather discuss deprecations/removals in the stable ABI in general – we obviously need to break PEP 384’s “until Python 4” promise, and that alone feels like a vat of worms.

1 Like
9

I think it’s in scope enough to discuss why we need three APIs - if the answer is, “we plan to not have three APIs” then that’s a good answer, but I don’t think it’s responsible to simply hope that one or two of the three will disappear. Think of it like a spending budget - free up some budget to pay for the new addition. (The budget is your time :wink: )

It’s especially relevant because the plan when we added the second API was for the first to go away. That hasn’t worked out, so there’s a higher bar for the next go at doing the same thing.

Did I say macro? I typed it at one point but thought I removed it while editing because I don’t think a macro can handle it. But it should be straightforward to have a function that reads the new slots and fills in a static PyModuleDef and returns it, without needing to customise it. I’m sure they’ll figure one out and do it themselves, but including our own that covers all bases makes the addition more palatable - something that’s clearly been an issue with the previous changes to this API.

4 Likes
10

Have you looked into adding an API to create a PyModuleDef object, something like:

PyModuleDef* PyModuleDef_Create(const char* name, const char* doc, /* other ModuleDef fields go here */);

This would fix most backward compatibility issues, although there are also disadvantages: the API would not be extensible, and there likely are memory management issues because PyModuleDef_Init (and hence PyInit_*) returns a borrowed reference (leaving no clean way to clean up the ModuleDef value).

1 Like
11

If it’s my time, I’d rather discuss stable ABI removals in general.

That sounds doable, thanks for the suggestion! There’ll be limitations but, yeah, they should be easy to explain. I’ll do this for the next update.

That function could be extensible if it took a slots array as argument.
But you’re right about the memory management issue; also the interpreter switch issue would remain.

3 Likes
12

I’ve merged an update (thanks to Hugo for proofreading):

Soft-deprecating PyInit_*

The PEP soft-deprecates the old way of doing things, and to help avoid knee-jerk reactions, it takes some care to explain what soft-deprecation means (PEP 387: “the API remains documented and tested, but will not be developed further [and] a soft deprecation does not issue a warning”).

As for planning any removals, it’s too soon to start.

Easier migration from PyModuleDef to tokens

The PEP now specifies that PyType_GetModuleByDef will match a module’s token – that is, it’ll be exactly equivalent to the new PyType_GetModuleByToken except for the signature.

An extension can use #ifdef to define a variable as either PyModuleDef or, for Python versions that don’t have that, a dummy char token, and the PyType_GetModuleByDef calls will not need a #ifdef – they’ll just need an ugly extra cast to void*.

To make this fully backwards compatible, the spec disallows using an unrelated PyModuleDef address as a token. That would be a silly thing to do anyway.

Backwards compatibility shim

The PEP now lists a copy-pastable PyInit_ function that calls PyModExport_.
If the PEP is accepted, I plan to add it to pythoncapi-compat. (Or a better home. No promises.)

2 Likes
13

The porting guide looks great!

No other comments right now, but only because I haven’t the bandwidth to go through it closely. Trusting in others to take a good look.

1 Like
14

Would it work to use intptr_t or uintptr_t instead? From what I can read, it’s guaranteed by the C standard to correctly roundtrip pointers.

15

FWIW, this recent paper targeting C2y covers the state of the art in some depth, and essentially proposes codifying existing practice.

16

This PEP explicitly sticks with the existing PyModuleDef_Slot.

Using intptr_t would mean either changing PyModuleDef_Slot, or making PyModExport_* use a new slot type.
I’d like to do the latter, and solve a few more issues with slots too, but it’s a more controversial proposal. See this topic.

1 Like