Pre-PEP: PyModuleExport -- a new export hook for modules

Core Development C API
1

Excuse me spamming “pre-PEPs” here…
I still think my previous one, PySlot, is a good idea, but doesn’t help its “immediate motivation” as well as I thought. When working on the implementation for modules, I felt like adding another layer of cruft.

So, here’s proposal that’s largely orthogonal (and would be helped by PySlot), but solves the “static PyObject” issue more directly, and adresses another pain point.

(Excuse the formatting, this is ReST interpreted as Markdown.)

cc @eric.snow @ncoghlan

Motivation

The memory layout of Python objects differs between regular and free-threading
builds.
So, an ABI that supports both regular and free-threading builds cannot include
the current PyObject memory layout. To stay compatible with existing ABI
(and API), it cannot support statically allocated Python objects.

There is one type of object that is needed in most extension modules
and is usually allocated statically: PyModuleDef returned from
the module export hooks (PyInit_* functions).

Module export hooks (PyInit_* functions) can return two kinds of objects:

  1. A fully initialized module object (for so-called
    single-phase initialization). This was the only option in 3.4 and below.
    Modules created this way have surprising (but backwards-compatible)
    behaviour around multiple interpreters or repeated loading.
    (Specifically, the contents of such a module’s __dict__ are shared
    across all instances of the module object.)

  2. A PyModuleDef object containing a description of how to create a module
    object. This option multi-phase initialization was introduced in
    :pep:489; see its motivation for why it exists.

The interpreter cannot distinguish between these cases before the export hook
is called.

The interpreter switch

Python 3.12 added a way for modules to mark whether they may be
loaded in a subinterpreter: the Py_mod_multiple_interpreters slot.
Setting it to the “not supported” value signals that an extension
can only be loaded in the main interpreter.

Unfortunately, Python can only get this information by calling the
module export hook.
For single-phase modules, that creates the module object and runs arbitrary
initialization code.
For modules that set Py_mod_multiple_interpreters to “not supported”,
the initialization needs to happen in the main interpreter.

To work around this, if a new module is loaded in a sub-interpreter, Python
temporarily switches to the main interpreter, calls the export hook
there, and then either switches back and redoes the import, or fails.

This is unnecessary and fragile extra work, but it is also highlights a
higher-level issue: Python has no way to query an extension before it can
potentially fully initialize itself.

Rationale

For avoiding the module export hook returning a statically allocated
PyObject*, two options come to mind:

  • Returning a dynamically allocated object, whose ownership is transferred
    to the interpreter. This could be the existing PyModuleDef, since it
    needs to contain the same data.
    Unlike the existing PyModuleDef, this one would need to be
    reference-counted so that it does not leak.
    That can break code that expects PyModuleDef\s to be static.

  • Adding a new export hook, which doesn’t return a PyObject*.

    This was considered for Python 3.5 in :pep:489, but rejected:

    Keeping only the PyInit hook name, even if it’s not entirely appropriate
    for exporting a definition, yielded a much simpler solution.

    After a decade of fixing the implications of this choice, the solution is no
    longer simple.

A new hook will allow Python to avoid the second issue mentioned above –
the interpreter switch.
Effectivelly, it’ll add a new phase to multi-phase init, in which Python can
check whether the module is compatible.

Using slots without a wrapper struct

The existing PyModuleDef is struct with some fixed fields and
a “slots” array.
Unlike slots, the fixed fields cannot be individually deprecated/replaced.
This proposal does away with them and proposes using a slots array directly,
without a wrapper struct.

The PyModuleDef_Slot struct does have some downsides to struct fields.
We believe these are fixable, but leave that out of scope of this PEP
(see “Improving slots in general” Rejected ideas).

Tokens

A static PyModuleDef has another purpose besides describing
how a module should be created.
As a statically allocated singleton that remains attached to the module object,
it allows extension authors to check whether a given Python module is “theirs”.
If a module object has a known PyModuleDef, its module state will have
a known memory layout.

An analogous issue was solved for types by adding Py_tp_token.
This proposal adds the same mechanism to modules.

Unlike types, the import mechanism often has a pointer that’s known to be
suitable as a token value; in these cases it can provide a default token.

Specification

When importing an extension module, Python will newly look for an export hook
like this::

int PyModuleExport_<NAME>(PyModuleDef_Slot **slots_p);

where <NAME> is the name of the module.
For non-ASCII names, it will instead look for PyModuleExportU_<NAME>,
with <NAME> encoded as for existing PyInitU_* hooks
(that is, punycode-encoded with hyphens replaced by underscores).

If not found, the import will continue as in previous Python versions (that is,
by looking up a PyInit_* or PyInitU_* function).

If found, Python will call the hook with slots_p as an “output argument” (a
pointer to memory to be filled).

On failure, the export hook must return -1 with an exception set.
This will cause the import to fail; Python will not fall back to PyInit_*.

On success, the hook must return 1 an set *slots_p to point to array of
PyModuleDef_Slot structs terminated by an entry with slot=0.
Python will then create a module based on the given slots.

The result and all data it points to (recursively) must remain valid and
constant until runtime shutdown.
Also, Python may cache a successful result until runtime shutdown.
(We expect functions to export a static constant, or one of several constants
chosen depending on Py_Version. Any dynamic behaviour should happen
elsewhere, typically in Py_mod_create and Py_mod_exec functions.)

Return values greater than 1 are reserved for future extensions.

Dynamic creation

A new function will be added to create a module from an array of slots::

PyObject *PyModule_FromSlotsAndSpec(PyModuleDef_Slot *, PyObject *spec)

Unlike with the export hook, the slot array passed to this function
may be destroyed after the PyModule_FromSlotsAndSpec call.

To simplify the implementation, the slots arrays for both
PyModule_FromSlotsAndSpec and the new export hook will only allow up to one
Py_mod_exec slot.
(Arrays in PyModuleDef.m_slots may have more; this will not change.)

A new function will be added to run the exec slots for a module – like
PyModule_ExecDef, but for modules created using slots::

int PyModule_ExecSlots(PyObject *, PyModuleDef_Slot*)

Tokens

Module objects will optionally store a “token”: a void* pointer
similar to Py_tp_token for types.

If specified, using a new Py_mod_token slot, the token must:

  • outlive the module, so it’s not reused for something else while the module
    exists; and
  • “belong” to the extension module where the module lives, so it will not
    clash with other extensions.

(Typically, it should point to a static constant.)

Modules created from a PyModuleDef will have the token set to that
definition. An explicit Py_tp_token slot will we rejected for these.
(Internally, the token and def will share storage.)

For modules created via a PyModuleSlots_* export function, the token
will be set to the slots array by default.
An explicit Py_tp_token slot will override this.
(This does not apply to modules created by PyModule_FromSlotsAndSpec,
as that function’s input is not expected to outlive the module.)

A PyModule_GetToken function will be added to get the token.

A new PyType_GetModuleByToken function will be added, with a signature
like PyType_GetModuleByDef but with a void *token argument,
and the same behaviour except matching tokens, rather than only defs.

New slots

For each field of the PyModuleDef struct, except ones from
PyModuleDef_HEAD_INIT, a new slot ID will be provided: Py_mod_name,
Py_mod_doc, Py_mod_clear, etc.
See :ref:api-summary for a full list.

All new slots – these and Py_tp_token discussed above – may not be
repeated in the slots array, and may not be used in a
PyModuleDef.m_slots array.
They may not have NULL value (instead a slot can be omitted entirely).

Bits & Pieces

A PyMODEXPORT_FUNC macro will be added, similar to PyMODINIT_FUNC
macro, but with int as the return type.

… _api-summary:

New API summary

The following functions will be added::

PyObject *PyModule_FromSlotsAndSpec(PyModuleDef_Slot *, PyObject *spec)
PyMODSLOTS_FUNC
int PyModule_ExecSlots(PyObject *, PyModuleDef_Slot*)
void *PyModule_GetToken(PyObject *)
PyObject *PyType_GetModuleByToken(PyTypeObject *type, void *token)

A new macro will be added::

PyMODEXPORT_FUNC

And new slot types (#define\d names for small integers):

  • Py_mod_name (equivalent to PyModuleDef.m_name)
  • Py_mod_doc (equivalent to PyModuleDef.m_doc)
  • Py_mod_size (equivalent to PyModuleDef.m_size)
  • Py_mod_methods (equivalent to PyModuleDef.m_methods)
  • Py_mod_traverse (equivalent to PyModuleDef.m_traverse)
  • Py_mod_clear (equivalent to PyModuleDef.m_clear)
  • Py_mod_free (equivalent to PyModuleDef.m_free)
  • Py_mod_token (see above)

All this will be added to the Limited API.

Backwards Compatibility

None known.
Only new API is added.

Security Implications

None known

How to Teach This

Rewrite the “Extending and Embedding” tutorial to use this.

Reference Implementation

Not yet.

Rejected Ideas

(Move Open issues here)

Open Issues

The inittab

We’ll need to allow PyModuleDef-less slots in the inittab –
that is, add a new variant of PyImport_ExtendInittab.
Should that be part of this PEP?

The inittab is used for embedding, where a common/stable ABI is not that
important. So, it might be OK to leave this to a later change.

Exporting a data pointer rather than a function

This proposes a new module export function, which is expected to
return static constant data.
That data could be exported directly as a data pointer.

With a function, we avoid dealing with a new kind of exported symbol.

A function also allows the extension to introspect its environment in a limited
way – for example, to adjust the returned data to the current Python version.

Improving slots in general (out of scope)

Slots – and specifically the existing PyModuleDef_Slot – do have a few
shortcomings; addressing them is left to a different PEP.
The most important are:

  • Type safety: void * is used for data poiners, function pointers
    and small integers, requiring casting that is technically undefined
    behaviour in C – but works in practice on all relevant architectures.
    (For example: Py_tp_doc marks a string; Py_mod_gil an integer.)

  • Limited forward compatibility: if an extension provides a slot ID that’s
    unknown to the current interpreter, module creation will fail.
    This makes it cumbersome to use “optional” features – ones that should only
    take effect if the interpreter supports them. (Recently added slots
    Py_mod_gil and Py_mod_multiple_interpreters would be great candidates
    for this.)

    A workaround is to check Py_Version in the export function,
    and return a slot array tailored to current interpreter.

Copyright

The usual.