Add stable API's for easy appending of frozen modules when embedding

Ideas
1

When embedding it is common to want to freeze custom modules without copying code from the python core itself related to also having the builtin frozen modules with them.

As such I propose API’s for appending frozen modules to these internal list structures from within the python core itself similar to AppendInitTab

Each of these apis would need to either insert the entries one place prior to the sentinels in each pointer/list the pointers point to or simply realloc them to extend the memory in them and then insert where the old sentinel was before inserting a new sentinel each time one is added.

New API’s (stable):

  • PyAPI_FUNC(int) PyImport_AppendFrozenModules(_frozen *);
    • Works exactly like PyImport_FrozenModules except the python core can handle the complex logic of setting it. Also setting a manual list for it and then setting the pointer would not be needed then when embedding. Allows appending both a single frozen entry along with a list of frozen entries to the existing list of frozen entries.
  • PyAPI_FUNC(int) PyImport_AppendFrozenAliases(_module_alias *);
    • Works exactly like the private _PyImport_FrozenAliases pointer except when one wants to add their own aliases to their frozen module they can avoid private API’s that can change or be replaced with something similar. This new API could replace this instead though as then there would be no point in depending on this private symbol from the python core when embedding. Allows appending both a single frozen alias entry along with a list of frozen alias entries to the existing list of frozen alias entries.
  • PyAPI_FUNC(int) PyImport_AppendFrozenTest(_frozen *);
    • Works exactly like the private _PyImport_FrozenTest pointer except that it allows one to append their own frozen test modules to the end of the builtin frozen test modules (without having to manually provide them yourself when embedding). Also allows appending both a single entry as well as a list of entries.
  • PyAPI_FUNC(int) PyImport_AppendFrozenBootstrap(_frozen *);
    • Works exactly like the private _PyImport_FrozenBootstrap pointer, except allows one to append their own frozen import hook to the list of default frozen bootstrap modules (currently _frozen_importlib, _frozen_importlib_external, and zipimport). Also allows appending both a single entry as well as a list of entries.
  • PyAPI_FUNC(int) PyImport_AppendFrozenStdlib(_frozen *);
    • Works for when one wants to freeze the entire default standard library instead of shipping their embedded interpreters with a zip file containing these modules or even freezing a specific few of them on top of the default frozen ones within the python core. This also allows to append a single entry as well as a list of entries.

Not only can this make embedding real world applications easier, it could also simplify some of the tests that might append to the frozen lists. This also makes it new C API user friendly as well.

Likewise with AppendInitTab this must error when one attempts to append entries to these after Py_Initialize.

Specific versions of Python I have in mind for this PEP:

  • Python 3.14.0a1+
2

[this thread probably belongs in Ideas to start with, with a C-API tag, we use the PEPs category for discussion of things that are more fleshed out and ready to be PEPs with a sponsor - moderators could move it]

Something I think that would help make the case for this are more fleshed out practical examples of how it would be used. Before and After scenarios for what you do today (which I expect… could be complex) vs what your C API code would look like after having this perhaps. As a way to practically demonstrate what it makes simpler.

Are there places internal to CPython where using these APIs would be beneficial? If so suggest some and explain how/why.

(disclaimer: I’m asking at an abstract level without the context for what all embedding requires in my head at the moment)

2 Likes
3

Would it be possible to extend the PyInitConfig API (PEP 741) with these functions, similar to PyInitConfig_AddModule()?

4

That could work as well.

5

There doesn’t seem to be any real need for the range of functions - once a module is available for importing, it’s available. There are some tricks related to how the stdlib ones get frozen, but for an embedder, simply being able to provide the name and the C function that creates/returns the module ought to be enough.

(Though being able to specify an import hook at initialization time would be interesting. I wouldn’t do it as a module, though. I’d honestly just prefer it as a C callback.)

6

True, but for most people they prefer to write the hook in python and then simply freeze it for use in their C embed code as it can be much simpler to do the hooks on the python side than it is in C depending on what it is they are doing. Something similar to how zipimport currently is compared to what it used to be (completely implemented in C).

Also likewise what if one wants their hook to subclass the zipimporter in the C type spec structure there is currently no clear way of telling python “this is a subclass of zipimport.zipimporter” as far as I am aware.

7

Sure, but PyImport_ExecCodeModule is real easy to use if you’ve got a C array containing the module. And having a slightly more generic hook allows for easier innovation (e.g. I can runtime decode or unpack the frozen module before calling exec on it).