New PEP: implementation independent native code invocation and data exchange ABI standard (not sure if accurate)

Ideas
1

Python is a “glue” language, its dynamic nature has programming advantage and performance disadvantage.

The best way to use Python is write Python code for high-level stuff and use native programming language like C, Rust, Zig, V for low level stuff. Currently, there is library like PyO3 and so on.

However there is a lot of limitation in those library.

  1. The “FFI” library is implementation specific. For example, PyO3, it highly depend on specific version of CPython. If you use a different version of CPython, there is some work for you to do. If you don’t use CPython, for example use another Python Implementation, This framework may not work.
  2. The project structure is rigid. When using something like PyO3. you must make your project structure following certain pattern, then finally produce a complete wheel package. The majority Python programmer does not write a whole Python project for a installable package and pack it up then install it. They are edit Python source file iteratively and run it locally.
  3. It is really ridiculous when you want to stick some item on the wall. You need to totally redesign this item and manufacture a new item to fit the glue you are going to use. As a glue language, Python should be designed to glue other native programming language as a feature of Python programming language itself not the tricks of certain variant of implementation.

It would be nice to add the feature in Python Standards (no matter what implementation is used) to satisfy the following capabilities:

  • The interface is universal across all variant and version of Python implementation (There might be protocol version update and may not backward-compatible, but it is not bounded with Python implementation). The overall effect is in some extent like JSON, but it is not a structured string, It is a lively data structure with in-memory representation, they are unified no matter what Python variant is used and what low level native language is used.
  • This mechanism is transparent to users, there is modules in standard library to support it. If user want, they can design a toy native programming language, and use Python to write a compiler for it. Then write a module with custom language then compile and import it. This mechanism provide user with maximum flexibility.
  • almost zero-cost abstraction. Even if it not depend on CPython tricks. But the central idea of this mechanism is still the dynamic linking feature provided by operating system. The detailed format will be slightly different. It doesn’t introduce new stuff in nature. It doesn’t spawn new process, nor launch a VM, and nor I/O operations is involved. It just make some basic data representation conversion and invoke the method in dynamic library.

This is just a raw idea. If it is valuable, it can take a discussion and make further steps.

2

This is a really awesome and very ambitious goal… that many people have been working toward for years. For example, are you aware of HPy?

1 Like
3

Thanks for letting me know.

But, does project only solve the goal of Python implementation independence.

For providing user the transparency and flexibility, it seems still a long way to go.

The overall effect is make native code development for Python language as easy as writing the language of what the native code itself. And provide user option for meta-programming. Same flexibility as what (Importing Modules — Python 3.11.3 documentation) and (Python Language Services — Python 3.11.3 documentation) do. It makes native extension development be Pythonic.

And make Python project with heterogeneous code base possible and portable.

4

I’m not sure I’m following, are you referring to some kind of an offical library like Cython?

5

I think “have a better C API” is something many people would like, but it is by no means easy. A PEP would be for a more specific topic, like “improve the C API by doing X and Y” or “add Z to the C API to solve Y” or “add an stdlib module that does this and that”. As far as I understand your original post, you have something that I would tend to call a wish more than an idea; to make a PEP I believe (NB I’m not in the core Python team) that it needs to be much more specific on how you want to achieve your goals, and have a reference implementation. There are great technical challenges related to the Python C API. I would expect that you need to propose a concrete plan in order to help improving it.

1 Like
6

I maintain PyCXX that tries to do just that if you code in C++, and there are other libs that aim to do the same. See https://cxx.sourceforge.net/ there are examples showing using python objects in C++ that looks like you ate writing python.

7

No, Cython is a specific language. And what I hope is something not limit on specific language, It provide a common API/ABI and a officially support mechanism to dynamically recompile and load the generated modules from native language source files.

User can invent their own toy language, and using the compiler entirely write in Python (the conventional interface stuff utilize the method in standard library) to generate loadable module.

And the user have more power to intervene the procedures. Which fully satisfy the meta-programming nature of Python.

8

Have you looked into what’s happening with WebAssembly? That’s a machine code spec for a VM that’s designed to be sandboxable (hence the application to the web), and it’s language-independent. Search this Discourse for previous threads on the topic; IMO it’s one of the most exciting web technologies under development, check it out!