dlech
(David Lechner)
August 10, 2023, 12:10am
1
I’m working on updating a binary extension for 3.12 and ran in to the new restrictions of not being allowed to use a metaclass with a custom tp_new in a C type created with PyType_From*()
.
I have some types that are various collections (e.g. Mapping, Sequence, etc.) and I was using the abc.collections
abstract classes as a base class so that I didn’t have to implement all of the mixin methods they provide in C. However, now this code is failing in Python 3.12 due to the new restriction since abc.ABCMeta
has a custom tp_new
.
Is there a different way I can get the mixin methods without using the abstract base classes?
dlech
(David Lechner)
August 10, 2023, 12:35am
2
opened 02:53PM - 09 Mar 23 UTC
python
22.x
<!--
NOTE: this form is for bug reports only.
For questions or troubleshooti… ng, please post on the protobuf mailing list:
https://groups.google.com/forum/#!forum/protobuf
Stack Overflow is also a useful if unofficial resource https://stackoverflow.com/questions/tagged/protocol-buffers
-->
**What version of protobuf and what language are you using?**
Version: 3.19.6 – but I searched the source code and verified that the current release (22.1) and `main` should also be affected
Language: Python
**What operating system (Linux, Windows, ...) and version?**
Fedora Linux Rawhide/39, plus experimental Python 3.12: https://copr.fedorainfracloud.org/coprs/g/python/python3.12/
**What runtime / compiler are you using (e.g., python version or gcc version)**
Python 3.12.0a5, GCC 13.0.1
**What did you do?**
Steps to reproduce the behavior:
1. Build the C++ version of the Python protobuf extension.
2. `from google.protobuf.pyext import _message`
This import occurred in the generated proto bindings in `googleapis-common-protos`; see downstream bug https://bugzilla.redhat.com/show_bug.cgi?id=2176158.
**What did you expect to see**
Successful import with no output.
**What did you see instead?**
```
[…]
File "/builddir/build/BUILDROOT/python-googleapis-common-protos-1.58.0-2.fc39~bootstrap.x86_64/usr/lib/python3.12/site-packages/google/api/annotations_pb2.py", line 20, in <module>
from google.protobuf import descriptor as _descriptor
File "/usr/lib64/python3.12/site-packages/google/protobuf/descriptor.py", line 47, in <module>
from google.protobuf.pyext import _message
TypeError: Metaclasses with custom tp_new are not supported.
Added the new limited C API function PyType_FromMetaclass(), which generalizes the existing PyType_FromModuleAndSpec() using an additional metaclass argument. (Contributed by Wenzel Jakob in gh-93012.)
The metaclass is used to construct the resulting type object. When metaclass is NULL, the metaclass is derived from bases (or Py_tp_base[s] slots if bases is NULL, see below). Note that metaclasses that override tp_new are not supported.
https://github.com/python/cpython/issues/93012
https://docs.python.org/3.12/whatsnew/3.12.html
```
Make sure you include information that can help us debug (full error message, exception listing, stack trace, logs).
**Anything else we should know about your project / environment**
See https://github.com/python/cpython/issues/93012.
Grepping for `tp_new` in `main` gives:
```
python/google/protobuf/pyext/descriptor.cc
1600: nullptr, // tp_new
python/google/protobuf/pyext/unknown_field_set.cc
197: unknown_field_set::New, // tp_new
python/google/protobuf/pyext/message_factory.cc
293: message_factory::New, // tp_new
python/google/protobuf/pyext/field.cc
126: nullptr, // tp_new
python/google/protobuf/pyext/message.cc
275: ScopedPyObjectPtr result(PyType_Type.tp_new(type, new_args.get(), nullptr));
502: message_meta::New, // tp_new
2762: cmessage::New, // tp_new
python/google/protobuf/pyext/descriptor_pool.cc
750: cdescriptor_pool::New, // tp_new
python/google/protobuf/pyext/descriptor_containers.cc
596: nullptr, // tp_new
783: nullptr, // tp_new
929: nullptr, // tp_new
```
…suggests adding a new type. For example, if you define Spam
that inherits from abc.collections.Mapping
in C using PyType_From...
, change it to BaseSpam
, remove the interitance and then call the metaclass to create a new class Spam
that inherits from both.
dlech
(David Lechner)
August 10, 2023, 12:41am
3
opened 02:53PM - 09 Mar 23 UTC
python
22.x
<!--
NOTE: this form is for bug reports only.
For questions or troubleshooti… ng, please post on the protobuf mailing list:
https://groups.google.com/forum/#!forum/protobuf
Stack Overflow is also a useful if unofficial resource https://stackoverflow.com/questions/tagged/protocol-buffers
-->
**What version of protobuf and what language are you using?**
Version: 3.19.6 – but I searched the source code and verified that the current release (22.1) and `main` should also be affected
Language: Python
**What operating system (Linux, Windows, ...) and version?**
Fedora Linux Rawhide/39, plus experimental Python 3.12: https://copr.fedorainfracloud.org/coprs/g/python/python3.12/
**What runtime / compiler are you using (e.g., python version or gcc version)**
Python 3.12.0a5, GCC 13.0.1
**What did you do?**
Steps to reproduce the behavior:
1. Build the C++ version of the Python protobuf extension.
2. `from google.protobuf.pyext import _message`
This import occurred in the generated proto bindings in `googleapis-common-protos`; see downstream bug https://bugzilla.redhat.com/show_bug.cgi?id=2176158.
**What did you expect to see**
Successful import with no output.
**What did you see instead?**
```
[…]
File "/builddir/build/BUILDROOT/python-googleapis-common-protos-1.58.0-2.fc39~bootstrap.x86_64/usr/lib/python3.12/site-packages/google/api/annotations_pb2.py", line 20, in <module>
from google.protobuf import descriptor as _descriptor
File "/usr/lib64/python3.12/site-packages/google/protobuf/descriptor.py", line 47, in <module>
from google.protobuf.pyext import _message
TypeError: Metaclasses with custom tp_new are not supported.
Added the new limited C API function PyType_FromMetaclass(), which generalizes the existing PyType_FromModuleAndSpec() using an additional metaclass argument. (Contributed by Wenzel Jakob in gh-93012.)
The metaclass is used to construct the resulting type object. When metaclass is NULL, the metaclass is derived from bases (or Py_tp_base[s] slots if bases is NULL, see below). Note that metaclasses that override tp_new are not supported.
https://github.com/python/cpython/issues/93012
https://docs.python.org/3.12/whatsnew/3.12.html
```
Make sure you include information that can help us debug (full error message, exception listing, stack trace, logs).
**Anything else we should know about your project / environment**
See https://github.com/python/cpython/issues/93012.
Grepping for `tp_new` in `main` gives:
```
python/google/protobuf/pyext/descriptor.cc
1600: nullptr, // tp_new
python/google/protobuf/pyext/unknown_field_set.cc
197: unknown_field_set::New, // tp_new
python/google/protobuf/pyext/message_factory.cc
293: message_factory::New, // tp_new
python/google/protobuf/pyext/field.cc
126: nullptr, // tp_new
python/google/protobuf/pyext/message.cc
275: ScopedPyObjectPtr result(PyType_Type.tp_new(type, new_args.get(), nullptr));
502: message_meta::New, // tp_new
2762: cmessage::New, // tp_new
python/google/protobuf/pyext/descriptor_pool.cc
750: cdescriptor_pool::New, // tp_new
python/google/protobuf/pyext/descriptor_containers.cc
596: nullptr, // tp_new
783: nullptr, // tp_new
929: nullptr, // tp_new
```
…suggests to manually register the class and add the methods later
MutableMapping.register(ScalarMapContainer)
# Mixin the methods manually.
ScalarMapContainer.__contains__ = MutableMapping.__contains__
ScalarMapContainer.keys = MutableMapping.keys
ScalarMapContainer.items = MutableMapping.items
ScalarMapContainer.values = MutableMapping.values