_ProtocolMeta: why? Can it be removed?

Having a look at typing.Protocol, this seems to depend on a metatype _ProtocolMeta.

From the main branch of the CPython repository, this is what it’s reported:

class _ProtocolMeta(ABCMeta):
    # This metaclass is somewhat unfortunate,
    # but is necessary for several reasons...

I haven’t looked too far in depth (or rather I did not search thoroughly) but: what are these reasons exactly? Up to Python 3.12 (I think), that comment was actually different:

class _ProtocolMeta(ABCMeta):
    # This metaclass is really unfortunate and exists only because of
    # the lack of __instancehook__.

What changed exactly? Can this class be removed, if it’s that unfortunate?

The reason I’m asking this is: I’m thinking of the possibility of a package like zope.interfaces, which was one of the case studies brought up by PEP 544 in the first place for structural subtyping.

As far as I recall, Protocols only check for the existence of attributes, and they don’t provide strong runtime checking reliability (that’s usually delegated to runtime type checkers). But at the same time the spec seems quite restrictive on determining the very baseline of protocols as well.

Supposing one wants to implement a runtime type checker that ships its own protocol implementation which is stricter; it would require to inherit _ProtocolMeta and maybe would bar from optimizing further (as in use some C-extension code).

So I’m curious why _ProtocolMeta still exist.

Yes, I think that comment was revised because we now need it for more reasons than just the lack of __instancehook__. We (mostly I) did a lot of refactors to the internals of _ProtocolMeta in 3.12 to address longstanding complaints about how slow isinstance() checks are against runtime-checkable protocols. We now cache a lot more information about protocols at class-creation time, which necessitates defining _ProtocolMeta.__new__, _ProtocolMeta.__init__, and _ProtocolMeta.__subclasscheck__ as well as _ProtocolMeta.__instancecheck__ (the last of these has always existed).

2 Likes

It looks like I actually edited that comment to its current form in a PR to fix some pretty cursed bugs due to an interaction with the abc-module cache: gh-104555: Runtime-checkable protocols: Don't let previous calls to `isinstance()` influence whether `issubclass()` raises an exception by AlexWaygood · Pull Request #104559 · python/cpython · GitHub

I can’t remember the details, but I’m guessing that bug was triggered by one of my optimisations earlier on in the 3.12 development cycle.

@AlexWaygood thanks for taking the time to answer.

So what would be the concept behind __instancehook__? What’s the intended behaviour at runtime?

From what you’re writing, decoupling the implementation of _ProtocolMeta from the abc module could simplify its management, would it not?