Make abc.ABC a regular class by making __instancecheck__ and __subclasscheck__ class methods

[Daverball (David Salvisberg)]

@alicederyn

I think that was more a problem with the other suggested solution of adding these methods to the class as well as the metaclass. My suggestion adds a single dictionary lookup; any smarts need to happen in the logic that registers classes in that dictionary.

__instancecheck__ and __subclasscheck__ on ABCMeta already does something like that, except in C, so it’s fast, all you’re really changing is moving the code from ABCMeta^[1] to isinstance/issubclass, which is fine if you’re allowing all classes to register virtual subclasses, but it will add static overhead to all isinstance calls, there’s not really a way to avoid that or trade it against the dynamic dispatch overhead, since that is part of Python’s data model.

---

1. or rather _abc if we’re talking about the C portion ↩︎

[alicederyn (Alice)]

@Daverball

that is part of Python’s data model

You mean by this that __instancecheck__ and __subclasscheck__ on metaclasses are an agreed protocol, right?

[Daverball (David Salvisberg)]

Yes: 3. Data model — Python 3.12.2 documentation

[alicederyn (Alice)]

Python APIs that have been used by more code than that have been removed in the past. It just takes good reasons, deprecation and will. (not saying that this would happen here!)

[blhsing (Ben Hsing)]

@mikeshardmind

I generally think the ABC method is worse than using protocols (runtime checkable ones, even, if needed), but ABC came first, and comes with runtime enforcement that subclasses implemented the abstract methods.

ABC and Protocol serve entirely different use cases. ABC enforces structure of a subclass, while Protocol enforces structure of an instance. Moreover, ABC implements actual logical behaviors while Protocol does none.

I think my recent use case of ABC demonstrates its purpose well:

Context manager protocol extension (2nd attempt)

Note that we can make it a mixin class too for easier usage: from abc import ABC, abstractmethod from contextlib import contextmanager class ContextManagerMixin(ABC): def __init__(self): self._contexts = [] def __init_subclass__(cls): cls.__with__ = contextmanager(cls.__with__) @abstractmethod def __with__(self): ... def __enter__(self): context = self.__with__() self._contexts.append(context) return context.__enter__() def __…

But now imagine how someone who wishes to use this mixin class to easily gain this small capability of a context manager, only to end up with a metaclass conflict with a big metaclass-based base class that actually legitimately needs a metaclass, all because of __instancecheck__ and __subclasscheck__ that aren’t even used.

[chepner (Clint Hepner)]

@MegaIng

think most people would be surprised if isinstance(obj, cls) doesn’t guarantee that obj is an instance of cls.

But that’s the entire point of providing __instancecheck__. I’m asking why that can’t be a method that is either provided by object and does the “expected” thing unless overriden by ABC, or simply not provided at all except by ABC.

[MegaIng (Cornelius Krupp)]

So you are suggesting moving __instancecheck__ from the metaclass to the class? If so, go back to the beginning of the thread and read the counter arguments there. I thought you wanted to move the implementation of ABCMeta into type, which would add the ability to have virtual subclasses to all types in python.

[chepner (Clint Hepner)]

@MegaIng

So you are suggesting moving __instancecheck__ from the metaclass to the class? If so, go back to the beginning of the thread and read the counter arguments there.

I think I was anticipating that the necessary distinction between “subclass of abstract base class” and “subclass of ABC” could be handled by the definition of __subclasscheck__, and ABCMeta was defined because it was the simpler option. But if the distinction is truly impossible (or at least undesirably messy), then yes, I would argue for moving the machinery into type itself.

[gerardw (Gerardwx)]

@blhsing

This is really unfortunate since the risk of a metaclass conflict effectively discourages developers from using abstract classes even when it makes total sense to do so, affecting both code reusability and readability in the long run.

Abstract classes can easily be implemented using protocols:

from abc import abstractmethod
from typing import Protocol, runtime_checkable


@runtime_checkable
class ExampleA(Protocol):

    @abstractmethod
    def x(self) -> int:
        pass


@runtime_checkable
class ExampleB(Protocol):
    @abstractmethod
    def y(self) -> int:
        pass


class Implement(ExampleA, ExampleB):
    def x(self):
        return 3

    def y(self):
        return 9


impl = Implement()
print(f"{impl.x()} {impl.y()}")
print(isinstance(impl, ExampleA))
print(isinstance(impl, ExampleB))

[MegaIng (Cornelius Krupp)]

• Protocols still have a metaclass
• Protocols imply structural subtyping, which isn’t applicable to quite a few usecases of ABCs (see this post)

[mikeshardmind (Michael H)]

The example linked for “not applicable” is using an ABC as a base class for a mixin Simply remove the ABC from it if it’s meant to be a concretely reusable mixin, and have a single ABC or protocol that represents the totality of things people need to implement. The replies to that in that thread also call out that that just isn’t the way ABC was designed to be used.

Note that the abstractmethod decorator works without ABC as a base

[MegaIng (Cornelius Krupp)]

So ABCs are completely obsolete, and everything should just use Protocol^[1]? Because if this isn’t a valid use of ABCs, I have no idea what is.

• It can’t be just defining a structure that types should match.
• It can’t be to be used as a MixIn, since as you pointed out there are virtual subclasses which don’t have to use it.

@mikeshardmind

Note that the abstractmethod decorator works without ABC as a base

For typecheckers maybe (even that isn’t consistent, IIRC), but not at runtime.^[2]

---

1. Not that would solve the problem mentioned in OP ↩︎

2. Despite the facilities being there since they are part of object.__new__ ↩︎

[mikeshardmind (Michael H)]

I don’t think ABCs are obsolete, but I do think they shouldn’t be used for reusable code snippets like what was linked. They can still be useful for defining a total needed interface, but this only works well in cases where the provider of the ABC knows what needs to exist in the final class.

The design of the linked mixin requires first implementing the mixin by subclassing the mixin to provide __with__ and only then using the mixin as a mixin*, this leads to deep inheritance trees. We have a better way to augment types with reusable functionality like this with class decorators. Compare functools.total_ordering for augmenting an existing type with functionality as long as it provides a specific set of initial capability.

* Technically you can use the class provided directly as a mixin, but then that’s an even stronger reason to just document it, remove the ABC base and the abstract implementation of __with__ if it isn’t abstract anymore and provides all the functionality it is intended to. The abstract non-implementation of __with__ here in a mixin makes the order of the mixin in MRO more important than it should be, and why the reasonable use of it requires subclassing it first, then using it as a mixin. As a mixin, the order could cause it to clobber a __with__ provided in another base.

[gerardw (Gerardwx)]

I’m not following. What’s different from the linked example that makes it not structural subtyping?

from abc import abstractmethod
from typing import Protocol, runtime_checkable


class Animal(Protocol):

    @abstractmethod
    def name(self)->str:
        pass

    @abstractmethod
    def legs(self)->int:
        pass

    def walks(self)->str:
        return f"A {self.name()} walks on {self.legs()} legs."


class Dog(Animal):
    def name(self) -> str:
        return "dog"

    def legs(self) -> int:
        return 4

class Chicken(Animal):
    def name(self) -> str:
        return "chicken"

    def legs(self) -> int:
        return 0

class Snake(Animal):
    def name(self) -> str:
        return "snake"

    def legs(self) -> int:
        return 0

for animal in (Dog(),Chicken(),Snake()):
    print(animal.walks())

Interesting The ABC PEP, in 2007, noted

ABCs are not intrinsically incompatible with Interfaces, but there is considerable overlap. For now, I’ll leave it to proponents of Interfaces to explain why Interfaces are better. I expect that much of the work that went into e.g. defining the various shades of “mapping-ness” and the nomenclature could easily be adapted for a proposal to use Interfaces instead of ABCs.

which seems to have happend in the 2017 Protocols PEP

[ruro]

If the only reason why we can’t implement ABC without metaclasses is because it’s needed for virtual inheritance, then why don’t we start by separating virtual inheritance and abstract method support.

It’s not immediately clear to me, why was virtual inheritance bundled together with abstract classes in the first place. The motivating examples for adding virtual inheritance in PEP-3119 seem to me like they should be represented by Protocols rather than Abstract classes. So, it might just be historical happenstance that ABC includes virtual inheritance support rather than an intentional design decision.

So here is my proposal:

1. Extract just the virtual inheritance parts of ABCMeta into a new meta class VirtualMeta. Also provide a helper Virtual class similar to the ABC helper for ABCMeta.

2. Extract just the abstract class parts of ABCMeta (without virtual inheritance support) into a new normal class Abstract (implemented using __init_subclass__).

3. Re-implement ABCMeta and ABC in terms of Abstract, Virtual and VirtualMeta classes, keeping the current behaviour. The new inheritance hierarchy will be an extension of the old hierarchy, so every existing piece of code should continue working.

4. (optional) Emit deprecation warnings whenever someone imports ABC or ABCMeta, suggesting that they should use Abstract/Virtual/VritualMeta directly instead.

With these changes, we would have the following:

• Abstract - a normal class that implements just the abstract base class functionality of ABC (i.e. setting __abstractmethods__). It doesn’t use metaclasses. Most people that don’t need virtual inheritance can just replace ABC with Abstract.

• VirtualMeta and Virtual implement just the virtual inheritance part of ABC (i.e. register, isinstance and issubclass). As a bonus, people can now create non-abstract virtual inheritance hierarchies.

• ABCMeta inherits from VirtualMeta and automatically adds Abstract and Virtual as base classes in its __new__ method. ABC is still just a helper class that does metaclass=ABCMeta.

I implemented a proof-of-concept version of this idea here: GitHub - RuRo/python-abstract-virtual-poc
In particular, see commit f64ac203 for the proposed changes to the abc module

[alicederyn (Alice)]

@ruro

If the only reason why we can’t implement ABC without metaclasses is because it’s needed for virtual inheritance, then why don’t we start by separating virtual inheritance and abstract method support

Correct me if I’m wrong, but as much as I like the idea, won’t this be a major break to the language since we have some pretty critical classes in collections.abc that do both?

[ruro]

Only if we decide to deprecate ABC and even if we do, I think that it should be possible to incrementally transition to Abstract+Virtual and/or Abstract. The trick is that with my proposed implementation, ABC itself still inherits from Abstract and Virtual.

There would be a transition period, where ABC/ABCMeta are deprecated for external use, but the stdlib still uses ABC/ABCMeta internally. During this period, users are encouraged to switch to using Abstract+Virtual and/or Abstract and since ABC inherits from those classes, they will interoperate seamlessly (hopefully).

At the same time, we can also add deprecation warnings to .register/isinstance/issubclass methods on ABC classes that we’d like to eventually transition to be just Abstract (without Virtual). This probably doesn’t apply to a lot of classes in collections.abc, but there are still some abstract classes in the stdlib that don’t seem like they need to be virtual (for example, contextlib.AbstractContextManager).

After this grace period is over, hopefully all user code should have switched over to Abstract+Virtual and/or Abstract at which point the stdlib can also switch from ABC to Abstract+Vritual and/or Abstract and even eventually remove ABC/ABCMeta completely, if we wish to do so.

Now, I’ll admit that this all is quite involved, but I don’t expect most user code to actually be affected by this. After all, most users of collections.abc just inherit from (or register) those classes without mentioning ABC and ABCMeta themselves. And code that does mention ABC/ABCMeta explicitly should be quite easy to upgrade (the hardest part is deciding whether to s/ABC/Abstract/g or s/ABC/Abstract, Virtual/g).

---

Edit: it’s probably quite a bit harder to do than what I am letting on, but I’d say that the biggest roadblock in this case is the sheer amount of churn that deprecating ABC would produce rather than the compatibility breakage on its own.

So unfortunately, completely deprecaintg ABC isn’t very likely to succeed. But adding the simple Abstract class to the stdlib as a lightweight alternative to ABC might still be desireable.

[alicederyn (Alice)]

Would you also want to deprecate direct inheritance from Protocols as a way to inherit default implementations (and to indicate intended compatibility to type checkers)?

[ruro]

I am honestly not sure if I understand your question. My (and seemingly OPs) primary agenda here is to find a way to provide useful development tools such as abstract base classes, virtual inheritance, protocols, etc while avoiding the use of metaclasses wherever possible. I think the first post in this thread explains the motivation for this goal pretty well – metaclasses can be quite clunky and there are a lot of restriction and hidden footguns associated with them.

I was not suggesting to deprecate ABC/ABCMeta because I am against abstract virtual classes, but rather because I believe that the fact that ABC provides two features (abstract classes and virtual inheritance) to be mostly a historical accident. So if one of these features (virtual inheritance) could not be implemented without the use of metaclasses, then IMO we ought to decouple these features so that the people who don’t need virtual inheritance don’t have to incur the associated costs.

Notice here, that I was suggesting to deprecate an imperfect implementation (ABC), not the idea of abstract classes with virtual inheritance. If you want to have abstract classes WITH virtual inheritance, that’s perfectly fine and nobody is stopping you from doing that (as long as you are explicitly opting in to these features by inheriting from Abstract and from Virtual).

Similarly, I have nothing against protocols (or even Protocols). Whether they are used as simple type hints or with @runtime_checkable or whether you are inheriting directly from them (and therefore getting all the features and restrictions of ABCMeta).

All I am really saying is this:

1. if we can provide a subset of such development features/tools without using metaclasses, we should
2. if some of these features were previously implemented with metaclasses, we should try to encourage users to migrate to the non-metaclass alternatives, and we should try to avoid “locking in” the users by overusing these metaclass-based features in the stdlib and other libraries

Of course, retroactively applying item 2 to pre-existing stdlib code leads to migrations, deprecations and possibly breaking backwards compatibility, so I’m not that strongly invested in “deprecating everything”.

Truth be told, I’ve implemented the ABC → Abstract + Virtual proposal as a fun distraction from work and some of the problems with completely replacing ABC only became apparent once I’ve already finished with the implementation.

Still, the non-metaclass Abstract class could still be potentially useful as a lightweight alternative to ABC rather than a complete replacement.

[blhsing (Ben Hsing)]

@ruro

With these changes, we would have the following:

• Abstract - a normal class that implements just the abstract base class functionality of ABC (i.e. setting __abstractmethods__). It doesn’t use metaclasses. Most people that don’t need virtual inheritance can just replace ABC with Abstract.
• VirtualMeta and Virtual implement just the virtual inheritance part of ABC (i.e. register, isinstance and issubclass). As a bonus, people can now create non-abstract virtual inheritance hierarchies.
• ABCMeta inherits from VirtualMeta and automatically adds Abstract and Virtual as base classes in its __new__ method. ABC is still just a helper class that does metaclass=ABCMeta.

Phenomenal idea! Re-implementing ABCMeta with two base classes is what I should’ve thought of when @MegaIng suggested a separate ABCWithVirtual class, to which I replied with a concern about backward compatibility.

The only thing I want to point out is that Virtual can just be a helper class that does metaclass=VirtualMeta, to keep the logics in one place.

Such an elegant solution this is. Can we go forward with a PEP?

@MegaIng

If the primary reason for ABC to have a metaclass is to allow having a registry of virtual subclasses, wouldn’t it be an option to deprecate this and have a separate ABCWithVirtual class that keeps the current behavior (and metaclass)? I think that most users of ABC are perfectly happy without having the ability to register virtual subclasses. This would remove the need for the two magic methods.