Abstract Base Class for `queue.Queue` and `multiprocessing.Queue`

[ebonnal (Enzo Bonnal)]

The interfaces of queue.Queue and multiprocessing.Queue have been designed consistently, but one still needs to use Union[queue.Queue, multiprocessing.Queue] to accept either one. Could we make it more elegant by adding something like the following to collections.abc?

class Queue(metaclass=ABCMeta):
    @abstractmethod
    def __init__(self, maxsize: int): ...
    @abstractmethod
    def qsize(self): ...
    @abstractmethod
    def empty(self): ...
    @abstractmethod
    def full(self): ...
    @abstractmethod
    def put(self, item, block=True, timeout=None): ...
    @abstractmethod
    def get(self, block=True, timeout=None): ...
    @abstractmethod
    def put_nowait(self, item): ...
    @abstractmethod
    def get_nowait(self): ...

[ajoino (Jacob Nilsson)]

I think the better way would ve to create some sort of Queue Protocol in this case, just like the other container types

[NeilGirdhar]

If you’re going to use it with isinstance, it’s superior to have a common base class the way he has it (but with the missing output types filled in). That way there will be:

• no false positives,
• type checking the whole interface,
• faster type checking, and
• more obvious declared intent when subclassing

Protocols are a great idea, and they have their place, but I don’t think this is it.

[xitop]

@ebonnal

but one still needs to use Union[queue.Queue, multiprocessing.Queue] to accept either one

Yes, but it isn’t that bad when one defines it as a type and then uses the short name.

type QueueType = queue.Queue | multiprocessing.Queue

AFAIK this syntax is 3.12+, but without the type keyword it works on older version as well.

[ajoino (Jacob Nilsson)]

I think we really don’t see eye to eye on ABCs versus Protocols. My thinking is that if Protocols are good enough for most of the other types that implement an interface it’s good enough for the Queues. Make it runtime checkable if using isinstance is a very important use case. I suspect it isn’t because I see the regular Queue all the time but multiprocessing Queue very rarely, and you almost certainly know if you’re using one or the other.

[NeilGirdhar]

@ajoino

if Protocols are good enough for most of the other types that implement an interface it’s good enough for the Queues.

They’re not used just because they’re “good enough”. Protocols are often used because client code can’t be easily changed, or people don’t want to force big changes. This would be a small change that’s in the library alone.

And I don’t see why you would aim for a solution that’s “good enough” as opposed to the more ideal solution that has many advantages. The only disadvantage to ABCs is that it makes the MRO one entry bigger.

@ajoino

Make it runtime checkable if using isinstance is a very important use case.

Yes, I’m assuming you’re making it runtime checkable. ABCs are already “runtime checkable”. Please see the list of disadvantages.

[NeilGirdhar]

Yeah that’s reasonable. But unfortunately, if you use a type alias, you won’t be able to use it in isinstance(x, QueueType) like you would if you used a union or tuple. As far as I know, there’s no style guideline on what to do here.

Should you have:

type QueueType = queue.Queue | multiprocessing.Queue
QueueTypeForIsinstance = (queue.Queue, multiprocessing.Queue)

even if it’s annoying to keep them synchronized, or just a single union type?

QueueType = queue.Queue | multiprocessing.Queue  # Can't be a type alias since it's used in isinstance.  Ruff may rightly warn you when you use this as an annotation.

[ajoino (Jacob Nilsson)]

ABCs have more disadvantages, like not being able to do multiple inheritance from another class with an ABCMeta mtea class. Overall we just disagree over which approach is the best.

Also, the “good enough” was not meant literally, I think it’s the “best enough” solution.

[NeilGirdhar]

@ajoino

like not being able to do multiple inheritance from another class with an ABCMeta mtea class.

These ABCs have metaclass type, so if another ABC comes along (theoretically) with another metaclass, there won’t be any inheritance failure.

@ajoino

I think it’s the “best enough” solution.

I see that, but I don’t think you’ve made a very good argument for that since your argument boils down to “there are other protocols”.

[ajoino (Jacob Nilsson)]

My full argument is this: I think ABCs make sense mostly when there is some shared functionality where the details are implemented in a subclass. Gor example, a generic HTML parser might expect to be subclassed by a class that implements the HTTP functionality. However, when it comes to just a name for classes that fulfill an interface, like the Queues, the Mappings, Iterators etc. That’s just structure, and I don’t think ABCs should be used anymore for just structure.

If you need to know exactly what implementation of the interface you’re using in a function, which I find is very rare, then I think the correct type in the function signature is A | B and not the full Protocoll. If you need to know the exact type you already know what types to look for.

Also, most uses of ABCs and mixins in general are better served by using composition instead, in which case Protocols becomes extra helpful since you probably don’t care if you mapping is a dict or a special mapping. If you do you just specify the type.

The one place I can say that yes, ABCs are superior is if you don’t use a type checker because Python will fail if you don’t implement the interface correctly. Luckily, that does not apply here since they are provided by the language. OP does not speak about user extensibility, only elegance (italic for extre elegance), in which case I think Protocols, being structural, are the most elegant solution.

Lasty, maybe things have changed in later versions of Python or my memory is going bad, but IIRC last time I tried to inherit from both an ABC and another class that already had a metaclass Python said no and failed. There were workarounds that involved writing a metaclass for accepting multiple metaclasses but that was a pain and I did things differently.

I don’t think any of the things I described are new to you, it’s the second time we’ve argued about Protocols vs ABCs so I know your stance and didn’t want to repeat the same discussion endlessly.

[NeilGirdhar]

@ajoino

I think ABCs make sense mostly when there is some shared functionality where the details are implemented in a subclass.

I think you’re talking about mixins, which are a type of ABC. However, there are plenty of ABCs that are not mixins. For example Iterator.

@ajoino

I don’t think ABCs should be used anymore for just structure.

Do you have a reason for that? Or is just that you think protocols are exciting because they’re newer?

@ajoino

Also, most uses of ABCs and mixins in general are better served by using composition instead,

I don’t think this has anything to do with the topic in this post.

@ajoino

in which case I think Protocols, being structural, are the most elegant solution.

You haven’t given one reason why. I’ve given four reasons why they’re not.

@ajoino

IIRC last time I tried to inherit from both an ABC and another class that already had a metaclass Python said no and failed.

That only happens if the two parent classes have metaclasses that don’t inherit from each other, but since all metaclasses inherit from type, it can’t happen here.

@ajoino

, it’s the second time we’ve argued about Protocols vs ABCs so

Yes, but this is a different debate. The other debate was about multiple inheritance in general. This is multiple interface inheritance, which has none of the drawbacks. Multiple interface inheritance is almost universally accepted as good design.

I think this is why you’re confused. It seems like you though: “Multiple inheritance is dangerous, so I should avoid it as bad design”, and you let that thought generalize to multiple interface inheritance. This is why you should go back to the actual reasons that you think something is good or bad design to verify that your conclusion still holds.

[ajoino (Jacob Nilsson)]

The reason I prefer Protocols is not that they are newer, it’s that they are a way to formalize duck-typing in a way that align with how duck-typing works in Python without needing to inherit from a common ABC. This makes Protocols more flexible IMO, not because you can do more things with but because they do less. Thus, I don’t agree with your list of reasons why Protocols are less flexible than ABCs. I think that if you find yourself in a situation where you need a runtime chackable protocol your API design is probably not so great IMO. In general, using inheritance instead of composition is almost always the wrong design choice IMO.

[NeilGirdhar]

@ajoino

The reason I prefer Protocols is not that they are newer,

I thought it was something like that, but I don’t see why that would matter.

Everything has its place. Protocols are not a replacement for inheritance.

@ajoino

it’s that they are a way to formalize duck-typing in a way that align with how duck-typing works in Python without needing to inherit from a common ABC.

Yes, that’s exactly what they’re for. But how does that apply to this thread? There are exactly two subclasses and they’re both in the standard library.

@ajoino

Thus, I don’t agree with your list of reasons why Protocols are less flexible than ABCs.

I didn’t say they were "less flexible. I said:

@NeilGirdhar

That way there will be:

• no false positives,
• type checking the whole interface,
• faster type checking, and
• more obvious declared intent when subclassing

@ajoino

I think that if you find yourself in a situation where you need a runtime chackable protocol your API design is probably not so great IMO.

Okay, I agree with that.

So look at the class defined in this thread. He wants a base class that will very easily be checked at runtime (used for more than just annotations).

Your ideas seem to be about a different issue than the one you’re replying to, wouldn’t you agree?

@ajoino

. In general, using inheritance instead of composition is almost always the wrong design choice IMO.

100% agree with you that composition should be preferred when it’s possible, but that has nothing to do with this post, so I’m not sure why you keep bringing this up.

[ajoino (Jacob Nilsson)]

@NeilGirdhar

@ajoino

it’s that they are a way to formalize duck-typing in a way that align with how duck-typing works in Python without needing to inherit from a common ABC.

Yes, that’s exactly what they’re for. But how does that apply to this thread? There are exactly two subclasses and they’re both in the standard library.

They are two different classes that implements a queue interface that is currently duck typed. It’s the prototypical example of where Protocols should be used IMO. Hence, according to the question posed in this thread, i.e. “Wouldn’t a Queue ABC be more elegant here?” my answer is no, a Protocol would be more elegant here.

[pf_moore (Paul Moore)]

@NeilGirdhar

So look at the class defined in this thread. He wants a base class that will very easily be checked at runtime (used for more than just annotations).

And there’s no explanation of why the OP wants such a base class. They didn’t even say that runtime checking mattered to them (that’s an assumption you’re making). They say it would “make it more elegant” when “accepting either one”, but there’s no explanation of why elegance is important here, nor is there any example of what “accepting” means in practice.

In the abstract, I agree with @ajoino - ABCs are unnecessary machinery if there’s no compelling use case for them. Specifically, requiring classes to inherit from such an ABC (or register with them as virtual subclasses) is additional overhead which needs justification.

I also don’t see the need for a protocol, but it has the advantage that it can be defined with no impact on the existing classes, so it’s a lower-impact option. Without a use case, it’s impossible to know whether it would help, but that’s just as true of an ABC.

So maybe we should stop the abstract ABC vs Protocol debate and wait for the OP to justify their proposal?

[ebonnal (Enzo Bonnal)]

Thanks a lot for these interesting arguments! Sorry if the vagueness of my post has led to misunderstandings.

To be more concrete, I need a ??? type that allows:

q: ???
q = queue.Queue()
assert isinstance(q, ???)
q = multiprocessing.Queue()
assert isinstance(q, ???)

Edit: And ??? must expose the methods common to both

So for my specific need, both an ABC and a runtime-checkable Protocol would work, right? Which approach would be more consistent with existing collections typing?

[ajoino (Jacob Nilsson)]

I think both a runtime checkable protocol and an ABC will be equally good in that case. Hell, I don’t even think you need an ABC, an empty QueueBase class that both Queues inherit from would suffice for isinstance checks.

Out of curiosity, what is your use case for knowing which kind of queue you’re using?

[ebonnal (Enzo Bonnal)]

Haha sorry my snippet is not the complete constraint, ??? = Any would work… I of course need ??? to expose the common methods mentioned in my post

Use case: streamable/streamable/iterators.py at f55496d703ca34206f30a232e0b89c637983b867 · ebonnal/streamable · GitHub

[ajoino (Jacob Nilsson)]

The code you’ve linked, specifically the FDFOOSFutureResultCollection looks from the callsite like it relies on duck typing to use either kind of queue. You’re not doing an isinstance check so this is where a Protocol would fit best I think.

[ebonnal (Enzo Bonnal)]

Indeed, this is not a client-facing part, and static typing is sufficient here. However, beyond my current personal use case, runtime checkability seems like a reasonable requirement, doesn’t it? Other container types are all runtime checkable, right?
If we go with a Protocol, are there any downsides to making it @runtime_checkable?