Namedtuple instance check

Ideas
1

I would like a way to be able to check if an object is an instance of a type that was created by the namedtuple function. Usually I would think this kind of introspection is not necessary, but in the case of serializing/deserializing and preserving the type, it can end up being helpful. Currently in the Python ecosystem, this is implemented with myriad error-prone heuristics:

  • pytorch/pytorc (pytorch/pytorch/blob/4dd73e659a8fd4872e5f49cfd72e420fa7c4e6c9/torch/utils/_pytree.py#L675-L684): checks for _fields and that _make and _asdict are callable
  • metaopt/optree (metaopt/optree/blob/e0e772ea3bb4ab551cb6f102b8074022bb8542ec/include/optree/pytypes.h#L217-L254): same as pytorch, but implemented in C++ with a custom weakref LRU cache in front
  • jax-ml/jax (jax-ml/jax/blob/792577ff9ee8a55968baa02719e3c84e518cdf35/jaxlib/pytree.cc#L216-L219): checks only for the presence of _fields
  • pytorch/tensordict (pytorch/tensordict/blob/785349d82c6e17928cfb4581e73dab16156c8a6c/tensordict/utils.py#L2378-L2381): checks for _fields, _replace, and _asdict
  • numba/numba (numba/numba/blob/5e0ceb077fb219a21ac8a50e5234ca643aaf96f9/numba/core/ir_utils.py#L1962-L1977): checks that tuple is the only superclass, as well as _make and _fields

and many, many more: Code search results · GitHub. These kinds of heuristics are still propagating out today, as seen from this 15-year old stackoverflow post.

In terms of implementation, I would think it would either involve adding a marker type into the hierarchy for named tuples so that they do not directly inherit from tuple, or adding a querying function to the collections module that would reliably be able to check this.

P.S., apologies for the formatting. This is my first post, and it will not let me post more than 2 links. All of the code examples are links from GitHub.

5 Likes
2

Yes, this is the most obvious solution. I think some people just hate tiny costs more than they hate creating issues like the ones you’ve linked.

1 Like
3

One officially documented workaround is to test if the factory function typing.NamedTuple is the class’ original base:

from typing import NamedTuple
from types import get_original_bases

class Point(NamedTuple):
    x: float
    y: float

assert get_original_bases(Point) == (NamedTuple,) # passes

or to account for typing.Generic as a possible original base:

assert NamedTuple in get_original_base(Point)

I suggest that this workaround be included in NamedTuple’s documentation.

Furthermore, I think it makes sense to make issubclass(A, B) perform an additional search for B in A.__orig_bases__, so the more readable and idiomatic approach of issubclass(Point, NamedTuple) can just intuitively work. After all, when one reads class A(B): they should expect issubclass(A, B) to return True.

Same goes for TypedDict.

2 Likes
4

@kddnewton asked for test for an instance of a class created by the namedtuple function. This is not it.

import collections, types, typing

Point = collections.namedtuple('Point', ['x', 'y'])
assert typing.NamedTuple in types.get_original_bases(Point) # fails
2 Likes
5

Indeed @blhsing’s solution works as demonstrated for typing.NamedTuple, but not for collections.namedtuple. As typing remains optional, it’s a partial solution at best, alas

2 Likes
6

For comparison, dataclasses have dataclasses.is_dataclass() and useful_types.experimental.DataclassLike. These both test for presence of attribute __dataclass_fields__.
is_namedtuple() could do the same by checking for any subset of attributes mentioned by OP.

7

Oops I totally misread the OP and thought it was for typing.NamedTuple, probably due to the capitalized letter N in the title.

Have there been incidents where such heuristics caused actual errors/false positives in the real world? In the spirit of Python’s tradition of duck typing I think the heuristics of testing for the presence of those well-documented methods are an acceptable practice. You can even bundle them as a protocol if you’d like.

If you’re really concerned about the heuristics being error-prone, how about the following test as a more error-proof heuristic?

import collections

Point = collections.namedtuple('Point', ['x', 'y'])
assert Point._make.__func__.__code__.co_filename == collections.__file__
1 Like
8

AFAIK, it was intentional design decision that isinstance() and issublass() don’t work for named tuples. Named tuples are not part of specific class hierarhy, this is a class of types that implements some interface (access to fields by attribute name and by index). You can implement a named tuple without using namedtuple(). For example, by using the PyStructSequence_* C API.

cc @rhettinger

4 Likes
9

Even with duck typing, I think you should be able to easily ask questions about a type if you want to—even if this happened to be a conscious design decision.

In this case, you can’t do that.

Yuck though, right :slight_smile:

2 Likes
10

This makes a lot of sense. I actually think it’s rather strange that this doesn’t work already, and I can see how someone might even classify this as a bug.

The workarounds that have been proposed here work at runtime, but not for static typing, i.e. type-narrowing.

2 Likes
11

True.

The best way forward is for sure to make issubclass work for namedtuple-created classes, but adding a dummy marker class to the bases may break some of the existing codes that explicitly check that tuple is the only superclass, e.g.:

  • numba/numba (numba/numba/blob/5e0ceb077fb219a21ac8a50e5234ca643aaf96f9/numba/core/ir_utils.py#L1962-L1977): checks that tuple is the only superclass, as well as _make and _fields

So it might make sense to make collections.namedtuple follow the footsteps of typing.NamedTuple and add itself to the new class’ __orig_bases__ and make its __mro_entries__ return (tuple,), and make issubclass search in __orig_bases__, so issubclass(Point, namedtuple) can work without a dummy superclass.

On the other hand, codes that explicitly check that tuple is the only superclass would already fail to recognize a generic typing.NamedTuple class as a valid namedtuple class (because typing.Generic would be the second base class), so it’s probably good in the long term to break them and force them to adopt a better check.

2 Likes
12

It would be much simpler to just put NamedTuple into the __mro__ like any other base class.

13

But NamedTuple is a function, not a class, so it can’t be in __mro__, which is why we need __orig_bases__ and __mro_entries__.

1 Like
14

I mean to create a class that would be added to all named tuples.

1 Like
15

Yes, if we are willing to break some existing codes, like I mentioned.

1 Like
16

How does having namedtuple return a class having an extra base class breaks any explicit promises in the documentation?

17

Nothing promised is broken. It’s just that some existing codes (such as the example from numba) makes the false (well, true at the time, but fragile) assumption that tuple is always the only superclass of a namedtuple class. Adding a new base class will break those codes.

18

So what would be next steps here? It seems like this has some for and some against. Is this the kind of thing where I should put together a commit that we could look at together? (Sorry still my first post here, so trying to determine the process.)

19

Maybe something along the lines…

In collections.__init__.py:

class _NAMEDTUPLE: pass

def namedtuple(...):
    ...
    class_namespace = {
        ...
        __namedtuple__: _NAMEDTUPLE
    }

In _collections_abc.py:

class NamedTuple(metaclass=ABCMeta):
    @classmethod
    def __subclasshook__(cls, C):
        return (
            C.__mro__[1] is tuple
            and getattr(C, '__namedtuple__', None) is collections._NAMEDTUPLE
        )

Then:

MyNamedTuple = collections.namedtuple('MNT', [])
print(isinstance(MyNamedTuple, NamedTuple))
# Hopefully True
20

In general I don’t see why it is necessary to use isinstance here. A class created by namedtuple is just a class and different classes created by namedtuple are just different classes. There is no need for there to be any common superclass besides tuple.

Maybe a function is_namedtuple could be added like is_dataclass (although is_dataclass has a weird dual meaning).

For serialisation you can use __getnewargs__:

In [1]: from collections import namedtuple

In [2]: A = namedtuple('A', ['x', 'y'])

In [3]: a = (1, 2)

In [4]: type(a)(*a.__getnewargs__())
Out[4]: (1, 2)

In [5]: a = A(1,2)

In [6]: type(a)(*a.__getnewargs__())
Out[6]: A(x=1, y=2)

Of course you would need to be able to serialise the class A and pickle has a fragile mechanism for doing this that fails for dynamically generated namedtuple classes.

For manual serialisation it is important to check exact types like:

# Don't use isinstance(obj, tuple):
if type(obj) is tuple:
    # Handle the tuple case
    # Any subclass of tuple may have a different constructor
    # and the class also needs to be serialised so we don't
    # handle those cases here.