PEP 661: Sentinel Values

  • I wouldn’t worry too much about users of if value instead of if value is not MySentinel. Those who do likely also use if value when they should be using if value is not None, which is much more common. A short explainer in the doc can be more than enough, IMHO.
  • In the overwhelming majority of cases you know if a sentinel represents something empty during sentinel declaration. If there’s a case where it isn’t, the bool value probably shouldn’t be used at all as changing it dynamically could break someone else’s code.

Sorry for a late reply. I saw this brought up on typing-sig and realized that maybe we don’t need a way to define a custom sentinel. Maybe we just need one new value in the stdlib that developers can use as a sentinel in a variety of cases where the sentinel must be distinguished from None? If three different libraries define a MISSING sentinel, why couldn’t they share it?

(I can see an answer to that, but I tend to think that it’s such a small second-order effect that we should be able to ignore it. After all any library that makes up a MISSING sentinel can still be fooled by a user who digs through the code to find the MISSING object (dig through the function defaults if you have to) and pass it explicitly.)

3 Likes

If I understood it correctly, this PEP is just a shortcut/convenience for the following practice:

from enum import Enum

NotGiven = Enum("NotGiven", "NOT_GIVEN")
NOT_GIVEN = NotGiven.NOT_GIVEN

def func(arg : NotGiven | OtherType = NOT_GIVEN):
    ...

The main advantage here is not requiring NotGiven and NOT_GIVEN to be defined at the same time (in the same way we don’t need to have a NoneType = type(None) defined to be able to use type hints).

Is this interpretation correct?

From earlier in the thread, I think the two reasons that “just one more” sentinel object (e.g. Ellipsis, or another singleton object beyond that) isn’t good enough are:

  1. The situation where two different code bases each use the extra sentinel type with their own internal meaning, and a sentinel object gets passed between them with unexpected consequences. This could break working code without anyone noticing if one library uses the other as part of its implementation, and the second library adds a meaning to the shared sentinel in an update. If the libraries could each make their own sentinels, that wouldn’t happen.
  2. The repr of a generic sentinel wouldn’t be as self-documenting as a custom sentinel could be. If you declare a function that uses a sentinel as the default value for some argument, it can help your users if the sentinel’s name in the function signature describes what it means to the function. What makes sense for one function may not for another one.

On a much less serious note, I was fascinated by @njs’s decidedly brain bending implementation from back in June and made my own version that works as a base class for class-based sentinels:

class _SentinelMeta(type):
    def __new__(mcls, name, bases, namespace, repr=None):
        def __new__(self, *args, **kwargs):
            raise TypeError(f"{self!r} is not callable")
        namespace["__new__"] = __new__
        if "__repr__" not in namespace:
            if repr is None:
                repr = name
            def __repr__(cls):
                return repr 
            namespace["__repr__"] = __repr__               
        cls = super().__new__(mcls, name, bases, namespace)
        cls.__class__ = cls
        return cls

class Sentinel(_SentinelMeta, metaclass=_SentinelMeta): pass
del Sentinel.__new__ # allow subclassing

You use it like this:

class MISSING(Sentinel): pass

You can test for the sentinel with either is or isinstance, since every sentinel is an instance of itself. MISSING is MISSING and isinstance(MISSING, MISSING) are both True.

Or if you want a custom repr for your sentinel.

class MISSING(Sentinel, repr="<MISSING>"): pass

I also tried a version that looks for a custom repr string inside the class namespace, rather than as a metaclass argument (e.g. replace the pass with repr="<MISSING>"). I’m not sure if one is notably better than the other. You can also write your own __repr__ method in the class body.

3 Likes

I really like this approach.

These are a few minor things that catch my eye:

Observation 1. SentinelMeta probably shouldn’t be private
Currently you’re implying that the metaclass, SentinelMeta, should be private by prefixing it with an underscore, however I don’t feel this should be the case (E.g. enum.EnumMeta isn’t private in this way). I can’t imagine there’s any reason to explicitly discourage users from subclassing SentinelMeta if they so desire (e.g. to change the default __repr__).

Observation 2. I’m not fully convinced accepting additional keyword arguments in SentinelMeta.__new__ is the correct approach
As you mention, sentinel classes can be created with a custom __repr__ method, which personally feels much more intuitive as the user is creating a class. It would be interesting to explore the possibility of implementing a Functional API (much like Enums have), e.g. something along the lines of Sentinel('MISSING', repr='!missing!').

Observation 3. The TypeError message is currently not strictly correct.
Sentinel classes can be made callable by implementing a __call__ method. Instead, by neutering the __new__ method, an attempt is being made to prevent them from being initialised/subclassed.

Observation 4. A default __repr__ is currently defined in SentinelMeta.__new__, however as the metaclass is subclassed, a default SentinelMeta.__repr__ can be implemented which feels a bit tidier.

With the above observations considered, here’s a minorly tweaked version of the code you provided:

class SentinelMeta(type):
    """ Metaclass for Sentinel """

    def __new__(metaclass, name, bases, namespace):
        def __new__(*args, **kwargs):
            raise TypeError(f'Classes derived from {type.__repr__(metaclass)} cannot be initialised or subclassed')

        namespace['__new__'] = __new__

        cls = super().__new__(metaclass, name, bases, namespace)

        cls.__class__ = cls

        return cls

    def __repr__(cls):
        return cls.__name__

    @property
    def name(cls):
        return cls.__name__

class Sentinel(SentinelMeta, metaclass=SentinelMeta): pass

del Sentinel.__new__ # Allow subclassing

I also added a name property, as fetching the sentinel’s name feels like a relatively standard behaviour, and it felt slightly more appropriate than expecting the user to acquire it through the __name__ attribute.

Here’s a quick and dirty example to show some of the behaviour supported by this approach:

class MISSING(Sentinel):
    message = 'A value was not provided'

    def __repr__(cls):
        return f'<--{cls.name}-->'

    def __bool__(cls):
        return False

    def __call__(cls, *args, **kwargs):
        print('__call__:', dict(args=args, kwargs=kwargs))

    @staticmethod
    def staticmethod(*args, **kwargs):
        print('staticmethod:', dict(args=args, kwargs=kwargs))

    @classmethod
    def classmethod(*args, **kwargs):
        print('classmethod:', dict(args=args, kwargs=kwargs))

On a side note, sentinels also feel like they should probably be frozen which is something the PEP doesn’t seem to explicitly mention (e.g. __setattr__ could be neutered as well).

Is there any reason to explicitly encourage users to subclass SentinelMeta? It’s much easier to make something public later than it is to take it away or to make changes if the design happens to be wrong. And with anything in the stdlib, changing things is extremely risky due to backwards-compatibility.

That’s an interesting question. Personally I neither feel users should be explicitly encouraged nor discouraged from this behaviour, however it should be an option that is there for those that need it. For example, EnumMeta is mentioned in the docs here without encouraging/discouraging user’s usage of it.

Having the ability to influence sentinel creation seems like an important factor as the user may want to alter default behaviour to suit their needs.

For example, here’s a few reasons a user may want to subclass SentinelMeta that come to mind:

  1. Change the default __repr__, e.g. to return f'<{cls.__name__}>'
  2. Change the default __bool__, e.g. to return the opposite of the existing implementation
  3. Implement functionality (attributes/methods) they would like all of their derived sentinel classes to have

Maybe that’s just me, however I can personally see reasons I would want to achieve such behaviour.

[tombulled]

I can’t imagine there’s any reason to explicitly discourage users from subclassing |SentinelMeta| if
they so desire (e.g. to change the default |repr| ).

[brettcannon]

Is there any reason to explicitly /encourage/ users to subclass |SentinelMeta|?

I’m going to say no. SentinelMeta is not as complex as EnumMeta, and there are good reasons to allow EnumMeta to
be subclassed, but the general EnumMeta subclassing advice is: don’t.

Much easier to make it public later rather than deal with error reports and Stackoverflow questions about why isn’t it
working.

Thank you for pointing that out. I’ll have to think about an appropriate way to caution against subclassing EnumMeta.

My intent with regards to _SentinelMeta is that it’s an implementation detail, the public interface of this system is the Sentinel class. I’m not sure I gave much thought to the underscore in its name, I just copied that from Nathaniel’s version (which he called _SentinelBase which is also not wrong since it’s both the base type and the metatype of Sentinel).

I did try making Sentinel callable, in addition to being used as a base class. It was very messy. One problem is that if you can call Sentinel, its subclasses will also be callable by default, and we probably want MISSING to raise a TypeError. Maybe we could inject a __call__ method into the class namespace the same way we do a __new__ method…

As for letting the __repr__ method be inherited, rather than injected into the namespace, I mostly did that so a metaclass argument could be used (which is only easily accessible from __new__).

Thanks both for your replies, I’ll work off of the assumption that SentinelMeta should be made private going forward.

Maybe we could inject a __call__ method into the class namespace the same way we do a __new__ method…

That’s a really interesting idea. I feel like a Functional API would be an awesome addition, so it’s definitely something worth exploring.

I’ve had a little fiddle and here’s what I’ve been able to come up with:

class _SentinelMeta(type):
    """ Metaclass for Sentinel """

    def __new__(metaclass, name, bases, namespace):
        def __new__(cls, *args, **kwargs):
            raise TypeError(f'cannot initialise or subclass sentinel {cls.__name__!r}')

        cls = super().__new__(metaclass, name, bases, namespace)

        # We are creating a sentinel, neuter it appropriately
        if type(metaclass) is metaclass:
            cls_call = getattr(cls, '__call__', None)
            metaclass_call = getattr(metaclass, '__call__', None)

            # If the class did not provide it's own `__call__`
            # and therefore inherited the `__call__` belongining
            # to it's metaclass, get rid of it.
            # This prevents sentinels inheriting the Functional API.
            if cls_call is not None and cls_call is metaclass_call:
                cls.__call__ = super().__call__

            # Neuter the sentinel's `__new__` to prevent it
            # from being initialised or subclassed
            cls.__new__ = __new__

        # Sentinel classes must derive from their metaclass,
        # otherwise the object layout will differ
        if not issubclass(cls, metaclass):
            raise TypeError(f'{metaclass.__name__!r} must also be derived from when provided as a metaclass')

        cls.__class__ = cls

        return cls

    def __call__(cls, name, bases=None, namespace=None, /, *, repr=None):
        # Attempts to subclass/initialise derived classes will end up
        # arriving here.
        # In these cases, we simply redirect to `__new__`
        if bases is not None:
            return cls.__new__(cls, name, bases, namespace)

        bases = (cls,)
        namespace = {}

        # If a custom `repr` was provided, create an appropriate
        # `__repr__` method to be added to the sentinel class
        if repr is not None:
            def __repr__(cls):
                return repr

            namespace['__repr__'] =__repr__

        return cls.__new__(cls, name, bases, namespace)

    def __repr__(cls):
        return cls.__name__

class Sentinel(_SentinelMeta, metaclass=_SentinelMeta): pass

Here’s a few quick usage examples to demonstrate functionality:
Example 1: Default __repr__

>>> class A(Sentinel): pass
>>> A
A
>>> # The functional equivalent:
>>> A = Sentinel('A')
>>> A
A

Example 2: Custom __repr__

>>> class B(Sentinel):
    def __repr__(cls):
        return f'<{cls.__name__}>'
>>> B
<B>
>>> # The functional equivalent:
>>> B = Sentinel('B', repr='<B>')
>>> B
<B>

Importantly, we’ll most likely want to provide other parameters alongside repr for the Functional API (e.g. module and qualname) much like EnumMeta.__call__.

The discussion may have moved on from this point, but I’d like to address it. In particular: why not both approaches?
There are cases where a library author doesn’t want “their sentinel” to be something which their users may already be using. But there are also lots of cases in which users would like to have readily-available, self-documenting (by name) sentinels. e.g.

from sentinels import DEFAULT

def foo(
   *, myparam: None | DEFAULT | str = DEFAULT
): ...

where foo(myparam=None) means something different from foo().

If a library provides an API which says it can take any value from a user, it may need a sentinel value which is explicitly not in the stdlib. Otherwise, the library authors cannot safely assume that user applications are not already using the sentinel for something different.

Why not provide the basis for implementing your own sentinels, understood by type-checkers to have an ergonomic syntax for annotations, but also provide a few more “standard” ones with no prescriptive meaning? These aren’t language-level constants like None and Ellipsis – they’re just documented values in the stdlib.

This is interesting! We’re used to writing methods that run on instances, but they’ll actually be called from the class.
@taleinat this strawman from the PEP is just broken:

class NotGiven: 
    def __repr__(self): 
        return '<NotGiven>'

>>> NotGiven
<class '__main__.NotGiven'>
>>> NotGiven()  # but you're not supposed to do this
<NotGiven>

even prepending @classmethod does not help.
But with @BlckKnght’s magic, it works :tada:, because the class is its own instance.

BTW defining __repr__ is poor example for the goal of class extensibility, as IMHO half the attraction of using class statement is its magic knowledge of its name. But suppose you want other custom methods…

If the PEP still rejects using class objects, I recommend updating the rationate there.
The stronger objection is that the naive pattern of “just write of class statement” is a footgun because it’s tempting to add method but they won’t do what you actually want.
Arguably class-is-its-own-instance magic is “minimum viable” requirement to make class statements convenient. (But the objection “Additionally, using classes this way is unusual and could be confusing” applies doubly to such magic.)

FWIW, if sentinels defined __set_name__ then you could avoid duplication:

# The example from the PEP:
class MyClass:
    NotGiven = sentinel('MyClass.NotGiven')

# using __set_class__():
class MyClass:
    NotGiven = sentinel()

However, it’s tricky dealing no-argument usage at the module level (since there’s no module-level equivalent for __set_name__).

After another long delay (sorry!), I’ve cleaned up the PEP and reference implementation. I’d like to submit the PEP as a proposal soon, as I feel that the reasoning and implementation are sound.

I’ve decided to forego type signatures specific to each sentinel, and make do with simply Sentinel. I spent far too much time and energy in an effort to achieve specific type signatures. I now realize that it’s not worth the extra complexity; in my mind this feature must be as simple as possible.

I’ve also made it possible to sub-class the Sentinel class, and so avoid the need for a parameter to set the boolean value (“truthiness”).

Final reviews and comments would be most welcome before I go ahead with proposing this.

3 Likes

That’s a shame that Literal[MISSING] is not something that’s supported by type checker.
Personally it would basically eliminate the use case for Sentinel, since I don’t agree with the conclusion in the pep mentioning that the missing support for specific type signatures is a minor issue. It’s one more paper cut in the python typing that already has plenty of others.

Without proper typing support I personally don’t see the advantage over just formalizing

class NotGiven(Enum):
    NOT_GIVEN = auto()
NOT_GIVEN = NotGiven.NOT_GIVEN

as proposed above

It is important to hash out in detail what exactly is “supported”, and what is not. Literal[MISSING] has been “supported” by typing ever since Literal been introduced; from the perspective of Python core developers (who actually implement the sentinel module), it most certainly has “proper typing support”. This semantic is not supported by type checkers (e.g. Mypy), but since they can be upgraded separately (not being a part of the core Python runtime and stdlib), IMO it makes more sense to standardise the “best” syntax and anticipate them to catch on, instead of settling for a compromise (which, I should note, also introduces tasks to type checkers, just different ones).

By “supported” above I meant supported by type checkers. I’ve clarified the comment.
I agree that from a Python typing module point of view Literal[MISSING] is already supported, but for all practical purposes it cannot be used to type code until type checkers also support it.

If type checker support for Sentinel should not be considered in this PEP, I personally don’t see much value in it compared to existing alternative already supported by python. Again these are just my 2 cents.

Maybe I misunderstand you, but isn’t this true for all new features that has typing usages?

It’s surprising that PEP decides not to specify the way type checkers should behave with this feature. It doesn’t have to (and probably shouldn’t) affect the design but it could still just specify the expected way of typing a specific MISSING sentinel. Once this feature becomes available, the users will most definitely want to use it in their typed code instead of the other patterns such as the enum one. There surely will be people asking for it on the issue trackers of type checkers and other places and so the typing community will have to end up making new specifications for it anyway. Why not get ahead of this and just specify author’s preferred way of typing this at least in a non-normative section so that there’s something that type checker contributors can refer to when they inevitably get requests for it? Otherwise, this may just prevent the adoption for many people for another year or two.


IMO the better syntax would be MISSING as the type hint rather than Literal[MISSING] though there may be some other syntaxes worth considering as well that I’m not thinking of. Due to the updated design in the PEP, there’s a global registry of sentinels which to me is quite a surprising choice but ignoring the probably rare case of someone getting sentinel with Sentinel("...") rather than just using the constant name it’s defined under in the module, the simple usage of MISSING constant as the type hint should be fine. It would make sense to either require that Sentinel instances are typed with Final for it to work as a type hint or just have the type checkers treat it as final implicitly similarly to type vars (I would probably just suggest the latter)

2 Likes