Improve typing, with ! to force not None value

Consider the following example

from typing import Optional


def foo(name: str) -> Optional[str]:
    if name == "foo":
        return "foo"
    return None


bar: str = foo("foo")

With mypy the errors out with

main.py:10: error: Incompatible types in assignment (expression has type "Optional[str]", variable has type "str")
Found 1 error in 1 file (checked 1 source file)

It would be great if ! could be used at the end to force the type checker that the value is not None
like this

from typing import Optional


def foo(name: str) -> Optional[str]:
    if name == "foo":
        return "foo"
    return None


bar: str = foo("foo")!

3 Likes

Does this do anything more than cast(str, foo("foo"))? Not really a fan since cast does all the things it does and express the intent clearer.

1 Like

Ok consider this exmaple:

from typing import Optional


class FooBar:
    name: Optional[str]

    def __init__(self, name: Optional[str] = None) -> None:
        self.name = name


def foo(name: str) -> Optional[FooBar]:
    if name == "foo":
        return FooBar()
    return None


bar: str = foo("foo").name

It errors out with

main.py:13: error: Missing positional argument "name" in call to "FooBar"
main.py:17: error: Item "None" of "Optional[FooBar]" has no attribute "name"
main.py:17: error: Incompatible types in assignment (expression has type "Union[str, None, Any]", variable has type "str")
Found 3 errors in 1 file (checked 1 source file)

It would require multiple cast to do so where as all can be done with adding a ! at the end @uranusjr

The exact syntax you propose is available in Hypertag (http://hypertag.io/), a Python-inspired language for templating and document generation. There are two different postfix operators (“qualifiers”) that mark a subexpression as either “optional” (?) or “obligatory” (!). The latter enforces a true (non-empty, not-None) value of the expression, otherwise an exception is raised.

Obviously, this operator is applied to ordinary expressions (type declarations don’t exist in Hypertag), still it might do as a related implementation for what you propose. Details can be found here:

http://hypertag.io/#non-standard-postfix-operators

1 Like

I would also see benefit in this:

It’s more or less shorter syntax:

cast(SomeSlightlyLengthierClassName, get_it_somewhere()).some_prop  # awful
get_it_somewhere()!.some_prop  # nice!

While cast() is a super-set, it’s much more verbose.

Also, what if I rename SomeSlightlyLengthierClassName? Fair enough, my IDE’s refactoring features should also rename the reference.

But what if I change get_it_somewhere() to return some other type? I need to manually iterate the code (let’s say I add a child class like SomeSlightlyLengthierChildClassName and return that instead).

If we don’t want to add new syntax to the language, how about a new function for this. It could be called something like unwrap() and would basically be sugar for the long form cast().

unwrap(get_it_somewhere()).prop

Or maybe I’ve been looking at Rust code for too long.

1 Like

I would be happy with Rust-like unwrap() or Swift-like !. It really would be nice if there were some conventional way of doing this.

What does “forcing” a not-None value mean, though? Mypy and other type checkers all have reliable type narrowing behavior around if x is None and similar checks.

I think a more useful family of new syntax elements would be “null-short-circuiting”, such as func_or_none?(x, y, z) and a?.b and u ?+ v. PEP 505 attempted to specify something like this (albeit a limited subset) and was rejected.

I understand why PEP 505 was rejected. It introduces ugly syntax for a narrow set of cases. Using ! would be more general and cleaner. It would, for example, enable type narrowing of the sort we want in comprehensions and in function arguments where doing the whole

if x is None:
    raise Exception

dance wouldn’t fit.

I suspect that the underlying disagreement in this discussion is about why some people (like me) really like the optionals of Rust and Swift (and also are into static type checking). I can elaborate on why I feel this would be a good thing and would not undermine anything Pythonic if asked.

I understand that Optional[T] is really just Union[T, None] and that the analogy to optionals in Rust or Swift is a limited analogy. But I want to point out why the kind of type narrowing I and others would like is distinct from other type narrowing.

Epoch Fails and the Problem with Zero

Let’s take a look at a kind of common bug that I like to call an “Epoch Fail”. This is where where some process tries to read a Unix time stamp and fails. Instead of handling that error, the buggy code treats the value as zero. But a zero Unix timestamp is the last second of 1969. We get results like

5C57DE5D-6AE3-4F51-9792-8B0FF9612134_4_5005_c

Using 0 to indicate a special case or the absence of a value leads to errors when 0 can also be a coherent value. So it is very useful to have a way to indicate that there is no value for something. None is perfect for that.

When None is not an error

Now in that example, failing to read a timestamp is almost certainly some error that should be handled as an error. But there are cases where having no value (aka None) is not an error.

Suppose you have an object for some sort of game state or simulation that gets updated each round of the simulation. Some object variables will depend on things from the previous round. Some might even depend on the previous two rounds. Thus those variables cannot be set to anything meaningful during __init()__. So during init, we’d want to set those variables to None.

During each round of our game or simulation there will be some methods called that need to behave is special ways. There will be a test if a variable is None. That case is fine and it doesn’t get helped by having an inline TypeGuard-like thing.

But there will be other functions that should assume that the parameters it is given are not not None. Suppose a probability gets updated, but doing so depends on certain search parameters and results from a previous round. Suppose we have something like

def updated_prob(probability: float, search_effectiveness: float) -> float: 
    ...

class SearchArea:
   def __init__(initial_probability: float):
       self.p: float = initial_probability
       self.search_effectiveness: Optional[float] = None

   def conduct_search(self) -> None:
      ...
      self.search_effectiveness = self.compute_a_lot_stuff()
      ...
      self.p = updated_prob(self.p, self.search_effectiveness)
      ...

If updated_prob() is a fairly general function that really just takes float arguments then it should be declared with the type hints of float, float for its arguments. And while we could always add a

if self.search_effectiveness is None:
    raise Exception

before the call to update_prob(), suppose we want to call update_prob() in a list comprehension or in an an argument to some function.

Being able to say things like

p = ...
probs_by_se = [updated_prob(p, se!) for se in effectiveness_list]

with the ! makes it much easier to write clean code and to communicate what we want to the type checker.

[Note. All of the code here is something I entered directly into the web interface for the discussion. It probably has numerous syntactic errors]

hello!
Is there any advance on this suggestion?
I believe it would very seriously improve the Python experience regarding typing, I cannot stop writing code where the variable has a possibility of None but we know at runtime, it would be filled.

Considering the following:

class MyFile:
  destination: pathlib.Path
  content: str | None

  def __init__(self):
    self.destination = pathlib.Path(".")
    self.content = None

  def fill_content(self, content: str) -> typing.Self:
    self.content = content
    return content

  def validate_requirements(self) -> bool:
    if not self.content:
      raise ValueError()
    ...
    return True

  def execute():
    self.validate_requirements()
    self.content  # This is still None to mypy! But we know it is a "str" now.

    # Old
    if not self.content:
        # Do something so that self.content is considered filled
    self.content  # Not None, but we waste space

    # New
    self.content!  # It's just one character away, nice and easy, and no condition required

This is the dumbest example I can give for myself.
To counter this problem, I just use raise keywords to ensure my variable is not None, but it gets very overwhelming, and I don’t believe that using typing.cast is something I’d like to use because I would have to include way too many typing.cast in my codebase.

Being able to indicate easily that my variable are not None is probably one of the greatest typing addition we could obtain.

It’s an issue when “None” as a value has a different meaning than “Not yet defined.” It would be great to have something like NotImplemented, but in case of an object that is REQUIRED to be set before the moment, it will be called for the read.

IMO this is something type checkers should be able to deduce that validate_requirements did not raise, so self.content is not None. So I’d start by raising an issue with mypy for this. It may be that it’s more complex than I’m assuming, though, so they may not be able to make this automatic. Even then, adding a simple assert self.content is not None should be enough to allow mypy to know that afterwards, self.content is definitely a str.

So I don’t think this example is a compelling argument for new syntax.

Apologize, I did not mean to send my post earlier.

I assume this is a decision made to optimize performance of mypy which can’t really be considered.
I am actually editing a project containing a lot of code (Up to +5500 lines a file, if not more), and mypy is already taking a long time already to finish its analysis. Making mypy doing deeper analysis would probably be a huge issue if it needed to analyze all called functions to determine every variable’s types.

Sorry, but I’m quite confused by this. Both example (From me and @kumaraditya303) achieve the exact same goal, the modified object is the only difference.
If this is accepted, I assume anything (Function calls, variables, etc.) will be affected by this new syntax.


Another applicable case is when properties are getting chained, but one property is None, and it would be cleaner to just indicate it’s not-None.
For instance:

class Color:
  rgb: tuple
  hexadecimal: str | None

class Url:
  name: str
  color: Color | None
  href: str

class Embed:
  name: str | None
  url: Url | None

class Embeds(typing.TypedDict):
  embeds: list[Embed]

embeds: Embeds = get_embeds_all_attributes()
# We're fetching our embeds, but we know the one we're fetching
# has every attributes filled

print(embeds)
# {
#  "embeds": [
#   Embed(
#     name="Name",
#     url=Url(
#      name="Link to example",
#      href="https://example.com",
#      color=Color(rgb=(255, 255, 0), hexadecimal="#ffff00")
#    )
#   )
# ]}

# Using assert, we would need to check every attributes to not
# trigger mypy, this would be long and painful
if embeds[0].url is None:
  raise ValueError
if embeds[0].url.color is None:
  raise ValueError
if embeds[0].url.color.hexadecimal is None:
  raise ValueError
# long, and can be hard to understand

# With the "!" syntax, it would be easy as:
rgb_color = embeds[0].url!.color!.hexadecimal!

This example is a bit over the top, but it truly illustrates how checking all chained attributes can be more painful than practical and smart, most especially when we know None will never be present in our context.

This could be even more useful in event-based application. For example:

def on_all_embeds_url_color(embeds: Embeds):
  """Called whenever we have an embed that has some color in its URL"""
  for embed in embeds:
    color = embed.url!.color!.hexadecimal!
    ...

I wished to contact you personally through this platform, but I am not yet capable to, if you accept, could you please drop me an email at pro.julien.mauroy@gmail.com so we can discuss this together? I’d like to hear more of your thought, for clarification, but this might get off-topic of this thread.

The ! suggestion isn’t just a Typing change, that would be a new feature in the language and require a PEP. It would be a postfix operator (the only one in the language).

And also, the unwrap function can already be defined that would allow basically exactly the correct behavior:

def unwrap(val: T | None) -> T:
    assert val is not None
    return val
    

v: int | None
w: int


w = unwrap(v)

Works as far as I can tell. So the discussion question would be that value! is significantly better, so much so that it requires language support, compared to unwrap(value). I haven’t seen an argument for that.

I think ? would actually be far more useful as far as postfix operators go, to allow things like foo?.bar returning None if foo is None, rather than having to write None if foo is None else foo.bar.

But neither ? nor ! really have that much to do with typing, they just make dealing with None easier, but excluding None from an optional type is far from the only type narrowing across method boundaries that will get missed by type checkers, it just happens to be a very common case, so you still will need to write assertions and TypeGuard functions anyways.

That being said I find it’s rare that I actually ever have to write things like assert foo is not None and foo.bar is not None and foo.bar.baz is not None and when I do it’s usually a sign I should refactor my code. assert foo is not None is quite common, but I personally am not that bothered by it.

2 Likes

That has been proposed, but it’s very VERY different from the current proposal, which is just a type-checker feature - equivalent to assert thing is not None (which is something all type checkers should already recognize).

(Side note: This wouldn’t be a “postfix” operator, since thing? wouldn’t really have any meaning on its own. It’s a variant form of attribute lookup ?. rather than being a separate operator that is handled first. Not that postfix operators are inherently bad or anything.)

It could be a postfix operator that turns None into a NoneProducer instance that returns None from all relevant magic methods. This could then be a single new magic method that has a default implemention for object that returns self and NoneType would overwrite that. But ofcourse, that isn’t the proposal in PEP 505 and I am not really sure if it has any real benefits.

1 Like

Closing as this specific discussion died out 2 years ago, and new comments since it was resurrected are just going in circles with the same suggestions made here and elsewhere about coalescing, maybe, and other similar ideas. At this point, if you want to discuss it, probably best to go through the PEP process and get an official decision.