Controlling mutability with static types

I’d like to receive some feedback about a little project of mine involving mutability and static types.

2 Likes

You might want to read this tutorial on packaging and watch this (a bit outdated) video on the same topic.

Is there a way to make this work without defining generic aliases for all types?

While I appreciate the effort put into this and implementing using it purely with what we have already available to us, I don’t think it is useful enough at this point for me to ever consider using it.

The main reason for this is that the variance of the generic does not change in accordance with its mutability. list_r should technically be covariant, but it is still invariant, because list_ itself is invariant. So the relatively small additional safety does not really warrant the additional mental overhead.

Especially because, and this is the other reason I wouldn’t use this, even your most simple solution using locks, seems clumsy and error-prone. Type errors would also become difficult to read, since you have to do the mental conversion between the type error given to you and this mutability model.


I would like to see something more simple, that doesn’t care about lifetimes. With proper type checker support, so the variance of a generic is dependent on its mutability. A borrow-checker to enforce lifetimes, is always something you can implement as an additional static analysis tool that works on top of this, especially since whether or not it keeps the reference can easily be inferred.

But either way we’d need a PEP to get something that is more generally useful and less clumsy to use. I personally think an Immutable modifier in addition to a @idempotent decorator to explicitly mark the methods that are still safe to use on an Immutable variant of the type[1], would be enough to get us most of the benefit, with relatively little additional effort required[2].


  1. self on an idempotent method implicitly changes to Immutable[Self], so all attributes become read-only ↩︎

  2. apart from the initial effort spent on marking idempotent methods ↩︎

3 Likes

Thank you, but my little project is not ready to be a package. I still need to assess whether using it is worth the effort. For now, it’s just an experiment. I posted it hoping some of you can think of ways to improve it.

If you are referring to the lifted versions list_, set_, etc…, then I don’t think it’s currently possible.

That should be easy enough to do. I’ll work on it and see how it goes.

This is probably the easiest solution:

T = TypeVar('T')
T_co = TypeVar('T_co', covariant=True)

class X_r(Generic[T_co]):
    # read-only interface here
    ...
    
class X_w(Generic[T, Mut_L], X_r[T]):
    # remaining interface here
    ...

class X_rk(Generic[T_co, Mut_L], X_r[T_co]):
    pass

class X_wk(Generic[T, Mut_L], X_w[T, Mut_L], X_rk[T, Mut_L]):
    pass

By keeping them separate we get rid of a type parameter and all those self: X_[..., W, L]. I could just keep X_r and group the other 3 but then we’d go back to 3 type parameters. Dropping rk and wk would also simplify things.

The locks are not so bad as you can’t duplicate them by accident, and they’re introduced right before their use, so it’s hard to mismatch them.

You can make the points of definition and use even closer:

class _L_f: ...
def f[L: _L_f](x: X_w[int, L], y: X_w[int, L]): ...

There’s only so much I can do with what I have :disappointed:

Edit: I’ll think about the errors, but no promises.

That’s kind of what I was trying to get at, yes. It’s cool as an experiment, but if you tried to use this in production code that other people have to either maintain as well or use downstream, they will be very upset with you, because there are just too many sharp corners and edges and it’s essentially impossible to understand unless you spend some time reading the documentation on how this is all modeled.

Additionally this will not really play well with existing nominal types, so you need to create dummy converters all over the place when interfacing with other code, e.g. the standard library.

I’ve played around with encoding concepts into generics that the type system doesn’t natively understand in the past myself and it never amounted to more than fun experiments for the same reason.

Mutability needs to be a first class citizen in the type system in order to really be useful, although I don’t have very high hopes for a PEP succeeding here, since a mutability type modifier is only really useful in patterns that are considered unpythonic. You are generally expected to use duck typing in parameter types in public APIs. So this only really makes sense in internal code, where you’re trying to optimize code by getting rid of useless copies.

1 Like

Also see PEP 351 – The freeze protocol | peps.python.org.

I’m not sure I follow. You don’t have to apply my method to every single nominal type to make it useful. Can you give me an example?

My approach only uses static types. There’s no copying or freezing.

I’m not saying you have to do that, in order for it to be useful, I’m just saying that anywhere you get a list from a standard library or some third library function you will have to convert using some intermediary converter function, like your lift before you can pass it into a function that requires list_. But the same goes the other way, so anywhere you export these types, people now will have to convert them back to regular types before they can be used in functions, that aren’t aware of these types, that’s an incredibly painful developer experience.

It’s like you have an alternate type system along the regular one that you constantly have to convert to and from. This wouldn’t at all be the case if mutability was an actual concept in the type system.

Also I don’t believe that your lift solution using overloads will scale beyond a handful of types, since, if I’m not mistaken, the time complexity for overload resolution in type checkers is cubic.

1 Like
  • I can eliminate lift by turning list_* into protocols.
    • The “nominality” can be preserved by adding an __unlifted_type field.
  • The overloads can be avoided by inserting do_conv into the lifted types as a __do_conv field.
    • All those fields will be used only by r, w, rk, and wk.
      • This means that the lifted type can still be just the regular type, at runtime.

EDIT: Mhm… I’ll think about it some more. I’m sure I can avoid the overloads, but maybe not the manual lifting. It all depends on whether I can trick the type system… We’ll see.

  • The other direction is even easier:
    • If you use the proper definitions you have the full functionality.
    • If you don’t care, just use the version with dummy definitions such as type list_r[T] = list[T].
      • No rewrite, nor manual un-lift, nor double interface needed.

I’m afraid I don’t really follow, while this is true for functions that were written with duck typing in mind, it is not at all true for nominal typing, also dummy definitions still don’t help unless I overwrite someone else’s definition using a stubs folder.

I’m talking about the case where someone decides to use your type definitions and writes a library with it. Now in order to interface with that library and use it in conjunction with another library or the standard library, that wasn’t nice enough to use duck types for all its argument types, I will now have to tediously convert back and forth or write wrappers to make the two APIs compose nicely.

You will also have to be incredibly careful with isinstance checks since your alternative types will result in what the type checker considers to be unreachable branches if you happen to use isinstance(x, list) instead of isinstance(x, list_) and vice versa.

I’m not trying to point you in the direction of an implementation that works well enough to be production ready, because I frankly don’t think it exists within the confines of the current type system. I don’t mean this in a discouraging way, experiments like these can yield useful results and guide future extensions to the type system, by demonstrating what we can’t easily express.

Currently I’d still be much better served by using Sequence and MutableSequence and using something like type Keep[T] = Annotated[T, "keep"] to signal intent with the possibility of in the future writing a static analysis tool that validates lifetimes.

2 Likes

While your approach is only static, the reason for not accepting a means to freeze arbitrary objects is actually extremely pertinent here, and you may want to read through the history.

Have you considered using more immutable data structures where you want actual immutability? immutables (pypi library) provides an immutable map with a mutation api returning a new object, tuple provides a reasonable option for a frozen sequence, and frozenset is also available. There are also frozen dataclasses.

1 Like

I don’t want to freeze objects nor do I want to use actual immutable objects.

What I mean is something like this:

# --------- my library ---------
from typing import TYPE_CHECKING

if TYPE_CHECKING:
    if OPT_OUT:
        type list_r[T] = list[T]
    else:
        # full mechanism here
        ...
else:
    list_r = list

def fun(x: list_r[int]) -> list_r[str]: ...

# --------- your code ---------
# If you opt_out:
x: list[str] = fun([1, 2, 3])

Maybe something like OPT_OUT is not possible at the moment, but, anyway, all we need to do is change an import in the library. When the library imports the proper defs the full mechanism is in place, whereas when it imports the opt_out defs, the mechanism is deactivated.

In other words, this works:

from typing import TYPE_CHECKING

if TYPE_CHECKING:
    type list_r[T] = list[T]
else:
    list_r = list

def fun(x: list_r[int]) -> list_r[str]: ...

x: list[str] = fun([1, 2, 3])

EDIT: There’s also no problem with isinstance.

Too late: I’ve just gotten rid of the overloads, removed the need for lift in many cases, and even improved a lot the “lock situation”: no more “class _L1: …”.