A feature that went away

This is my second and last thread (see first thread) about a library about mutability I was working on.

After receiving feedback on this forum (thank you, @Daverball) I solved most of the problems and I was about to publish the second version, but today Pyright in VSCode was updated from v. 1.1.363 to v. 1.1.364, breaking my code.

I won’t open an issue on this because it’s clear Eric Traut is not a fan of my work.

I’ll just document what happened, technically, and move on.

My library introduces read and write qualifiers so that one can control mutability. For instance:

def f1(xs: list_r[int]):
    # the list CANNOT be modified here...
    xs.sort()          # type error
    xs[0] = 4          # type error

def f2(xs: list_w[int]):
    # the list can be modified here...
    xs.sort()

list_r[T] is covariant in T, of course. With a little bit of work, one can support other types such as set and dict (Indeed, the library already supports them).

I think write permissions should be explicitly granted:

def f1(xs: list_r[int]) -> int: ...

def f2(xs: list_w[int]): ...

def f3(xs: list_w[int]):
    res = f1(xs)
    f2(w(xs))
    ...

Since xs is of type list_w[int], we can pass it to f2, but f2(xs) won’t work: we have to be explicit and write f2(w(xs)), where w is a little function that gives write permission. Of course, w(xs) will raise a type error when xs is read-only.

This is not a simple thing to do, though. This was my first attempt:

class _LK1: ...
def f[L: _LK1](xs: list_w[int, L]):
    ...

class _LK2: ...
def f[L: _LK2](xs: list_r[int], ys: list_w[int, L]) -> list_w[int, None]:
    f(ys)               # type error
    f(w(ys))            # OK

How does it work?

Here’s the idea:

  • list_w has an additional type parameter for the “lock”.
  • when list_w is first created, the lock is None
  • w changes the lock from None to Any.
  • f(ys) doesn’t work because None is incompatible with _LK1.
  • f(w(ys)) works because Any is compatible with _LK1.

Note that the return type should have a lock equal to None or the caller might pass the returned value back to the same function without having to use w.

This is not ideal, but it works.

Is there another way?

If only we could manipulate function signatures!

Well, it turns out we can…or, better, we COULD :frowning:

First, the idea:

@lock
def f[L](_: L,
         xs: list_r[int],
         ys: list_w[int, L],
         zs: list_w[int, L]) -> list_w[int, L]:
    ...

Up to version 1.1.363, lock transforms that function into the following:

def f(xs: list_r[int],
      ys: list_w[int, _Unlocked],
      zs: list_w[int, _Unlocked]) -> list_w[int, None]:
    ...

Try the (stripped-down) code for yourself on Pyright Playground.

Up to v. 1.1.363 we get

Type of f is (x: int, y: X_[_Unlocked], z: str) -> X_[None]

From 1.1.364 onwards we just get

Type of f is (x: int, y: X_[LK@f], z: str) -> X_[LK@f]

The main idea is this:

  • we add a “temporary” _: LK to get a hold of the type paramater LK
  • we force Pyright to replace LK with _Unlocked and None.
  • we remove _: LK from the final signature

If some of you are thinking “Why don’t you do the transformation by hand?”, keep in mind that the original function still sees the original signature. Only the callers see the modified signature. That’s the whole point! We can do this transformation by hand not by modifying the signature, but by adding an “external” signature through @overload. Basically, @lock automates that.

I was thrilled when I discovered this trick.

I had already solved the lifting problem. I was able to replace all the overloads (slow) with a type-level binary search (100x faster), and then even moved to small local lifts (much lighter and even faster).

I’ve learned a lot in these few weeks and I believe I can do pretty interesting things with the Python type system, but it seems my energy would be better spent elsewhere, so I’m going back to (ethical) hacking where my “abuses” are rewarded :wink:

After @erictraut [1] answered you saying you were “abusing the type system”, I feel that you read that in the wrong way. Being a non-native english speaker myself, I can see how that could happen – maybe words similar to “abuse” in your language have connotations that suggest a personal attack or something like that. I’m pretty sure, however, that he was only making the objective assertions that (1) Python’s type system was not designed to handle your problem; and (2) you are very likely to keep struggling and fighting against type checkers if you want to depend on them for what you’re doing.

Python’s type system is a new endeavor, and has a lot of poorly specified behavior when it comes to edge cases. (Just take a look at the other threads in this Discourse category!) To deal with that, type checkers are always constantly changing. You should never depend on their behavior just because “it worked when I twisted it just right”. You should instead depend only on documented behavior – at least if you want to be able to update their versions without having too much trouble.

If you want a type checker that does reliably and precisely what you want in specific cases that you care about but are under-specified – well, I guess your best bet is to either write your own type checker, or try to convince the rest of the community that supporting these cases officially would be nice for them.

If you just want to hack around and see how much you can twist other people’s software for your own amusement and objectives – then that’s awesome! It’s Free Software after all. You just shouldn’t feel the right to complain when the next version of the Free Software you’re using breaks something they never promised to support in the first place.


  1. At-mentioning him since his name was mentioned above. ↩︎

9 Likes

I don’t think it was an abuse at all. I just wondered why you were putting so much focus into preventing mutability of lists, instead passing in an iterator, or even just using tuples.

No, I read it the right way. “Abuse” means outside the supported use cases.

Let me point out, though, that we’re talking about two types of “abuses”: one is undefined behavior (the feature that went away), but the other is a technical limitation as it only depends on the complexity of the problem. It’s analogous to calling a 50-parameter optimization problem an abuse because the optimizer only supports 10 parameters. That’s a little improper, in my opinion. That’s why I wrote “abuse” between quotation marks.

As for complaints… I can’t see any in my post. I only stated facts.

The hacking part wasn’t a joke: I use Python (+ OCaml + …) to develop code analyzers to find bugs in software. I took a short break to try to extend Python for my needs. That’s all.

I usually work with big graph-like data structures that I access in complicated ways. Those data structures are accessed in read-only mode most of the time, with mutations sprinkled here and there.

While my examples were about lists, one can support any type. You just need to split your interface/implementation into read-only and write parts, and you’re ready to go. You decide what’s considered read-only and write. For instance, you can allow “internal mutability” if you want.

1 Like