PEP 805: Safe Parallel Python

PEPs
41

I’m not following this part, but I think this might be clearer with more details that just aren’t present yet.

FT has to pay costs for synchronization even when not needed: GitHub - facebookexperimental/free-threading-benchmarking: Benchmark results for free-threaded builds of Python · GitHub

The proposal also increases the size of every python object

Only non GC objects (ints, floats, etc) increase in size. GC objects remain the same size.
FT also increases the size of the PyObject header, yet you seem happy with that.

1 Like
42

It does, but this increases it further with some rather hard-to-evaluate claims as reasons for it. The increases needed for freethreading were clearly beneficial, but these ones aren’t as clearly going to be beneficial. It’s hard to be sure if both together are necessary without an implementation showing that it actually results in performance gains that are proportionate to the increased memory use.

I’m aware, but I also think that’s possible to improve without this proposal, and said as much:

One of the hard parts for this proposal is that there’s definitely some potential wins here, but the specific proposed way of getting them seems to be to call freethreading into question, rather than to make the interpreter smarter about freethreading.

This proposal also pushes some things onto users, which is a fine tradeoff to propose, but I would hope that we could come up with something better where the interpreter can do more of the right thing for users automatically. One way is to just use whether or not something is shared and mutable to decide what protections the interpreter has for itself, and to be smarter about applying those protections.

With that route, deep-freezing becomes more useful in general, and would become a performance choice, but that also probably leads to more correct programs when it comes to people not reaching for 3rd party specialized data structures, so I don’t actually mind if people decide to deepfreeze everything they don’t need mutable.

43

PEP 703 makes race conditions more likely, but in the general case it doesn’t introduce a risk that didn’t exist already.

I would contend that it does introduce a risk that didn’t exist already:
There are plenty of simple if that: do this checks in code that are safe with the GIL (whether accidentally or deliberately) that would be unsafe with FT.

But, you are right, the GIL isn’t magic.

If if’s “in the same way” as with the GIL, then ThreadGroup is affected by the issue explained above.

Threads within a ThreadGroup get GIL-like protection But ThreadGroups themselves get strong protection from race conditions. Not quite as strong as multiple interpreters, but close.

You can’t just serialize threads and then decide you have no synchronization to care about.

True, but we already have the GIL. We can’t just throw it away and pretend that nothing will break.
We want to add parallelism, not lessen safety.

This seems to favour task-based parallelism rather than data-based parallelism

Can you more specific about how you are distinguishing the two, and why you think that is?
If by data-based parallelism, you mean operating on large arrays of numbers in parallel, that’s probably best left to numpy and friends.

The PEP explicitly discourages sharing mutable data. Parallel operations on mutable data can’t scale due to physical limitations, as well as being very easy to get wrong. Immutable can be shared among as many threads as you want, safely.

How is it possible to implement this in a Python implementation that doesn’t use reference counting?

With difficulty, but I have a fix for that. Watch this space…

This is neat, but orthogonal to this PEP, right?

Not orthogonal, not a core concept either, but a necessary add-on.
If the aim of the PEP is to enforce necessary locking and claim safety, we don’t want to be replacing races with deadlocks

Sure, but multi-threading bugs in collections have also existed in spite of the GIL. They just have been painfully squashed, one by one, along multiple decades.

There haven’t been that many, I think (largely due to the GIL). There have been plenty of re-entrancy bugs though, which do look similar.

2 Likes
44

It does, but this increases it further

No, it doesn’t. What are you basing this on?

I’m aware, but I also think that’s possible to improve without this proposal, and said as much

How? I’m genuinely interesting to hear how you would do this.

One way is to just use whether or not something is shared and mutable to decide what protections the interpreter has for itself, and to be smarter about applying those protections.

That’s not easy to do efficiently, if at all. Again, if you have a suggestion on how to do what you suggest, I’m eager to know more.

45

I recall that the GIL code can delay the switching to another thread for various reasons.
Which made it more likely that threading races where seen less often.
Some code may work for a “deliberate” reason.

46

They’re actually not safe under the GIL, since you could find the thread being interrupted in between the check and the use.

But overall I’m with Mark on this one - FT encourages coding patterns that make if that: do this an issue, whereas without it basically nobody would’ve written it because it wasn’t worth it. If you wanted efficient parallel operations before, then you set them up in ways where thread switching was never going to be an issue, and so we never really worried about thread switching.

However, FT now makes it relevant, because there is now value in doing Python-level parallelism (as opposed to data level in a C extension or process/machine-level), and so all the places where we never worried about if that: <interruption> do this before we will have to start worrying about. A proposal that avoids drastically increasing that worry[1] is worthwhile.

  1. And I’m not saying this is one, but FT isn’t even pretending to be one. ↩︎

5 Likes
47

I would be curious to see such a snippet, because it’s not obvious to me why that would be safe (or why it would be less safe in free-threaded mode, for that matter).

48

The proposal says the size increases, eg. yet another increase. It doesn’t replace the prior increases, so this is a further increase.

Well, part of the problem is that we don’t have an easy way for users to create objects that are actually immutable. Even this proposal only offers shallow freezing. Once users have such an object, much of the work the interpreter needs protect state no longer applies to such objects.

I’m of the opinion that working on making it easier to have objects that the interpreter doesn’t need to do as much work to protect is a more obvious win here, and that even if this proposal ends up neccessary, it would be better to prove that making immutability easier is not enough on it’s own given that in most good uses of concurrency, shared data doesn’t need to be mutable.

This proposal asks users to declare that objects are safe, but for refcounted objects, there’s already a mechanism for determining if the objects are shared, it was neccessary that this happen for biased refcounting.

Since the dangerous case is mutable, with mutation occuring, and shared, if we can make it easier for users to either:

  • Not share
  • Not have the shared objects be mutable

Rather than declare an object is safe, it’s safe by properties that can actually be checked.

This slims the field of what the interpreter needs to protect significantly. After that, there are ways to go further here, I can’t immediately come up with one that would work in as many cases as this proposal covers, but I can think of a few that work for all python code, leaving extension code to handle their own synchronization needs (this is the current status quo for extension code already).

One would be optimizing out these checks when the only concurrent use is in functions that don’t mutate the shared objects passed to them. That is something possible to determine from bytecode.

I’m going to give this some more thought here. This proposal feels like a large addition that sits ontop of an already provisional large proposal.

1 Like
49

Most aren’t safe with the GIL, like if self.attr is None: self.attr = initial_value() in this case, the race is present with the GIL, but it is much more likely in FT.
If the race manifests as a leak, or some other small effect, then the difference between the GIL build and FT could be running to completion vs not.

However, there are some cases where the GIL does prevent a race.

The “best” example I have found is this:

where (provided the keys are strings or other simple objects) the GIL prevents the race with __setitem__

50

The proposal says the size increases, eg. yet another increase. It doesn’t replace the prior increases, so this is a further increase.

Prior increases? FT isn’t prior though, it is an alternative ATM.
Here’s the proposed struct:

This proposal asks users to declare that objects are safe,

Not true. All objects are safe, the user gets to choose how.

but for refcounted objects, there’s already a mechanism for determining if the objects are shared, it was neccessary that this happen for biased refcounting.

FT doesn’t have a mechanism to determine if objects are shared, as it would be prone to race conditions.It would be nice if it did.

I’m going to give this some more thought here. This proposal feels like a large addition that sits ontop of an already provisional large proposal.

Please do. I’d recommend reading the OCaml and Verona papers if you are serious about this

51

Your OP says that this PEP builds on PEP 703, and you have a section on how it’s compatible? I think many people interpreted that to mean that it was not going to remove the changes that come from FT. If you’re proposing this as an alternative I think that’s not clear.

1 Like
52

I’ve read them, but definitely should re-review them when I get the chance and prior to more in depth thought about this.

I was under the impression that since this wasn’t explicitly saying it replaces anything going on as a result of FT, that the end state of accepting this is having both. If you mean for this to replace FT, then I think the bar is actually higher for other reasons and there’s other lines of discussion required.

Sorry, I could have phrased that better, it requires object declare whether or not they provide their own internal safety or require the interpreter to provide additional checks.

53

Builds on, in the sense that it uses much of the work done for FT. Not that it is a branch of it, or that it inherits the semantics.

In terms of specification, the PEP should stand alone, not be relative to FT. If anything isn’t clear, do please let me know so I can clarify the PEP.

54

it requires object declare whether or not they provide their own internal safety

Only C extensions can declare that they provide their own internal safety.
Python objects are made safe by the VM.

We have to trust C extensions, as we have no way to verify whether or not they are safe.

55

Even if we go this route to improve the safety offered by the VM, I think there’s a lot of room to explore options with fewer points of synchronization, and with more of the safety actually extending to users. None of the provided synchronization helps user code be more correct, it only protects the interpreter state. That’s definitely something that’s worth protecting, but this won’t fix issues where users have shared mutable data and have data races that don’t corrupt state.

Immutability (actual immutability, not shallow freezing) helps users as well as reducing the work to protect state.

I’ve got one more detail to note that I think is possible here before giving this more in-depth thought about the actual performance implications, though the performance other languages that do this get is encouraging:

when it comes to python functions, the VM can determine if a function is able to modify an object. If, instead of marking objects, functions (and effectively method receivers) are marked for extension code, the interpreter would be able to know when there’s even the potential to need synchronization.

56

I think that for python-flint it would be easier to adapt to this version of parallel Python than freethreading. Currently under free-threading it is possible to get a segfault/corruption if mutating some python-flint matrix types while sharing them across threads. I don’t think that there are particularly good use cases for doing the kinds of things that could lead to a segfault but ideally we should get to a situation where caveats are not needed to explain exactly what is or is not safe.

The obvious thing to do is to add immutable matrix types but then that still leaves the question of what to do with the existing mutable matrix types (as used by all current users). Making the matrix types safe under free-threading requires choosing between unfavourable options:

  • Add locks to every single operation harming both single and multithreaded performance (for users who mostly don’t need the locking and mostly don’t even use threads).
  • Set Py_MOD_GIL_USED so that import flint immediately brings back the GIL (defeating the point for anyone who wants to use the free-threading build right now and clearly not long-term desirable either).
  • Make a backwards incompatible change that makes the default matrix types immutable (so users have to switch to using slower-but-safe mutable matrices if that is what they want).

With this proposal (PEP 805) the forward path seems clearer:

  • First declare the current mutable matrix types as local.
  • Then add new immutable matrix types.

I imagine this would have minimal impact for performance of existing single-threaded code compared to adding locks everywhere (although I don’t know how significant PEP 805’s general overheads would be). It would mean that if someone tries to do the potentially unsafe thing of sharing a mutable matrix across ThreadGroups they would get an exception. That would not be a backwards incompatible change for users because ThreadGroups would be new so no one is currently using them. Then if someone wants to share matrices across ThreadGroups to benefit from shared parallelism they can switch to using the new immutable matrix types.

8 Likes
57

Out of curiosity, does your package support FT now?

Just a small note: you don’t need to declare types as local because that’s the default.

You can also do the __class__ switching for immutable matrices as I was noting previously, which might help you with code maintenance: PEP 805: Safe Parallel Python - #14 by dpdani

1 Like
58

Yes, but in the consenting adults sense: you should not mutate a matrix that is shared between threads. If two threads try to mutate the same element of a matrix (M[1, 2] = 3) then you can get a segfault. No one has complained about this though and I’ve only seen it happen in contrived situations that deliberately provoke the problem.

If you stick to the “don’t mutate shared objects” rule then everything is fine. In other words you should respect the same constraints that this proposal (PEP 805) imposes but those are not enforced currently.

2 Likes
59

There were some issues that came up with the aforementioned previous attempt at making it possible to convert module and class dicts into frozendicts that caused the idea to be abandoned that would need to be resolved. This isn’t intended to be a comprehensive list, but these were the big ones in that case.

  1. PEP-810 lazy imports rely on replacing a lazy_import object in the module dict when accessed. Under that implementation this caused TypeErrors if a lazy_import object was accessed after a module was frozen.
    • Under -X lazy_imports=all this is (almost) every import
    • A caveat was that the implementation from that attempt didn’t manage to replace the reference to the original unfrozen dict held by functions so lazy imports did work inside functions, though not intentionally and not correctly.
    • Both lazy and regular imports can also add attributes to a module dict that aren’t there when the module is created
  2. __annotations__ is lazily created and added to class and module dicts only when first successfully accessed under PEP-649/749 annotations.
  3. Using freeze(__module__) inside a module prevents monkey patching, which is not uncommon (think unittest.mock.patch or what gevent does to the stdlib).

Note that this is specific to modules and to a lesser extent, classes. I’d love a proper way to make instances frozen - frozen dataclasses are slow[1].

I’ll also note that the PEP asserts:

Almost all objects in Python have a __dict__ attribute. Freezing an object will convert its __dict__ into a frozendict.

There’s now the fairly common case via @dataclass(slots=True) or @attrs.define of objects with slots and potentially without a __dict__. I do think it’s probably necessary to explain how such objects will (or won’t) be handled.

  1. compared to non-frozen classes, about twice as slow to make the class and 3.5x as slow to make each instance ↩︎

1 Like
60

I’ll answer you point by point:

  1. Any module with lazy imports would not be a good candidate for freezing as is. Modules can be synchronized, which would allow lazy imports and sharing.
    An alternative might be change the implementation of lazy imports to use in-place replacement of the PyLazyImportObject with a module, not changing the module itself. This would only work for lazy import foo not lazy import foo from bar.

  2. We can eagerly add empty __annotations__ and lazily initialize that, not changing the module.

  3. That it is true. Making something immutable prevents you mutating it. It is up to the developer to choose. You can always synchronize if you want mutation and sharing and are willing to loose some safety.