Per some discussion with @eric.snow at the PyCon US sprints, I decided to change the PEP to specify only the SharedObjectProxy type, without including the share() function or the __share__ protocol. The plan is to implement these in a future PEP, but they should include a C layer for implementing natively shareable objects in C extensions.
As such, I’ve decided to submit PEP 797 to the SC. There are likely still some sharp edges here, but I think that’s just the nature of proxy types. I’m not entirely convinced that more discussion will be of much benefit.
FYI, It looks like the current text of the PEP still contains mentions of interpreters.share.
Also, I think that we kind of got distracted (and then I forgot about this thread, lol). We were in the middle of a discussion about how to facilitate “unwrapping” of SharedObjectProxy objects. I guess, it still feels a bit weird to me that there is a way to wrap an object in a SharedObjectProxy, but no way to unwrap it (which should basically be a no-op if the original object belongs to the current interpreter and a simple pickle/unpickle, if it was created in another interpreter).
Especially given that wrapping also happens implicitly when objects are passed between interpreters via other already shared objects.
Unless you’ve changed the PEP significantly since I last looked, the issue comes from the fact that:
An unshareable object might end up back in it’s original interpreter. Currently, such an object would remain a SharedObjectProxy even though it doesn’t technically need to be (there is no need to switch contexts or share arguments/return values, if the underlying object already belongs to the current interpreter).
“Copiable” objects will still end up being “shared” when crossing interpreter boundaries via another shared object. It’s totally conceivable that a consumer of such a large object might want to make a local copy before working with it.
Imagine, if I get an open file foo passed to me via a SharedObjectProxy (so some other interpreter owns the opened file). If I call lines = foo.readlines() on this proxy, I would receive a SharedObjectProxy that wraps a large list of strings. Every interaction with this list requires a context switch and each argument / return value gets automatically wrapped in a SharedObjectProxy (unless they are already shareable).
So, if I know that I am going to be interacting with that object a lot, I might prefer to obtain a local copy of this list of strings (by using the standard pickle/unpickle mechanism that allows sending “copiable” objects between interpreters).
I think an interpreters.dont_share/interpreters.materialize function would somewhat defeat the point of a shared object proxy. It is not designed to be efficient. It is, by nature, an expensive object meant only for when no other option is available. It is also designed to be effectively invisible at runtime (apart from using type()), so explicit APIs that work on an object proxy also hurt that notion.
But anyway, context-specific copying APIs will still work on an object proxy. In your example, the simple solution would be to just do lines = list(foo.readlines()), which is perfectly explicit and doesn’t involve a new API.
I think that’s a bit of a shame, and we don’t really need a PEP to just add that type.
The protocol is what makes this extensible and interesting for others to participate in. Without that, it’s almost a bug that we don’t have a generic proxy that works across subinterpreters.
I could see a little bit of value in an unwrap_if_local type function (that fails if it’s remote, forcing you to use deep copy via the proxy), but mostly for debugging. The perf advantage would likely only matter in a very tight loop, and if you don’t already know the source of the objects you’re using in a tight loop, you’re probably better off with a slightly different algorithm for the case where you’re (unexpectedly) dealing with actual proxies anyway.
FWIW, I’m finally getting around to writing my PEP about object sharing. It will include a rough equivalent of what Peter had proposed (minus the proxy), plus C-API.
I also think the proxy doesn’t need a PEP. A variety of such proxies would be useful and shouldn’t require steering council involvement.
That said, I do think we need to be sure we really want to introduce a proxy object that implements all special methods. I suspect there are some cases where such a proxy will prove problematic. Furthermore, we may want to consider the general question of proxies outside the context of cross-interpreter sharing.
Yeah, Eric brought this up too. I’m not really sure what to do at this point; withdrawing the PEP and implementing it on my own will look like I’m trying to bypass the SC, at least to those without context.
Hm, maybe not a bad idea, but at this stage is there any actual harm in just keeping this as a (small) PEP, and then just letting the SC rubber-stamp it? I recognize that it’s not ideal to make small PEPs, but this has already gone through discussion and been submitted, so trying to undo that seems a bit overkill.
More generally, I’m just not very comfortable with the idea of writing a PEP for a feature, removing that feature and masking/deferring/withdrawing the PEP, and then implementing it in CPython without acceptance. It feels like a large breach of process and not something that we want to set precedent for doing. If an object proxy is universally seen as a good idea, then I don’t see why the SC would have any trouble approving it.
This only works because readlines happens to return a flat structure. Replace file.readlines with request_data.parse_json() and suddenly the “solution” isn’t so simple anymore.
But even worse – doing list(shared_proxy) (or something like copy.deepcopy for non-flat objects) actually ends up performing O(N) context switches and data transfers between interpreters. Instead of serializing the whole object in the original interpreter, sending the serialized data in one piece and deserializing in the current interpreter you basically end up iterating over the list (calling __iter__ and __next__ a bunch of times). Except the iteration loop is happening in the current interpreter, while the object that is being iterated over lives in another interpreter, so each __next__ call switches to a different interpreter and back.
A “proper” implementation would need to 0) switch to the interpreter that “owns” this object, 1) “unwrap” the SharedObjectProxy in the “owning” interpreter, 2) serialize that object in the “owning” interpreter and 3) deserialize it in the “target” interpreter. Or alternatively – replace “serialize/deserialize” in steps 2/3 with “put/get via a shared queue”. It might be possible to implement this in “userspace” if there was a way to do steps 0 and 1.
A trivial example of a “tight loop” would be exactly something like list(shared_object_proxy) or copy.deepcopy(shared_object_proxy). Also, I’m not sure, what you mean by “a slightly different algorithm”?
Also, you say “(unexpectedly) dealing with actual proxies” – but I’m not talking about “unexpected” shared proxies. In the situation I am describing, I know perfectly well that the object I’m working with will be a proxy.
Quoting from the “Motivation” section in the PEP itself:
one of the primary applications of subinterpreters is to bypass the global interpreter lock, it is fairly common for programs to require highly-complex data structures that are not easily shareable. In turn, this damages the practicality of subinterpreters for concurrency.
From this quote, one would expect that SharedObjectProxy should allow me to share some “highly-complex data structure” that isn’t easily shareable (either because it’s unpicklable, or because it maintains a shared mutable state that needs to be “owned” by a single interpreter and can’t simply be copied per-interpreter).
A shared object proxy is designed to be a fallback for sharing an object between interpreters. A shared object proxy should only be used as a last-resort for highly complex objects that cannot be serialized or shared in any other way.
This quote implies that the SharedObjectProxy mechanism should be avoided whenever possible. I agree. This makes perfect sense – you should use the SharedObjectProxy to only wrap the “highly-complex” object that can’t be copied, but then as soon as you are done with the part that is “unpicklable”, you should switch back to using other (better) communication methods such as “native” sharing or “copying”.
But with the current implementation, “once you go SharedObjectProxy, you can’t go back”. Everything[1] that is touched by a SharedObjectProxy also becomes a SharedObjectProxy and there is no way to switch back to the “more efficient” method (copying).
with the notable exception of “natively shareable” immutable objects, but most python objects don’t fall into that category ↩︎
Sure it is – just serialize the data and send it to the other interpreter instead of making a proxy. JSON in particular is pretty easy to deal with because it’s comprised of only shareable data.
Generally speaking, I’m not convinced that this isn’t an artificial use case. For this to be useful, you would have to know enough about an object to know that it’s an instance of SharedObjectProxy, but not enough to know how to copy it. I can’t think of an example where that would both be true and not easily solvable through other means.
Yeah, this is the key point that I was also getting at with my “unexpected proxy” comment, and add on that you also need to know which interpreter you created it in and that it’s the same as the current one, but apparently you don’t have another way to get back to it?
In any case, the SharedObjectProxy is meant to be a fallback - functional, but not ideal performance. The reason I’m so keen on the __proxy__ protocol is because it allows any object type to provide an optimised proxy for its own behaviours, and so we don’t have to build every possible optimisation into the fallback
Me too, but we won’t get it in this PEP. And, if the SC decides to reject __share__/__proxy__ for whatever reason, it’d be nice to have the proxy around anyway.
Thank you for submitting PEP 797 (Shared Object Proxies) to the SC. After careful consideration, we have decided to reject the proposal.
The primary reason for our rejection is the lack of a clear, coherent use-case and accompanying support from users. The Motivation section describes the gap in sharing objects between subinterpreters, but doesn’t provide any background on why this is a problem for existing users of subinterpreters, and gives no practical, real-world examples to test the proposed solution against. This makes it very difficult to determine whether the solution’s semantics are appropriate, and whether the cost in complexity, performance and maintainability are worth it. And those costs are, in our view, quite significant.
Even setting that aside, we have other concerns with this specific proposal that we want to highlight, as they would need to be addressed in a future resubmission (or other proposals):
The proposed solution is extremely complex, involving interactions between different subinterpreters in essentially uncontrolled and unknowable situations. This can easily create new, undesirable interactions between PEP 797’s shared object proxies and other parts of the interpreter, standard library, or third-party code.
The shared object proxy’s handling of methods, especially dunder methods, make its behaviour very hard to predict and reason about.
The attempt at a transparent sharing of objects between subinterpreters is at odds with the original design of subinterpreters, especially subinterpreters with their own GIL. Sharing data at the Python object level is very difficult when the interpreter internals, as well as the standard library and third-party code, never expected such sharing to occur.
The thread state swaps the shared object proxy uses are incredibly inefficient when contention occurs, much more so than the GIL would have been.
All of those technical concerns together make shared object proxies a dangerous attractive nuisance. They seem simple but have tremendous impact on semantics, a lot of complex pitfalls, problematic action at a distance, and bad performance.
Even setting aside the lack of real-world users, there seems to be no consensus on this approach even among the few people actively involved with subinterpreters. Consensus is an important part of PEP discussions.
We don’t want to discourage further discussion and development of these ideas, but we do urge you to keep all those points in mind as you do so.