PEP 797: Shared Object Proxies

PEPs
1

Hi everyone,

After extensive discussion and delays, I’m excited to present PEP 797, which proposes introducing an object proxy to share arbitrary objects between subinterpreters.

Here’s the full text:

8 Likes
2

I think that the “Interpreter Switching” section is a bit vague at the moment.

For example, what happens if I do this:

def foo(target):
    # This code is supposedly run in the main interpreter,
    # but `unshareable` was originally owned by `interp`
    target.write("Our chief weapon is surprise...")

interp = interpreters.create()
proxy = interpreters.share(foo)
interp.prepare_main(foo=proxy)
interp.exec("""
with open("spanish_inquisition.txt") as unshareable:
    foo(unshareable)
""")

Is the unshareable/target argument implicitly wrapped in its own SharedObjectProxy (so the target.write call actually ends up executing in the subinterpreter)?

3

Thanks for the feedback.

Yes, that’s correct. Arguments and return values from functions are implicitly wrapped in a SharedObjectProxy, unless the object is natively shareable. So, for example, if target were to be a string, then that wouldn’t require the creation of a new proxy. I’ll add a section clarifying this.

1 Like
4

Follow up question:

I am slightly concerned that SharedObjectProxy objects might be “viral”. They are seemingly intended to be used “transparently” in place of the wrapped object/type. But since arguments / return values are shared automatically, it might be exceedingly easy to accidentally “infect” a performance critical part of the program with shared proxies.

For example, a shared object might be accidentally passed to a library function that performs caching (or other operations that affect the global state). This would then “poison” the cache / global state, causing the library function to start returning values that actually live in a different interpreter.

One mitigation for this might be to automatically “unwrap” the proxies when they are passed / returned back to their originating interpreter. But that doesn’t quite solve all potential “virality” issues.

1 Like
5

I don’t think “virality” is the right term, because objects referenced by shared proxies aren’t also wrapped in a proxy. But anyway, this is why proxies are supposed to be a fallback mechanism. If they’re causing problems, use serialization in your app instead. I don’t expect proxies to be very useful in performance-critical code paths.

I’m not sure I understand your example. A function returning a shared object proxy isn’t inherently bad, especially if all you’re doing is looking at a cache. They’re slower when accessed from multiple threads, but that’s true for lots of objects.

More generally, though, global state is a bad idea in concurrency situations, and care is going to be needed regardless of what you’re using.

Hm, what does “unwrap” mean here? If you mean unpacking the underlying object, then that’s not possible, because it isn’t thread-safe to access the object from another interpreter.

2 Likes
6

Just a small comment: the specification for share should probably explicitly say that it returns the proxy, or else people may think it somehow converts the object into a proxy.[1] Perhaps something like:

If obj is natively shareable, return obj. Otherwise, returns a SharedObjectProxy that wraps obj.

Also, does “natively shareable” include SharedObjectProxy objects themselves? As in, what happens if someone calls share(obj) where obj is already a SharedObjectProxy?

  1. Or maybe I just think that after reading too much of the other thread about frozen objects! :slight_smile: ↩︎

1 Like
7

Yes, by “unwrap” I meant to unpack the underlying object. But I am only suggesting to perform this unwrapping, when the SharedObjectProxy ends up back in it’s original interpreter.

def round_trip(target):
    # target is a SharedObjectProxy here
    return target

interp = interpreters.create()
proxy = interpreters.share(round_trip)
interp.prepare_main(round_trip=proxy)
interp.exec("""
with open("spanish_inquisition.txt") as unshareable:
    result = round_trip(unshareable)

    # result is automatically unwrapped, because the underlying
    # `unshareable` object is owned by the current interpreter
    assert result is unshareable
""")

I don’t see, why this wouldn’t be thread-safe. Any reference to unshareable is either wrapped in a SharedObjectProxy (that grabs the GIL of the correct interpreter when interacting with it) or is unwrapped in the interpreter that owns it (just like any normal python object).

Can you clarify, which exact methods / properties / behaviours of the underlying object are proxied / emulated. For example:

  1. Are __hash__ and __eq__ proxied?

    • is share(foo) indistinguishable from foo when used as a dict key?

  2. Does SharedObjectProxy proxy the underlying __str__ and __repr__ or does it have its own __repr__?

    • is share(foo) indistinguishable from foo when printed / displayed in the REPL?

  3. Does SharedObjectProxy use __instancecheck__ / __getattribute__ magic to pretend to be an instance of the correct class?

    • does share(foo).__class__ return SharedObjectProxy or share(type(foo))?
    • is isinstance(share(foo), SharedObjectProxy) true?
    • is isinstance(share(foo), share(type(foo))) true?
    • is isinstance(share(foo), type(foo)) true?

Also, if non-natively-shareable arguments / return values are automatically shared, then there is currently an asymmetry between the two sharing methods (for non-natively-shareable objects).

  • when explicitly passing objects between interpreters (via Interpreter.call or interpreters.Queue or similar), you can either

    • share the object, sending a thin wrapper
    • pass the object as is, sending it via pickle

  • when implicitly passing objects (as arguments / return values of shared functions)

    • the objects are implicitly shared, sending thin wrappers
    • (no way to send “normally” via the pickle/unpickle method)

I suppose, you could always manually pickle.dumps / pickle.loads the objects, but that’s a bit unwieldy. It would be nice to have a way to either “materialize” SharedObjectProxy objects, or to disable the automatic shareing of arguments / return values.

8

Okay, I’ll include this in my revision.

Yeah, share() on a SharedObjectProxy will just recreate the proxy.

Edit: I don’t know what I was thinking here. A SharedObjectProxy is “natively shareable”, so the proxy is just returned.

I guess that’s plausible. I feel that it does create some counterintuitive behavior, but practicality beats purity. Can you provide an example where unwrapping in the original interpreter would be useful?

Yes, every method is proxied.

It copies the __repr__. I’ll clarify this in the PEP.

Well, there’s not really any magic involved. The code roughly looks like this:

# This code is actually in C, but I'll use Python for example's sake
def __getattribute__(self, name):
    with self.switch_interpreter():
        attribute = self.value.__getattribute__(name)
        return share(attribute)

To the wrapped value, there’s no discernible difference between calling __getattribute__ in the main interpreter and calling it from a SharedObjectProxy. This applies to pretty much every other dunder method.

Objects that are natively shareable, such as bytes objects from pickle.dumps, won’t be wrapped in a SharedObjectProxy. You can still use the pickle/unpickle method, as far I can tell.

I see what you’re getting at, but I’m worried that will get really annoying. For example, if we required return values to explicitly be shareable (either natively or via a proxy), then that would break any use-case involving unshareable attributes, because __getattribute__ methods won’t return proxy objects. In other words, it will be really frustrating to expect objects to be written in a special way to support shared object proxies.

Stepping back a little bit, I think it would be a good idea to expose a protocol for serializing/deserializing an object for use across multiple interpreters. Eric has already expressed a desire to have this in C, but a Python-level protocol seems much simpler and perhaps feasible for this PEP. If an object implements such a protocol, then share() can consider that object “natively shareable” and skip the creation of a proxy.

1 Like
9

Well, the first thing that comes to mind is for passing basically any kind of “callback” function that mutates its arguments. But I’ll admit that’s fairly vague. I’ll try thinking of some more practical / concrete examples for this idea and for my concerns about “proxy poisoning/virality” once I am 100% sure that I completely understand the expected semantics of SharedObjectProxy objects.

TBH, it’s been a while since I touched these parts of python, so I might be misremembering, but IIRC overloading just the __getattribute__ method is not sufficient to forward all dunder methods, because in a lot of cases the dunder methods are looked up on the type of the object, not on its instance.

Case in point:

class Proxy:
    def __init__(self, wrapped):
        self._wrapped = wrapped

    def __getattribute__(self, name):
        return getattr(super()._wrapped, name)

hello = Proxy("hello")

# this prints <__main__.Proxy object at 0x...> instead of 'hello'
print(repr(hello))

# this fails with TypeError: unsupported operand type(s) for +: 'Proxy' and 'str'
hello + " string concatenation"

This applies to a bunch of other dunder methods too. Actually, I was surprised that isinstance(hello, str) works out of the box in this case. Looking at projects like wrapt, implementing “full” object proxies is actually quite involved, because the ObjectProxy type itself actually has to overload almost all of the dunders, so that type(hello).__any_dunder__ will actually delegate to the wrapped value.

idk, if this also applies to the “real” C implementation, but I think that the actual semantics should be spelled out explicitly, just to avoid any potential confusion.

Hmmm… I agree that explicit / opt-in shareing of arguments / return values would be very annoying. What about making it opt-out, though? I see two potential ways to implement such an “opt-out” mechanism.

Either we can opt-out before the object is passed into / returned from the shared function:

def example(argument):
    # here, `argument` is a copied (pickled-unpickled) dict,
    # NOT a SharedObjectProxy, because the caller "opted out"
    value = [1, 2, 3]
    return interpreters.dont_share(value)

interp = interpreters.create()
interp.prepare_main(example=interpreters.share(example))
interp.exec("""
argument = {"key": "value"}
value = example(interpreters.dont_share(argument))
# here, `value` is a copied (pickled-unpickled) list,
# NOT a SharedObjectProxy, because the example function "opted out"
""")

Honestly, this option is quite ugly and I am mostly including it here for completeness sake. It relies on the caller knowing that it is interacting with a shared function and presumably dont_share would also need to return some kind of special “marked” / “unshared” object.

Or we can “opt-out” after the proxy object is already in another interpreter:

def example(argument):
    local_argument = interpreters.materialize(argument)
    return [1, 2, 3]

interp = interpreters.create()
interp.prepare_main(example=interpreters.share(example))
interp.exec("""
value = example({"key": "value"})
local_value = interpreters.materialize(value)
""")

# interpreters.materialize(x) is rougly equivalent to
def materialize(x):
    if type(x) != SharedObjectProxy:
        return x

    # the code below should basically do whatever the current "normal"
    # pickle-based sharing does when sending data between interpreters
    with x.switch_interpreter():
        wrapped = x._wrapped
        data = pickle.dumps(wrapped)
    return pickle.loads(data)

IMHO, the second option is significantly more elegant. The only downside of this option is that it still has an overhead for initially creating and sending the shared object (on top of the “normal” overhead of pickling / unpickling).

Yes. I am essentially proposing to add a convenient wrapper around pickle.loads(pickle.dumps(x)) with a few asterisks:

  • no-op for objects that are natively shareable (or were already materialized)

  • the actual implementation of interpreters.materialize might be slightly more efficient than the “naive” loads(dumps(x))

  • the “naive” implementation actually has a subtle bug, because it uses “pickle.dumps” from the current interpreter, rather than from the interpreter that owns x._wrapped (this might have some performance implications and also might produce different results if copyreg or monkey-patching of the pickle module was used)

Implementing this at the type/class/protocol level unfortunately means that it can’t be easily applied to third-party (or even builtin) types. What if I have a shared function/method (that I want to be executed in the main interpreter), but I need to pass it a long list/dict/set with data to be processed.

In this case, it would make more sense to copy this list/dict/set once using the “normal” sharing mechanism instead of switching between interpreters and creating further SharedObjectProxy objects on every operation in the main interpreter.

10

Oh, I was using __getattribute__ as an example. This exact sort of function is replicated for every dunder method. For example, the __add__ method is roughly equivalent to:

def __add__(self, other):
    with self.switch_interpreter():
        result = self.value.__add__(share(other))
        return share(result)

I still feel that this is going to become a huge headache, because now you need materialize calls everywhere. For example, to simply call a method on an object and get the result, you need three materialize calls:

# Materialize the proxy
proxy = materialize(shared)
# Materialize the method descriptor
method = materialize(shared.method)
# Materialize the result
result = materialize(method())

This is an awful lot of boilerplate compared to result = shared.method(), and I’m also struggling to see the benefit of having a “materialize” proxy everywhere vs an on object proxy everywhere.

You would just ask the maintainer(s) nicely to add support for the protocol, or otherwise inherit from the type yourself and implement it. This problem would apply to pickle too, so I don’t think it’s too much of an issue in practice.

As an alternative to something like materialize (though I don’t understand how that would solve this issue, since the object would still be an “unmaterialized” proxy object rather the list), we could add something like a PickleWrapper to explicitly opt-in to sharing via pickle.

For example:

def example(data):
    for item in data:
        print(data)

shared = share(example)
# Pretend this is in another interpreter
shared(PickleWrapper([1, 2, 3]))

PickleWrapper will be natively shareable (and thus won’t trigger the creation of a proxy) and will simply calls loads/dumps as part of its crossinterpreter protocol.

1 Like
11

I think, you completely misunderstood what I am proposing. interpreters.materialize is an opt-out mechanism. So the arguments / return values of shared functions will still be implicitly shared by default.

interpreters.materialize is supposed to take existing SharedObjectProxy objects and return regular non-proxied python object (by “sending” them to the current interpreter “normally” like via Interpreter.call or interpreters.Queue - either by pickling/unpickling or “natively sharing” them).

So, for example

def do_work_in_main_interpreter(shared_from_subinterpreter):
    # The `shared_from_subinterpreter` object is owned by the
    # subinterpreter and it might not even be pickleable
    assert type(shared_from_subinterpreter) == SharedObjectProxy
    data = shared_from_subinterpreter.method_that_returns_a_huge_list()

    # `data` is a proxy, the real "huge list" is owned by the sub-interpreter
    assert type(data) == SharedObjectProxy
    assert type(data[0]) == SharedObjectProxy

    # You can interact with this proxy as expected, but it may be slow
    # due to all the interpreter switching and SharedObjectProxy creation.
    print(data[0] + data[1])

    # Now, we send the data from the sub-interpreter to the main interpreter.
    # Of course, the "huge list" must be pickleable or natively shareable.
    local_data = interpreters.materialize(data)

    # `local_data` is just a list of normal python objects
    assert type(local_data) == list
    # Working with local_data (or its elements) is now blazingly fast (tm),
    # because it's just a normal python list of normal python objects,
    # owned by the current interpreter.
    for element in local_data:
        ...


interp = interpreters.create()
proxy = interpreters.share(do_work_in_main_interpreter)
interp.prepare_main(do_work_in_main_interpreter=proxy)
# Sub-interpreter creates an unshareable class.
# Calling the shared `do_work_in_main_interpreter` function
# implicitly calls `interpreter.share()` on its arguments
interp.exec("""
unshareable = MyUnshareableClass(...)
do_work_in_main_interpreter(unshareable)
""")

Your PickleWrapper is what I called interpreters.dont_share (the “opt-out before” option). But I still think that the interpreters.materialize option has better UX in this case.

12

I think there is a terminology issue in the current wording of this PEP. Time for some bikeshedding.

Here’s the problem. Before this PEP, there were 2 ways to “use python objects in multiple interpreters”:

  1. Pass a “natively shareable” object to Interpreter.call or similar
  2. Pass a pickleable object to Interpreter.call or similar

The current python docs and the original PEP 734 Multiple Interpreters in the Stdlib called these two ways “sharing” and the relevant objects “shareable”. Using that terminology, “to share an object” meant “to make it available to another interpreter” and “shareable” objects were those that can be passed to methods like Interpreter.call and Queue.put.

Now, PEP 797 introduces a third way of using python objects across multiple interpreters:

  1. Pass a SharedObjectProxy-wrapped object to Interpreter.call or similar

In practice, these 3 approaches are quite different, but right now they all are called “sharing”. As a consequence, even the PEP itself often ends up using terms like “shared”, “shareable”, “unshareable” and “to share” in inconsistent or contradictory ways.

Sometimes “share” is used to refer (exclusively) to the first 2 kinds of sharing. For example, the PEP claims that “Many Objects Cannot be Shared Between Subinterpreters” (and gives an example of file objects returned by open()).

But then in the main body of the PEP, “share” is used to refer (exclusively) to the new third kind of sharing. The name of the new interpreters.share function implies that it shares objects, but that’s not what it actually does. It doesn’t share objects, it makes them shareable. Recall that “to share” was supposed to mean “to make available to another interpreter”, but interpreter.share by itself only prepares a proxy that might be eventually shared.

Additionally, even if it was called something like interpreters.make_shareable, it still wouldn’t be accurate for what it does, because calling it with a pickleable object (which is supposed to already be “shareable” according to the established terminology) “changes” how it behaves in terms of mutability. Why is calling share/make_shareable on an already shareable object change its sharing behaviour from copy/value semantics to reference semantics? To be clear, IMHO, the problem is not in the behaviour of the function, but in its name.

All of this makes talking about these 3 different “sharing” mechanisms extremely annoying. You basically have to always clarify whether you mean “natively shared”, “non-natively shared” or “shared via proxy”. Without these clarifications, you either end up making contradictory statements (the very first code example in the PEP talks about how to “share” an “unshareable” object) or meaningless statements (literally all objects are “shareable” now, so saying that an object is “shareable” doesn’t convey any information).

I think that it might have been a mistake to call the 2 original PEP 734 ways of passing objects between interpreters “sharing”. It really should have been called “sending” or something, but unless we are prepared to change the terminology that was established in PEP 734, we are stuck with the current meaning of “share”.

Here’s a proposal to change the wording of this PEP to (hopefully) make it a bit less contradictory:

  1. Let the words “shareable” / “unshareable” keep their PEP 734 meaning (“shareable” means either “natively shareable” or “non-natively shareable” aka pickleable). File objects and other non-pickleable objects are still considered “unshareable”.

    “To share” still means refers to what happens when a shareable object gets passed to Interpreter.call or Queue.put or similar communication method.

  2. Change the PEP name from “Shared Object Proxies” to “Shareable Object Proxies” (or maybe even just “Shareable Proxies”). The implication is that the Proxies themselves are shareable, even if the objects that they are wrapping might not be.

    Also, this means that an object “gets shared” only when that object is actually passed to another interpreter. Creating a proxy isn’t “sharing”, because no exchange between interpreters is performed. Passing this proxy to another interpreter is “sharing”, but only the proxy itself is “shared”, not the underlying object.

  3. Change the name of the interpreters.share function to interpreters.proxy or something similar. Some other candidates might be interpreters.make_proxy, interpreters.wrap, interpreters.delegate, interpreters.expose or interpreters.publish.

    Not too happy with any of those names, so further bikeshedding is welcome. I think that ideally, the function name should still be an obvious verb, but I couldn’t come up with anything better right now.

1 Like
13

interpreters.dont_share takes an existing SharedObjectProxy and creates a copy of the wrapped object – PickleWrapper avoids the creation entirely. I’m not a big fan of giving proxy objects any special behavior to reconstruct an object; that’s something concurrent.interpreters should have a mechanism for, not a proxy.

I think there’s some confusion about what a SharedObjectProxy is. An object proxy is natively shareable. There’s no real difference between sharing a str and a SharedObjectProxy, except that the latter is a more complicated object.

This basically applies to the rest of your post; there’s no point in any distinction between “native sharing” and “proxy sharing”, because they’re fundamentally the same thing and use the same protocol.

I’m open to renaming share. Perhaps ensure_shareable?

I think it’s really nitpicky to say that the wrapped object might not be “shared”, because I can’t think of a use case where one would want to create a proxy without ever sending it to another interpreter. But, if others express support for this idea, I’ll defer.

14

Uh… No it doesn’t? At least not the interpreters.dont_share that I suggested. I don’t want to be rude, but please consider re-reading my comment. This is the second time you are making incorrect assertions about what I said in that comment.

I proposed two opt-out mechanisms:

  1. interpreters.dont_share, which prevents the creation of a SharedObjectProxy. It is intended to be called before using the object in a context that would normally create a proxy (ie passing it as an argument or returning it from a function).
  2. interpreters.materialize, which coerces an already existing SharedObjectProxy to a “proper” local object by doing approximately what loads(dumps(x)) would do (with a few caveats).

No, no. I understand that SharedObjectProxy is natively shareable.

I am probably being extremely pedantic, but I really think that establishing these distinctions in terminology is extremely important, if we don’t want to end up with ambiguous / contradictory terminology.

Consider this piece of code

foo = open("somefile")
proxy = interpreters.share(foo) # 2
interp.prep_main(whatever=proxy) # 3

  1. The name of the interpreters.share function and the wording of the PEP makes it seem like this code “shares” the foo object. It doesn’t. It shares the proxy object.

    In fact, foo is an open file, which is unshareable, so it can’t be shared (this was true before this PEP and is still true after it). Only the proxy object is shareable.

  2. The name of the SharedObjectProxy class and the wording of the PEP makes it seem like the “sharing” occurs on line 2 (we called the share function and got a SharedSomething object in return, so it seems like “sharing” was performed).

    But according to the definition from the original PEP / current docs, an object “gets shared” when it is “made available to another interpreter”. So in this case, sharing actually occurs during the prep_main call on line 3, not in the share call on line 2.

    The class name of proxy shouldn’t be “Shared…” but “Shareable…”, and the function to create an instance of this class shouldn’t be named “share”. Because “sharing” happens when objects get sent to interpreters / queues, not when the proxy is created.

Again, sorry for the pedantry.

Calling the function ensure_shareable also doesn’t quite fix the issue, because in this example

list_of_ints = [1, 2, 3]
proxy = interpreters.ensure_shareable(list_of_ints)

list_of_ints is already shareable (because it’s pickleable). So going by this name, one might expect that ensure_shareable should be a no-op in this case.

I guess, it could be called something like ensure_natively_shareable. But that’s just horrible.

Actually, I think I just had a great idea! Instead of “Object Proxies”, we can call the new objects “References” (ShareableReference) and then name the function interpreters.share_reference. These names even automatically imply that this new method of sharing uses reference semantics instead of copy/value semantics.

Edit: Although this still kind of implies that share_reference is the function that performs the sharing. Damn.

Edit 2: Ah! I think I got it. How about a interpreters.make_reference function that returns a ShareableReference.

2 Likes
15

Yeah, I misunderstood, my bad! I’ll try to pay closer attention in the future. But, please try to be a little more direct in your examples. I think I got tripped up by seeing dont_share in both interpreters, so I assumed there was some sort of unwrapping going on; it’d have been enough to see the call in only one interpreter to understand the behavior.

Okay, I’m personally much more fond of dont_share/PickleWrapper, because I don’t think the proxy should have anything to do with manipulating the wrapped object. (The proxy is supposed to feel like it is the wrapped object, and adding a mechanism to magically invoke pickle hurts that intuition.)

I’m not entirely disagreeing, but again, it’s not really a useful distinction to make because the underlying object will technically be “shared” anyway. (Here with the terminology again; should the object be considered “shared”, or only the proxy? The proxy is supposed to be transparent, so one could argue the former, but physically it’s only the latter. I think this is sort of gray area at the moment.)

I’m not opposed to renaming to Shareable, but I’d like to hear someone else voice the same opinion, because I’ve been using SharedObjectProxy for however long I’ve worked on this proposal and nobody else has had an issue with it.

It’s not clear to me whether “pickleable” implies “shareable”. PEP 734 uses that terminology, but the documentation does not, as pickleable objects are referred to as being “copied” while natively shareable objects use the term “share”.

This is also reflected in the _interpreters module:

>>> import _interpreters
>>> list_of_ints = [1, 2, 3]
>>> _interpreters.is_shareable(list_of_ints)
False

I see where you’re coming from, but “reference” is already a term in Python that means something else. In C, a reference means PyObject *, which isn’t shareable unless the object is immutable and immortal (static types, for example, are shareable references). At the C level, a PyObject * pointing to an instance of a SharedObjectProxy is not shareable, which is confusing.

1 Like
16

Oh, wow. I originally learned about interpreters from the PEP, so I have just assumed that the docs kept using the same terminology. But you’re totally right. Now that I am taking a closer look, the current docs consistently use “shared and/or copied” where the PEP used just “shared”. The only exception is the NotShareableError exception (ha!), which still kind of uses the old meaning of “shareable”.

Now that I noticed this change in terminology, I have much less of a problem with calling the proxy objects Shared and the function - share. It might still be benifitial to call the function ensure_shareable, but that’s much less of a concern if pickleable objects are “copiable”, not “shareable”. Actually, excluding pickleable objects from the “shareable” category makes much more sense.

Using the new terminology, we can just say that “sharing” always follows reference semantics, because you can only “share” immutable types (None, bool, bytes, str, int, float, tuple) and “special” types that explicitly support reference-style mutability (memoryview, Queue and now SharedObjectProxy). All other objects can’t be shared directly, but some of them can be “copied” (if they are pickleable) and you can “indirectly share” any object by wrapping it in a proxy.

Consider my complaints about naming withdrawn.

1 Like
17

For what it’s worth (vaguely referring back to the fine-grained opt-out mechanisms for this particular proxy type), I’m very much in favour of this being a simple fallback proxy, and if you want to opt out, you do it by implementing a more specific proxy for your type.

As it stands, I’m happy with this PEP. The one thing I think we could add here is a protocol for .share() so that types can provide their own proxy. e.g. if the argument provides .__proxy__() then that gets called and the result must be natively shareable (and natively shareable objects can be defined as “returns self from .__proxy__()”, kind of like what we do for iterators). This gives us space to grow in the future, and hopefully provides an obvious path for how to handle opt-outs.

We might also need something like the pickle shareable as well, to make this a usefully functional proposal. Then custom proxy implementations are probably going to specify a type that can be discovered and instantiated in the other interpreter with the (natively shareable) arguments necessary to call back into the original.

Thinking about that now, it may mean we also need a way to get the fallback proxy type when __proxy__() is overridden - perhaps instantiating SharedObjectProxy directly needs to be allowed. I’m thinking of a case like this:

class MyType:
    def __init__(self, a, b, c):
        self.a, self.b, self.c = a, b, c

    class _Proxy:
        def __init__(self, remote_self):
            self.remote_self = remote_self
            # 'a' and 'b' are assumed constant, so copy them once
            self.a, self.b = self.remote_self.a, self.remote_self.b

        # Always query remote object for 'c', and disallow setting
        @property
        def c(self):
            return self.remote_self.c

    def __proxy__(self):
        return SomeProxyTypeThatNeedsAName(MyType._Proxy, SharedObjectProxy(self))

IMHO, it’s that very last function that needs figuring out before object proxies are going to be broadly useful. We can probably do the bare basics in this PEP, since it’s introducing the .share() function.

But overall, +1 on what’s here. This is a critical building block.

2 Likes
18

I like that idea a lot and I think it makes sense to include it in this proposal. Bikeshedding: It’s probably better to call it __share__, since the result of __proxy__ might not actually be a proxy.

Agreed as well – I initially disallowed it because it would be no different than share(), but it makes sense to allow it if that’s no longer true.

2 Likes
19

I have updated the PEP. The proposal now has the __share__ method that was discussed above.

1 Like