Is there a way to access parent nested namespaces?

Question

Is there a reliable way to get a namespace containing Bar in the following example?

Ideally I would like a way to get all parent nested namespaces between sys._getframe(1) and the global namespace.

import sys

class MyBaseClass:
    def __init_subclass__(cls, **kwargs):
        class_local_frame = sys._getframe(1)
        class_local_ns = {
            k: v for k, v in class_local_frame.f_locals.items()
            if not k.startswith('_')
        }
        print(class_local_ns)
        return super().__init_subclass__(**kwargs)

# easy to get a namespace including Foo as it's global
Foo = 'this is Foo'

def nested1():
    # is there a way to get a namespace containing Bar?
    Bar = 'this is Bar'

    def nested2():
        # relatively easy to get a namespace containing Spam
        # via sys._getframe(1).f_locals
        Spam = 'this is Spam'

        class MyClass(MyBaseClass):
            b: Foo

        return MyClass

    return nested2

def calling_it():
    Baz = 'this is baz'
    return nested1()()

calling_it()

Note: this is NOT possible with class_local_frame.f_back.f_locals - it would work in some scenarios, but in this case it would be the namespace inside calling_it, not as desired the namespace the ns inside nested2.

I’m basically looking for a way to get a namespace that accurately shadows how attributes would be accessible at runtime - e.g. when defining MyClass I could use Foo, Bar and Spam, but not Baz.

Context

Some context for my question:

As per PEP 563: “Annotations can only use names present in the module scope…”.

Thus, if I call get_type_hints() in __init_subclass__ with either from __future__ import annotations or string annotations, by default it would raise an error if type hints referenced Bar or Spam since they’re not in the module scope.

@guido, mentioned here that I could use sys._getframe(1) to access the frame where the class is defined, and that seems to be working well (implemented in pydantic#4663), thanks so much.

But that leaves a very weird edge case as demonstrated by the example above: Foo will work fine, Spam will work fine, but Bar won’t. Even though at runtime (or with annotations as python objects) using Bar would work fine.

To be clear: I understand this will come up every rarely, but when it does, it’ll be very unintuitive for users, hence I would love a way to get get_type_hints to work as near as possible identically whether or not type hints are python objects.

Thanks in advance.

Correct me if I am wrong, but I am assuming that the only thing you are allowed to modify is the definition for MyBaseClass. Everything else in the code is not allowed to be modified.

The issue here is that nested2 is not actually called within the scope of nested1. So it is a bit misleading to say that nested1 is a “parent nested namespace” of nested2. By the time nested2 is called, nested1’s namespace will have been garbage collected already. So if Bar does not have any references to it elsewhere, then it will go away. TLDR: by the time nested2 is called, Bar does not exist!

So without any additional modifications to the problem, I would say there is no sane way to do what you want because the variable you want isn’t technically in a parent scope.

If we are allowed to, say, give the user a contextmanager (and/or decorator) that they can wrap calling_it in, then it becomes possible. One way to do this would be using ast.NodeVisitor, and save a reference to each namespace on each visit_Call. Another approach is to use the builtin gc module, and to persist objects with 0 references up until the contextmanager is exited. Both of these approaches, especially the latter, may have unintended consequences for other things called within the context.

My $0.02 is-- anyone who is writing functions that return functions is probably an advanced enough Python coder that they should not find this too confusing. Alternatively that user should come to understand and accept that the scope is closed out and garbage collected by the time their subclass is created because the function nested2 is called outside the scope of nested1 (and then said user should find a way around it, which is as simple as bringing Bar into nested2 via nonlocal Bar so it isn’t garbage collected).

Thanks for your response.

But if I change the code to remove class MyClass..., and instead insert print(Bar), then Bar is correctly printed.

Similarly, if Bar where a valid type annotation used in MyClass, say:

Bar = Annotated[int, Gt(0)]
...
def nested2():
    class MyClass(MyBaseClass):
        b: Bar

Then get_type_hints could successfully resolve Bar if neither of the following are the case:

  • Bar is not defined as a string annotation - b: 'Bar'
  • from __future__ import annotations is not defined

I assume this is some magic scoping logic that means the definition of nested2 and therefore MyClass somehow has access to Bar, even when nested2 is called outside nested1.

I guess I’m asking if there is some way to use/access that magic?

I may be misunderstanding something here. But, I think replacing class MyClass(MyBaseClass): ... with print(Bar) and then having that work isn’t magic, it’s an expected behavior of garbage collection in Python. On creation of the function nested2, the Python interpreter sees that there is is a reference to Bar, so it increases the reference counter for Bar. Then when the call for nested1 closes out and garbage collection happens, the object isn’t GC’d because the nested2 object is keeping it alive.

It doesn’t work with strings and future annotations because those aren’t real references to the Bar object.

Here is your code example, but slimmed down. If you comment out the line print("Bar in nested2:", Bar), the calls Refs to Bar and Total scopes ... change. That is because commenting out that line removes the reference to Bar in the scope of nested2’s function definition.

import sys
import gc

def nested1():
    Bar = 'this is Bar'

    def nested2():
        print("Bar in nested2:", Bar)
        return

    print('Refs to Bar:', gc.get_referrers(Bar))
    print('Total scopes referencing Bar:', len(gc.get_referrers(Bar)))
    return nested2

def calling_it():
    Baz = 'this is baz'
    return nested1()()

calling_it()
1 Like

That makes sense, thanks so much for helping.

The same is true if I change print(...) to

class MyClass:
    bar: Bar
  • the ref count increases with this code, but goes back down if I either use from __future__ import annotations or equivalently define the annotation to bar as a string.
1 Like

Yeah. My understanding of from __future__ import annotations is that they’re basically strings / forward refs under the hood (Or at least, that’s what I vaguely remember from the kerfuffle where Sebastian encouraged people to raise concerns about changes to annotations with the Python core team). So that makes sense that using it means that it no longer works.

To give a TLDR here + my interpretations of the implications for what you’re doing:

  • the behavior is expected because nested2 is called outside of nested1’s scope
    • nested1 is a parent scope for the creation of nested2, but not for the call of nested2.
  • if the user does not do some sort of action that prevents GCing the Bar object, then there isn’t much that can be (sanely) done about it if we want to resolve forwardrefs to Bar

Thanks, I’ve commented on the PEP 563 discussion with a link to this case.

I’m afraid that was me as well, sorry about that. :person_facepalming:

1 Like