Functions variable binding and `exec`

Hello,

Could someone please help me understand the binding mechanism behind this behaviour?

>>> def f(): print(x)
...
>>> x=1
>>> exec("x=2; f()", {}, {'f': f})
1

I would like to know how to somehow get exec to print 2 here (injecting it into exec arguments is totally fine).

EDIT: I’m trying to clarify the constraints, sorry for them not being very well defined. The ideal goal here is to emulate x being a local variable for f, whose value we can control. Exactly as if x was part of f’s signature and that we called f(2).

EDIT 2: Some references that may help:

EDIT 3: So far the best approach seems to be using

f_with_x = FunctionType(f.__code__, f.__globals__ | {'x': x})

which also provide easy retrieval of return value if needed. It’s still not perfect as statements such as x += 1 would still fail without a prior global x statement.

Thanks!
python 3.13

You can’t get this to work without modifying f in some way.

x is looked up in the dictionary pointed to by f.__globals__.[1] This is set upon definition time of f (i.e. when the line def f():... gets executed). Callng exec creates a new globals and locals dictionary for the code run within - but this doesn’t affect preexisting function objects, similar to how imported functions use the original modules globals.

How this can be fixed depends on what exactly you want to do. The safest solution is to move the definition of f into the code executed by exec.


  1. In this simple case ↩︎

1 Like
def f():
    print('printing:')
    print(x)

x=1
exec("x=42; f()")

Result:

printing:
42

Note that in @elis.byberi’s example, x is being modified in the original global scope - the change isn’t restricted to a new scope anymore.

@MegaIng’s answer assumes that limiting the scope of the rebinding is desired, which is what imposes the requirement that f be defined differently to handle the use case.

4 Likes

Thank you for your proposition. Sadly, I will not accept it as solutionsince it “cheats” by emulating exec happening at the module level (i.e. with locals=globals(), and the implication that).

I’m sorry this is was not very well described in my opening post.

x is looked up in the dictionary pointed to by f.__globals__.

Could you elaborate a bit more on this?

For example, I don’t expect this to be true in general if x were defined directly inside fright? So would you know how to describe the step-by-step mechanism for f looking up for x?

It’s not really clear what your constraints are. I think the point of the exercise is what @MegaIng explains, that f is bound to x in the scope where it was defined, and so you have to change x in that scope somehow:

>>> exec("f.__globals__['x']=2; f()", {}, {'f': f})
2
>>> x
2

In terms of language semantics, f looks for x in its local scope, enclosing scopes, the global (i.e. module) scope, then builtins. The compiler, however, already knows x is not local (since not assigned) or enclosing (since there isn’t one).

1 Like

I agree and will discard this approach indeed.

@MegaIng’s answer assumes that limiting the scope of the rebinding is desired, which is what imposes the requirement that f be defined differently to handle the use case.

Well, this is where I’m not entirely sure of what’s reasonable to strive for. But ideally indeed, in a perfect world, I would like to emulate adding x for the signature of f, and then calling f(2). I’ll clarify this in the opening post.

I guess a “bruteforce” aprpoach would be to (when possible) use inspect.getsource(), manually add x to the signature and carry on. But even then, I’m pretty sure nonlcal/global variables binding will arise.

Here’s a bit more context.

Thank you for your answer. I’m sorry for this and I agree that the constraints are not well defined by me, for the lack of completely knowing the mechanisms of scopes.

The “ideal goal” is to emulate xbeing a local variable for f and assigning it desired value (like if it were originating for f’s signature and if fwere caled with a desired value).

Is this expression of the constraints making sense to you?

Oh I see. He’s executing the code in a different context from the one that the object f would supply if you called it. (I was almost there.)

>>> exec(f.__code__, {'x':2}, {'f': f})
2

I think this will only work if f has no arguments or local variables.

1 Like

I think this may be close to the wanted solution, but not quite yet. Look at this for example.

# some_file.py
a = 1

def f():
    print(a + b)

Then both of the following would fail.

>>> exec(f.__code__, {'b': 2}, {'f': f})
NameError: name 'a' is not defined
>>> exec("f()", {'b': 2}, {'f': f})
NameError: name 'b' is not defined

However, this works

>>> exec(f.__code__, f.__globals__ | {'b': 2}, {})
3

And I think it’s pretty good no? :slight_smile:

Does anyone know if this approach has any unexepected side effect? Especially, I’m thinking about the following:

  • if f were to interact with locals() and globals() in its body;
  • if f were using nonlocal or global statements in its body.
    Thanks for you help, once more!

You’re definitely on the right path. The locals of a function seem to be a dictionary returned by locals() but really they are accessed from an array as far as the actual code is concerned. There is a subtle mechanism to keep the two in sync that has evolved through versions.

A function object contains several structures that support the eventual exec of the __code__ by holding values captured from the definition, including the default values of regular and keyword arguments. This is another way one might manipulate values apertaining at the call. But then you could just use the parameters.

Now I’m struggling with extracting return value and also passing arguments to f when calling exec with code object directly.

However I found this that can help a lot I think.

# some_file.py
a = 1

def f(c):
    return a + b + c
# other_file.py
from types import FunctionType

from some_file import f

f_with_b = FunctionType(f.__code__, f.__globals__ | {'b': 2})
print(f_with_b(3))

Giving the following

$ run python other_file.py
6

I has the drawback of defining a new function from the code object (I don’t know the cost of that but I guess not extreme as it’s already compiled from what I understood). But most importantly, it treats b as a “global” variable, meaning that modifying f to the following will fail!

def f(c):  # Version with assignment
    b += 1
    return a + b + c
$ run python other_file.py
UnboundLocalError: cannot access local variable 'b' where it is not associated with a value

If, however, the user defining f knew this in advance, they could still work around this by adding a prior global b statement.

def f(c):  # Version with assignment and `global` statement
    global b
    b += 1
    return a + b + c
$ run python other_file.py
7

It really depends on if f is an existing function defined elsewhere that you can’t change, or if you can change the way it is defined.

If it’s the former, then you’re pretty much reduced to cloning the original function object and replacing its global variables: python - Override globals in function imported from another module - Stack Overflow

If you’re responsible for defining f though, then there are lots of things you can do:

  • make f take an x argument, then use functools.partial to build zero-arg callables with different values of x
  • make f a nested function returned from a factory function that takes an x argument, defines f, and then returns f, so the returned zero-argument callable sees the value of x that was passed to the factory function
  • make f an instance of a class that defines __call__ to retrieve x from an instance attribute instead of the module globals

With the latter two options, you can manipulate the value of x on the class instance, or via the cell in f.__closure__, to change its state between calls. Using a callable class for that purpose would even be a particularly unusual thing to do - having state changes affect behaviour is a normal situation for mutable class instances.

1 Like

Thank you for your answer! I’m in the case where I don’t have control over f definition (user-side), but I do load it from a file, which makes essentially everything possible since I can always inspect.getsource. However, for obvious reasons, I would like to stay as far as possible from altering the code directly.

I thought of this approach of encapsulating f inside a function that exposes the added variable but didn’t manage to make it work properly.

It’s funny though, I think the message I posted minutes ago is essentially the answer to your linked post (minus a few wrapping considerations it seems).

you can manipulate the value of x […] via the cell in f.__closure__.

I was looking at this for a while but couldn’t gather sufficient knowledge about closure objects and if it’s possible to properly engineer them.

I’m not sure what you’re asking. It would be clearer if you explained the actual procedure you’re trying to accomplish without suggesting the use of exec or similar features.

Indeed I don’t think it should necessarily revolve around exec.

The actual perfectbehaviour I would have liked is the following.

Imagine someone defines, somewhere in uncontrolled source,

def f(b):
    a += 1
    return a + b

In general, this function might not be executable unless a was somehow bound at function definition. The goal, is to make such function executable for me, with a that I define.

For more context, you can refer to this embryo of pytest PR. The idea is to be able to use autouse fixtures as if they were injected into the function’s local namespace.

I’ve come to understand that this would essentially be very hard, so the PR revolved around a looser approach of making it available as global variable, independently for each test.

Note that there is a very fundamental difference between the above and

def f(b):
    return a + b

The latter is possible by just faking a __globals__ dict - there are various possible approaches for this.

The first function definition cannot reasonably be “fixed” since a is actually a local variable within the function - these cannot be set before the function is called and even then are pretty hard to modify.

This can be done via byte code rewriting, or a debugger attached to function entry that then sets the variable. The byte code rewriting is unstable between python versions, but would be pretty performant. The second option could have a pretty large performance impact, but should be reasonably stable between python versions.

Alternatively, require people to use global a/nonlocal a if they want to set the variable within the function. I don’t believe this to be that unreasonable.

1 Like

Note that there is a very fundamental difference between the above and

Yes I know, hence the “looser” version mentioned above.

I think the most “stable” way of doing this would “simply” be by accessing the source with inspect.getsource if we suppose that it is always available (pretty reasonable in the case of pytest files) and then manually adding the initialized variable, via ast.

This is quite tricky. I did previous create a POC for a similar issue: Proof-of-concept `@ast_transformer` decorator that allows somewhat easy AST based transformation of functions · GitHub

But note that this also depends on the internal details of Code Objects and to a less extend the ast, and I don’t believe it’s possible to do without, so it’s still not perfectly stable, probably about as stable as the debugger based approach.

I would suggest to try a debugger based approach and try and see if the performance issue is relevant - it should be the smallest amount of code. (If you don’t understand what I mean, I can try to provide an example)

1 Like