Building some code on the fly - is ast the best approach?

I’m writing a library that builds code on the fly - it’s another variation on “anonymous functions”, where x + 2 generates the same code as lambda x: x + 2 if x is a magical variable from my library. I’m aware that a lot of such libraries exist, so while I wouldn’t mind seeing pointers to “prior art”, I’m definitely not looking to switch to using an existing library. This is a learning exercise as much as anything.

What I want to do is to generate the same bytecode as a lambda does. I’m doing that by building up an AST using the ast module, and then compiling that. It works pretty well, and the resulting library is fine. On Python 3.9. The problem is that the ast library changes between Python versions. That’s documented, and so I know I’m working with something pretty unstable, but I don’t know of any other way of doing something like this.

To give a concrete example, subscripting has changed between 3.8 and 3.9. In 3.9, I can do something like

ast.Subscript(
    value=self.ast,
    slice=ast.Slice(lower=..., upper=..., step=...),
    ctx=ast.Load()
)

But if I run that in 3.8, I get “TypeError: expected some sort of slice, but got _ast.Constant object”.

I can probably fix this, but I think it’ll need a lot of experimentation with various Python versions, and it’ll be challenging to keep up to date as new versions are released.

Has anyone else tried to build code using the AST like this? Is it a viable approach to take? If not, are there any other good ways of building code objects “on the fly” like this? (Let’s stipulate that building code as a string and then using eval on it isn’t what I’d call a “good way” :slightly_smiling_face:)

2 Likes

Does this help?
https://bytecode.readthedocs.io/en/latest/

Thanks, but I don’t really want to go down to the bytecode level, as I’d have to handle code generation from the program structure myself, which looks hard…

Yes, AST is your best bet. It will need constant maintenance, and it will be specific to CPython, but CPython can only generate bytecode from AST (or strings, through AST). (Or CST, up until 3.10, but that’s no easier.)

Also consider pushing a policy change PEP to start documenting AST incompatibilities better – e.g. describe them in all the What’s New document :‍)

1 Like