Adding conditional compilation to Python

Introduction

There has been lots of changes introduced to the python language in a period of a few years. Some of these changes are internal for example switching from the LL(1) parser to the PEG parser, other changes are more public for example the numerous modifications to the builtin objects and many other added feature.

My main focus however is the changes made to the python syntax. There have been a few modifications to the syntax of python, but these modifications are quite huge when it gets to adoption. The most recent syntax change was the addition of the match and case soft keywords which without a doubt have numerous applications.
Not to mention the modification of union type hints to be created with |, the addition of the walrus operator in python 3.8 and many other examples.

Problem

These changes are sources of evolution to the language, which is a good thing, but we increasingly see code which would want to adopt to this new syntax but cannot change as they would introduce syntax errors for other teams relying on their code and yet running lower versions of python. Great examples are big projects like django and the like, which still use the minimum supported python syntax. And if they’d want to try out something new provided by a new version of python, they’d have to drop support for an older version. Currently Django supports 3.8, 3.9 and 3.10, but still cannot use 3.10 specific features as 3.8 is the minimum version supported, and in doing so, it would introduce syntax errors.

solution

So in accordance to the problem above. I was suggesting the introduction of conditional compilation in python. Conditional compilation in python is not new as stated by Andre in his idea for Python 101 here. My suggestion was to be able to extend the existing machinery for conditional compilation to support this feature.

How it would work

So I was suggesting that a compiler feature be added to the __future__ module. Once someone imports compiler at the top level of the module, then the python compiler would be prepped to be ready for conditional compilation.

Then for the actual conditions. I was suggesting we extend the existing ability of using comments to switch the default encoding of python to support # -*- compileif:version -*- and # -*- endcompileif -*-. With this, someone can be able to activate conditional compilation for a specific section of python code in their module.

example usage
from __future__ import compiler

def check (val):
    # -*- compileif:3.10 -*-
    match val:
        case 1:
            print("got one")
        case _:
            print(" got no one")
    # -*- compileif:3.8.7 -*-
    if val == 1:
        print("got one")
    else:
        print("got no one")
    # -*- endcompileif -*-

so that’s how the code would kinda look. The compilerif comment could also support comparative operations such as # -*- compileif: <= 3.9 -*-.
The other idea I thought about was to be able to extend the compile builtin function to support conditional compilation. I’ve not thought out much how this would compile sections of code as yet, though I’m sure it’s possible.

pros
  1. enables quick adoption of new python features
  2. allows for the framework, library or module to evolve along with the python language
cons
  1. The new workflow might be hard or confusing to adopt
  2. I don’t think it can be easily added to more stable versions of python.

So I’m not sure if this is something worth undertaking, I invite your criticisms for this idea.

An interesting idea.

During the transition between Python 2 and 3, most of the code I wrote had to support versions 2.4 through to 3.3 and beyond. There were quite a few syntactic changes between them. Simple things like

except ValueError as err:

don’t work in Python 2.4. I mostly was able to work around the differences in functionality pretty easily. Differences in syntax were harder to deal with.

I played around with conditional compilation using compile and exec:

try:
    code = compile("""block of code""", "", "exec")
except SyntaxError:
    code = compile("""alternative block of code""", "", "exec")

exec(code)

but generally speaking I found it unsatisfactory. It worked okay if the code block was very small, but otherwise it wasn’t ideal. It added a lot of complexity and complication to my code, and made it difficult or impossible to test the implementations separately.

The trick I found was to factor my code so that the syntax differences were containing in the smallest possible unit (say, a single function), then write two versions of that function, and use importing to choose between them:

try:
    from timerlib import Timer
except SyntaxError:
    from timerlib24 import Timer
obj = Timer(*args)

That worked pretty well up to the point that I know long needed to care about 2.4 and 2.5 and could abandon the practice :slight_smile:

Being in separate file, I could test both versions of the function:

python2.4 timerlib24.py
python3.3 timerlib.py

(unit test that specific function) independently, which was great for finding bugs, as well as the full app:

python2.4 myapp.py
python3.3 myapp.py

(sort of like integration testing).

So based on my experience, I would not use conditional compilation.

Some more points.

Since from __future__ import compiler doesn’t exist now and cannot be back-ported to 3.8, your examples are impossible. If we add this feature to 3.11, no library can use it until it has abandoned support for versions below 3.11. So in practical terms that probably means that if we add it now, people who need it won’t be able to use it until 3.14 or 3.15 or so.

Using comments for code is an anti-pattern (citation required). We can get away with it for the coding cookie because the interpreter cannot even begin to parse the source text until it knows what encoding to use, so the encoding line cannot be anything that is read at runtime. It has to be something that is skipped at runtime, hence a comment. But we should not emulate that with new features that occur at runtime.

There are languages with conditional compilation (although not usually for syntactic features, more like platform features). You should survey those languages to see whether conditional compilation is considered a good feature or a source of excessive complexity.

My instinct is to think that even if we wanted this, it would be too hard to implement. The from __future__ import feature is more of a compiler instruction than a real import, and it must come first, before any executable code. It is too hard for the compiler to jump back and forth between

“this code can be compiled”

and

“this code cannot be compiled, but don’t raise an error, just skip it”.

Not quite impossible though: we can do something similar in Python today, with the compile function, which can be told to ignore or use future imports. So maybe this is plausible.

But I still wouldn’t use it :slight_smile: I would refactor out the code into separate modules and import the required one.

5 Likes

Taken into account what @steven.daprano already mentioned, syntax wise, a context manager would be a natural fit IMHO, or rather an if/else syntax:

with python-39:
    my-special-python 3.9 code
with python-38:
    my other special code

if python-version > 3.5:
    do something
else:
    do something else

Though the problem is backporting this to older versions :thinking:

:point_up_2: A tried and true technique when porting/writing straddling code for Python 2 and 3.

Related: PEP 638: Syntactic Macros. While also only useful a few versions after implementation, it could allow for 2to3-style libraries allowing for sytactic backward compatibility

1 Like

This idea here reminded me of those projects:

All these projects try to modify the ASTsource in order to modify"extend" Python syntax. Maybe a similar technique could be used to remove whole branches of the treecode (unless I misunderstood things, I did not dig deep).

But yes, that is probably more for intellectual curiosity, because for the actual purpose it seems like the techniques presented here by others are more than good enough (and already used extensively).

Ideas includes the possibility to do source transformations prior to creating an AST. To create an AST, you need valid Python syntax in the first place .

5 Likes